vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Bugfix][TurboQuant] Fix dangling decode scratch when workspace grows after cudagraph capture
## Summary Fixes a CUDA "illegal memory access" that kills the engine on the **first decode step** when serving with `--kv-cache-dtype turboquant_*` (see issue: #45670) and CUDA graphs enabled (the default `FULL_AND_PIECEWISE`). ## Root cause TurboQuant decode (`_decode_attention`) sourced its scratch buffers (`mid_o`/`output`/`lse`) from the **growable** `W
pull request
[Profiler] Add execution trace capture to torch profiler config
## Summary Adds a `torch_profiler_execution_trace` option to `ProfilerConfig` that captures a PyTorch **execution trace (ET)** alongside the existing Kineto trace, wired through the existing `--profiler-config` flow. Previously vLLM's profiler config could only collect the Kineto/Chrome trace via `tensorboard_trace_handler` — `TorchProfilerWrapper` built `to
pull request
[Bugfix] Drop incomplete tool-call markup in non-streaming to match streaming
## Purpose Fixes #47137 When generation terminates (`max_tokens`/`length` or a `stop` string) inside a `<tool_call>` opener that has not been promoted to a tool call, the streaming path drops the buffered markup but the non-streaming path returned it as `content` — the two paths disagreed for identical model output. Contrary to the hope in the issue thread t
pull request
[Bugfix] Fix `--data-parallel-start-rank 0` being treated as unset in `create_engine_config`
## Summary `EngineArgs.create_engine_config` uses Python truthiness (`if self.data_parallel_start_rank`) instead of `is not None` to detect whether `--data-parallel-start-rank` was explicitly set. Since `0` is a valid, meaningful starting rank (the node owning the first slice of global DP ranks), an explicit `--data-parallel-start-rank 0` is silently treated
pull request
[Refactor]: StructuredOutputManager x Speculative Decoding Refactor
## Purpose See https://github.com/vllm-project/vllm/issues/48197. ~100 insertions, ~200 deletions of non-test changes. Then ~400 insertions, ~600 deletions for test consolidation. Moves `tests/v1/spec_decode/test_mtp_structured_output.py` and `tests/v1/structured_output/test_reasoning_structured_output.py` into `tests/v1/structured_output/test_structured_out
pull request
[Bugfix] Make Gemma 4 suppress-token masking CUDA-graph safe
## Purpose Gemma 4 CUDA graph capture used Python suppress-token IDs and advanced indexed assignment with a Python scalar. Both can introduce host-to-device work during capture. Store validated IDs as nonpersistent model buffers and use `index_fill_` for graph-safe masking in the target, unified, and MTP paths. The centroid MTP proposer also returned sparse
pull request
[CI/Build][The Rock] Use model_class_overrides so spawned worker can use test PredictableLlamaForCausalLM class when worker spawned using Python 3.14
## Purpose This PR changes the test to use the "`module:class`" string format with `model_class_override`, which lets `ModelConfig.model_class_override` re-register the model in each worker process. The following warning is seen when running this test: WARNING 07-20 15:41:58 [system_utils.py:157] We must use the `spawn` multiprocessing start method. Overridi
pull request
Detect ROCm wheel variant from environment for precompiled wheels.
Fixes **AMD: Python-only Installation** failing because ROCm precompiled wheels on `wheels.vllm.ai` use a different path layout than CUDA. - **`setup.py`**: Detect installed ROCm at runtime, match against published variants on `wheels.vllm.ai/rocm/{commit}/`, fall back to AMD PyPI. - **`python_only_compile.sh`**: Same variant resolution for the preflight met
pull request
[KV Offload] Carry chunk index in OffloadKey
## Purpose Re-scopes this PR to a single, minimal change agreed with the maintainer in review: carry the chunk index inside the `OffloadKey` so the self-describing KV-event path can recover a chunk's provenance from the key alone. `OffloadKey` becomes: ``` block_hash | group_idx:u32 (big-endian) | chunk_idx:u32 (big-endian) ``` where `chunk_idx` is the absol
pull request
fix(kernel): restore scalar_t RMSNorm intermediate rounding boundary (#49616)
## Purpose Fixes #49616. ### Problem and root cause The RMSNorm CUDA kernels kept the normalized value in `float` through the weight multiplication. That changed the rounding boundary relative to the unfused `scalar_t` path and produced bit-level differences during greedy NGRAM/speculative verification. In our nightly comparison, the unpatched path differed
pull request
[ROCm][AITER] Add GDN long-prefill split-QKV fast path
## Motivation ROCm/aiter#4299 adds a channel-last HIP path for `causal_conv1d_fwd_split_qkv`, which can consume Qwen3.5's contiguous `[T, D]` projection through its logical `[D, T]` transpose without materializing a copy. vLLM already fuses its packed post-conv split, normalization, and gating, so dispatching the AITER op unconditionally regresses short prom
pull request
[Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP
## Summary This PR adds the scheduler and KV-cache-manager support required for multi-module MTP (one MTP module per speculative step, e.g. Inkling's 8-depth checkpoint). It is the companion to #48892, which introduced the speculator itself to Model Runner V2. The core property this PR protects: the multi-module drafter reads **ahead** of the computed tokens
pull request
[Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch
# [Bugfix][MRV2] Require all requests to be decoding for uniform-decode dispatch ## Purpose Split out of #50488 at a maintainer's request, so each fix is reviewable on its own. V2 classified a batch as uniform decode from its shape alone (`num_tokens == num_reqs * max_query_len`) and nothing about whether the requests were actually decoding. A prompt chunk o
pull request
[Bugfix][Gemma4] Keep image bidirectional attention within the sliding window on the V2 model runner
## Purpose #47217 fixed Gemma4 vision models silently falling back to **causal-only** image attention whenever a single image's soft-token span exceeded the text sliding window (e.g. ~1120 soft tokens at `max_soft_tokens=1120` vs `sliding_window=1024`). The runner was dropping the *entire* bidirectional `mm_prefix` range for such an image; the fix gated that
pull request
[Bugfix][Rust] MiniMax-M3 tool parser: accept elided first-parameter opening tag
## Purpose Fixes #51073 Deployed MiniMax-M3 checkpoints (`MiniMaxAI/MiniMax-M3-MXFP8`; chat template byte-identical on `olka-fi/MiniMax-M3-MXFP4`) deterministically emit tool-call invokes whose **first parameter is missing its opening tag**: the value sits directly between the namespace sentinel and a closing tag carrying the parameter name. Real capture (no
pull request
[Kernel] Hoist tensor-descriptor build out of unified-attention TD ti…
## Summary Part of the TD adoption strategy proposed in #42545. `USE_TD`'s K/V tile loader rebuilt the tensor descriptor every tile iteration, since the descriptor's `base` baked in the per-tile physical block index. That forces a `tensormap_create` per iteration, which the pipeliner can't predicate: - On Hopper (sm_90), this only compiles at `num_stages=1`,
pull request
[Platform] Add check_runner_kv_caches_multi_layer
## Purpose Add `check_runner_kv_caches_multi_layer` interface to avoid platform hardcode in bind_kv_cache. So that oot platform can override it to avoid error raising. ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing i
pull request
replace batch_norm to numerically identical without cudnn
Fixes #51717 ## Purpose With `--mm-device-do-normalize` (added in #50411), image normalization runs on device through `FusedInputNorm`. That module implements the per-channel affine `output = (input * rescale_factor - mean) / std` by calling `F.batch_norm` with `running_mean=0`, `running_var=1`, `eps=0`, `weight=1/std`, `bias=-mean/std`. On CUDA, `F.batch_no
pull request
[Quantization] Remove dead `QuantizationConfig.is_mxfp4_quant`
## Purpose This is dead code following https://github.com/vllm-project/vllm/pull/37128. This was originally added in https://github.com/vllm-project/vllm/pull/29008 that supported padding for gpt-oss / certain MXFP4 backends, see: https://github.com/xuebwang-amd/vllm/blob/c62f664e97977ee54ab1d1c77604ebb45081bc06/vllm/model_executor/layers/fused_moe/layer.py#
pull request
[XPU] Enable Kimi K3 KDA kernel tests on XPU
## Purpose `gather_initial_states()` asserted `state.is_cuda`, which is False for XPU tensors, so it blew up before reaching the Triton kernel; widened to `state.is_cuda or state.is_xpu`. `test_kda.py` hardcoded `DEVICE = "cuda"`, now taken from `current_platform.device_type` - the CUDA-only cases already self-gate. This gives XPU coverage of the KDA Triton
pull request
fix(pooling): validate BGE-M3 combined task ownership
## Purpose Fixes #51801. BGE-M3 can resolve to the internal `embed&token_classify` task. That combined task belongs to the sparse IO-processor plugin, which also converts the combined model output into the public dense/sparse response. Registering a generic pass-through processor would avoid the original `KeyError`, but it would not perform that conversion.
pull request
[Bugfix] Disable fine-grained prefix-cache hits for incompatible hybrid KV layouts
## Summary Fine-grained prefix-cache hits are enabled for hybrid models containing Mamba `align` groups. However other groups, such as a sliding-window DSpark drafter, may use KV cache managers that only support block-aligned lookups. This previously caused an assertion when prefix caching was enabled. This change: - Enables fine-grained hits only when every
pull request
[ROCm][Model][Bugfix] Enable GLM-5.2-MXFP4 on the deepseek_v32 path and fix sparse attention correctness
## Purpose Enables GLM-5.2 (`GlmMoeDsaForCausalLM`) end-to-end on `vllm/models/deepseek_v32/amd/` for gfx942/gfx950. Routing is opt-in via `--model-class-overrides`; the registry entry is unchanged, so the default path for GLM-5.2 and DeepSeek-V3.2 is untouched. The following issues were also fixed as a result of this overall enablement since it surfaced dor
pull request
[Perf][DSV4] Optimize global top-k index kernel with compile-time constants
## Purpose Optimize global top-k index kernel with compile-time constants ## Test Plan ``` vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --enable-expert-parallel --tensor-parallel-size 8 --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v
pull request
[Bugfix][ROCm][MoE] Update AITER MXFP4 W4A16 tests to the renamed expert_mask
## Purpose Fixes two errors in `tests/kernels/moe/test_rocm_aiter_moe.py` on gfx950: ``` TypeError: rocm_aiter_fused_experts() got an unexpected keyword argument 'expert_map' ``` #49758 renamed that keyword from `expert_map` to `expert_mask` and updated the production callers (`AiterExperts.apply`, quark), but not these two test call sites. This renames them
pull request
[Bugfix][CPU] Workaround contradictory mamba validation after CPU MLA disables prefix caching
## Purpose Partial workaround for #52008. **Not the full fix.** #52008 asks for one of: - **A.** Keep prefix caching and chunked prefill enabled on CPU when KimiLinear requires them - **B.** Fail fast with a clear “KimiLinearForCausalLM is not supported on CPU” - **C.** Honor `--hf-overrides '{"use_mla": false}'` before the CPU platform check **This PR does
pull request
[Frontend][Rust] Bound chat-template evaluation to prevent DoS (#52025)
## Purpose Fixes #52025. The Rust frontend renders a caller-supplied (or model-supplied) Jinja `chat_template` from `POST /v1/chat/completions` with **no evaluation budget**. Rendering cost is `O(N^depth)` in caller-controlled loop bounds, and minijinja's per-`range()` element limit is trivially walked around by nesting three individually-legal `range()` cal
issue
[Usage]: 启动 qwen3 vl 超级超级超级慢，sglang 启动很快，可能的原因是什么？
### Your current environment 连执行 python collect_env.py 都很慢，环境是直接 uv 安装的 ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.2 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version : version 4.1.2 Libc version : glib
issue
Document Ray Serve LLM incompatibility with vLLM v1 engine for PD disaggregation
## Summary vLLM v1 engine with `RayDistributedExecutor` is incompatible with Ray Serve LLM's `build_pd_openai_app` due to nested placement group conflicts. Users attempting PD (prefill/decode) disaggregation with Ray Serve LLM encounter silent failures. This should be documented to guide users toward working deployment patterns. ## Environment - **vLLM**: 0.
issue
TP > 1 with Ray Serve: Use Multiprocessing Executor (Not Ray Executor)
# TP > 1 with Ray Serve: Use Multiprocessing Executor (Not Ray Executor) ## Summary When deploying vLLM with `tensor_parallel_size > 1` on Ray Serve, use the **multiprocessing executor** (`distributed_executor_backend="mp"`) instead of the Ray executor. This avoids placement group context issues with vLLM v1's subprocess architecture. ## Problem Description
issue
[Bug]: Language Models Test (Extended Generation) test_models[False-False-5-32-bigcode/starcoder2-3b] test issue
### Your current environment main ### 🐛 Describe the bug I got the test to fail under the following two situations: 1. We're trying to upgrade PyTorch from 2.10 to 2.11. The test succeeds in PyTorch 2.10, but fails in PyTorch 2.11 ([logs](https://buildkite.com/vllm/ci/builds/56028#019ce7ae-eb9b-41be-90ae-310763ec6438)) 2. In PyTorch 2.10, if I add [--enforce
issue
[Bug/Perf]: hybrid-SWA prefix caching collapses to zero for ALL requests in multi-session round-robin at ~25% pool occupancy (Gemma-4-31B; eager-freed SWA tails recycled tail-first)
## Summary For hybrid-SWA models (tested: Gemma-4-31B, 10 full-attention / 50 sliding-window(1024) layers), **cross-request prefix-cache reuse collapses to exactly zero for every request in a multi-session round-robin workload once the combined working set exceeds a sharp threshold far below pool capacity** (~34–38k tokens vs a 153k-token pool — ~25% occupan
issue
[Bug][Spec Decode] Drafter attention backend auto-selection picks FlashInfer on SM90 for a sliding-window drafter, hitting the guard crash
### Environment Same as #48494 (v0.25.0, single H100 NVL 94GB SM90, Gemma4 MTP drafter with 3 sliding-attention layers + 1 full-attention layer), **without** `--attention-backend`. ### Behavior - Target model backend selection: `Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN']` — fine (heterogeneous head dims exclude others). -
issue
[Bug]: With qwen3.5-35b-a3b, the performance is relatively poor both when using dflash and when not using it, but the accepted length of dflash is around 5–6.
### Your current environment <details> ```text vllm 0.26.0 +12.9 torch 2.11+12.6 { "architectures": [ "DFlashDraftModel" ], "auto_map": { "": "config.DFlashSpeculatorConfig" }, "aux_hidden_state_layer_ids": [ 1, 6, 11, 16, 22, 27, 32, 37 ], "block_size": 8, "draft_vocab_size": 24576, "dtype": "bfloat16", "mask_token_id": 248077, "sample_from_anchor": false,
issue
[Feature]: Populate completion_tokens_details in streaming and non-streaming usage responses
### 🚀 The feature, motivation and pitch The OpenAI chat/completions spec defines `completion_tokens_details` as part of `CompletionUsage`, which is shared between streaming and non-streaming responses. vLLM does not populate this field in either mode. vLLM 0.23+ fixed this for `prompt_tokens_details` (#44961): both streaming and non-streaming responses now i
issue
[Bug]: online FP8 (--quantization fp8) produces corrupted, non-EOS-terminating output on Qwen2.5-1.5B-Instruct
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version
issue
[Bug] DeepSeek-V4-Flash-0731 intermittently emits malformed DSML tool-call start wrapper on v0.27.1 + DSpark
## Summary With **DeepSeek-V4-Flash-0731** on **vLLM v0.27.1** with **DSpark enabled**, we have intermittently observed malformed DSML tool-call output where the opening wrapper is corrupted from: ```text <｜DSML｜tool_calls> ``` to something like: ```text <｜DSML｜toolcalls> ``` while the rest of the DSML block remains structurally close to valid and the closin
issue
[Bug]: speculative decoding under pipeline parallelism produces wrong output with --no-async-scheduling
### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Reproduced at `pipeline_parallel_size` 2, 4 and 8, with two different speculative methods and two model families. Code references are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug With `--no-async-scheduling`, spe
pull request
[ROCm][Kernel] Extend skinny gemm N=5 to N=8 cases on GFX12 (RDNA4) using SWMMAC optimization
## Purpose This PR adds an optimized ROCm GFX12 (RDNA4) SWMMAC-based skinny GEMM path. The new path targets serving workloads with low max concurrency, matching the benchmarked decode GEMM range of N=5-8. The implementation uses sparse wave matrix multiply-accumulate instructions on GFX12 (RDNA4) to improve throughput for these skinny GEMM cases, while keepi
pull request
[ROCm][Perf] Enable fused indexer-Q RoPE+quant kernel for DeepSeek/GLM sparse attention
## Summary Enables the existing fused indexer-Q kernel (`fused_indexer_q_rope_quant`) on ROCm for the DeepSeek Sparse Attention (DSA) indexer used by GLM-5.2 / DeepSeek-V3.2-style models. On CUDA this kernel already collapses the indexer's query-side RoPE + FP8 quantization + weight-scale fold into a single launch; on ROCm the same work was falling back to a
pull request
[Misc] Remove `override_attention_dtype`
## Purpose `override_attention_dtype` is only used for V0 and has been removd from https://github.com/vllm-project/vllm/pull/25351/ long time ago. It's safe to remove it now. ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link exi
pull request
[Bugfix][ROCm] Only run FP8 AITER MLA prefill when using FP8 KV
## Purpose Even when KV is not FP8, the FP8 PS ASM prefill kernel's persistent buffers are both (a) initialized/reserved and (b) filled with metadata (through `_build_fp8_prefill_ps_metadata`) in every forward. Specifically the latter leads to performance degradation when using BF16 as `_build_fp8_prefill_ps_metadata` is a host-side function with several syn
pull request
[Spec Decode][V1] Warm Eagle and DFlash/DSpark spec-decode Triton kernels at startup
## Summary Triton specializes integer arguments whose runtime value is 1 into compile-time constants, producing a separate cubin per "which params are 1" combination. Triton also specializes `tl.constexpr` arguments (one cubin per distinct value) and `tl.num_programs` grid axes when they equal 1. Without warmup, the first request in each shape pays a JIT lat
pull request
[Bugfix] Guard tool call argument JSON parsing in chat message postprocessing
## Purpose Fixes #47761. `_postprocess_messages` in `vllm/entrypoints/chat_utils.py` calls `json.loads()` on the assistant tool-call `arguments` string that clients replay in conversation history, with no error handling: ```python if content := function.get("arguments"): if not isinstance(content, (dict, list)): parsed = json.loads(content) # raises on
pull request
[Test] e2e hybrid-Mamba prefix-cache corruption regression tests (#43559)
## Purpose End-to-end output-correctness regression tests for **#43559** — hybrid-Mamba prefix caching silently corrupting output under `mamba_cache_mode="align"` + MTP/EAGLE speculative decoding. These are the e2e tests split out of #48361 (per review) and supersede the test-only **#48198**. Two graded arms, each compared against an `enable_prefix_caching=F
pull request
[XPU][UT] Fix OOM and skip graph case
Fix UT on XPU: 1. test_mamba_cache_cg_padding can only run with graph enabled. Otherwise, UT will fall into an infinite loop. 2. To address the issue of insufficient VRAM caused by consecutive use cases (where there is insufficient free memory upon engine startup), the approach is to reuse the existing ROCm VRAM waiting mechanism and provide an appropriate d
pull request
[EC Connector] Added Build Connector Worker Meta for EC Connector
## Purpose **Why this is still needed after #38390.** The PR implemented the V2 model runner EC Connector, but EC still has no worker -> scheduler metadata channel. KV connectors have a complete one: `build_connector_worker_meta()` -> `KVConnectorOutput.kv_connector_worker_meta` -> `KVOutputAggregator` -> scheduler-side KV connector. EC had none of those thr
pull request
[Model] Add Cohere2MoE Eagle3 auxiliary hidden states
## Purpose Enable Cohere2MoE target models to serve with compatible Eagle3 and DFlash draft models by exposing the auxiliary hidden states required by the speculator. I ~am working on training~ have released a DFlash speculator for North-Mini-Code-1.0 and realized that this was missing! Hopefully it helps others, ~especially when I upload the checkpoints.~ :
pull request
[Bugfix] Bound accepted-token state lookups in GDN/KDA spec decode
## Summary On a hybrid GDN model (Qwen3.5 / Qwen3.6) with MTP speculative decoding and prefix caching (`--mamba-cache-mode align`), the engine dies with `CUDA error: unspecified launch failure` within 7-10 requests of agent-shaped traffic. The GPU faults, not the runtime: the signature is an SM address exception (`Xid 13`, `ESR 0x404000`) or an MMU fault (`X
pull request
[Hardware][AMD] Enable fused bf16→fp32 router GEMM on ROCm
## Purpose This pull request addresses issue #50267 by enabling fused bf16→fp32 GEMM operations on AMD ROCm hardware for MoE router gates. ## Problem Statement The MoE router gate requires `out_dtype=torch.float32` for the `grouped_topk` operation. However, on ROCm, the fused fp32-output GEMM implementation was gated exclusively to CUDA platforms. This cause