vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Bugfix][MoE] Plumb swigluoai activation into FlashInfer b12x MoE
## Purpose Models with a clamped SwiGLU-OAI MoE activation (`swigluoai_uninterleave`, e.g. [nvidia/MiniMax-M3-NVFP4](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)) cannot use the FlashInfer b12x MoE backend: `FlashInferB12xExperts` handles only silu and relu2. This matters on SM120/SM121: FlashInfer TRT-LLM MoE requires SM100-family GPUs and FlashInfer CUT
pull request
[Bugfix] Emit added/done lifecycle events for zero-delta streaming items
Fixes #48274 ## What's going on There's a known bug in `emit_previous_item_done_events` where items that complete with zero deltas get silently dropped from the SSE stream. The code already acknowledges it: ```python if not state.sent_output_item_added and not state.is_first_function_call_delta: # Suppress done events for items had no delta and thus had no #
pull request
Bump astral-sh/setup-uv from 7.6.0 to 9.0.0
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 7.6.0 to 9.0.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/astral-sh/setup-uv/releases">astral-sh/setup-uv's releases</a>.</em></p> <blockquote> <h2>v9.0.0 🌈 Change <code>prune-cache</code> default to <code>false</code></h2> <h2>Changes</h2>
pull request
Bump fsspec from 2024.12.0 to 2026.6.0
Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2024.12.0 to 2026.6.0. <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/fsspec/filesystem_spec/commit/a2457004d03e0312f715f90f58873de5ab195a37"><code>a245700</code></a> changelog (<a href="https://redirect.github.com/fsspec/filesystem_spec/issues/2049">#2049</a>)</li> <li
pull request
[KV Connector] Validate HMA children for MultiConnector subclasses
## Summary - Validate HMA support from child connectors for every `MultiConnector` subclass. - Add auto-configuration coverage for custom wrapper subclasses with all-HMA and mixed-HMA child configurations. ## Root cause `KVConnectorFactory.supports_hma_config()` special-cased the literal connector name `"MultiConnector"`. A subclass inherited `SupportsHMA`,
pull request
[XPU] Support sequence parallelism for block fp8 on XPU
## Purpose Enable sequence parallelism (SP) support on XPU for the block-quant W8A8 FP8 model SP is enabled with: ```json {"pass_config":{"enable_sp":true},"use_inductor_graph_partition":true} ``` ## Test Plan Model: `RedHatAI/Qwen3-32B-FP8-block` (dense, block-quant W8A8 FP8, `weight_block_size=[128,128]`), tp=4, `--linear-backend xpu`. ### Server (one of t
pull request
[Bugfix][Quantization] Keep unquantized routed experts unquantized under AutoGPTQ
# Issue #49141 — Fused_moe dimension mismatch for Qwen MXFP4 model on ROCm ## 1. Root cause The issue has two reports. Both are the same underlying defect: **a MoE layer whose expert weights are stored *unquantized* in the checkpoint is still given a *quantized* MoE method**, so the allocated parameters do not match the checkpoint tensors. The trigger in bot
pull request
[BugFix] HunyuanOCR - handle duplicate image_start tokens in xdrope positions
## Purpose Fix IndexError crash in HunyuanOCR multimodal inference. `get_xdrope_input_positions` loops over `<image_start>` token count but indexes actual image tensor array. HunyuanOCR tokenizer inserts duplicate tokens → out-of-bounds access → EngineCore crash on first multimodal request. ## Test Plan **Reproduce (before fix):** ```bash # Start vLLM server
pull request
[Frontend] strict=false in response_format json_schema
Per OpenAI semantics, strict=false means best-effort: the schema may guide the model via the prompt, but the output is not grammar constrained. vLLM previously ignored the flag and always applied guided decoding. Now skip the json structured-output override when strict=false, on both Chat Completions and Responses APIs. Default (absent) and strict=true keep
pull request
[ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends
# [ROCm][MoE] Split AITER CK and Triton MXFP4 W4A16 into separate backends ## Summary Split overloaded `AITER_MXFP4_BF16` into distinct backends: - **`AITER_MXFP4_BF16`** → CK only (`AiterExperts`), behavior unchanged (gfx950). - **`AITER_TRITON_MXFP4_BF16`** (new) → aiter Triton `moe_gemm_a16w4` (`AiterW4A16ExpertsMonolithic`), available on **gfx942 / gfx95
pull request
[Bugfix] Fix DeepSeek MoE sequence-parallel layout tracking
## Purpose `deepseek_v2.py` decides whether `hidden_states` are sequence-parallel by comparing the first dimension against the full token count. That inference is ambiguous, and single-token decode is exactly where it breaks. With TP=2 and one decode token, `ceil(1 / 2) == 1`, so a full input and a padded local shard both have one row: | Value | First dimens
pull request
[Bugfix][TurboQuant] Resolve the cache preset when get_kv_cache_shape gets no dtype hint
# Issue #50709 — TurboQuant hybrid model crashes with `Unknown cache dtype: auto` Upstream issue: https://github.com/vllm-project/vllm/issues/50709 Reported symptom: serving a hybrid model (Qwen3.5-35B-A3B: full attention + GDN linear attention + MoE) with `--kv-cache-dtype turboquant_4bit_nc` aborts during startup (`determine_available_memory()`) with ``` U
pull request
[Bugfix][Model] deepseek-vl2: restore original DeepseekV2Config defaults for omitted language_config fields
## Purpose Loading `deepseek-ai/deepseek-vl2-small` (and `deepseek-ai/deepseek-vl2`) currently crashes during model initialization: ``` TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' ``` at `DeepseekV2Attention.__init__` (`self.kv_lora_rank + self.qk_rope_head_dim`). Working around that surfaces a second failure while loading `embed_token
pull request
[ROCm][Perf] Skip redundant sparse index remap on non-indexer layers
## Summary With cross-layer index sharing, only some attention layers update the shared top-k buffer. The old path still remapped that buffer on every layer. This PR runs the remap only when the layer owns an indexer. For GLM-5.2 with `index_topk_freq = 4`, this reduces remaps from 78 to 21 per forward pass. ## Safety - Layer 0 always owns an indexer, so the
pull request
[Quantization][Humming] Support MXFP4 weight + block-FP8 activation for MoE
## Purpose Enable Humming MoE GEMM with **MXFP4 weights + block-FP8 (group-128) activations**, and — on the expert-parallel (DeepEP) path — quantize activations to block FP8 **before** the all-to-all dispatch so FP8 rather than BF16 crosses the interconnect. The Humming kernel already implements MXFP4 weights (group-32 `e8m0`) with a grouped FP8 activation v
pull request
[XPU] Add sequence parallelism support for DeepSeek V4
## Purpose Add sequence parallelism (SP) support to the XPU DeepSeek V4 model path. Attention activations are kept sharded along the sequence dimension across TP ranks, so each rank only materializes `num_tokens / tp_size` rows for the MoE and hyper-connection stages. This reduces activation memory and the volume moved by the post-attention collective on XPU
pull request
[Memory] Measure complete CUDA graph capture footprint for KV budgeting
## Purpose Fix CUDA graph memory undercounting in `profile_cudagraph_memory()`. The old path profiled only two descriptors per graph mode and extrapolated the rest. It also summed per-mode deltas even though FULL and PIECEWISE share one runtime pool, and it did not budget memory allocated during profiling setup or retained after profiling cleanup. The shortf
pull request
Fix weight tying
Fixes #51063. vLLM decides whether to tie `lm_head` purely from `tie_word_embeddings` and never looks at the checkpoint, so a config claiming tied weights against a checkpoint with a real `lm_head` silently discards it and generates gibberish. `ModelConfig.maybe_untie_word_embeddings` now checks the safetensors metadata and builds the model untied so the hea
pull request
[ROCm][Perf] Add bpreshuffled blockscaled fp8 GEMM
## Purpose Activated when shapes allow for it and configs are tuned. **Implications (DSv3 1k/1K):** - TP8+DPA: +4-8% QPS - TP8+EP: +0-4% QPS ## Test Plan Bench serve & accuracy validation with DSv3 on (1) TP8+DPA (2) TP8+EP on 8xMI350. (note need to run with `VLLM_ROCM_USE_AITER_FP8BMM=0` until https://github.com/vllm-project/vllm/issues/51957 is resolved) `
pull request
[Humming] Fuse Kimi SiTU activation + per-token FP8 quant on w2 path
Draft. Stacked on #51332 (Humming MXFP4 w + block-FP8 activation) and #51673 (SiTU `valid_rows` early-exit) — only the top commit is this PR's contribution; the rest are those two open PRs. Do not merge until both land. Fuses the Kimi SiTU (SituGLU) activation and the following per-token dynamic FP8 (e4m3) quantization of the Humming down-projection (w2) inp
pull request
[ROCm][DSV4] Opt-in AITER gluon kernel for sparse-MLA decode on gfx950
## Purpose Adds an opt-in gfx950 gluon replacement for the Triton sparse-MLA decode pair (`_sparse_attn_decode_partial_kernel` + `_sparse_attn_decode_reduce_kernel`), provided by AITER ([ROCm/aiter#4382](https://github.com/ROCm/aiter/pull/4382), merged). ## Changes Single file, `v1/attention/ops/rocm_aiter_mla_sparse.py`, additive only (+78 / -0): - `VLLM_RO
pull request
[BugFix][EPLB] Log correct avg/max token counts and balancedness
## Purpose Here is a fix for a small but confusing logging bug that I ran into: the EPLB balancedness log line reduced `avg_tokens`/`max_tokens` over the layer axis instead of the rank axis, so logged values were wrong whenever `num_moe_layers != num_ranks` leading to wrong conclusions about expert balance. Now both reduce the trailing rank axis. Log-only, a
pull request
[ROCm][Perf] Enable CSA multi-stream overlap for DeepSeek-V4
## Purpose > Part of https://github.com/vllm-project/vllm/issues/41820. > Continue working on https://github.com/vllm-project/vllm/pull/43718. > Based on https://github.com/vllm-project/vllm/pull/50866. > Added https://github.com/vllm-project/vllm/pull/52093 and https://github.com/vllm-project/vllm/pull/52094. To be completed... ## Test Plan - Benchmark with
pull request
[qwen3] correct linear_attn norm and a_log parameter dtype to F32 for qwen 3.5 series
## Purpose For Qwen 3.5 series (Gated Delta Net) models, the parameter `language_model.model.layers.{i}.linear_attn.norm.weight` is stored in **fp32** in the [HF checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B/tree/main?show_file_info=model.safetensors.index.json), but vLLM currently allocates it in **bf16** (for bf16 models) and silently casts the loade
pull request
Bump the minor-update group across 1 directory with 174 updates
Bumps the minor-update group with 174 updates in the / directory: | Package | From | To | | --- | --- | --- | | [regex](https://github.com/mrabarnett/mrab-regex) | `2026.2.28` | `2026.7.19` | | [requests](https://github.com/psf/requests) | `2.32.3` | `2.34.2` | | [tqdm](https://github.com/tqdm/tqdm) | `4.67.3` | `4.70.0` | | [tokenizers](https://github.com/h
pull request
[Bugfix] Give CPU attention layers their own metadata when head counts differ
## Purpose The CPU backend sizes one scheduler metadata blob per KV cache group from the model-wide query head count, so models that vary it per layer (e.g. Laguna) overrun the split-KV scratchpad and either segfault or hang. Build metadata per distinct head count and let each layer select the one matching its own. Change-Id: I671eadc2b5601f1a3af12f39391657a
pull request
[Core] Fix MultiConnector returning duplicate finished_recving signals
## Purpose Resolves #51846 Fixes a scheduler crash (`assert req.status == RequestStatus.WAITING_FOR_REMOTE_KVS`) that occurs in Disaggregated Prefill/Decode (PD) setups when multiple consumer KV connectors are configured (e.g., `NixlConnector` + `LMCacheMPConnector`). ### Root Cause When multiple KV connectors are configured as consumers, `MultiConnector.get
pull request
[KVConnector] Add retention interval to OffloadingConnector
## Purpose Add retention interval support to `OffloadingConnector`. The Mooncake KV connector supports retention interval support via the `envs.VLLM_PREFIX_CACHE_RETENTION_INTERVAL` env variable. It is used to avoid storing/loading blocks that are outside of the sliding window of the attention mechanism. This is implemented via block masks. The implementatio
pull request
[Quantization][CT] fix the mxfp4 inference for MiniMax-M3 with CT format.
## Description Running **MiniMax-M3** MXFP4 checkpoints quantized with the **compressed-tensors (CT)** `mixed-precision` format currently fails at model load with a `KeyError` on the vision tower. In these checkpoints the **vision module is not quantized** (it is listed in the CT `ignore` list and stored as plain `.weight`), but vLLM wrongly treats the visio
pull request
[Perf] Move context_lens compute into prefill path for Minimax
## Purpose Move `compute_num_computed_tokens()` from the top of the method into the `if num_prefills > 0` branch because it's only used in prefill. Since context_lens tensor is not used in decode path, this change avoids the tensor computed and discarded in decode path. Similar as #51913 ## Profiling Main: <img width="1900" height="495" alt="Screenshot 2026-
pull request
[Bugfix][Kimi-K3] Use MTP draft model config
## Bug Kimi-K3 constructs its MTP model with the target model's `hf_text_config` instead of the speculative draft model's config. The target config can legitimately have `num_nextn_predict_layers = 0`, while the MTP draft config has the actual draft layers (5 in our setup). Kimi therefore initializes `self.num_mtp_layers` to zero. During the profile run, MTP
pull request
[Bugfix][KVConnector][MoRIIO] Fix MoRIIO port collisions for deployments using both DP and TP
## Purpose Fix MoRIIO port collisions for deployments using both DP and TP. Port offsets now use `dp_rank * tp_size + tp_rank`, where `tp_size` belongs to the listening peer. This also handles heterogeneous TP, asynchronous WRITE completion, and the unknown-TP sentinel correctly. ## Test Plan Run the focused MoRIIO unit tests ## Test Result Unit tests passed
pull request
[Model][MRV2] Support pipeline parallelism for DiffusionGemma
## Purpose - Support pipeline parallelism for DiffusionGemma (follow-up to #45828) - Broadcast the sampler-owned per-step state to the other ranks; the initial canvas is seeded per request, so all ranks agree without a broadcast - Fix a PP deadlock in the need-sampled mask for models that roll back `num_computed_tokens` - Harden `PPHandler.broadcast` against
pull request
[Attention] Skip stage2 for single-split Triton MLA decode
## Purpose When grouped Triton MLA uses one KV split, its second-stage reduction has one input and is mathematically an identity. This PR lets stage1 write the final output and LSE directly and skips the stage2 launch. The fast path is restricted to MLA shapes for which the normal performance heuristic selects one split. Multi-split attention, MHA/GQA, and l
pull request
[Bugfix][Frontend] Import parser plugins under the vllm.* namespace so their logs survive
Fixes #52027 ## Purpose `ToolParserManager.import_tool_parser` and `ReasoningParserManager.import_reasoning_parser` import a plugin under a module name taken from its file name, so a plugin doing what every in-tree parser does - `logger = init_logger(__name__)` - ends up with a logger named after its file. Only the `vllm` logger is configured in `DEFAULT_LOG
pull request
[Feature] Add opt-in dynamic NVFP4 MoE GEMM2 quantization
### Summary This adds a default-off runtime option for ordinary per-routed-token NVFP4 quantization of the post-activation GEMM2 input in the FlashInfer TRTLLM MoE backend: ```bash VLLM_FLASHINFER_MOE_NVFP4_DYNAMIC_GEMM2=1 \ vllm serve MODEL --moe-backend flashinfer_trtllm ... ``` The checkpoint's GEMM1 input remains quantized with its existing static outer
pull request
[Bugfix][AMD] Annotate draft KV cache groups on the hybrid grouping path
## Purpose Serving **Kimi-K3** (`KimiLinearForCausalLM` — 93 layers: 24 full-attention MLA + 69 KDA linear-attention) with the **DSpark MTP draft** (`Inferact/Kimi-K3-DSpark`) and a CPU KV offload tier, the external prefix cache never serves anything: `vllm:external_prefix_cache_hits_total` stays pinned at exactly 0 for the life of the server while `vllm:kv_
pull request
[ROCm] Use silu_and_mul_with_clamp's torch._C op
This is limited to alpha=1.0 and beta=0.0 as a safety consideration due to MiniMax previously avoiding this kernel in other cases. During DeepSeekV4, day 0 support, forward_cuda was disabled in favor of forward_native for silu_and_mul_with_clamp. This can be reverted for similar accuracy and around a 6% speedup in cases without speculative decode. Example Co
pull request
[Bugfix][V1] Chunk prompt-logprobs logits to bound the activation peak
## Purpose `_get_prompt_logprobs_dict` materializes the full `[num_scheduled_tokens, vocab_size]` logits tensor in one shot and then upcasts it to float32 via `log_softmax`. Both tensors are live simultaneously, so the transient peak is roughly: ``` num_scheduled_tokens * vocab_size * (2 bytes bf16 + 4 bytes fp32) ``` At 8192 scheduled tokens and a 152k voca
pull request
[ROCm][Perf] Split MiniMax-M3 prefill index-score K loop
> **Note:** Validation required cherry-picking two fixes on top of the nightly image: > - [PR #51585](https://github.com/vllm-project/vllm/pull/51585) — Fix CUDA graph capture failure (zeroed CPU query offsets broke mixed-attention metadata builders) > - [PR #51632](https://github.com/vllm-project/vllm/pull/51632) — Fix Triton fused shared expert alignment (
pull request
[ROCm][Perf] Fold scalar KV scales in MiniMax-M3 sparse attention
> **Note:** Validation required cherry-picking two fixes on top of the nightly image: > - [PR #51585](https://github.com/vllm-project/vllm/pull/51585) — Fix CUDA graph capture failure (zeroed CPU query offsets broke mixed-attention metadata builders) > - [PR #51632](https://github.com/vllm-project/vllm/pull/51632) — Fix Triton fused shared expert alignment (
pull request
[XPU] Fix ragged-N FP8 block-scale GEMM and sparse-MLA metadata sync
Two XPU-only fixes needed to serve sparse-MLA FP8 MoE models (e.g. GLM-5.2, DeepSeek DSA) on Intel GPUs. 1. Ragged-N FP8 block-scale GEMM (scaled_mm/xpu.py): oneDNN derives the weight block-group width as wei_group_n = N/ceil(N/128), so when the output dim N is not a multiple of 128 it is < 128 and oneDNN cannot create the matmul primitive ("could not create
pull request
[KV Offload] Forward ownership in KV cache events
## Purpose Add optional ownership information to KV cache events so events emitted by a KV secondary tier can be distinguished from framework-owned events. No conflicting PRs. ## Test Plan pytest -q tests/v1/kv_connector/unit/offloading_connector/test_events.py
pull request
[KV Offload] Preserve KV event metadata until final residency removal
## Purpose Track offloaded-block residencies by medium and ownership so removing one residency does not discard metadata still needed by another. Depends on PR #52067, no conflicting PRs. ## Test Plan pytest -q tests/v1/kv_connector/unit/offloading_connector/test_events.py
pull request
[Perf][Spec Decode] Disable EAGLE cache-hit block drop for always-K=0 dynamic DSD
## Purpose Opt-in skip of EAGLE-style last-block drop on prefix-cache hits when dynamic speculative decoding is **always K=0**. Default is off, so stock cache-hit behavior and observed greedy tokens are unchanged. Always-K=0 DSD (`num_speculative_tokens_per_batch_size=[[1,16,0]]`) produces no draft tokens, but `use_eagle=True` still drops the last matched KV
pull request
[Bugfix][Spec Decode] Apply suppress_tokens on the Gemma 4 MTP sparse path
## Purpose When `use_ordered_embeddings` is set on a Gemma 4 MTP drafter, `Gemma4Proposer._greedy_sample` samples through `get_top_tokens` rather than `compute_logits`: ```python # vllm/v1/spec_decode/gemma4.py:113 return self.model.get_top_tokens(hidden_states) ``` That reaches `Gemma4MTPMaskedEmbedder.get_top_tokens`, which takes a sparse argmax directly o
pull request
[Spec Decode] Test extract_hidden_states on NemotronH hybrid models
## Purpose `NemotronHForCausalLM` is the architecture vLLM's own contributor docs point to as the reference implementation for Mamba2 + attention hybrids, and it already implements the `EagleModelMixin` / `SupportsEagle3` hooks that `extract_hidden_states` consumes. It has no coverage in the `extract_hidden_states` integration tests. The only hybrid verifier
pull request
[Bugfix] Improve context-parallel backend error guidance
## Summary Improve context-parallel compatibility errors so they give users concrete recovery steps. - tell DCP users to select another backend with `--attention-backend`, or disable DCP with `--decode-context-parallel-size 1` - add equivalent actionable guidance to the PCP compatibility error - add regression tests for both messages ## Related issue Closes
pull request
[Bugfix] Handle empty YAML config in `--config` parsing
## Description Passing an empty or comments-only YAML file to `--config` caused `yaml.safe_load()` to return `None`, which then crashed with `AttributeError: 'NoneType' object has no attribute 'items'` in `FlexibleArgumentParser.load_config_file`. This PR: - Treats `None` (empty/comments-only YAML) as an empty configuration so the parser proceeds with defaul
pull request
[Attention] Avoid redundant mask compute in GDN metadata build
## Purpose The spec decode detection in GDNAttentionMetadataBuilder.build() computed tensors multiple times: 1. computed the `num_decode_draft_tokens_cpu >= 0` mask twice in line 192 and 200. Each >= 0 creates a fresh boolean tensor. 2. computed `~spec_sequence_masks_cpu` 4 times This PR avoid redundant compute: 1. Compute `num_decode_draft_tokens_cpu` only