vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Bugfix][Spec Decode] Make EAGLE weight sharing TP-consistent
## Purpose Addresses the rank-divergent EAGLE weight-sharing decision described in #50158. The issue stays open: real-hardware runs (see discussion below) show the acceptance collapse reported there persists after this fix and has a separate root cause that is still under investigation. EAGLE currently decides whether to share the target model's `embed_token
pull request
[Bugfix] Stop the Granite reasoning marker leaking into streamed reasoning
## Summary `GraniteReasoningParser` leaks the tail of `Here is my thought process:` into the streamed `reasoning` field when a single delta closes that marker and opens the reasoning at the same time. `_get_delta_message_with_no_response_bounds` reconstructs "the reasoning before this delta" as: ```python previous_text = reasoning[: -len(delta_text)] ``` Tha
pull request
DCP: consume owner-sharded Top-K candidates through symmetric memory
# DCP: consume owner-sharded Top-K candidates through symmetric memory ## Summary This change replaces the sparse-indexer candidate AllGather with direct consumption from one symmetric-memory candidate shard per DCP rank. Each producer stores only its local `(rows, candidates, score_and_token)` shard. Symmetric-memory rendezvous gives every consumer a peer t
pull request
[sleep functionality] code refactor about sleep/wake_up
## Purpose 1. For now, frontend multiprocessing the sleep and wake_up are utility commands and in `async_llm` the call will await the request future. So, it is a sync operation. The log should be deleted as v0 has been completely deleted. 2. remove hard code strings 3. more detailed logs about the scheduler state change ## Test Plan NA ## Test Result NA ---
pull request
[XPU] update UMD to 26.27
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
pull request
[CPU] Fix NUMA node binding for multi-TP CPU workers
## Purpose On CPU, each tensor-parallel worker bound its memory to the NUMA node of the **first allowed CPU** (`get_allowed_cpu_list()[0]`), so every rank in a multi-TP run landed on the **same** NUMA node. This defeats NUMA locality: all workers compete for one node's memory bandwidth while other nodes sit idle, and the per-worker KV cache sizing is compute
pull request
[ROCm][Perf] Use AITER tuned GEMM for the MoE router gate
## Purpose The MoE gate linear layer was set to use a bf16×bf16→fp32 GEMM on tier 5 for faster performance, but it is conditioned on Hopper and Blackwell. This enables it for AITER when the underlying tuned GEMM shapes exist. ### Trace comparison <img width="1840" height="1180" alt="pr_fig1_cpu_stack" src="https://github.com/user-attachments/assets/35a59fe3-
pull request
[Kimi-K3][ROCm] Fuse KDA upper-triangle zeroing
Avoid a separate full `Akk` buffer memset by writing the required zeros in the existing fused solve kernel. ## Purpose Kimi-K3's ROCm KDA prefill path allocates `Akk` with `torch.zeros`, even though the fused inter/solve Triton kernel already writes its lower triangle. This change makes the fused kernel explicitly write the required upper-triangle zeros, all
pull request
[ROCm][Perf] Fuse Kimi-K3 FP8 pre-route projections
## Purpose Integrate the Kimi-K3 gfx950 pre-route primitives through one AMD-only, fail-closed ownership boundary: - prepack routed-down and shared-expert weights after checkpoint loading; - produce routed inputs, shared-expert output, and FP32 router logits in the AITER mixed-precision grid; and - pass the precomputed shared output through a scoped seam so
pull request
[PD] Emit inactive KV blocks for decode affinity
Publish GPU block inactivity when the last request reference is released so external schedulers can maintain accurate decode load signals. ## Purpose Add a new KV cache event, `BlockInactive`, so external PD schedulers can track when a GPU prefix-cached block is no longer actively referenced. Today, KV event consumers can observe `BlockStored` and `BlockRemo
pull request
[Bugfix][KV Offload] Scale UniformTypeKVCacheSpecs groups by DCP
## Purpose #49964 gated the DCP scaling of a group's token span on `isinstance(group.kv_cache_spec, AttentionSpec)` so that Mamba groups stay unscaled. `UniformTypeKVCacheSpecs` is a container rather than an `AttentionSpec` subclass, so DSA models — whose indexer and MLA layers merge into a single uniform group — lost the DCP factor too: - `build_offloading_
pull request
[Bugfix] Forward SwiGLU clamp/alpha/beta in compressed-tensors W4A4 MXFP4 MoE
## Purpose Fixes #51028 Models whose MoE uses the clamped SwiGLU-OAI activation (e.g. MiniMax-M3 in compressed-tensors MXFP4 W4A4 form, `olka-fi/MiniMax-M3-MXFP4`) crash at startup with: ``` RuntimeError: Worker failed with error 'SWIGLUOAI_UNINTERLEAVE requires clamp_limit' ``` `CompressedTensorsW4A4Mxfp4MoEMethod.get_fused_moe_quant_config` never forwards
pull request
Enable Speculative Decoding for NVIDIA-Nemotron-Parse-2.0
co autored with: @DenisOvchinnikov93 [NVIDIA-Nemotron-Parse-2.0](https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-2.0) ships: > Auxiliary Prediction Head: One training-time decoder prediction head is preserved separately in auxiliary_prediction_heads.safetensors.extra for future multi-token prediction research. Standard generation uses the tied decoder in
pull request
[Bugfix] Disable async scheduling when VLLM_BATCH_INVARIANT=1
## Purpose `VLLM_BATCH_INVARIANT=1` pins the *kernels* so a row's result no longer depends on batch size or composition. It does nothing about **which batch the scheduler puts a request in**. Async scheduling is enabled by default (`SchedulerConfig.async_scheduling = None` -> `True`) and composes the next step's batch before the current step has retired, so
pull request
[ROCm][Bugfix] Restore AITER-compatible Triton 3.7 pin
## Purpose Restore an AITER-compatible ROCm Triton 3.7 pin. [vLLM #50607](https://github.com/vllm-project/vllm/pull/50607) moved the ROCm image to Triton `0263a6a` from `release/internal/3.7.x`. That revision rejects the `DistributedLinearLayout` offsets produced by AITER's CDNA4 Gluon MLA kernel: ```text expected offsets type layout to be BlockedLayout or S
pull request
[ROCm][Perf] Fuse the DSA indexer prologue with AITER
## Summary The DSA indexer prologue uses four kernels for K normalization, Q/K RoPE, FP8 quantization, and the K-cache write. This PR uses AITER's fused `indexer_qk_rope_quant_and_cache` kernel instead. The fused path is enabled only for its supported BF16/FP8 shape; other configurations keep the existing path. ## Results MI355X, TP=4, rank 0: | indexer stag
pull request
[Core] Declare EAGLE cache-peek capability for KV cache managers
## Summary - declare whether each KV cache manager can safely peek past an EAGLE/MTP candidate boundary - resolve EAGLE cache-peek groups through the shared KV cache spec registry - exclude recurrent and otherwise unsupported cache managers from the fallback - fail closed when an unsupported group is explicitly annotated - apply the shared capability in the
pull request
[Bugfix][ROCm] Route non-divisor small-head MLA decode to Gluon on gfx950
Fixes #51232. Kimi-K3 has 96 MLA query heads; at TP8 each rank gets 12 local heads. 12 is not a divisor of 16, so `use_gluon_decode` returned `False` in "auto" mode, routing single-token decode to the padded ASM persistent path. That path pads 12 heads to 16 and asks AITER for a `{bf16, fp8, gqa=16, qseqlen=1}` kernel, which does not exist on gfx950 — every
pull request
[ROCm] Enable fused QK-norm+RoPE+gate Triton kernel for Qwen3-Next/Qwen3.5
## Purpose `Qwen3NextAttention` fuses the gated QKV split + QK-RMSNorm + (partial) NeoX RoPE + gate copy into a single Triton kernel (`fused_qk_rmsnorm_rope_gate`, added in https://github.com/vllm-project/vllm/pull/44176). The kernel is pure Triton and backend-agnostic, but it is gated behind `current_platform.is_cuda()`, so ROCm falls back to the slow multi
pull request
[Bugfix][MRV2] Respect dynamic K=0 from scheduler in AutoRegressiveSpeculator
## Problem When Dynamic Speculative Decoding (DSD) selects K=0 at higher batch sizes, `AutoRegressiveSpeculator` ignores the scheduler's runtime K and executes the configured maximum number of draft steps. Those drafts are not used, so this wastes GPU work. The scheduler already publishes the selected K in `SchedulerOutput.num_spec_tokens_to_schedule`; MRV2
pull request
[ROCm][RFC] Enable scoped AITER W8A8 support on gfx1100
## Summary Enable scoped gfx1100-compatible AITER W8A8, GDN decode/causal-conv1d single-token, and sampler paths without widening the CDNA/RDNA4 master gates to unsupported CK/ASM operations. Addresses vllm-project/vllm#51136. ## Changes - Add independent fail-closed gates for gfx1100 W8A8, GDN decode, and lazy sampling. - Route W8A8 through public `aiter.ge
pull request
[2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (DBO)
<!-- markdownlint-disable --> DBO for Model Runner V2 (RFC: #50738 ) is two PRs: #50945 [1/2][Model Runner V2] DBO support, eager mode (P0–P2) -> #51700 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (P3–P4) DBO for Model Runner V2 ([RFC #50738](https://github.com/vllm-project/vllm/issues/50738)) is two PRs: Stacked on top of #50945; t
pull request
[ROCm][DSpark][DCP] Support decode context parallelism for Kimi-K3 DSpark
## Purpose Enable decode context parallelism (DCP) for Kimi-K3 with the DSpark drafter on ROCm. Two things stood in the way: 1. **Hybrid KV cache grouping was not drafter-invariant.** The DSpark draft adds an MLA-only attention type with far fewer layers than the target. Since group size was `min(bucket_sizes)`, that small bucket shrank `group_size` and over
pull request
[Fix] Support heterogeneous models (Gemma 4 / Qwen) failing with AmbiguousGlobalPerLayerAttributeError
### What this PR does This PR fixes severe crashes that occur when loading heterogeneous models (such as **Gemma 4** and **Qwen MTP**) with recent versions of `transformers` (>=5.15.0). These models possess varying `head_dim` and `num_key_value_heads` across layers. The new `transformers` library raises an `AmbiguousGlobalPerLayerAttributeError` (which does
pull request
[FS-Offloading] : Simplify fs manager
## Purpose The tracking of failed keys on `main` is handled in the fs/manager. This makes the manager more complicated than necessary. This PR makes failed_keys a first class citizen in the DualQueueThreadPool. The DualQueueThreadPool already maintains a JobState to track a Job's progress and reasons about failure/success based on results directly from the t
pull request
[Docker] Update to nixl-1.3.2
## Purpose The KV-connectors layer installs `nixl` from requirements/kv_connectors.txt, then force-reinstalls the CUDA-matched backend wheel so the correct nixl_ep_cpp.so is present. That second install passes `--no-deps` and no version, which bypasses the `nixl` meta package's `nixl-cu*==<version>` constraint and resolves whatever nixl-cu${CUDA_MAJOR} is ne
pull request
[Model] Match Qwen3.5 GDN BF16 semantics
## Summary - preserve activation-dtype Q/K normalization and beta semantics for Qwen3.5 BF16 activations - cover fused prefill, generic/speculative decode, and packed decode with the same model-scoped gate - propagate the gate through Triton warmup so compiled variants match runtime selection - add sensitivity tests for prefill and both incremental-decode pa
pull request
fix and test EPLB balancedness calculation
## Purpose Fix EPLB balancedness logging to aggregate rank load within each MoE layer. The current reduction uses the layer axis, so it can report perfect balance when one EP rank receives all tokens in every layer. ## Test Plan ```bash pytest -q tests/distributed/test_eplb_algo.py ``` ## Test Result The new regression case has equal token totals per layer w
pull request
[Model] Add DeepGrove Maple (MapleForCausalLM)
## Purpose Adds support for [`deepgrove/maple-preview`](https://huggingface.co/deepgrove/maple-preview) (`MapleForCausalLM`), a 20B-A1B MoE reasoning model (MIT). Architecture specifics, none of which an existing model file combines: - `layer_types` interleaves sliding-window and global attention 3:1. The checkpoint runs FlashAttention with `window_size=(sli
pull request
[Bugfix][Frontend] Fix prompt_logprobs=0 bypassing admission guards and chat echo
## Summary `prompt_logprobs=0` is a valid, meaningful `SamplingParams` value ("return the prompt token's own logprob, no alternatives"), but several checks along the request path test it with plain truthiness instead of `is not None`, so a value of `0` silently slips through: - `ChatCompletionRequest.to_sampling_params` inherits `top_logprobs` (which default
pull request
Cached blocks never hit eviction first
## Purpose Most cached blocks with block_hash that never hit. Now they are appended to the free list mixed with fewer hit blocks. We should differentiate the both cases because of the non-hit blocks amount that is huge and far greater than hit blocks. So we should append the non-hit blocks first and then append the hit blocks so that non-hit blocks are alloc
pull request
[Refactor] Use common sp utils for Qwen3.5 MoE
## Summary This PR enables the common sequence-parallel path for Qwen3.5 MoE, keeps sequence-sharded tensors across attention/MoE boundaries, and restores full-token outputs only where required. It also preserves the Qwen3.5 MTP path by normalizing the local shard before gathering the final draft-model output. ## End-to-end flow ```text Qwen3_5ForConditional
pull request
[Misc] Use VLLMValidationError in pooling input validation
## Purpose Part of #48227. Like #51753, this is an independent file-level Step 5 migration. Migrate four caller-caused validation errors in `vllm/entrypoints/pooling/base/io_processor.py` from raw `ValueError` to `VLLMValidationError`: - conflicting offline pooling tasks - untrusted request-level chat templates - mismatched prompt and pooling parameter count
pull request
NIXL: Use int32 array for indices to avoid intermediate conversion
## Purpose As documented NIXL accepts KV indices as `int32` array, while vLLM creates and passes `int64` array. This results in extra allocation and conversion on NIXL side, which is redundant. Removing this conversion + adding zero-copy on NIXL side improves performance https://github.com/ai-dynamo/nixl/pull/1718 ## Test Result <img width="2250" height="150
pull request
[Mypy Fix] Mypy fix for "vllm/model_executor/models/[cC][dD]"
## Purpose Mypy fix for "vllm/model_executor/models/[cC][dD]" ## Test ```bash pre-commit run --hook-stage manual mypy-3.13 -a Run mypy for Python 3.13.................................................Passed ```
pull request
[Bugfix] Fix .../mrope.py::apply_interleaved_rope() when torch.compile is used in torch==2.13
## Purpose This PR fixes incorrect outputs of `torch.compile` `.../mrope.py::apply_interleaved_rope()` when it's used with `torch==2.13` (which newest vLLM uses), In `torch==2.11` it worked correctly. We fix it by computing the same output in a way that `torch.compile` doesn't break . This method does only indexing and assigning so compiling it should not in
pull request
[Bugfix][ROCm] Stop running shared experts on the aux stream
## Purpose `LM Eval Qwen3-5 Models` on MI355 fails on `Qwen3.5-35B-A3B-DEP2`: ``` FAILED evals/gsm8k/test_gsm8k_correctness.py::test_gsm8k_correctness[Qwen3.5-35B-A3B-DEP2] AssertionError: GSM8K metric too low: 0.0182 < 0.8400 - 0.0300 = 0.8100 ``` Bisect over the window in which the job turned red lands on 47ececb5, "[Perf][ROCm] Dual-stream decode with hip
pull request
[Kernel] Add B12X causal paged attention backend
## Purpose Depends on #52016 for the optional B12X dependency and shared lazy-import integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) causal paged-attention backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing attention backend interface. It does not modify generic model-runner behavior or introduc
pull request
[Kernel] Add B12X FP4 MoE backend
## Purpose Depends on #52016 for the optional B12X dependency, shared lazy imports, packed-storage reuse, and warmup integration. This PR adds an explicitly selected [B12X](https://github.com/local-inference-lab/b12x) FP4 MoE backend for NVIDIA SM120 and SM121 GPUs using vLLM's existing fused-MoE backend interfaces. It does not introduce a new MoE abstractio
pull request
[KV Offload] Fix store threshold admission counting
## Purpose Fix `CPUOffloadingManager.store_threshold` admission so it measures distinct requests that actually produce storable KV blocks. Previously, the reuse counter was updated by `lookup()`. Prefix lookup stops at the first miss, so later store candidates in the same request were not observed. With `store_threshold >= 2`, reusable blocks could repeatedl
pull request
[Perf][ROCm] Dual-stream decode with hipgraphs
## Changes since reverted in https://github.com/vllm-project/vllm/pull/52024 Re-opened https://github.com/vllm-project/vllm/pull/48223 with fix for Qwen3.5-style models (CI failed test_gsm8k_correctness[Qwen3.5-35B-A3B-DEP2] after merging this PR previously). **Changes made since revert:** https://github.com/vllm-project/vllm/pull/52033/changes/50e80bb721d43
pull request
[CI] Force source builds for hybrid dependencies
Hybrid language-model CI began failing before pytest while `mamba-ssm` and `causal-conv1d` probed guessed GitHub release-wheel URLs. The requested CUDA/Torch and ROCm/Torch wheels do not exist, so the normal path is a caught HTTP 404 followed by a source build. During today's intermittent GitHub connectivity problems, some requests instead ended with `Remote
pull request
[Bugfix] Fall back to FA2 for Blackwell head-dim-256 paged attention
## Purpose I came across this issue while enabling ColPali for model runner v2 pooling migration. ColPali MRV2 exposed the issue because its decoder attention supplies `seqused_k,` while FA4’s specialized SM100 head-dim-256 kernel currently rejects `seqused_q/k`. This caused the inference to abort after selecting FA4. This PR consistently resolves FA2 when s
pull request
[Bugfix] Bound KV block zeroing launch geometry
## Purpose Fix the `KVBlockZeroer` launch overflow reproduced on [`main` nightly #83443](https://buildkite.com/vllm/ci/builds/83443/canvas?jid=019ff2a1-641e-4916-9149-30a5074c8a9c&tab=output), at commit [`3e372c5ff2`](https://github.com/vllm-project/vllm/commit/3e372c5ff23438eeeafc86c7d8d51026f3dacb6a): ``` OverflowError: signed integer is greater than maxim
pull request
[Feature] Add native forward-pass metrics emission
## Purpose Related to #38760. Add an opt-in native vLLM forward-pass metrics (FPM) pipeline for production per-iteration observability. This moves production FPM emission out of an external custom scheduler while keeping the disabled path lightweight. The change: - defines the FPM v1 wire contract and utilities together in `vllm/v1/metrics/forward_pass_metri
pull request
[Docs] extract_hidden_states supports chunked prefill
## Summary The `extract_hidden_states` docs state: > Chunked prefill is not compatible with this feature and must be disabled. That appears to be stale. The feature's own integration test exercises chunked prefill deliberately, in `tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py`: ``` 2. **Chunked prefill**: max_num_batched_tokens=
pull request
[Core] Clearer comments in `BlockPool.free_blocks()`
The comments explaining eviction precedence in `BlockPool.free_blocks()` were ambiguous/confusing. Make them clearer / more explicit.
pull request
[Bugfix] Warn when DeepSeek-V4 silently defaults reasoning_effort to "high"
## Purpose Fixes #52083. #50580 changed the DeepSeek-V4 chat encoding (both the Python tokenizer wrapper `vllm/tokenizers/deepseek_v4.py` and the Rust renderer `rust/src/chat/src/renderer/deepseek_v4/encoding.rs`) so that any request which ends up with thinking mode enabled but no explicit `reasoning_effort` now defaults to `reasoning_effort="high"`. That ha
pull request
Auto-ping Cohere on related issues
## Purpose As discussed offline ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results c
pull request
[CPU] Ship triton-cpu wheel and fix several hardcoded pin_memory=True
## Summary - Build and install a pre-built `triton-cpu` wheel in the CPU build/test images instead of `pip install`-ing it from source inside CI, unblocking the Triton topk-topp kernel to run as a normal (non-soft-fail) test. - Move the topk-topp Triton kernel test out of the soft-fail `CPU-ModelRunnerV2 Tests` suite into `CPU-Kernel Tests`, and the linear-a