vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Perf][DSV4] Optimize sparse top-k metadata kernels for higher prefill throughput
## Purpose Optimize sparse top-k metadata kernels for higher prefill throughput ## Test ``` vllm bench serve \ --backend vllm \ --base-url http://localhost:8000 \ --model deepseek-ai/DeepSeek-V4-Flash-0731 \ --dataset-name random \ --random-input-len 1024 \ --random-output-len 64 \ --num-prompts 128 \ --num-warmups 8 \ --request-rate inf \ --ignore-eos \ --t
pull request
Fix: Deprecated pylance warnings on using AsyncIterator with asynccontextmanager
## Purpose Fix the deprecated pylance warnings ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pastin
pull request
Revert "[Bugfix] Guard DeepSeek V4 MRV1 piecewise CUDA graphs" (#51768)
## Summary - Restore Model Runner V1 as the DeepSeek-V4 default. - Remove the MRV1 piecewise-graph rejection added by [#51768](https://github.com/vllm-project/vllm/pull/51768). - Stack this change on [#52093](https://github.com/vllm-project/vllm/pull/52093), which first restores the wider eager attention region required for correctness. ## Dependency Stacked
pull request
[ROCm] Reduce AITER MLA FP8 BMM warmup sizes
## Purpose Reduce ROCm AITER MLA FP8 BMM startup work by precompiling only the real decode sizes that are covered by the configured CUDA graph limit. The previous path precompiled every FP8 BMM size from `1..1024` for every MLA layer. This patch keeps a contiguous warmup range, because `MLAAttention.forward_impl` strips CUDA-graph padding before this BMM pat
pull request
[Core][Spec Decode] Add suffix_gpu drafter for Async Scheduling
## Purpose Suffix decoding (`method="suffix"`, #25784) has the best acceptance lengths among model-free drafters on repetitive/agentic traffic, but it is CPU-side and rejected by the async-scheduling whitelist, so it gives up CPU/GPU overlap exactly where spec decode helps most (high-concurrency TPOT). This PR adds `method="suffix_gpu"`: a GPU-resident suffi
pull request
[Core][MoE] Add explicit expert-parallel topology
## Summary - add an experimental `--expert-parallel-size` independent of tensor parallel size - construct fixed-TP-lane EP groups, e.g. TP4/DP2 gives `[0,4]`, `[1,5]`, `[2,6]`, `[3,7]` - preserve existing flattened EP behavior when the option is unset - fail closed for PP/PCP, EPLB, elastic EP, non-AG/RS backends, and EP sizes other than DP This is the topol
pull request
[MoE] Preserve TP sharding with explicit EP
## Summary - preserve physical TP size/rank inside experts when explicit EP is configured - use the fixed-lane EP group for expert mapping and pre-I/O weight filtering - keep AG/RS dispatch/combine over EP and the existing late TP all-reduce - add coverage for TP4/EP2 config, Kimi expert ownership, loader filtering, final TP reduction, and the ROCm latent ta
pull request
[Distributed][MoonEP] BF16 PoC integration of MoonEP balanced EP backend
## Purpose First item of the MoonEP integration roadmap RFC #52095: a correctness-first BF16 proof of concept of MoonEP (https://github.com/MoonshotAI/MoonEP) as a vLLM all2all backend. Included: - `"moonep"` `All2AllBackend` registration and `has_moonep()` probe - `MoonEPAll2AllManager` owning the cached `moonep.Buffer` - `MoonEPPrepareAndFinalize` conformi
pull request
[Bugfix][Multimodal] Guard read_frames against empty frame indices
## Summary `Qwen3VLVideoBackend.compute_frames_index_to_sample()` returns an empty list for a 0-frame video (it lacks the base class's `max(1, ...)` floor), which then crashed `VideoBackend.read_frames()`: - **frame-recovery path** (`_read_frames_with_recovery`): `IndexError: list index out of range` at `next_target_map[frame_indices[-1]] = total_frames` — t
pull request
[Bugfix][Core] Restore incremental thinking-budget marker scans
## Purpose PR #46425 made thinking-budget marker searches incremental by advancing per-search cursors after unsuccessful scans. PR #45984 later removed those cursors while fixing natural `</think>` re-entry and replaced them with a shared `scan_offset` that only advances when a thinking section exits. As a result, while a single long thinking section remains
pull request
[XPU][CI/Release][3/N] Add xpu wheel release to release pipeline
# Summary This PR adds XPU wheel building and publishing support to the Buildkite release pipeline, enabling pre-built XPU wheels to be distributed via wheels.vllm.ai. # Changes Release pipeline (release-pipeline.yaml) - Add a new `Build wheel - x86_64 - XPU` step that builds the XPU wheel using `Dockerfile.xpu` and uploads it to S3. - Rename the existing `P
pull request
[sharded state loader] support pp in sharded state loader
## Purpose For now, sharded state loader does not support pipeline parallel. Add pipeline parallel support for sharded state loader and keep backward compatibility, i.e., tensor parallel only. ## Test Plan Add some ut to cover newly added functionalities. 1. TP only, backward compatibility 2. PP only, save sharded state + load sharded state + generate 3. TP+
pull request
[Bugfix][ROCm] Fix a few int4/int8 quantization errors
## Purpose Fix a few quantization bugs on ROCm introduced by #44120 to enable int4/int8 quantized models like cyankiwi/MiniMax-M3-AWQ-INT4, QuantTrio/Qwen3-235B-A22B-GPTQ-Int8. 1. Add asym quantization support for TRITON moe backend; 2. Add SWIGLUOAI_UNINTERLEAVE activation for TRITON moe backend; 3. Remove wrong assertion (GROUP_SIZE==-1) from MoeWNA16Metho
pull request
[Bugfix][Model] Fix DualChunkRotaryEmbedding hard-coded cuda device crash
## Purpose `DualChunkRotaryEmbedding.__init__` hard-codes `self.device = torch.device(f"cuda:{device_idx}")` alongside the portable `torch.accelerator.current_device_index()`. On any non-CUDA backend the subsequent `.to(device=self.device)` inside `_compute_cos_sin_cache` triggers CUDA lazy-init and crashes model loading with "Torch not compiled with CUDA en
pull request
[Security] Kill regex compilation subprocess on timeout
Replace the thread-based compile_regex_with_timeout() with a fork-based subprocess that is SIGKILL'd on deadline expiry. The previous ThreadPoolExecutor approach could not cancel already-running callables, allowing adversarial regex patterns to accumulate CPU/memory-consuming compiler workers after each timeout error. - Use multiprocessing fork context for f
pull request
[Bugfix] Fall back to native rotary when flash-attn is installed but fails to load
## Purpose `ApplyRotaryEmb.__init__` guards the optional flash-attn rotary import with `suppress(ModuleNotFoundError)`. That covers a *missing* module, but not an *installed but unloadable* one: when the `flash_attn` C extension cannot be loaded, Python raises `ImportError` — the parent class — which escapes the narrower guard and aborts engine startup durin
pull request
fix: prevent PyNvVideoCodec decoder slot limit bypass via ClassVar shadowing
Move mutable decoder-pool state (_decoder_slots, _active_decoder_slots, _decoder_slot_cond, _max_decoder_slots) out of ClassVar attributes on PyNvVideoCodecVideoBackendMixin into a module-level _PyNvDecoderPool singleton. This prevents subclass augmented assignment (cls._active += 1) from creating independent shadow counters per concrete class, which allowed
pull request
[Rust Frontend] Add Hunyuan A13B tool parser
## Purpose The Rust frontend exposes parser selection and automatic model matching, but it did not provide the Python frontend's `hunyuan_a13b` tool parser. Serving `tencent/Hunyuan-A13B-Instruct` through the Rust frontend therefore could not parse the model's `<tool_calls>[...]</tool_calls>` JSON-array format into OpenAI tool-call deltas. This PR adds a Hun
pull request
[Docs] Fix `WhisperEncoderLayer.forward` docstring in `dots3_note`
## Purpose The docs build emits griffe warnings for `vllm/models/dots3_note/nvidia/audio_encoder.py`: ``` WARNING - griffe: audio_encoder.py:351: Parameter 'attention_mask' does not appear in the function signature WARNING - griffe: audio_encoder.py:353: Parameter 'layer_head_mask' does not appear in the function signature ``` The `WhisperEncoderLayer.forwar
pull request
[XPU]bump up vllm_xpu_kernels to 0.1.13.1
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
pull request
[Model][Quantization] Add Ling 3.0 Flash FP4 support
## Purpose Add loading and serving support for [inclusionAI/Ling-3.0-flash-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-fp4). Ling FP4 uses mixed FP8/MXFP4 quantization metadata: ```json { "quant_method": "fp8", "routed_experts_quant_method": "mxfp4" } ``` This PR: - adds a model-specific configuration hook that keeps `Fp8Config` while selecting MX
pull request
[Bugfix] Fix standalone torch.compile cache loading after relocation
## Purpose The standalone Inductor compile cache persists artifact handles as `(key, absolute_path)`. vLLM supports reusing its `torch.compile` cache after copying it to another machine or baking it into a container image. If the cache directory changes location in that process, the artifact exists under the new cache directory, but `InductorStandaloneAdapto
pull request
Fix Inkling structured output stop tokens
<!-- markdownlint-disable --> ## Purpose Fixes #51693. Inkling can emit structural response tokens after completing a JSON-schema structured output. The reported failure token `200028` is `<|begin_of_text|>` in the public `thinkingmachines/Inkling-Small-NVFP4` tokenizer metadata. Because these Inkling structural tokens are not declared through generation con
pull request
[Test] Add pause/resume E2E tests
## Purpose One part of the RL CI Matrix for vLLM proposed by #45585. This PR adds end-to-end tests for the pause/resume API used in RL rollout, covering: 1. is_paused state & idempotency: repeated pause/resume, multiple cycles 2. Invalid mode behavior — return 400; state unchanged 3. Behavior of keep/wait/abort on in-flight and new requests, and behavior aft
pull request
[Bugfix] Handle persistent top-k candidate overflow
## Purpose Fixes #51782. `persistent_topk` uses fixed-size candidate buffers in several optimized dispatch paths. When many scores map to the same coarse histogram bin, those buffers can overflow. The counters keep increasing while excess candidates are dropped, so the kernel returns valid, unique indices that are not the true top-k values. This change: - de
pull request
Fix fused add RMSNorm residual-sum variance
## Purpose Fixes #52104. The CUDA `vllm_c` fused-add RMSNorm kernel was computing variance from the low-precision rounded `input + residual` value. The native IR implementation keeps `input.float() + residual.float()` in FP32 through variance calculation, then rounds only the published residual output. This PR aligns the vectorized and generic `fused_add_rms
pull request
Pass model EOS ids to xgrammar tokenizer info
## Purpose Fixes #52146. `XgrammarBackend.__post_init__` built xgrammar `TokenizerInfo` from the tokenizer's single `eos_token_id`. Models can declare multiple EOS ids in `generation_config.json`, and vLLM's sampling path can stop on those ids. If xgrammar does not know about the full EOS set at tokenizer-info construction time, one of those model EOS tokens
pull request
[Attention][Spec Decode] Support varlen trtllm-gen decode for adaptive verification
## Purpose - Resolves #51871: adaptive verification needs decode batches with per-request query lengths - Wire FlashInfer trtllm-gen decode to `cum_seq_lens_q`/`max_q_len`; `q_len_per_req` must be `None` for varlen - Flip `supports_device_cpu_query_lens_mismatch` on SM100: device `qo_indptr` is the source of truth, CPU lengths only an upper bound - Guard: ad
pull request
[Bugfix][Hardware][Intel] Disable compiled grouped_topk on XPU
## Purpose Unconditional `@torch.compile` on `grouped_topk` emits an XPU Inductor kernel that gathers `scores[topk_ids]` with **out-of-range expert ids**: ``` Assertion `index out of bounds: 0 <= tmp0 < 128` failed. ``` This fires at engine-init warmup for any path that uses the Python grouped-topk router on XPU (NemotronH / Nemotron-3.5 Lightning MTP draft
pull request
[Doc] Fix group numbering in Case 3 of hybrid_kv_cache_manager.md
## Purpose Fix a self-inconsistent group numbering in `docs/design/hybrid_kv_cache_manager.md` Case 3 (lines 111-124). The doc says Gemma-3-27b has 52 sliding-window layers, but lists: - Group 6: 10 sliding window attention layers (sw.40 - sw.49) - Group 7: 2 sliding window attention layers (sw.50 - sw.51) and 8 padding layers With group_size=10, 52 sw layer
pull request
[Bugfix] Detect all attention-spelling variants in ModelConfig.is_hybrid
## Purpose `ModelConfig.is_hybrid`'s carve-out for granite-4.0-micro (a hybrid-capable architecture with zero actual mamba layers) only matched `layer_types` entries spelled `"attention"`. Transformers >= 5.13 normalizes pure-attention entries to `"full_attention"` instead (the model's own `granitemoehybrid.py` already handles both spellings in `ALL_DECODER_
pull request
[Misc][Spec Decode] Detect DeepSeek-V4 DSpark checkpoints from config
## Purpose To close https://github.com/vllm-project/vllm/issues/52111 ## Test Plan Add a test case `tests/config/test_deepseek_v4_dspark_config.py` (TODO) ## Test Result Coming soon. --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR
pull request
[Bugfix] Disable sequence parallelism for Dots3 NOTE
## Why The DeepSeek V3.2 sequence-parallel refactor changed the inherited forward paths to use `use_sequence_parallel`. Dots3 NOTE uses custom model and decoder initializers and has not adopted the new sequence-parallel execution path. This causes serving to fail during KV cache profiling with: `AttributeError: 'Dots3NoteModel' object has no attribute 'use_s
pull request
[Bugfix] Add forward_xpu to XDRotaryEmbedding for HunyuanOCR on XPU
## Purpose `XDRotaryEmbedding` (rope_type `xdrope`, used by `tencent/HunyuanOCR`, arch `HunYuanVLForConditionalGeneration`) overrides `forward_native` and `forward_cuda` but not `forward_xpu`. `CustomOp.dispatch_forward()` binds the method once at init based on `compilation_config.custom_ops`: - Default (Inductor on): `custom_ops=[]` -> `'none'` -> custom op
pull request
[Bugfix][DBO] Run the profile run without DBO when DBO is enabled.
## Purpose Running the profile run with DBO effectively sets the max batch size to `max_num_tokens // 2`. If vllm bails out of DBO for any reason, it can end up in a situation where it's running with a batch size that is larger than `max_num_tokens // 2`. Deep EP v2 crashes in this case because an internal buffer wasn't sized correctly, but it's plausible th
pull request
[Bugfix] Fix PP speculative decode cadence with sync scheduling
## Purpose Fixes #52071. When using the V2 model runner with pipeline parallelism and speculative decoding, the base `Scheduler` can schedule a request again before the sampled token from the previous PP step has been relayed back. The scheduler already checks `request.next_decode_eligible_step`, but this value was only being updated in `AsyncScheduler`. Wit
pull request
feat(inkling): add Eagle3 hidden state extraction support
## Summary Enable speculative decoding training for Inkling models by adding Eagle3 hidden state extraction support. This PR addresses three interconnected issues that prevented hidden state collection from working end-to-end: - **`vllm/models/inkling/nvidia/model.py`** -- Added `EagleModelMixin` to `InklingModel` and `SupportsEagle3` to `_TmlForCausalLMBase
pull request
[Model] Pixtral: use packed multimodal encoder attention
## Purpose Fixes #52180 Pixtral currently concatenates all image patch sequences in an encoder batch. When xFormers is unavailable, it constructs a dense block-diagonal mask and applies SDPA to the combined sequence, causing latency and memory use to scale poorly with the number of images. This PR routes both Pixtral vision implementations through vLLM's `MM
pull request
[XPU][Bugfix] Fix XPU crash with speculative decoding + prefix caching in Mamba models
## Purpose Model tested: cyankiwi/Qwen3.6-27B-AWQ-INT4 HW Backend: XPU Hardware: Intel ARC Pro B70 vllm version: 0.27.1 If I only enable prefix caching, I see no error and vllm can process requests. If I only use spec decoding(MTP) that works fine too. But if I use both together I get following error: ``` ValueError: Overflow when unpacking long long File "v
pull request
[Core][Perf] Zero new KV blocks by cache group
## Purpose #51749 correctly generalized recycled KV-page zeroing to every allocating `AttentionSpec`, preventing stale FP8 sliding-window pages. However, the scheduler currently flattens the newly allocated block IDs from every cache manager, and `KVBlockZeroer` applies that union to every attention-cache segment. On a hybrid or otherwise multi-group layout
pull request
[Spec decode] Support Kimi-K3 DCP with DSpark
## Purpose This PR adds support for running Kimi-K3 decode context parallel with DSpark with FlashinferMLA and Tokenspeed as target causal attention backend and Tokenspeed as the draft non-causal backend. ## Test Plan Kimi K3 GSM8k with the different backend combination. ## Test Result Default (no backend specified): ``` vllm serve moonshotai/Kimi-K3 \ --ten
pull request
[Bugfix][CPU] Fail fast when MLA head dimensions are not supported by the CPU decode kernel
## Purpose Partial implementation of **option B** from #52008: fail fast with a clear error when an MLA model's head dimensions are not supported by the CPU MLA decode kernel, instead of crashing deep inside kernel dispatch after a confusing chain of contradictory mamba / prefix-caching validation errors. The CPU MLA decode kernel (`csrc/cpu/mla_decode.cpp`)
pull request
[CPU] Support FP16/BF16 persisted GDN state on AMX
## Purpose BF16/FP16 GDN state reduces the aligned block size used by hybrid prefix caching and therefore reduce the prefix-caching glassjaw on CPU AMX backends. ## Test Plan Server ```bash vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \ --kv-cache-dtype {auto|fp8} \ --mamba-ssm-cache-dtype {auto|bfloat16} \ --language-model-only \ --enable-prefix-caching \ ``` Client
pull request
diagnostic patch created covering all workspace creation
## Purpose Checks #52023 ## Test Plan This is still in WIP, it is for logs from reporter. ## Test Result WIP --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [
pull request
[LoRA][MoE] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts
# [LoRA][MoE] Fix PEFT 0.18+ `target_parameters` LoRA loading for 3D MoE experts ## Summary PEFT 0.18+ can train a LoRA against a **fused 3D MoE expert parameter** via `target_parameters` (e.g. `mlp.experts.gate_up_proj` / `down_proj`) rather than against an `nn.Module`. When it does, it reads the parameter `[E, dim1, dim2]` positionally as `[E, in=dim1, out
pull request
[Quantization] Add Stage 1 (CPU-only, reference) BitNet ternary quantization backend
## Purpose Adds an initial CPU-only, reference implementation of a BitNet b1.58-style ternary ({-1, 0, 1}) weight quantization backend for linear layers (`vllm/model_executor/layers/quantization/bitnet.py`), registered through the existing `register_quantization_config` out-of-tree plugin mechanism (no core files touched). There is real, currently-unclaimed
pull request
[Mamba][Quantization] Add Stage 1 (CPU-only, reference) int8 quantization utilities for SSM state
## Purpose Adds a self-contained, pure-PyTorch reference implementation of per-channel dynamic int8 quantize/dequantize helpers for the Mamba/SSM recurrent state (`vllm/model_executor/layers/mamba/state_quant.py`), plus a `QuantizedSSMState` wrapper modeling the read/compute/write pattern a real integration would use each decode step. **Motivation:** vLLM al
pull request
[Bugfix][LoRA] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts
# [Bugfix][LoRA] Fix PEFT 0.18+ target_parameters LoRA loading for 3D MoE experts ## Problem PEFT 0.18+ can train a LoRA against a fused 3D MoE expert parameter via `target_parameters` (e.g. `mlp.experts.gate_up_proj` / `down_proj`), reading it `[E, dim1, dim2]` as `[E, in, out]`. vLLM's `add_lora_w13` / `add_lora_w2` punica kernels use the opposite `nn.Line
pull request
[Feature] Add local/external prefix-cache hit breakdown to prompt_tokens_details (additive)
## Summary A **strictly additive** change that exposes the **local vs external** prefix-cache hit breakdown per request in `prompt_tokens_details` (closes #52137). No existing field or behavior changes: `cached_tokens` keeps its exact meaning (total = `local + external`), and the two new fields are `None` by default / omitted from the response unless the fla
pull request
[Fix] Bump lmcache minimum version to 0.3.11
The `get_kv_events()` API used in `LMCacheConnectorV1` is only available starting from lmcache 0.3.11. Bump the minimum version requirement to prevent runtime AttributeError. <!-- markdownlint-disable --> ## Purpose - Bumps the minimum `lmcache` version from `>= 0.3.9` to `>= 0.3.11` in `requirements/kv_connectors.txt` - The `get_kv_events()` API used in `LM