vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
issue
[Bug]: openai_harmony.HarmonyError: unexpected tokens remaining in message header
### Your current environment <details> I keep hitting this error “ openai_harmony.HarmonyError: unexpected tokens remaining in message header” In multi-turn conversations when using gpt-oss-120b and with both vllm v0.10.1 and v0.10.1.1. I use the official docker image. Seems like a lot of users have this problem. Please also see this issue: https://github.co
pull request
[New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes
## Summary This PR enables DeepSeek V4 Flash on SM120/SM121 Blackwell client hardware by carrying the SM12x fallback and tuning stack needed for the current vLLM V1 path. It targets RTX PRO 6000 Blackwell Workstation Edition, RTX 5090-class SM120, and GB10 / DGX Spark SM121 users who cannot use SM100-only TMEM / `tcgen05` kernels. The branch is **reconciled
pull request
[Spec Decode] DSpark confidence-scheduled verification
Adaptively sizes the DSpark draft-verification budget from per-request confidence instead of always verifying every drafted token. Motivation: fixed-k speculation collapses at high concurrency — once the GPU saturates, verifying 7 drafts per request burns more compute than the accepted tokens return, dropping **below** non-speculative decoding (see table). #
issue
[Bug]: MTP speculative decoding crash with illegal memory access on long sequences (Qwen3.6-27B-FP8, v0.19.1)
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Your output of `python collect_env.py` here ``` </details> ### 🐛 Describe the bug Description When using MTP speculative decoding (num_spec_tokens=5) with the FP8‑quantized Qwen3.6‑27B model as both target and draft model, the engine crashes on
pull request
Add Muse Glimmer model support
Dense 29.6B vision-language model with a ViT-G/14 perception encoder and 128K context. Adds the model, its config and processor, channel-scoped reasoning and ATEM tool-call parsers, and DFlash speculative decoding support for its draft head. The model does not emit JSON tool calls and does not wrap reasoning in <think> tags. Every turn is a sequence of chann
pull request
[Model] Add Inkling multi-depth MTP support [5/N]
## Summary This draft now contains the multi-depth MTP work that remains after the following model slices landed: - #48799: core Inkling model and frontend support - #48822: breakable PIECEWISE CUDA graph support - #48858: Hopper FA4 relative attention - #48869: CUDA-graph-compatible MTP=1 support Additional independent changes were extracted into draft PRs:
pull request
[Feature] Enable AITER MXFP4 MoE on gfx942 and optimize tile configurations for MI325X Target Kimi K3 running on MI325X
## Summary Makes Kimi-K3 run on AMD MI325X (gfx942 / CDNA3). The model loaded but aborted in MLA decode; once past that it fell back to a dequantise-to-BF16 MoE emulation path. Both are fixed, and a third change addresses AITER MoE tile shapes being sized for CDNA4 on all architectures. Measured on 8x MI325X with the real 2.75T-parameter MXFP4 checkpoint: |
pull request
[Core] Extensible (growable) KV cache
> **Stacked on https://github.com/vllm-project/vllm/pull/51718. Only the commits above that PR's head belong to this one. > This is a **draft** — it cannot merge until https://github.com/vllm-project/vllm/pull/51718 lands, after which the base will be retargeted to `main`. ## Summary Opt-in growable KV cache (`--enable-extensible-kv-cache`) built on the stan
issue
[Performance]: Fully Async Spec-Decoding | Make `seq_lens_cpu` in CommonAttentionMetadata optional
### Proposal to improve performance Currently fully overlapping input-prep with model forward pass is blocked in the spec-decode case by the following Host<>GPU syncs: 1) `_get_valid_sampled_token_count` (ultimately needed to compute `seq_lens_cpu`): https://github.com/vllm-project/vllm/blob/fe25772aa97beb8bcb07ea49e06a2892b521a7ed/vllm/v1/worker/gpu_model_r
issue
[Performance]: Qwen 3.5 27B Prefix Caching
### Proposal to improve performance _No response_ ### Report of performance regression There have been discussions if Qwen 3.5 (27B) supports prefix caching in vLLM. - https://www.reddit.com/r/LocalLLaMA/comments/1rplb3r/vllm_prefix_caching_cannot_be_used_with_qwen_35/ - #36010 I would appreciate information on this issue. ### Misc discussion on performance
issue
[RFC]: Packed Variable Length Speculative Decoding
### Motivation. In speculative decoding, we can save verification cost by selectively verifying a different number of draft tokens per request in each step. There are various techniques of selecting and tuning which / how many tokens to submit for verification, but all share a common benefit: reducing verification overhead with per-request variable-length qu
pull request
[Feature][Whisper] Native word-level timestamps (cross-attention + DTW)
Whisper accepts `timestamp_granularities[]=word` today, but `words` always comes back `null`. This fills it in. ```bash vllm serve openai/whisper-large-v3-turbo --enable-word-timestamps curl localhost:8000/v1/audio/transcriptions \ -F file=@mary_had_lamb.ogg -F model=openai/whisper-large-v3-turbo \ -F response_format=verbose_json -F 'timestamp_granularities[
pull request
[Feature] Mask Replay
## Summary This PR adds experimental support for **sampling distribution replay**. A sampling mask represents the vocabulary support retained after top-k/top-p filtering. It is not an attention mask and does not affect causal attention or KV-cache behavior. When enabled, vLLM returns the sampling support for each generated token in a CSR-style representation
pull request
[Model] Apertus 1.5
## General Information This change enables support for serving Apertus v1.5 multimodal (hf tags apertus-ai/Apertus-v1.5-8B, apertus-ai/Apertus-v1.5-70B) Special thanks to @blancsw for creating a workable refactor from https://github.com/swiss-ai/vllm/tree/apertus_integration that optimizes the multimodal inference pipeline and provides a foundation for futur
pull request
[XPU] Add tuned Mamba SSU configs for Intel Arc Pro B70
## Purpose Add tuned `selective_state_update` configs for **Intel Arc Pro B70 Graphics**, and fix `benchmarks/kernels/benchmark_selective_state_update.py` so it runs on an XPU-only build. The config directory holds AMD and NVIDIA devices only, so Mamba/hybrid models on Intel GPUs fall back to the heuristic. Four files, two shapes: | shape | cache_dtype | exe
pull request
[Kimi-K3] Add GEMM-RS for sequence parallelism
## Purpose Add GEMM-RS kernel for Blackwell, based on https://github.com/NVIDIA/cutlass/blob/dcf215a/examples/python/CuTeDSL/cute/blackwell/kernel/distributed/distributed_gemm_reduce_scatter_blackwell.py (`multimem.ld_reduce`) - Supports any value of M e.g. M=1023. However, only uses GEMM-RS when M>=128 since the kernel was not optimized for small/medium M.
issue
[Bug]: vllm/vllm-openai:latest fails to start Gemma4 with Transformers 5.15.0
### Your current environment ```text Docker image: vllm/vllm-openai:latest vLLM: 0.27.0 Transformers: 5.15.0 Model: yasu-oh/gemma-4-31B-it-qat-NVFP4 Tensor parallel size: 2 KV cache dtype: fp8 ``` The Transformers version in the image can be confirmed with: ```bash docker run --rm \ --entrypoint python3 \ vllm/vllm-openai:latest \ -c 'import vllm, transforme
pull request
[Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring
## Summary Replaces the vanilla padded-bmm MaxSim path (introduced in #35330, re-enabled on GPU in #38620) with vendored flash-maxsim Triton kernels for ColBERT / ColPali document scoring. Addresses feature request #38282. ## Why this isn't a duplicate - Issue #38282 explicitly asks for this integration; no other open PR addresses it (searched `is:pr is:open
issue
[RFC]: Support ViT Full CUDA Graph (Tracker)
### Motivation. Multimodal large language models (e.g., Qwen3-VL, Qwen3.5, GLM-V, Kimi K2.5) rely on a Vision Transformer (ViT) encoder to process visual inputs before feeding them into the language model backbone. In production serving scenarios, the ViT forward pass involves launching a large number of small CUDA kernels — including patch embedding, layer
pull request
[Bugfix] Correct prompt lengths for timed_traces benchmark
## Purpose #39795 introduced `timed_traces` support for `vllm bench serve`. The traces look as follows: ``` {"timestamp": 0, "input_length": 6758, "output_length": 500, "hash_ids": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]} {"timestamp": 0, "input_length": 7322, "output_length": 490, "hash_ids": [0, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27
pull request
[Profiler] Add Proton CUDA graph attribution
## Purpose Extend the Proton backend merged in #48789 with opt-in CUDA graph replay attribution. Proton must observe graph creation using the same Data object that later receives replay events. Setting `proton_graph_attribution` to true therefore creates the user-configured Proton session immediately before vLLM enters CUDA graph capture, keeps that session
pull request
[Formatting] Collapse multi-line arg lists where possible
This is a very large PR because it adjusts a ruff formatting rule which affects a lot of files. This might be partially personal preference but I find it much cleaner when lists / method defs are collapsed onto a single line when they fit. It reduces the lines of code and how much vertical real estate you have to scan to understand things. And just looks cle
pull request
[Model][LoRA] Add tower/connector LoRA support for Ultravox
# Purpose Part of #31479: enable `enable_tower_connector_lora` for Ultravox. This is not duplicate work. The open-PR and issue checks for #31479 show this as the only Ultravox tower/connector LoRA implementation; the other open PRs cover different multimodal model families. Ultravox previously could not apply LoRA to its audio tower and connector: - The towe
pull request
[Bugfix] Add Kimi K3 MoE support to benchmark_moe.py
## Purpose `benchmarks/kernels/benchmark_moe.py` cannot tune the fused MoE Triton kernel for Kimi K3. `get_model_params()` does not recognize `KimiK3ForConditionalGeneration`, so it falls through to the Mixtral default branch and raises `AttributeError` (Kimi's config has no `num_local_experts`). Kimi K3 is multimodal and stores its MoE parameters in a neste
pull request
[Model][Spec Decode] Tap the pre-norm AttnRes mixture as the Kimi K3 DFlash aux state
## Purpose The DFlash drafter consumes auxiliary hidden states captured at a fixed set of target layers. K3 captures the post-mixture stream, which is not what the drafter was trained against: the AttnRes residual mixture is applied before the layer norm, and the current capture site reads the value after it. Tapping the pre-norm mixture instead recovers the
pull request
Manual silu quant fusion
Adding more quant types to SiLU+quant activation. Made with the assistance of Claude, all generated code has been human-reviewed. ### Testing: #### Unit test ``` pytest tests/kernels/test_fused_quant_activation.py ``` #### E2E test example (fp8, static) - manually expect if outputs are sane ``` from vllm import LLM, SamplingParams # Sample prompts. prompts =
issue
[Performance]: Inconsistent speculative decoding acceptance metrics between vLLM and SpecForge on Qwen3-32B baselines
### Proposal to improve performance _No response_ ### Report of performance regression _No response_ ### Misc discussion on performance I am benchmarking speculative decoding baselines in vLLM and observed inconsistent acceptance metrics compared with SpecForge. I would like to understand whether this difference is expected due to different metric definition
pull request
[Core][WIP] Check for GPU<->CPU sync during CI
vLLM now uses asynchronous scheduling by default and in the majority of cases. Performance relies on the absence of any gpu<->cpu synchronizations on the main cuda stream, but such syncs can be opaque and it is easy for them to creep in accidentally. This change adds a `VLLM_GPU_SYNC_CHECK` env var which enables `torch.cuda.set_sync_debug_mode` for the model
pull request
[ModelRunner v2] Enable MRV2 for pooling models by default
## Purpose Upstream defaults every pooling model to Model Runner V1 even though the V2 pooling path now supports both encoder and decoder text tasks. This enables MRV2 by default for **native text-only pooling models**, including Qwen3 embeddings, rerankers, and reward/PRM models, while retaining V1 for multimodal models such as ColQwen3, the Transformers ba
pull request
[ROCm] Defer `tilelang` import through its import `from vllm.tilelang_utils import tilelang` and relaxed `has_tilelang`
## Purpose Fixes https://github.com/vllm-project/vllm/issues/51151 This PR introduces `vllm.tilelang_utils` and prevents direct `tilelang` imports. Avoid importing TileLang during ROCm module import, because importing TileLang can load **wrongful/bugged** TVM and HIP stub symbols into the global process scope before AITER loads its JIT modules. This changes
pull request
[Bugfix][Helm] Fix chart resource references
Assisted-by: Codex ## Purpose Fix inconsistent Helm chart resource references when custom labels and autoscaling are enabled. Before this change: - The Service selector used configured labels, while the Deployment selector and Pod labels were hard-coded to `test/test`. - The HPA targeted a non-existent Deployment named `vllm`. After this change, the Deployme
pull request
[K3] support recoverssm for K3
<!-- markdownlint-disable --> ## Purpose This PR adds ReplaySSM speculative decoding for the NVIDIA Kimi-K3 KDA path on Model Runner V2. It is intended for Kimi-K3 serving with DSpark and supports `mamba_cache_mode=align` prefix caching. Without ReplaySSM, KDA speculative decoding materializes a full recurrent state for every speculative position. ReplaySSM
pull request
[Bugfix] Handle DeepseekV4ForCausalLM in benchmark_moe get_model_params
## Summary Fixes [#52042](https://github.com/vllm-project/vllm/issues/52042): `benchmarks/kernels/benchmark_moe.py` crashes for `deepseek-ai/DeepSeek-V4-Flash-0731` with `AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts'`. While fixing it we audited every registered MoE architecture against `get_model_params()`'s dispatch table a
pull request
[Bugfix] Add DeepseekV4ForCausalLM to benchmark_moe.py model param dispatch
## Purpose Fixes #52042. `benchmark_moe.py --tune` crashes for DeepSeek-V4 models (e.g. `deepseek-ai/DeepSeek-V4-Flash-0731`): ```text AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' ``` `get_model_params()` dispatches on the architecture string. The DeepSeek-family branch reads `n_routed_experts` / `num_experts_per_tok` / `moe
pull request
Add `pydocstyle` to the `ruff` rules
Enables the `D` ruleset in ruff and fixes the resulting violations across the codebase. ### Config `D100`–`D107` are ignored, so this does **not** require a docstring on anything that lacks one today. Six more rules are ignored because their remaining violations were either impossible to autofix or actively wrong to autofix: | Rule | Why skipped | |---|---|
issue
[Bug]: vllm 0.23.0 and 0.24.0 - Qwen3.6-35B-A3B-FP8 - Fails generating code- "400 Unterminated string starting at"
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake versio
issue
[Performance]: Improve Pixtral vision attention scaling for batched images
### Proposal to improve performance Pixtral concatenates the patch sequences for all images in an encoder batch, while attention remains semantically independent per image. On the default installation, where xFormers is unavailable, the model constructs a dense block-diagonal mask and applies SDPA to the combined sequence. The resulting attention work and ma
pull request
[ModelRunner V2] Speculative Decoding NGram GPU Implementations
## Purpose 1. Added a new NGram GPU speculator. The main feature is a new implementation at: `vllm/v1/worker/gpu/spec_decode/ngram/speculator.py`, similiar to https://github.com/vllm-project/vllm/pull/29184. 2. Updated request state storage for NGram GPU. `vllm/v1/worker/gpu/states.py` `vllm/v1/worker/gpu/model_runner.py` This changes how RequestState is ini
pull request
[LoRA][Gemma4] Support vision tower LoRA
This PR adds the remaining LoRA plumbing needed for Gemma4 multimodal LoRA support. After #43798, Gemma4-MM vision linear layers are already converted through the Transformers backend path, so this PR no longer reimplements the Gemma4 vision tower. Instead, it focuses on the runtime LoRA mapping and token-counting pieces needed by Gemma4 image/video/audio in
pull request
[Bugfix] Fix MiniMax M3 prompt reasoning initialization
## Purpose Fixes #46042. Fixes #50549. MiniMax-M3 adaptive reasoning can leak a leading `</mm:think>` into streaming `delta.content`. The existing initialization scans the entire rendered prompt with `is_reasoning_end()`. Because the prompt can contain paired `<mm:think></mm:think>` markers as instructions or examples, the parser may incorrectly conclude tha
pull request
[Bugfix] Restore multimodal support on the plain "vllm" throughput backend
## Purpose `vllm bench throughput --backend vllm` crashes on any multimodal dataset (e.g. ASR/Whisper) with: ValueError: Multi-modal content is not supported on backend 'vllm'; use one of []. Regression from #50981. `get_requests()` computed the multimodal-backend allowlist from the backend already selected instead of from what the backend can actually handl
pull request
Feat/spec decode under pipeline parallel
## Purpose EAGLE3-style speculative decoding (`eagle3` / `dflash` / `dspark`) is rejected outright when pipeline parallelism is enabled: ``` ValueError: <method> with pipeline parallel is not supported. ``` The drafter runs on the last PP rank, but it consumes auxiliary hidden states tapped from target layers that may live on earlier stages, and those tensor
pull request
[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec
## Purpose Part of the KV-cache layout standardization series (RFC #42082). **Stacked on #51612** — the diff shown includes it until that lands and this retargets `main`. Attention specs today carry quant-format sizing knowledge inline: `nvfp4` / per-token-head branches in the page-size properties, a `TQFullAttentionSpec` subclass, and fp8_ds_mla constants i
issue
[Bug]: Decode Context Parallelism (`--decode-context-parallel-size`) output drift and gibberish in v0.21.0 and latest nightly
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake versio
issue
[Feature]: Kimi K3 Performance Optimization
### 🚀 The feature, motivation and pitch Tasks - [x] https://github.com/vllm-project/vllm/pull/50383 @jeejeelee - [x] https://github.com/vllm-project/vllm/pull/51146 @jeejeelee - [ ] https://github.com/vllm-project/vllm/pull/50478 @robertgshaw2-redhat - [x] https://github.com/vllm-project/vllm/pull/50484 @GirasoleY - [x] https://github.com/vllm-project/vllm/p
pull request
[Structured Output][Tool Calling] Use xgrammar's reasoning parser instead vLLM's
## Purpose Filter reasoning for structured output and tool calling on Xgrammar's side instead of vLLM's side to simplify logic of scheduler. Recently many bugs related to combinations of structured output/tool callings with reasoning were reported in vLLM. All these issues can be resolved on vLLM's side but requires code changes inside scheduler that makes t
pull request
[KV-offload][FS]: Batching for read/write threads
PR #49152 introduced batched C versions of load_block / store_block. The batching was request-level, i.e. each thread in the thread_pool receives all the keys for a request and loads/stores them one after another. If there are 16 requests, each with say 100 keys to load, and we have 16 threads. Each thread will work on a separate request. This is less desira
pull request
buffer size insuffient Dspark sd for FlashInfer MNNVL allreduce
fixes [#50877](https://github.com/vllm-project/vllm/issues/50877) ## Purpose FlashInferAllReduce.should_use_fi_ar gates on: self.max_num_tokens = max_workspace_size // (hidden_dim * element_size) max_workspace_size is the size of the whole MNNVL allocation (2 MB for TP8). But the MNNVL backend is Lamport-based and rotates through NUM_LAMPORT_BUFFERS=3 buffer
pull request
[Kernel][Perf] Add fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN
## Purpose Speed up Qwen3.5 (GDN linear attention) MTP speculative decode on Blackwell. During MTP decode, the Triton path launches a chain of small kernels per step (gating, delta-rule recurrence, state rewind/update, gated RMSNorm), which leaves the GPU latency-bound at decode batch sizes. This PR adds a single fused CUDA kernel, `fused_gdn_decode_post_con
pull request
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout
## Purpose The core of the KV-cache layout standardization series (RFC #42082). **Stacked on #51704 → #51612** — the diff shown includes them until they land and this retargets. Standardizes every KV cache allocation on the logical `[L, B, H, N, C]` vocabulary: - `KVCacheLayout` enumerates the physical stride permutations (`LBHNC`, `LBNHC`, `LHBNC`, `BLHNC`,