vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
Add vllm_enable_compile_cache config flag with backward compatibility
- Add vllm_enable_compile_cache boolean flag to CompilationConfig (defaults to True) - Keep VLLM_DISABLE_COMPILE_CACHE environment variable for backward compatibility - Environment variable overrides config flag when set via model_validator - Update is_compile_cache_enabled() to accept config flag parameter - Update all test files to use config flag instead
pull request
fused_moe: add VLLM_TRITON_USE_TD tensor-descriptor path
Working on perf optimizations for the Triton MoE kernels. Adds an opt-in `VLLM_TRITON_USE_TD` env var that switches the fused MoE kernel onto a tensor-descriptor based load/store path, mirroring `VLLM_TRITON_ATTN_USE_TD` (PR #40327). Auto-on for XPU; off by default on CUDA/ROCm (opt-in on Blackwell). The changes in this PR are scoped to the fused MoE kernel
pull request
[ROCm][CI] Gating more ROCm tests
Add 22 AMD mirrors for existing CI test groups as the Stage D gating expansion. ### `mi250_1` - Cudagraph - Docker Build Metadata - Kernels Mamba Test - Kernels Helion Test - Language Models Test (PPL) - Language Models Test (MTEB) ### `mi250_2` - Plugin Tests (2 GPUs) ### `mi300_1` - Model Runner V2 Spec Decode - PyTorch Compilation Unit Tests - Speculators
pull request
[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method
## TL;DR ModelOpt linear quantization is implemented today as **six near-duplicate `LinearMethod` classes**, one per format (FP8 per-tensor, FP8 per-channel/per-token, FP8 block-weight-only, NVFP4 W4A4, NVFP4 W4A16, MXFP8). This PR replaces all six with **one generic `ModelOptLinearMethod`**, composed from per-`QuantKey` schemes and driven by a `QuantSpec(we
pull request
[Bugfix][Build] Pin minimal DeepGEMM SM120 and SITU revision
## Purpose Addresses #47436 with a minimal DeepGEMM dependency revision that combines: - `a6b593d2826719dcf4892609af7b84ee23aaf32a`, the revision previously used by vLLM for SM120 support. - The SITU change from `f5a76426fa084087169693fd0cd815223576d6e9`, cherry-picked directly on top as `586032eba787dd99e3d1c069f8514ab01ca64906`. - A one-line follow-up impo
pull request
fix(v1): decouple async Mamba align D2H counts from InputBatch row shifts (#51571)
### Target Issue Closes #51571 ### Description When running speculative decoding / MTP in `align` mode with `use_async_scheduling=True`, `GPUModelRunner._update_states_after_model_execute()` passes `input_batch.num_accepted_tokens_cpu_tensor` as the D2H target for `postprocess_mamba_align_gpu()`. However, while the GPU D2H copy is in flight on the CUDA strea
pull request
[Bugfix][KV Offload] Emit self-describing CPU events at KV-group block granularity
## Purpose Fix self-describing CPU KV events for hybrid KV-cache layouts where request hashes are computed more frequently than the full-attention group's block size. For example, DeepSeek V4 may use: - tokens_per_hash = 4, derived from the GCD of its KV-group block sizes - tokens_per_block = 256 for the full MLA group The offloading event tracker currently
issue
[Feature]:[New Model] Gemma4UnifiedForConditionalGeneration (google/gemma-4-12B-it)
### The model to consider. `google/gemma-4-12B-it` — `model_type: gemma4_unified`, architecture `Gemma4UnifiedForConditionalGeneration` (Google's unified text+vision+audio "omni" Gemma 4). Public repo; ships no remote code (no `auto_map` / modeling `.py`). ### The closest model vLLM already supports. vLLM registers `Gemma4ForCausalLM` and `Gemma4ForCondition
issue
[RFC]: Context-length-aware speculative token scheduling — extending num_speculative_tokens_per_batch_size with a context-length axis
## Summary `num_speculative_tokens_per_batch_size` schedules the speculative depth K by batch size. We propose extending each entry with an optional, backward-compatible **context-length range**, so the runtime picks K from a `(batch, ctx)` table instead of a batch-only table: ```jsonc // today (unchanged, still valid): "num_speculative_tokens_per_batch_size
issue
[ROCm][AMD] Kimi-K3 Gap and Roadmap Tracking
Kimi-K3 is a very successful launch. This issue is to track vLLM upstream work for feature enablement and performance optimization on ROCm. <details><summary>Day 0 features/baselines</summary> - AITER fused-moe: a16w4 (GENERAL) and a8w4 (INTERLEAVE ) were integrated for performance. Flydsl/opus a8w4 is perf is 3.5% higher than flydsl a16w4. - AITER MLA atten
issue
[Bug]: upgrade vllm from 0.26.0 to 0.27.0 run deepseek v4 flash error
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text python collect_env.py Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : 18.1.3 (
pull request
[CI] Add persistent cache mounts and fix test download paths
This PR avoids cold network downloads during AMD CI test collection and execution. The immediate motivation was a Buildkite multimodal shard spending time downloading before the selected tests even ran: https://buildkite.com/vllm/ci/builds/76781/canvas?jid=019f3b96-c3dd-4f37-98b2-0599cfe73733&tab=output. The biggest issue was `test_phi4mm.py` resolving the 1
pull request
AMD remove sync visible devices
## Purpose Use HIP_VISIBLE_DEVICES and device control environment variable on ROCm platform. Aligning with Pytorch and Ray frameworks. Syncing with CUDA_VISIBLE_DEVICES is no longer necessary ## Test Plan vllm serve with ray backend `vllm serve Qwen/Qwen2-7B-Instruct --tensor-parallel-size 4 --pipeline-parallel-size 2 --distributed-executor-backend ray --gpu
pull request
[EPD] Add ECMooncakeConnector for encoder cache over Mooncake TransferEngine
Wire factory and ec_transfer config; add two-process e2e test, EPD full-pipeline script, and README notes. ## Purpose - Add **`ECMooncakeConnector`**: encoder-cache (EC) transfer over **Mooncake TransferEngine** (HTTP registry + ZMQ coordination + pull path), for disaggregated setups where consumers load EC tensors without relying on shared filesystem. - Reg
pull request
[Bugfix][Kernel] Fix persistent top-k histogram reuse after short rows
<!-- markdownlint-disable --> ## Purpose Fix a correctness bug in `persistent_topk` when one persistent CTA group processes a radix row (`seq_len > 32768`), followed by a short or medium row (`seq_len <= 32768`), and then another radix row. The kernel previously used the outer row iteration counter to rotate its triple-buffered radix histograms. Short rows a
pull request
fix(security): enforce audio decode duration limit in NanoNemotronVL
The _extract_audio_from_videos method called load_audio_pyav without max_duration_s, allowing a small compressed video to decompress into gigabytes of PCM and crash the server via OOM. Pass VLLM_MAX_AUDIO_DECODE_DURATION_S to match the safeguard already used by AudioMediaIO. This should be merged only when https://github.com/vllm-project/vllm/pull/49948 bran
pull request
[Core][Spec Decode] Add lookahead-aware prefix cache hashing for EAGLE-style draft models
## Purpose Implement https://github.com/vllm-project/vllm/issues/50438 EAGLE-style draft KV at a block boundary depends on both the target prefix through that block and the finalized token immediately following it. The existing cache key proves only the block prefix, so vLLM conservatively drops the last matched unit and recomputes it. That cost is especiall
pull request
[Bugfix][Kernel] Fix divergent warp collectives in partial NeoX QK-Norm+RoPE
## Summary ## Author: Drew Jin, SJTU & Qwen Team, <drewjin0827@sjtu.edu.cn> Solving Issue: https://github.com/vllm-project/vllm/issues/51049 This PR fixes silent Q/K corruption in the partial-NeoX path of `fused_qk_norm_rope`. When `rotary_dim < head_dim`, the multi-head-per-warp kernel produces incorrect Q/K values for every head after the first in each war
pull request
[Frontend] Move api_server.py out openai folder
## Purpose Following https://github.com/vllm-project/vllm/pull/41907 > In the early days of vLLM (2023), there was only the OpenAI, so vLLM online serving was referred to as the OpenAI-Compatible Server. We'd better move api_server.py out openai folder ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Check
issue
[Bug]: sleep level 2 causes gibberish outputs
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text ============================== System Info ============================== OS : Ubuntu 20.04.6 LTS (x86_64) GCC version : (Ubuntu 10.5.0-1ubuntu1~20.04) 10.5.0 Clang version : Could not collect CMake version : version 3.27.7 Libc version : glibc-
issue
[Bug]: cudaErrorIllegalAddress in gdn_attn.py:237 when using qwen3_next_mtp with num_speculative_tokens=5 under load
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake versio
issue
[RFC]: KV offload event path refactor — provenance-carrying events and key-only removals
### Motivation. > Code references are pinned to `main` at > `3d204dfdaaf09d67d49c7855630ef949754e0f8f`; implementation starts from > current `main`. [#48679](https://github.com/vllm-project/vllm/pull/48679) added self-describing KV-offload events as an initial implementation. This RFC finishes the event-path refactor in two PRs. Today, `OffloadKey` contains
issue
[Tracking][Spec Decode] Adaptive DSpark Bring-Up Tracker
Tracking issue to organize the adaptive verification support for DSpark speculative decoding ## Backend Support Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing
pull request
Change the default value of truncate_prompt_tokens in the embedding/rerank/pooling model to -1
Change the default value of truncate_prompt_tokens in the embedding model to -1，By default, the model is truncated according to its maximum length. ## Purpose The client no longer needs to worry about the maximum length supported by the model and will not report an error if the input text is too long ## Test Plan work wll on bge-m3 ## Test Result --- <detail
pull request
[Spec][V2] Support MTP speculative decoding under pipeline parallelism
## Purpose MTP speculative decoding does not currently work under pipeline parallelism on the V2 model runner. This PR makes it functional for DeepSeek-family MTP drafts (DeepSeek-V3, GLM-5.2, Qwen3.5/3.6, …). Five independent issues, all on the PP>1 path. Fixes #1-#3 are DeepSeek-family-specific; fix #4 applies to all models using sparse MLA attention; fix
pull request
[Spec Decode] Context-length-aware K in DSD (RFC #48627): extend num_speculative_tokens_per_batch_size with a ctx axis
## What this PR does Extends `num_speculative_tokens_per_batch_size` (Dynamic SD, #32374) with an optional ctx-length axis, so that a schedule can select K based on both batch size and per-request context length instead of batch size alone. The change is a backward-compatible schema extension — a 3-item entry `[bs_lo, bs_hi, K]` parses and behaves exactly as
pull request
[Bugfix][Spec Decode] Fix autoregressive draft decode capture with dynamic SD
## Purpose Fixes #48494. Dynamic speculative decoding (DSD) derives the decode query lengths for each scheduled speculative-token count from the manager's configured `decode_query_len`: ```text num_new_sampled_tokens = decode_query_len - max_speculative_tokens runtime_query_len = scheduled_speculative_tokens + num_new_sampled_tokens ``` This derivation is va
pull request
[Spec Decode][Perf] Fuse the MTP trailing all-reduce; local-argmax draft tokens
Two optimizations on the DeepSeek-V3.2 / GLM-5.2 MTP draft path. - **Fuse the trailing all-reduce into the final RMSNorm** on the non-sequence-parallel path, as the main model already does at layer boundaries. The sequence-parallel path is unchanged. - **Greedy draft tokens via vocab-parallel local argmax** (`get_top_tokens`), skipping the full-vocab all-gat
pull request
[MRV2][Multimodal] Enable encoder cuda graph for model runner v2
## Purpose - Enable encoder cuda graph on model runner v2. ## Test Plan ``` pytest -s -v tests/models/multimodal/generation/test_vit_cudagraph.py ``` ## Test Result All tests should pass --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [x] The purpose of the PR, such as "Fix some issue (link existing issues thi
pull request
[ROCm]Remove special-case SiTU support model-specific gating
## Purpose Mxfp4MoEMethod previously contained a model-specific predicate (_use_k3_situ_aiter) that special-cased the Kimi-K3 SiTU activation, gating three separate code paths — backend selection in __init__, size round-up bypass in maybe_roundup_sizes, and an entirely separate weight-shuffle method _setup_kernel_k3_situ in process_weights_after_loading. Thi
pull request
[Bugfix] Keep Qwen3Next layer boundaries sequence parallel
<!-- markdownlint-disable --> ## Purpose Fixes #50681. Qwen3.6-35B-A3B produces corrupted output during single-token decode when expert parallelism and MoE sequence parallelism are enabled with DP2 and TP2. The Qwen3Next model path infers the hidden-state layout from the first tensor dimension, but during TP2 single-token decode the full input and each padde
pull request
[Bugfix][Mamba] Fix overlapping state copy race
[PR #30877](https://github.com/vllm-project/vllm/pull/30877) introduced generic Mamba state copies, and [PR #40172](https://github.com/vllm-project/vllm/pull/40172) added the fused GPU copy. A speculative-decode convolution-state shift can copy within the same physical block with overlapping source and destination ranges. Parallel memcpy-style loads/stores d
pull request
[Bugfix] Report FULL_ATTENTION for uniform-base UniformTypeKVCacheSpecs groups instead of UNKNOWN
## Problem `get_kv_cache_spec_kind()` returns `KVCacheSpecKind.UNKNOWN` for a `UniformTypeKVCacheSpecs` whose members have more than one *inner* kind: ```python if isinstance(kv_cache_spec, UniformTypeKVCacheSpecs): inner_kinds = {get_kv_cache_spec_kind(spec) for spec in kv_cache_spec.kv_cache_specs.values()} if len(inner_kinds) == 1: return next(iter(inner_
pull request
[K3 Perf] Flash kda out kernel for prefill, 1.1~1.4x kernel performance improvement
## Purpose Use workspace manager to preallocate the memory for `_flashkda_prefill`, avoid re-allocate each time when we call the kernel. ## Test Acc covered in unit tests Perf can be seen in this AI generated script ```py import statistics import time import torch import vllm._flashkda_C # noqa: F401 from vllm.models.kimi_k3.nvidia.kda import _flashkda_prefi
pull request
[Bugfix] Make DSV4 sparse MLA work end-to-end for plain decode, MTP, and DSpark
## Purpose DeepSeek-V4-Flash-0731 could not run reliably through the SM120 sparse MLA backend. This fixes the seven defects that blocked it across all three decode modes -- plain decode, MTP, and DSpark -- verified end-to-end on 8xRTX PRO 6000 Blackwell across in-flight batching and prefill/decode disaggregation. Commits 1-5 unblock DSpark. Commits 6-7 fix a
pull request
[Build] Update DeepGEMM pin to deepseek-ai nv_dev tip
## Purpose Update both DeepGEMM pins (`cmake/external_projects/deepgemm.cmake` and `tools/install_deepgemm.sh`, which are documented to stay in sync) from the `vllm-project/DeepGEMM` fork at `e21c821f` to upstream `deepseek-ai/DeepGEMM` at `8b1392b978f5a03c828dd1711090d7fb50958b8a`, the current tip of the `nv_dev` branch. The fork pin was kept because the pl
issue
[Bug]: draft_model speculative decoding crashes at init under TP>1 when draft hidden_size > target (TRT-LLM fused allreduce+RMSNorm workspace sized from target only)
### Summary With `draft_model` speculative decoding under tensor parallelism (`TP>1`), the engine crashes at init when the **draft model's `hidden_size` is larger than the target model's**. The `fuse_allreduce_rms` compilation pass builds the FlashInfer TRT-LLM fused allreduce+RMSNorm workspace with a token budget derived from the **target** model's `hidden_
issue
[Bug]: Rust frontend renders caller-supplied chat templates with no evaluation budget: a 116-byte request body costs 55 CPU-seconds
### Your current environment vllm/collect_env.py output is not included: this reproduces entirely against the Rust workspace under rust/, driven by cargo test and the in-tree mock engine. The vLLM Python package is not installed on the repro machine and there is no GPU, so collect_env.py would report N/A for essentially every field it exists to collect. Noth
pull request
[Responses API] Fix truncation: auto returning 400 instead of truncating
## Summary Fixes #38132 When using the Responses API with `truncation: "auto"`, sending input that exceeds the model's context window returns a 400 error instead of truncating the input to fit. **Root cause**: The Harmony code path (used by `gpt-oss` models) bypasses the standard renderer/`TokenizeParams` pipeline and directly tokenizes via `render_for_compl
pull request
[MM][CG] Enable encoder CUDA Graph for MiniCPM-V
## Purpose Add encoder CUDA Graph support for MiniCPM-V 2.5, 2.6, 4.0 as part of tracker #38175. This implementation follows the existing workflow introduced in #38061. The captured graph covers both the ViT encoder (VPM) and the resampler. MiniCPM-V 2.0 is not included, as it predates the slice-based vision architecture required by this implementation. Mini
pull request
Fix sparse BlockStored event token/hash mapping
## Purpose Fixes #44451. `BlockStored` events can become ambiguous when a KV cache group skips logical blocks, such as Mamba groups with `--mamba-cache-mode align`. Before this change, `block_hashes` and `extra_keys` only included emitted non-null blocks, while `token_ids` still covered the full logical token range. This could produce sparse events like one
pull request
[XPU] eplb
Enable XPU eplb and add TorchDistXCCLStagedEplbCommunicator for this. Torch_xccl and torch_gloo can both be used on XPU. ``` model=Qwen/Qwen3-30B-A3B VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model $model --enforce-eager --port 30088 --trust-remote-code --gpu-memory-util=0.9 --no-enable
pull request
[Perf][ROCm] Add AITER custom AG/RS
## Purpose Part of #48255. Only activated under uniform batches. Enabled by default when using DP attention. Same as ATOM. Disable with `VLLM_ROCM_USE_AITER_CUSTOM_AR=0`. **Perf gain:** **~3% improved TPOT on 1k/1k** (1-256 conc) ### Technical details - AG: Runs as 3 separate calls for hidden, topk_weights, and topk_ids. AITER doesn't have a grouped AG unfor
pull request
[Kernel] Gemma-4 FA4 FP8 Kernel
<!-- markdownlint-disable --> ## Purpose Gemma-4 uses 256-wide heads in `sliding_attention` and 512-wide heads in `full_attention`. On SM90, the full-attention layers upgrade from FA3 to the FA4 CuTeDSL kernel. This PR wires the FA4 FP8-KV-dequant path from [vllm-project/flash-attention#164](https://github.com/vllm-project/flash-attention/pull/164) into vLLM
pull request
[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses
## Purpose Speculative decoding guesses several tokens ahead, then checks them. Some guesses get accepted, some don't. How many get accepted tells you how well spec decode is working. Today you can only see this as a server-wide average on the `/metrics` page. You can't tell how any single request did. Tools like AIPerf want that per-request number, and righ
pull request
[ROCm] Switch to the Rock, Keep Python 3.12 and Ubuntu 22.04
## Purpose This PR switches to using The Rock 7.14 with wheels provided by The Rock while keeping Python 3.12 and Ubuntu 22.04 as-is. Need to wait for @Rohan138 to bump to torch 2.12, then will need to rerun testing. We could possibly add` /ready` to this and retry as needed when it becomes convenient. ## Test Plan Full CI runs. ## Test Result Currently fail
pull request
[ROCm] [bugfix] Chunked prefill paged decode masked load perf
## Purpose Observed 10-15% latency performance regression for serving Qwen/Qwen3-30B-A3B-Thinking-2507; trace revealed that `kernel_paged_attention_2d` was the culprit, taking 1.33x as long on average on v0.25.0 vs v0.24.0. #47305 fixed correctness but introduced performance drop because of universally applied masking. Edited masking so that it is only enact
pull request
[SM120] Add NVFP4 KV cache support for consumer Blackwell (RTX 5090)
## Summary - Fix V scale swizzle mismatch: the NVFP4 KV store kernel unconditionally swizzled V block scales for the SM100 trtllm-gen MHA kernel, but FlashInfer FA2 (used on SM120) reads scales linearly. Add runtime SM detection so SM≥120 writes V scales without swizzle, resolving garbled output on RTX 5090. - Enable FlashInfer NVFP4 KV cache on SM120: `supp
pull request
[Core] Fix ValueError on KV load failure with a hybrid KV cache
## Purpose `_update_requests_with_invalid_blocks` unpacks `get_block_ids()` -- which returns one block-id list per KV cache group -- into a single name: # TODO (davidb): add support for hybrid memory allocator (req_block_ids,) = self.kv_cache_manager.get_block_ids(req_id) so the first reported invalid block on a hybrid (Mamba/SSM plus attention) model raises
pull request
[Kimi-K3][AMD] Return KDA and MLA projection outputs directly
## Purpose Kimi-K3's AMD attention paths copied each output projection into a caller-owned buffer even though the projection already returns a tensor with the required shape, dtype, layout, and lifetime. This PR removes that redundant post-projection allocation and copy from both attention families: - 69 KDA layers - 24 MLA layers - 93 post-attention-project