vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[DCP] Default query replication for GLM sparse attention
## Summary - add `--dcp-q-replicate` and `--no-dcp-q-replicate` as explicit CLI options; - preserve `VLLM_DCP_Q_REPLICATE` as the highest-precedence override; - enable query replication automatically for `glm_moe_dsa` models when decode DCP is enabled and PCP is not enabled; - keep query replication opt-in for other MLA model families. The automatic default
pull request
[Bugfix][Structured Output] Mask request stop tokens in xgrammar until grammar terminates
## Purpose Following #49227's merge, remove the patch used in HarmonyParser and update test to mirror production path. ## Test Plan `pytest tests/parser/test_harmony.py` ## Test Result `64 passed, 38 warnings in 26.74s` --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix som
pull request
[Docs] Fix dead class references in MoE design docs
## Purpose Six symbol references in the two MoE design docs no longer resolve against main. `docs/design/moe_kernel_features.md`: - The all2all backend table lists a `naive` backend whose Subclass column is an autoref to `vllm.model_executor.layers.fused_moe.layer.FusedMoE`. That class no longer exists, so the cross-reference cannot resolve. `naive` is also
pull request
docs(serving): note OpenAI client base_url for multi-model gateways
## Summary The OpenAI-compatible server docs already show the OpenAI Python client via `base_url`. This PR adds a one-line tip that the same client pattern works with OpenAI-compatible multi-model gateways when not running vLLM locally, using [DaoXE](https://daoxe.com) (`https://api.daoxe.com/v1`) as one concrete example. Docs only — no runtime behavior chan
pull request
[Bugfix][R3] Size monolithic routing replay buffer for DP
## Purpose Fix routing-replay capture for the FlashInfer monolithic MoE kernel under naive data parallelism, including padded sequence-parallel shards when expert parallelism is enabled. Two related assumptions fail in a TP2/DP2 deployment: 1. **Replay buffer capacity.** `max_num_tokens` is a per-rank scheduler limit, while the naive dispatch path all-gather
pull request
[Bugfix] Preserve the audio duration-limit error through the PyAV fallback
## Purpose load_audio wraps every failure on its PyAV fallback path as "Invalid or unsupported audio file." including the rejection raised by the VLLM_MAX_AUDIO_DECODE_DURATION_S guard itself. For containers soundfile cannot probe (like WebM), audio over the duration limit therefore reports a misleading invalid-file error, while soundfile-readable formats (m
pull request
[ROCm][K3] Extend FP8 asm MLA prefill to non-divisor small head counts
## Context The AITER FP8 MLA **prefill** path (`mla_prefill_ps_asm_fwd` + `mla_reduce_v1`) is gated on `num_heads % 16 == 0`. Kimi-K3 has 96 heads over `kv_lora_rank=512` → **12 heads/rank at TP8**, a non-divisor of 16, so its FP8 prefill falls back to the BF16 FMHA decompress path. That fallback builds a bf16 working set not covered by the FP8 KV-pool accou
pull request
[Bugfix] [Build] Fix DeepGEMM SM90 paged mqa `prefix_sum` out of range
## Purpose Fix #51181. DeepGEMM SM90 paged mqa logit contains an out-of-index error which is easily hit in common device setup (DeepSeek V4 DSpark on H100 and common `max-num-seqs`). This PR prevents the out-of-index error (`CUDBG_EXCEPTION_WARP_OUT_OF_RANGE_ADDRESS` on CUDA coredump) by building DeepGEMM from bugfixed branch (vllm-project/DeepGEMM#2). **TOD
pull request
[Feature] IndexCache for DeepSeek-V4 (validated on V4-Flash-0731, including DSpark on to accelerate on top of
## Summary: This work is built to support Dspark + IndexerCache to achieve best deepseek v4 flash serving performance. ## Purpose Adds DSA IndexCache support to DeepSeek-V4, so C4A layers marked shared reuse the top-k the previous C4A layer left in the shared `topk_indices_buffer` instead of running their own indexer. Same config surface as the DeepSeek-V3.2
pull request
[Rust Frontend][gRPC] Add RL lifecycle control
## Purpose Add reinforcement-learning lifecycle operations to the Rust frontend's existing gRPC `Control` service. - Add pause and resume RPCs for scheduler control. - Add sleep and wake RPCs for GPU memory management. - Add weight-transfer initialization, start, update, finish, and version RPCs. - Advertise RL capabilities through the EngineCore ready hands
pull request
[Bugfix][DSv4] Make the C128A decode topk row stride capture-stable
# [Bugfix][DSv4] Make the C128A decode topk row stride capture-stable Fixes the intermittent output corruption on DeepSeek-V4-Flash under concurrent long-context load reported in the #41834 thread (mid-generation `<|begin_of_sentence|>` bursts, multilingual token salad — one request per batch, never the first). Root-cause analysis and A/B evidence: https://g
pull request
[Bugfix][Spec Decode] Fix DSD K-lookup to count sampling-only requests, preventing K thrashing at batch boundaries
> [!IMPORTANT] > **Withdrawn after controlled follow-up (2026-08-13).** This PR correctly > demonstrates that total-request and sampling-request lookup can choose > different K values, but I had not established that sampling-only was the > correct policy or that it reduced K thrashing. A later 25-pair controlled > experiment found no target workload meeting
pull request
[Perf] Avoid a per-prefill-step GPU->CPU sync in the KDA chunk kernels
`prepare_chunk_indices` derives per-sequence chunk counts with `.tolist()`, so passing it a device `cu_seqlens` blocks the caller. The count also fixes the Triton grid, so it has to reach the host either way; compute it from the host-side cu_seqlens we already have and copy the result across, as `gdn_attn` already does for its own prefill slice. Models from
pull request
[Doc] Fix stale rejection_sample_method and synthetic_acceptance_rate
## Purpose Fixes #51609 Sync the `--speculative-config` table in `docs/features/speculative_decoding/README.md` with the current code: 1. `rejection_sample_method` (line 87): `strict, probabilistic, synthetic` (default `strict`) -> `standard, synthetic, block` (default `standard`); `probabilistic` now belongs to `draft_sample_method`. See #40651. 2. `synthet
pull request
[Bugfix] Clamp block table indices in align mode to prevent OOB gather
In mamba_cache_mode='align' with speculative decoding, sequences with seq_len near max_model_len caused indices_to_gather to exceed the block table column count, triggering a CUDA device-side assert in torch.gather. Fixes #42084 ## Purpose Fix an out-of-bounds `torch.gather` in `mamba_get_block_table_tensor` when `mamba_cache_mode='align'` is used together w
pull request
[Bugfix][ROCm][CI] Restore the DeepSeek-V4 input GEMM override point
## Purpose The GSM8K accuracy job for `amd/DeepSeek-V4-Flash-NVFP4` on gfx950 reports 0.0000 against a threshold of 0.92. It is not a near miss: the server comes up clean and then answers every one of the questions with an unparsable run of repeated tokens, so the failure is in the numerics rather than in the harness. A bisect over the window in which the jo
pull request
[ROCm][K3] Dequantize the fp8 decode query for MLA backends without quant-query support - TRITON_MLA
## Purpose When enabling fp8 kv-cache dtype in DSpark speculative decoding, we got an assert error. THis PR is to put out a minimal fix to unblock this path. On ROCm, the DSpark draft auto-selects TRITON_MLA, the only backend supporting its non-causal multi-token blocks atm. TRITON_MLA dequantizes fp8 KV on load and takes a bf16 query, so Kimi-K3's `_decode_
pull request
[ROCm][CI] Speed Up ROCm Skinny GEMM Tests (reduced parameterizations,
This PR reduces the `test_rocm_skinny_gemm` suite from a ~2 hour runtime to about 10 seconds by reducing the number of parameterizations from 11040 to 2644, and removing the unnecessary environment cleanup between tests. We now just cleanup the environment one time at the end of the entire module rather than needlessly eating the 0.3s cost to cleanup after e
pull request
Reject oversized media before fully downloading it
# Reject oversized media before fully downloading it ## What this fixes Several media paths checked size only after a remote response or inline audio value had already been turned into one complete `bytes` object. Batch speech requests could also download several oversized rows concurrently before applying their per-row limit. For example, with a 1 MiB limit
pull request
[CI/Build] Add warning for unsupported global PTX architecture requests in...
... CMake configuration. ## Purpose Implements one item from #9129: Warn that PTX builds are not currently supported (post [CI/Build] Per file CUDA Archs (improve wheel size and dev build times) #8845), currently if there is a +PTX in TORCH_CUDA_ARCH_LIST this will be ignored. We should warn when this is the case Users can request PTX through `TORCH_CUDA_ARC
pull request
[Frontend] Add routed-experts prompt offset
## Summary - Add `routed_experts_prompt_start` to OpenAI chat/completion requests and `SamplingParams`, allowing clients to omit an already-known prompt prefix from returned R3. - Centralize NumPy-to-base64 serialization used by existing R3 responses and document the `int32` expert-ID representation. - Keep OpenAI streaming behavior unchanged: R3 remains sup
pull request
[Model] Enable LoRA support for tower and connector in Cosmos3-Edge
## Purpose An operator serving `nvidia/Cosmos3-Edge` can attach a LoRA adapter to the language model, but not to the vision **tower** or the **connector**, so visual feature extraction and its projection into the language model's space are frozen. Adapting the model to a new visual domain — a different imaging modality, an unusual camera, a specialised docum
pull request
[Build] DeepGEMM pin has no SM120 kernels: family-12 Blackwell cannot run hyperconnections
## The problem DeepGEMM dispatches the TF32 hyperconnection pre-norm GEMM on the device arch major, in `csrc/apis/hyperconnection.hpp`. At the revision this repository pins, `vllm-project/DeepGEMM` `e21c821`, that dispatch handles `arch_major` 9 and 10 only: ```cpp const auto arch_major = device_runtime->get_arch_major(); if (arch_major == 9) { sm90_tf32_hc_
pull request
[Bugfix][Structured Outputs] Reject empty JSON schemas
## Purpose Fix #52011 by rejecting an empty JSON Schema before structured-output backend selection. The empty schema `{}` is a valid universal JSON Schema, but it leaves number generation unconstrained. If decoding enters the number branch, it may continue producing digits until `max_tokens` and return an unusable response with `finish_reason="length"`. This
pull request
[Bugfix] Pin DeepEP by its full commit hash
## Purpose `tools/ep_kernels/install_python_libraries.sh` pins DeepEP by a 10-character abbreviation: ```bash DEEPEP_COMMIT_HASH=${DEEPEP_COMMIT_HASH:-"d4f41e4e93"} ``` An abbreviated object ID is not a ref, so it cannot be fetched directly. GitHub serves any *complete* commit via `allowAnySHA1InWant`, but an abbreviation is never a valid want: ```console $
pull request
[Bugfix] VLLM 0.27.0 has not been fixed MiniCPM-V-4.6 startup fails
## Purpose Fixes #51842. Serving `openbmb/MiniCPM-V-4_6` fails during weight loading, before the server starts: ValueError: There is no module or parameter named 'k_proj' in MiniCPMV4_6ViTWindowAttentionSelfAttn. The available parameters ... are: {'qkv_proj.weight', 'qkv_proj.bias', 'out_proj.weight', 'out_proj.bias'} **Root cause.** The MiniCPM-V-4.6 ViT at
pull request
[Build][AMD] Update triton_kernels pin for RDNA MXFP4 matmul_ogs config
Advances the ROCm triton_kernels pin to 74e4569a70, which backports upstream Triton PR #10810 ("[AMD][kernels] Improve MXFP4 matmul_ogs config on RDNA"). Avoids register spilling in the MXFP4 MoE GEMMs on RDNA. ## Purpose Advances the ROCm `triton_kernels` pin in `cmake/external_projects/triton_kernels.cmake` from `0f380657` to `74e4569a70`, picking up the R
pull request
[Frontend] Log output token IDs at DEBUG level
## Purpose Allow operators to keep human-readable generated output logs without emitting output token IDs at the default INFO level. Following maintainer feedback, this now mirrors the existing request-input logging split instead of adding a new CLI flag: - INFO keeps generated text and the finish reason. - DEBUG additionally logs output token IDs. - `--max-
pull request
[Bugfix] Do not require SupportsPP of draft models under pipeline parallelism
Fixes #52069. `create_draft_parallel_config` copies the target's `pipeline_parallel_size` into the draft's parallel config, so `_verify_args` demands `SupportsPP` from the draft model. No head in `MTPModelTypes` implements it, so every MTP speculative config dies in `create_engine_config` under PP > 1. A draft model always runs whole on a single PP rank (the
pull request
[Core] Update PyTorch to 2.14.0, torchvision to 0.29.0, triton to 3.8.0 (test channel)
## Purpose Update the PyTorch ecosystem to the **2.14.0** release candidate, resolving wheels from the PyTorch **test channel** (`download.pytorch.org/whl/test/...`): - `torch`: → **2.14.0** - `torchvision`: → **0.29.0** - `triton`: → **3.8.0** 2.14.0 is published on `download.pytorch.org/whl/test/` but not yet on the release index / PyPI, so the index URLs
issue
[Bug]: AttributeError: 'Qwen3_5TextConfig' object has no attribute 'max_window_layers'
### Your current environment vllm 0.17.0 lastest. ### 🐛 Describe the bug CUDA_VISIBLE_DEVICES=1,2 vllm serve /home/ub3960x/model_data/Qwen3.5-9B \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.80 \ --port 30005 \ --served-model-name qwen3.5-9b \ --max-model-len 16384 \ --reasoning-parser qwen3 \ --enable-auto-tool-choice \ --tool-call-parser hermes \
issue
NixlConnector hardcodes backends=["UCX"] default; no env-var override path; LIBFABRIC/EFA operators must discover kv_connector_extra_config.backends from source
### Your current environment - vLLM 0.17.1 (bundled in `nvcr.io/nvidia/ai-dynamo/vllm-runtime:1.1.0`) - Dynamo 1.1.0 runtime (for disaggregated prefill/decode serving) - NIXL `nixl_cu12` 1.0.1 with LIBFABRIC + UCX plugins both present on disk - AWS EFA hardware (SRD transport, libfabric provider `efa`) - 2× P5.48xlarge H100 HyperPod nodes - `libplugin_LIBFAB
issue
[ROCm/MI325X] DeepSeek-V4-Flash: Triton fp8_mqa_logits kernel requires 96KB shared memory, MI325X limit is 64KB
## Environment - **Hardware**: AMD MI325X × 8 (255.98 GiB HBM each) - **vLLM version**: 0.20.2rc1.dev90+g7a576e2c7 (rocm/vllm-dev:nightly, 2026-05-06) - **Model**: deepseek-ai/DeepSeek-V4-Flash - **Config**: TP=8, EP=8, `--kv-cache-dtype fp8`, `--enforce-eager` ## Bug At inference time, the first request fails with a Triton shared memory OOM: ``` triton.runt
issue
[Bug]: vLLM wheel version mismatch
### Your current environment vLLM wheel that is getting installed is different from that of the branch used. Example Docker is generated using the vllm branch - 0.19.0 , https://github.com/vllm-project/vllm/tree/releases/v0.19.0 But while building the wheel/docker, it generates the wheel which has version 0.19.1 pip3 list -l | grep vllm vllm 0.19.1.dev0+g2a6
issue
[Bug]: nvfp4 reshape_and_cache_flash assumes NHD layout — silently mis-swizzles HND caches when num_kv_heads % 4 == 0
### Environment ``` GPU: RTX 5090 (SM120, 32 GB) | driver 595.71.05, CUDA 13.2 vLLM 0.25.1 | torch 2.11.0 ``` ### Bug `reshape_and_cache_nvfp4_dispatch` (`csrc/.../nvfp4_kv_cache_kernels.cu:210`) reads `block_size` from tensor dim-1, which is only correct for NHD caches. With an HND cache it picks up `num_kv_heads` instead: - if `num_kv_heads % 4 != 0` you g
issue
[Bug]: Engine core livelock (100% CPU, no crash) with MTP speculative decoding + xgrammar structured outputs - regression from v0.24.0
## Your current environment ``` vLLM version: 0.25.1 (pip install, venv) xgrammar version: 0.2.1 PyTorch: 2.11.0+cu130 CUDA: 13.0 GPU: 1x NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96 GB) Driver: 580.159.03 OS: Ubuntu 25.10, kernel 6.17.0-40-generic Model: nvidia/Qwen3.6-27B-NVFP4 (compressed-tensors, NVFP4 weights, FP8 KV cache) ``` Full serve comma
issue
[Bug][Spec Decode] DSpark speculative decoding broken on nightly
## Summary DSpark speculative decoding (`method: "dspark"`) is non-functional on current main. Multiple code paths assume only `"dflash"` when DSpark reuses the same proposer/speculator infrastructure (`DSparkSpeculator` extends `DFlashSpeculator`, both use `DFlashProposer`). The result is that DSpark either crashes at startup or produces wrong results at in
issue
EAGLE/MTP block drop + prefix caching is untested for hybrid models with ≥3 attention groups (DeepSeek-V4-Flash + DSpark lands there)
## Summary #33524 fixed the "0% prefix cache hits with hybrid attention + EAGLE" bug for the 2-attention-group (GPT-OSS) shape, and its author noted in the PR body that more complex models were not addressed: > However, it is worth noting that for more complicated models with multiple attention groups, this PR does not fully address the EAGLE spiral block dr
issue
[Bug]: CuTe FA4 forward (flash_fwd.py SM80/SM120) fails on first call: NameError 'mDynamicCausal' and missing self.is_split_kv
### Summary Two independent Python-level bugs make `vllm/vllm_flash_attn/cute/flash_fwd.py` (`FlashAttentionForwardSm80`, and by inheritance `FlashAttentionForwardSm120`) fail on the very first call, before any kernel runs. They look like the code path has never been executed: SM80-class GPUs use the C++ FA2/FA3 kernels, so this CuTe DSL path is effectively
issue
[Bug]: Kimi-K3-NVFP4 on 8xB300 produces degenerate, incoherent output in the reasoning channel on v0.27.0
### Your current environment We are not able to run `collect_env.py` against the affected build. It was a production deployment that has since been rolled back, so that container no longer exists. The information below is therefore split into the node-level facts (identical across both builds, since it is the same physical machine) and the identity of the af
issue
[Bug]:amd mi308x gpu, vllm 0.27.0~0.27.1, rocm 7.2.3, Kimi-K2.7-Coder start fails:AssertionError: mla_gluon requires gfx950 (CDNA4), got gfx942
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text (vllm) [fanlf@2025-h3c-83 tmp]$ python ./collect_env.py Collecting environment information... ============================== System Info ============================== OS : Rocky Linux 10.2 (Red Quartz) (x86_64) GCC version : (GCC) 14.3.1 202510
issue
[Bug]: security: bump minimum setuptools version in requirements/common.txt
### Your current environment <d<details> <summary>The output of <code>python collect_env.py</code></summary> ```text OS: Linux (Termux / Android) Python Version: 3.11+ vLLM Repository: main branch audit ### 🐛 Describe the bug A security audit on `requirements/common.txt` identified known security advisories affecting the pinned version of `setuptools`: - **s
issue
[Bug]: Mistral-Small-3.1 FP8 (Pixtral) returns NaN on image inputs with compilation enabled; works with --enforce-eager
### Your current environment <details> <summary>Environment</summary> - vLLM: v0.24.0 - Hardware: 1x NVIDIA H100 80GB (tensor-parallel-size 1) - Model: `RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic` (compressed-tensors FP8-dynamic; only the language-model `Linear` layers are quantized — `vision_tower`, `multi_modal_projector`, and `lm_head` are i
issue
[RFC]: LoRA adapter support for DFlash speculative decoding draft models
### Motivation. vLLM supports serving many LoRA adapters over a single base target model, but speculative decoding still requires a separate, fully-trained draft model per domain. When serving many domains, this is expensive: each domain needs its own drafter (e.g. 0.8B) held in memory, with corresponding I/O cost to load and swap. This RFC proposes applying
issue
[Performance]: MM input tensors are re-broadcast to all TP workers on every request, even when the prefix cache fully covers the image tokens (~19 ms per unique image)
## Environment - vLLM v0.25.1 (official `vllm/vllm-openai` Docker image), single API server process - TP=4 on one node (4 GPUs), `mm_processor_cache_type: lru` (behavior identical under `shm`) - Gemma-4-31B-IT with vision; `--mm-processor-kwargs {"max_soft_tokens": 560}` → `pixel_values` per image is `(1, 5040, 768)` fp32 ≈ **15.5 MB** - `--limit-mm-per-prom
issue
[Bug]: AttributeError: 'DeepseekV4Config' object has no attribute 'num_local_experts' / vllm 0.27.1 / DeepSeek-V4-Flash-0731
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (aarch64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version :
issue
[RFC]: Session-Aware KV Cache Hints for Agentic Workloads
### Motivation. Agentic applications often know more about future KV cache reuse than the inference engine: - a child agent is waiting for a tool and will probably resume soon; - several agents share a long system/tool prefix; - a branch has completed and its KV cache can be released; - an offloaded branch will be resumed and can be prefetched. Today, after
issue
OffloadingConnector AssertionError in _build_store_jobs under MultiConnector with high concurrency multi-turn
## Summary The OffloadingConnector's scheduler crashes with an `AssertionError` in `_build_store_jobs()` when running under a MultiConnector (NixlConnector + OffloadingConnector) with high-concurrency multi-turn agentic workloads. ## Environment - **vLLM**: v0.26.0 (`vllm/vllm-openai:v0.26.0`) - **Model**: RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-FP8-block
issue
[Feature]: FA2 is only supported on devices with compute capability >= 8
### 🚀 The feature, motivation and pitch I am trying to use the vllm for loading [QWEN3.6 model](https://huggingface.co/Qwen/Qwen3.6-27B) Operating system= rocky linux 9 GPU = 2X Quadro RTX 8000 48 GB As soon as I try to load with the command `vllm serve ./Qwen3.6-27B` there is error below. `(EngineCore pid=1673840) ERROR 08-13 16:42:32 [fa_utils.py:273] Cann
pull request
Llamas 3.1 405B fp4 changes upstreaming from 355_wip
Perf is the same: upstream tp1 ``` ============ Serving Benchmark Result ============ Successful requests: 320 Maximum request concurrency: 64 Benchmark duration (s): 413.07 Total input tokens: 326905 Total generated tokens: 327680 Request throughput (req/s): 0.77 Output token throughput (tok/s): 793.27 Total Token throughput (tok/s): 1584.66 ---------------