vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Bugfix] Fix collect_env.py crash when pip list command fails
## Purpose Fixes the `collect_env.py` crash noted in #52027: in a uv venv with `uv` not on `PATH`, the pip-list command fails, `run_and_read_all` returns `None`, and `run_with_pip()` calls `None.splitlines()` → `AttributeError`. Separate from #52026 (the logging fix for the same issue), which doesn't touch this file. ## Changes - Return `None` when the list
pull request
Revert KV block zeroing generalization due to ROCm launch overflow
## Summary - Revert #51749, which generalized KV block zeroing from `FullAttentionSpec` to every `AttentionSpec`. - On DeepSeek-V4-Pro with ROCm and 64 concurrent GSM8K requests, the broader zeroing path makes `_zero_kv_blocks_kernel` launch with a grid that exceeds the AMD Triton driver's signed integer limit. - The resulting `OverflowError: signed integer
pull request
[CI] Mirror external test assets in vLLM S3
## Summary - mirror externally hosted video, image, and GSM8K test assets into the public vLLM S3 bucket - update affected tests to use the shared `VLLM_S3_BUCKET_URL` constant ## Why [Buildkite build 83608](https://buildkite.com/vllm/ci/builds/83608) had multiple test failures caused by direct dependencies on third-party hosts, including `RemoteDisconnected
pull request
[BugFix] Fix memory pinning in Qwen3_VisionTransformer for non-gpu
Fix for CI failure introduced by https://github.com/vllm-project/vllm/pull/51841
pull request
[Bugfix][Spec Decode] Preserve constant effective K schedule semantics
## Purpose `num_speculative_tokens_per_batch_size` is currently treated as runtime-variable whenever the schedule is present. Some schedules are syntactically non-uniform but resolve to one positive effective K after the configured runtime maximum is applied: ```text num_speculative_tokens = 3 schedule = [(1, 8, 5), (9, 16, 4)] effective K values = {min(5, 3
pull request
[Bugfix][MiniCPM-V] Fix AssertionError in get_dummy_mm_data when passing VideoDummyOptions to _get_dummy_images
**Issue**: pytest tests/lora/test_minicpmv_tp.py::test_minicpmv_lora raises AssertionError on non-CUDA platforms (e.g., XPU). **Root Cause**: Commit 9a276d6375 added a runtime assertion to _get_dummy_images in dummy_inputs.py: assert overrides is None or isinstance(overrides, ImageDummyOptions) However, MiniCPMVDummyInputsBuilder.get_dummy_mm_data in minicpm
pull request
[Renderer] Shutdown the renderer properly.
## Purpose Currently, during online serving, the renderer is shutdown twice, while it is not shutdown during offline inference. ``` offline: python examples/basic/offline_inference/embed.py this pr: INFO 08-13 16:11:10 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore (EngineCore pid=19851) INFO 08-13 16:11:10 [core.py:1327] [shut
pull request
[Attention][DSA] Take the native decode path for MTP=3 on SM90
# [Attention][DSA] Take the native decode path for MTP=3 on SM90 ## Purpose Closes #35878. The DSA indexer flattens a spec-decode batch into one single-token row per query whenever `next_n` falls outside `{1, 2}`, so with MTP=3 (`next_n = 4`) each request's KV tile is read four times instead of once. DeepGEMM's `nv_dev` branch, which vLLM already pins (`cmak
issue
[Usage]: MoE flatten_tp_size should not unconditionally include dp_size — DP loses its original semantics for MoE layers
## Summary In FusedMoEParallelConfig.flatten_tp_across_dp_and_pcp (vllm/model_executor/layers/fused_moe/config.py:979-988), when EP is not enabled, dp_size is unconditionally folded into the flattened TP size: ``` flatten_tp_size = dp_size * pcp_size * tp_size flatten_tp_rank = dp_rank * pcp_size * tp_size + pcp_rank * tp_size + tp_rank ``` This means that f
issue
[Bug]: In_proj_ba of GDN in Qwen3Next use MergeColumnParallelLinear may cause accuracy decrease?
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Your output of `python collect_env.py` here ``` </details> ### 🐛 Describe the bug I found that in_proj_ba is a MergeColumnParallelLinear now, and it was a ColumnParallelLinear before. It leads to a different way of weight loading. And it will ca
issue
[Bug]: Segfault encountered in qwen3.5-27B long video (>2G)inference
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Your output of `python collect_env.py` here 2026-06-24 07:11:18 (110 B/s) - 'collect_env.py' saved [35090/35090] Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5
issue
[Bug]: gpt-oss chat completions return 500 "Unexpected token 200002 while expecting start token 200006" when ignore_eos=true
### Your current environment - vLLM **0.26.0** and **0.18.0** (both reproduce), official `vllm/vllm-openai` images - `openai-harmony` 0.0.8 - Model: `openai/gpt-oss-120b`, 1x H200, TP=1 - `vllm serve openai/gpt-oss-120b --max-model-len 32768 --tensor-parallel-size 1` ### 🐛 Describe the bug A Chat Completions request with `ignore_eos: true` returns **HTTP 500
issue
[Bug]: DiffusionGemma: runtime CUDA OOM under concurrent decode (fp32 canvas×vocab temporaries)
### Summary Block-diffusion decode produces `canvas_length` (default 256) logit rows **per request per step**, and `_compiled_sample_step` in `vllm/model_executor/models/diffusion_gemma.py` materializes ~6 fp32 temporaries of shape `[num_decode, canvas_length, vocab_size=262144]`. At `--max-num-seqs 8` each of these tensors is 2.1 GiB (>12 GiB total transien
issue
[RFC]: Race-free port management: pick ports at bind time, publish over an existing channel
## Motivation `get_open_port()` probes by binding a port, closes that socket immediately, and returns only the number. A later process or library performs the real bind. Another process can claim the port in the interval, producing ZMQ `Address already in use`, PyTorch `DistNetworkError` / `EADDRINUSE`, or Rust `AddrInUse` during concurrent cold starts. The
issue
[Fix]: MTP speculative decoding crashes with RuntimeError on heterogeneous per-layer head_dim, fixed below and created a PR
### Bug Description Serving Gemma 4 31B with its native assistant model (`google/gemma-4-31B-it-assistant`) on `vllm/vllm-openai:latest` crashes during parameter loading with `RuntimeError: start (0) + length (4096) exceeds dimension size (2048)`. ### Root Cause Gemma 4 uses heterogeneous attention layers (45 sliding layers with `head_dim=256`, 15 full layer
issue
[RFC]: Suffix decoding on GPU (`suffix_gpu`) — GPU-resident suffix decoding compatible with async scheduling
## Summary Suffix decoding gets the best acceptance lengths among model-free drafters on repetitive and agentic workloads, but the in-tree implementation (`method="suffix"`, #25784, from Arctic Inference) is CPU-side and rejected by the async-scheduling whitelist. This RFC proposes `suffix_gpu`: a GPU-resident suffix drafter that follows the same device-stat
issue
[Bug]: FP8 block-scaled weights fail on sm120 (RTX 5090) — DeepGEMM "Unknown SF transformation" during process_weights_after_loading
### Your current environment <details> <summary>Environment</summary> ``` vLLM: 0.26.1rc1.dev608+g99a10304d (docker.io/vllm/vllm-openai:muse-glimmer) GPU: 2x NVIDIA RTX 5090 (sm120 / device capability 12.0), PCIe Platform: Linux x86_64, CUDA 12.x Model: RedHatAI/Muse-Glimmer-30B-FP8-block (compressed-tensors, block-scaled FP8 weights) ``` </details> ### 🐛 De
issue
[Bug]: Qwen3 MoE GPTQ `qzeros` shape mismatch on ROCm gfx1201
### Your current environment <details> (Run from within remote container that I started via `docker --context` from host) ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : 22.0.0git (https://github.com/RadeonOpenCompute/llvm-project
issue
[Bug]: Bugs in kimi-k3 docker image
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text N/A - discussion about docker image ``` </details> ### 🐛 Describe the bug The [kimi-k3](https://hub.docker.com/layers/vllm/vllm-openai/kimi-k3/images/sha256-fb16b180bd9727600067e16fcd6a6de43fb4db1baf4298ef20b4dbdf6bfa5a0e) image and other [kimi
issue
[Bug]: mnnvl allreduce workspace init hangs 30s and leaks GPU memory on IB-only multi-node
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake versio
issue
[Bug]: DeepSeek-V4-Flash-0731 + DSpark fails to start on vLLM 0.27.0 / H100 sm90 (DeepGEMM CUDA_ERROR_ILLEGAL_ADDRESS); works on 0.26.0
### Your current environment - **GPU**: 8× NVIDIA H100 80GB HBM3 (sm90a), single node - **vLLM**: 0.27.0 (pip install into a clean venv) - **Model**: `deepseek-ai/DeepSeek-V4-Flash-0731` (local copy of the HF repo) - **Parallelism**: TP=8, expert-parallel, fp8 KV cache - **Also installed / compared against**: vLLM 0.26.0 in a separate venv on the *same* imag
issue
[Bug]: MTP speculative decoding cannot start under pipeline parallelism — SupportsPP demanded of the draft model
### Your current environment vLLM `0.26.1rc1.dev693+g7f7a32cfe` (nightly), 8× RTX 3090 (sm_86), CUDA 12.x, `VLLM_USE_V2_MODEL_RUNNER=1`. Also reproduced on `dev678` and `dev686`. Code references below are against `main` @ `98f86b9c0232`. ### 🐛 Describe the bug Speculative decoding with an MTP head cannot start at all under pipeline parallelism. It fails duri
issue
NemotronH MTP speculator: 0% draft acceptance on Intel XPU — V1 execution path bypasses standard sampling chain
# NemotronH MTP speculator: 0% draft acceptance on Intel XPU — V1 execution path bypasses standard sampling chain ## Summary Native MTP speculative decoding for Nemotron-3.5-Lightning-30B-A3B produces **0% draft acceptance** on Intel Arc Pro B70 (Xe2/Battlemage) via vLLM XPU nightly. After extensive debugging (9+ GPU cycles, 3 identified bugs patched, 7 inst
issue
[RFC]: vLLM Native Reusable Engine Snapshot Sleep (L3)
## Motivation. A vLLM cold start includes model weight loading, EngineCore initialization, CUDA context and allocator creation, `torch.compile`, CUDA Graph capture, KV cache profiling, and warmup. These steps are repeated for every new instance. This impacts responsiveness in elastic scaling and multi-model deployment scenarios. When an instance is released
issue
[Bug]: VLLM_BATCH_INVARIANT does not cover convolution, used in VAE blocks
### Your current environment OS : Ubuntu 22.04.5 LTS (x86_64) Clang version : 22.0.0git (roc-7.2.3 26084 f58b06dce1f9c15707c5f808fd002e18c2accf7e) Libc version : glibc-2.35 PyTorch version : 2.11.0+gitd0c8b1f ROCM used to build PyTorch : 7.2.53211 Python version : 3.12.13 (main, Mar 4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime) Python platform : Linux-5.15
pull request
Fix Quark OCP-MX W4A6 support: dequant dtype + apply_weights
<!-- markdownlint-disable --> Fixes two issues preventing Quark W4A6 models (MXFP4 weights / MXFP6 activations, e.g. `ziliangpeng/DeepSeek-V3-Quark-MXFP4-v4-w4a6`) from running on ROCm: 1. **MoE QuantType crash** (`rocm_aiter_moe.py`): W4A6 models set `weight_quant_dtype="mxfp4"` but not the `use_mxfp4_w4a4/w4a16` flags, causing AITER MoE to receive `QuantTy
pull request
Use physical device id for get_device_uuid
This PR updates the get_device_uuid to translate the device_id parameter to a physical gpu id before retrieving the gpu handle. `get_device_uuid` is used in the Verl application PPO and Fully Async uses cases which executes the code below when the vllm backend is being used: def get_device_uuid(device_id: int) -> str: from vllm.platforms import current_platf
pull request
[Model] Add MixtureOfExperts support for PhiMoE
This PR adds EPLB (Expert-Parallel Load Balancing) support for PhiMoE by implementing the `MixtureOfExperts` protocol. ## Changes - Add `PhiMoEMixtureOfExperts` mixin class with `extract_moe_parameters()` and `update_physical_experts_metadata()` - Thread `enable_eplb` from model config through `PhiMoEModel` → `PhiMoEDecoderLayer` → `PhiMoE` - Pass `enable_ep
pull request
[DO NOT MERGE][Perf][Feat] Add generic cuteDSL LL FP32 router (GEMM)
Depends on: https://github.com/vllm-project/vllm/pull/50174 ## Motivation This kernel is intended to be a generic low-latency fp32-router kernel. We should not expect it to outperform highly specialized PTX/CUDA kernels tuned for one exact router shape. The goal is different: make the CuTeDSL implementation competitive with those specialized kernels while ke
pull request
[Bugfix] EXAONE 4.5: trim trailing MTP entry from text_config.layer_types
## Purpose Let EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B-FP8`) and its self-MTP speculative-decode path load on mainline `transformers` v5 + vLLM. The LG-AI EXAONE 4.5 release configs publish `text_config.layer_types` of length `text_config.num_hidden_layers + 1`: the trailing entry is the MTP head registered alongside the regular transformer stack. (Stil
pull request
[Misc] Add unit test for _fwd_kernel_ep_scatter_1 and _fwd_kernel_ep_…
## Purpose Adds a unit test for the `_fwd_kernel_ep_scatter_1` and `_fwd_kernel_ep_scatter_2` Triton kernels (DeepGemm group) in `tests/kernels/moe/test_ep_scatter.py`. Both kernels run inside the `ep_scatter` wrapper and had no upstream coverage: - `_fwd_kernel_ep_scatter_1` computes per-expert region offsets and fills `m_indices` (expert label per output r
pull request
[Misc] Add unit test for merge_attn_states kernel
## Purpose Adds a device-agnostic unit test for the `merge_attn_states` Triton kernel (Attention group) in `tests/kernels/attention/test_merge_attn_states_torch_ref.py`. The existing `test_merge_attn_states.py` compares the Triton kernel against a custom CUDA op (`merge_attn_states_cuda`) and skips on non-CUDA platforms, so it cannot run on XPU. This test in
pull request
[Feature] Initial support for fault tolerant ep using scale-down
## Purpose This PR adds **scale-down** and **auto-recovery** support to the vLLM fault tolerance (FT) framework for DP+EP MoE deployments (external LB mode). The existing FT framework (`--enable-fault-tolerance`) supports `retry` — reinitializing the DP process group after a transient fault where all ranks survive. When a DP rank permanently dies (hardware f
pull request
[Spec Decode] Add FlashInfer CuteDSL non-causal decode path for DFlash
## Purpose Part of the DFlash bring-up tracker #46105 ("Attention backend → CuteDSL via FlashInfer", contribution-welcome item). vLLM currently runs non-causal DFlash queries through the FlashInfer *prefill* wrapper (`BatchPrefillWithPagedKVCacheWrapper` with `causal=False`), because the decode/TRTLLM paths cannot express non-causal attention. FlashInfer's C
pull request
[KERNEL][ROCm]Native HIP MXFP4(Compressed+Quark) (dense + MoE) for RDNA3
## What this is RDNA3 (the RX 7900 XTX and friends) currently has no way to run MXFP4 models in vLLM. Marlin is CUDA-only and AITER's MXFP4 path is CDNA4/gfx950-only, so when you point vLLM at an MXFP4 checkpoint on a 7900 XTX it either refuses to load (compressed-tensors models, which have no ROCm kernel to fall back on) or limps along on the Triton-unfused
pull request
[CPU] Add CPU-tuned autotune configs for FLA (GDN) Triton kernels
> **Depends on #49583.** That PR bumps the triton-cpu pin to a commit that includes triton-cpu #275 (the `CPUOptions.hash()` runtime-only cache-key fix). Without it, the configs here recompile once per `num_cpu_threads` candidate, so the warmup speedup is only realized after #49583 lands. ## Purpose The FLA op kernels used GPU-oriented autotune configs that
pull request
fix: reject required/named tool_choice for Harmony models without a tool parser
### Summary In the OpenAI-compatible chat path, when no tool-call parser is configured, `tool_choice="required"` and named tool choices were accepted for Harmony (gpt-oss) models and then produced empty or incorrect tool-call results. ### Fix Only exempt Harmony for `tool_choice="auto"` (best effort). Named and `required` now fail closed with a 400 when no t
pull request
test(quantization): cover tied lm_head/embed_tokens when lm_head excluded from ModelOpt
### Summary Adds regression coverage for tying an excluded `lm_head` to `embed_tokens` through the quantization method path (`QuantizeMethodBase.tie_weights`), the behavior that fixed the original NVFP4 crash on main. Test only. Related to #48238
pull request
Make `load_weights` completely optional
Part of https://github.com/vllm-project/vllm/issues/48972 Infrastructure changes: - Add `autoload_weights` and use it in weight loaders and runners. This creates it's own `AutoWeightsLoader` if the model doesn't have a `load_weights` method - When the model has no `load_weights` method, auto apply any `hf_to_vllm_mapper` in `AutoWeightsLoader._load_module` a
pull request
[ROCm][feature] Add new moe backends supporting int4/int8 weight-only…
## Purpose 1. Add Int8EmulationTritonExperts and TritonWNA16OTFExperts to EMULATION MoE backend to support int4/int8 quantization for WNA16; 1.1 Int8EmulationTritonExperts: dequant at load time; 1.2 TritonWNA16OTFExperts: dequant on-the-fly, supporint sym/asym/static/dynamic int4 quantization schemes, and also int8 quantization; 1.3 TritonWNA16OTFExperts is
pull request
[Misc] Bump `openai` to `>=2.25.0` to support namespace tools types
## Purpose Fix the dependency lower bound issue #49103. We have recently added OpenAI responses API namespace tools support, but the corresponding openai types are only supported for `openai>=2.25.0`. Hence the lower bound bump. Note that the dependency bump is necessary in the sense that any client with `openai<2.25.0` got an inevitable `ImportError` from t
pull request
[Hardware][XPU] Register matmul and linear batch-invariant kernels for XPU
## Purpose This is the second (2/2) PR introducing batch invariance to Intel XPU devices. You can find the first one [here](https://github.com/vllm-project/vllm/pull/41934). This PR registers the missing matmul and linear kernels. It also adds a new matmul_kernel_descriptor_persistent Triton kernel for better matmul performance on XPU. ## Test Plan Tested wi
pull request
[BugFix] bound FlashMLA sparse decode intermediate tensors size
## Purpose Fix #44545 - not in an ideal way though. When FlashMLA sparse kernel is selected with FP8 KV cache dtype, FlashMLA sparse decode kernel(https://github.com/vllm-project/FlashMLA/blob/a8f794d1251cbfd88a5011445dd5582289c727e4/csrc/api/sparse_decode.h#L184) is called in mixed batch mode. This kernel internally allocates intermediate tensors (`o_accum`
pull request
[Spec Decode] Add PARD-2 parallel draft model support
## Purpose This PR adds support for PARD-2 ([AMD-AGI/PARD](https://github.com/AMD-AGI/PARD)) target-aligned parallel draft models as a speculative-decoding method in vLLM. ## Test Plan **Serve** (PARD-2, target-dependent): ``` vllm serve <TARGET> --max-model-len 4096 --no-enable-prefix-caching --port 8008 --host 0.0.0.0 \ --speculative-config '{"model": "<PA
pull request
[Bugfix][MM] Profile the multimodal embedding merge during memory profiling
Automated draft from oss-work.
pull request
[Test] Cover detokenize=False in offline beam search
Offline counterpart of #46422. Sets `detokenize=False` on the per-step sampling params in the offline beam search path, and threads it through the structured-output beam path as well, so intermediate beam candidates are not detokenized on every step. This does not fix the Whisper slowdown reported in #49197. That one is the per-beam encoder recompute called
pull request
[Bugfix] Hoist $defs/definitions in Cohere parser tool schema composition
## Purpose Fix a bug where any `/v1/chat/completions` request with tools whose JSON Schema contains `$defs`/`definitions` fails with HTTP 500 on Cohere models served with `--reasoning-parser cohere_command3|cohere_command4`. Resolves the bug class reported in #16467 (closed as stale, never fixed) for the Cohere reasoning parser path specifically. Other parse
pull request
[Cohere][Spec Decode] Add CohereEagleProposer to support multi layer eagle drafts
Follow up to #42078 with model released. Cohere Eagle draft models have multipe layers and HybridKVCacheManager split these layers across multiple KV cache groups, which the base EagleProposer does not support (it assumes a single group with one shared block table / slot mapping). This PR adds CohereEagleProposer, which tracks per-group block tables and slot
pull request
[3/N][Feat][Perf] Add new warmup infrastructure for JITs. Add provider registry and orchestration for JIT warmup
## Description This PR extends the shared JIT warmup infrastructure with provider registration and centralized orchestration. It builds on #49315 and the contract described in #47456. For more details, see parent (draft) PR: https://github.com/vllm-project/vllm/pull/49627 and tracking list issue https://github.com/vllm-project/vllm/issues/49349 ``` JIT kerne
pull request
[Bugfix] Skip generic required/named tool grammar for parsers that set supports_required_and_named
# Issue #49981 — `tool_choice: "required"` xgrammar FSM crash / infinite hang (GLM-5.2) ## 1. Root cause `ToolParser.supports_required_and_named` is the flag a tool parser sets to say *"my model does not emit the standard tool-call JSON; do not use the generic JSON-based `required`/named handling for me."* GLM parsers (`Glm47MoeModelToolParser`, used for `--