vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
Revert "[Perf] Narrow DeepSeek V4 eager CUDA graph region" (#51430)
## Summary - Revert [#51430](https://github.com/vllm-project/vllm/pull/51430) and restore the wider DeepSeek-V4 eager attention region. - Preserve the later ROCm `_fused_wqa_wkv_gemm` override point while resolving the revert against current main. - This is the prerequisite for safely restoring MRV1 piecewise graphs in the stacked follow-up. ## Why On gfx950
pull request
[Model] [Quantization] Add Ling hybrid MXFP4 routed experts support
## Purpose Add support for Ling checkpoints that use hybrid quantization: block FP8 for dense and shared-expert projections, and MXFP4 for routed experts. This change reads Ling-specific quantization metadata and remaps routed-expert scale names to the convention expected by `Mxfp4MoEMethod` for both the main and MTP models
pull request
[XPU] [Bugfix] process ragged weights in xpu linear backend
python examples/basic/offline_inference/generate.py --model gaunernst/DeepSeek-V2-Lite-Chat-FP8 --enforce-eager --max-model-len 2048 --trust-remote-code Before: <img width="1688" height="171" alt="image" src="https://github.com/user-attachments/assets/82d1fcf4-19fd-48df-a105-ca1a77ca9db4" /> After: <img width="865" height="227" alt="image" src="https://githu
pull request
[Security] Fix ReDoS in validation error path sanitizer
## Summary - Fix CWE-1333 regex denial-of-service in `sanitize_message()` where a 100KB slash-delimited input (`"/a" * 50000`) could keep an API worker busy for 3+ seconds per rejected request. - Add a dot-character pre-check to skip the generic path regex entirely when no filename extension is possible, and use an atomic group `(?>...)` to prevent backtrack
pull request
Update CODEOWNERS
## Purpose As discussed offline with @Isotr0py ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pastin
pull request
[CI/Build][CPU] Shrink triton-cpu-build layer by dropping build artifacts
## Summary - The `vllm-triton-cpu-build` stage in `docker/Dockerfile.cpu` clones and builds `triton-lang/triton-cpu`, but only the resulting wheel is needed by later stages/the final image. - The cloned `triton-cpu` source/build tree and Triton's downloaded LLVM/MLIR toolchain (under `/root/.triton`) were being retained in the final layer, growing the stage
pull request
Enhance engine snapshot management and API lifecycle
## Purpose Add an opt-in Engine Snapshot deep-sleep path that keeps the API frontend, HTTP listener, and launcher alive while snapshotting and exiting the `EngineCoreProc` subtree. On wake, vLLM validates the snapshot and runtime environment, restores the EngineCore and CUDA state, adopts the restored process, reconnects the frontend, and reopens request adm
pull request
fix(missing_documentation): Class `RequestOutputKind` in vllm/sampling_params.py has no
## TLDR **Gap:** Class `RequestOutputKind` in vllm/sampling_params.py has no docstring — add parameter docs, return type, and example [BLOCKED:Tests failed after retry: ==================================== ERRORS ==================================== ___________________ ERROR collecting vllm/sampling_params.py _____ **Wedge type:** `missing_documentation` **I
pull request
[Bugfix][ROCm][CI] Give the AITER MLA decode metadata stub its MLA dims
# Purpose `tests/kernels/attention/test_rocm_aiter_mla_decode_metadata.py::test_persistent_decode_metadata_matches_fp8_golden` fails on main with `AttributeError: 'types.SimpleNamespace' object has no attribute 'q_lora_rank'`. Two jobs report it, the dedicated AITER MLA job and the sharded `kernels/attention` job, but it is the same test. "[Model] Add native
pull request
[Misc] Add missing return type annotations in outputs.py
### Purpose Add missing return type annotations to from_base() static methods and PoolingRequestOutput.__repr__() in vllm/outputs.py. All from_base() static methods on EmbeddingOutput, ClassificationOutput, ScoringOutput, EmbeddingRequestOutput, ClassificationRequestOutput, and ScoringRequestOutput lacked return type annotations. PoolingRequestOutput.__repr_
pull request
Standardise weight tying on `ParallelLMHead.tie_weights`
## Purpose vLLM expresses tied word embeddings in three different ways. Only one of them, `self.lm_head = self.lm_head.tie_weights(embed_tokens)`, dispatches through `quant_method.tie_weights`. This PR converts the other two so tying is expressed one way everywhere. - `self.lm_head.weight = embed_tokens.weight` (33 sites) bypassed the quant method entirely,
pull request
[Attention] Fix FlashInfer SM12x prefill with sinks
## Summary Use FlashInfer's sink-aware paged prefill wrapper on SM12x when attention sinks are enabled. The generic FA2 prefill path accepts a `sinks` argument but does not apply it, so #49718 can use XQA for decode while producing incorrect prefill output. The wrapper is specialized with the active dtypes, head dimensions, sliding window, and softmax scale.
pull request
[Docs] Note pooled aggregation semantics for mean_acceptance_length
## Purpose `SpecDecodingLogging._log` and the PromQL recipe in `SpecDecodingProm.__doc__` compute the `mean_acceptance_length` as a **pooled mean** (weighted by draft count, equivalent to `total_generated / total_verify_calls`), not an unweighted mean of per-request means. Some external benchmarks report the latter — SpecForge's `average_acceptance_length`,
pull request
[Bugfix] Declare SupportsEagle3 on KimiLinearForCausalLM
### Purpose `KimiK3ForConditionalGeneration` (multimodal) declares `SupportsEagle3`; the text-only `KimiLinearForCausalLM` does not — even though both serve the same inner `KimiLinearModel`, which already inherits `EagleModelMixin` and implements the aux-hidden-state tap machinery. Serving a text-only Kimi-K3 checkpoint with EAGLE3-family speculative decodin
pull request
Apply logit softcapping in Transformers modelling backend
This is mainly used by Gemma models and only affects workloads where the exact logit values are important. Generation is unaffected because it does not reorder anything.
pull request
Remove VLLM_TEST_FORCE_FP8_MARLIN to replace with linear_backend/moe_backend
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
issue
[Bug]: MLA chunked-context prefill crashes on sm80 with Marlin FP8: kv_c_normed cast to packed-int32 weight dtype (`unsupported \`a\` scalar_type`)
### Your current environment - vLLM 0.23.0 (official image), PyTorch 2.x, Triton, NCCL 2.28.9 - 12× NVIDIA A100-SXM4-80GB (compute capability **8.0**, no native FP8) across 3 nodes - Model: GLM-5.2-FP8 (fp8 weights → `MarlinFP8ScaledMMLinearKernel` / `MARLIN Fp8 MoE`), PP=3 × TP=4, `--enable-expert-parallel` - Attention: `TRITON_MLA` decode backend, `FLASH_A
issue
[Bug]: `--data-parallel-start-rank 0` is silently treated as unset due to truthiness checks in `create_engine_config`
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... uv is set ============================== System Info ============================== OS : Linux Mint 22.1 (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake
issue
[Bug][Spec Decode] num_speculative_tokens_per_batch_size + MTP speculator fails full CUDA graph decode capture (InputBatch.make_dummy assert)
### Environment - vLLM v0.25.0 (official docker image), single H100 NVL 94GB (SM90) - Target: `prithivMLmods/gemma-4-31B-it-qat-FP8` (compressed-tensors FP8) - Drafter: `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` (Gemma4 MTP, 4 layers: 3 sliding + 1 full attention) - `--kv-cache-dtype fp8 --async-scheduling --attention-backend TRITON_ATTN` (backen
issue
[Feature]: Expose startup status and health endpoint before model engine is ready
### 🚀 The feature, motivation and pitch vLLM currently exposes its HTTP API only after the model engine has completed initialization. For models with expensive startup phases, this can leave the service unreachable for a long time even though the process is alive and making progress. For example, startup may include: Loading model checkpoints Compiling execu
issue
[Bug][P/D]: NIXL handshake failure with asymmetric TP in PD disaggregation (pTP2 + dTP4)
### Your current environment - vLLM version: `0.1.dev1+g51f799c1a` - NVIDIA GPU: H200 (140GB) - CUDA: 13 - NIXL transport: UCX/RDMA (RoCE) ### Model google/gemma-4-26B-A4B (MoE, 26B total, ~4B active) ### Describe the bug NIXL KV transfer handshake fails with `NIXL_ERR_BACKEND` when using asymmetric tensor parallelism in PD disaggregation (prefill TP=2, deco
issue
[Bug]: Kimi-K3 with --kv-cache-dtype fp8 is unusable on H200/Hopper — assertion demands use_prefill_query_quantization, but that flag is silently ignored on non-Blackwell devices
## Summary Serving **Kimi-K3** with `--kv-cache-dtype fp8` on **H200 (Hopper, SM 9.0)** fails at runtime with an assertion telling the user to enable `--attention-config '{"use_prefill_query_quantization": true}'`. **Setting that flag does not resolve the error** — the same assertion fires again. On any non-Blackwell (non-capability-100) device the flag is s
issue
[Doc]: Stale rejection_sample_method and synthetic_acceptance_rate in speculative decoding README
### 📚 The doc issue The `--speculative-config` table in `docs/features/speculative_decoding/README.md` is out of sync with the current code: 1. `rejection_sample_method` (line 87): documented as `strict, probabilistic, synthetic` (default `strict`), but the current code is `Literal["standard", "synthetic", "block"]` (default `standard`) (`vllm/config/specula
issue
[Bug]: VLLM 0.27.0 has not been fixed MiniCPM-V-4.6 startup fails (because vLLM expects k_proj in a fused-qkv ViT attention layer)
### Your current environment vLLM: 0.27.0 Python: 3.10.19 PyTorch: 2.13.0+cu130 GPU: NVIDIA GeForce RTX 4070 Ti SUPER (16 GB) NVIDIA driver: 610.62 OS: Ubuntu 22.04.5 LTS under WSL2 Model: MiniCPM-V-4.6 Model path: /mnt/e/BaiduNetdiskDownload/MiniCPM-V-4.6 --trust-remote-code enabled ### 🐛 Describe the bug ### Description Starting `OpenBMB/MiniCPM-V-4.6` wit
issue
[Feature][DSpark]: Enable varlen for Nemotron-H (Mamba)
### 🚀 The feature, motivation and pitch Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches
issue
[Feature][DSpark]: Enable varlen for Qwen 3.x (GDN)
### 🚀 The feature, motivation and pitch Adaptive verification has a firm requirement that the attention backend must not use the CPU-side query lengths as a source-of-truth, instead using the gpu tensors (so that we can run the confidence-based assignment on the GPU without synchronizing the CPU). Additionally, kernels must be able to support decode batches
issue
[Feature][DSpark]: Enable logprobs with adaptive verification
### 🚀 The feature, motivation and pitch Follow-up leftover from #47808. See: https://github.com/vllm-project/vllm/pull/47808#discussion_r3723768178 ### Alternatives _No response_ ### Additional context _No response_ ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right
issue
[Feature][DSpark]: Polish DSpark adaptive timings
### 🚀 The feature, motivation and pitch Following #47808, the engine profiles the forward pass and proposer timings for use in estimating the batch runtime. The measurement facility could use a refactor. It is valuable to expose these metrics to the frontend, via /metrics endpoint and also compatibility with iteration-level logging so we can easily observe t
issue
[Feature]: Allow logging model output text without output token IDs
### 🚀 The feature, motivation and pitch In vLLM v0.27.1, `--enable-log-outputs` logs the generated text, `output_token_ids`, and finish reason together in the same INFO record: ```text Generated response <request_id>: output: '...', output_token_ids: [...], finish_reason: stop ``` For production troubleshooting, I want to keep the human-readable generated ou
issue
[Bug]: logs from a parser loaded with --tool-parser-plugin bypass vLLM's logging config
### Your current environment <details> <summary>The output of <code>python collect_env.py</code> (trimmed to the relevant parts)</summary> ```text vLLM Version : 0.27.1 PyTorch version : 2.13.0+cu130 CUDA used to build PyTorch : 13.0 Python version : 3.12.13 (64-bit runtime) Python platform : Linux-7.0.0-generic-x86_64-with-glibc2.43 OS : Ubuntu 26.04 LTS (x
issue
[Usage]: RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
I was trying to run a standard vllm code ``` python import os from dotenv import load_dotenv load_dotenv() import huggingface_hub HF_token = os.getenv("HF_token") huggingface_hub.login(token=HF_token) from vllm import LLM, SamplingParams prompts = [ "Hello, my name is", "The president of the United States is", "The capital of France is", "The future of AI is
issue
[Bug]: Gemma 4 31B MTP - Slower T/S at high context
### Your current environment ``` Collecting environment information... ============================== System Info ============================== OS : Ubuntu 26.04 LTS (x86_64) GCC version : (Ubuntu 15.2.0-16ubuntu1) 15.2.0 Clang version : Could not collect CMake version : version 4.2.3 Libc version : glibc-2.43 ============================== PyTorch Info ===
issue
[Bug]: gpt-oss-120b MXFP4 + --enable-expert-parallel crashes at startup: modular MoE finalize allocates padded (3072) output for an unpadded (2880) expert result
**Before submitting a new issue...** - [x] I have searched for existing and past issues, at four breadths: the symptom (padded/unpadded shape mismatch in MoE finalize), the subsystem (`gpt-oss` MXFP4 expert-parallel MoE), the faulting file/class (`modular_kernel.py`, `FusedMoEPrepareAndFinalize`), and the exact symbols (`has_unpadded_output`, `hidden_dim_unp
issue
[Feature][DSpark]: Evaluate STS for online DSpark confidence alignment
### 🚀 The feature, motivation and pitch The confidence scores from a DSpark head are not always perfectly calibrated. In particular, it seems like the deeper positions have higher variance and some bias in their expectation. The DSpark paper suggests [Sequential Temperature Scaling (STS)](https://arxiv.org/html/2607.05147v1) as an online solution to calibrat
issue
[Feature][DSpark]: Improve Adaptive DSpark Online Profiling
### 🚀 The feature, motivation and pitch The initial version does profiling at startup to estimate the verification step time by replaying the recorded CUDA graphs. This is known to have some drift, especially at higher batch sizes. Here are my results from initial study: <img width="3240" height="972" alt="Image" src="https://github.com/user-attachments/asse
issue
[Bug]: Breaking change to DeepSeek-V4-Pro default reasoning effort
### Your current environment vllm main, bisected to https://github.com/vllm-project/vllm/pull/50580 ### 🐛 Describe the bug This PR (https://github.com/vllm-project/vllm/pull/50580), which changes the DSV4 encoder for DSV4-Flash-0731 support, changes the default behaviour for DSV4-Pro. Previously, only reasoning_effort == "max" adds the reasoning blurb. See h
issue
[CI Failure][Anthropic] /v1/messages returns 500 when stop_sequences has more than 4 items
### Your current environment Observed in vLLM Buildkite CI while running PR #48215. ### Describe the bug The stateless OpenAPI Schemathesis test for `POST /v1/messages` can generate an Anthropic request containing more than four `stop_sequences`. The Anthropic request schema accepts the payload, but converting it to `ChatCompletionRequest` raises a Pydantic
issue
[Bug]: Continuous Host Memory Growth / Possible Memory Leak with V2 Runner on Qwen3-14B and Qwen3-Rerank-4B
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version
issue
[Bug]: vllm_c fused_add_rms_norm rounds the residual sum before variance
### Summary The CUDA `vllm_c` implementation of `fused_add_rms_norm` and the native vLLM IR implementation use different rounding orders for BF16/FP16 inputs. At upstream commit [`903d2efe`](https://github.com/vllm-project/vllm/commit/903d2efe7eb6d045d03c0a8192b1ac7bec50d168): - The [native IR implementation](https://github.com/vllm-project/vllm/blob/903d2ef
issue
[Bug]: sparse_mla_attention forward_mha falls through to unimplemented base class under low max-num-batched-tokens, surfaces as scheduler KeyError
## Environment - vLLM: v0.27.0, source-built with #51538's two commits (`47f6574`, `db39e67`) applied - Model: GLM-5.2 (compressed-tensors WNA16 Int4-Int8), `GlmMoeDsaForCausalLM` - Hardware: 4x DGX Spark GB10 (sm_121, capability 12), TP=4, `mp` executor - Attention backend: `FLASHINFER_MLA_SPARSE_SM120` (the only one that supports this hardware capability p
issue
[Bug][ROCm/gfx942]: DeepSeek-V4-Flash silent retrieval corruption for prompts ≥ ~4-5k tokens (AITER sparse indexer)
## Environment - 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14 - Image: `vllm/vllm-openai-rocm:nightly` (2026-08-12, v0.26.1rc1.dev668+g3ee2df303) - plus local backports of #51821 (merged 08-13) and #52058 / #51252 (open) — reproduces identically without the two open-PR backports - Model: `deepseek-ai/DeepSeek-V4-Flash-0731`, TP=8, `--kv-cache
issue
[Bug]: DeepSeek-V4-Flash-0731 MTP speculative decoding fails to load: KeyError 'model.layers.43.mtp_block.main_norm.weight'
## Environment - 8× MI325X (gfx942), `vllm/vllm-openai-rocm:nightly` (2026-08-12, g3ee2df303) - `deepseek-ai/DeepSeek-V4-Flash-0731` (config has `num_nextn_predict_layers: 1`), TP=8 ## 🐛 Describe the bug Enabling self-speculative MTP: ``` --speculative-config '{"method":"mtp","num_speculative_tokens":1}' ``` Architecture resolves to `DeepSeekV4MTPModel`, the
issue
[Feature]: split local/external prefix-cache hits in `prompt_tokens_details`
### 🚀 The feature, motivation and pitch ## Background With distributed prefix caching (KV connectors such as Mooncake / lmcache), cache hits in a single prefill can come from two different sources: - **local**: hits on this instance's GPU prefix cache; - **external**: hits fetched from a remote KV store / another instance via the KV connector. The scheduler
issue
[Bug] xgrammar backend ignores the model's other EOS ids, so guided decoding can stop mid-string
`XgrammarBackend.__post_init__` builds the `TokenizerInfo` from `tokenizer.eos_token_id` alone: ```python tokenizer_info = xgr.TokenizerInfo.from_huggingface( self.tokenizer, vocab_size=self.vocab_size, ) ``` `from_huggingface` accepts `stop_token_ids`; omitted, it derives a single id. A model whose `generation_config.json` declares several EOS ids ends up w
issue
[Bug][ROCm/gfx942]: GLM-5.2-FP8 — first request after GPU idle emits garbage; piecewise CUDA graph cold replay corrupts the request's own prefill (workaround: cudagraph_mode=FULL_DECODE_ONLY)
## Environment - 8× AMD Instinct MI325X (gfx942), ROCm 7.14.0, amdgpu 6.19.14 - Images: `vllm/vllm-openai-rocm:nightly` 2026-08-12 (v0.26.1rc1.dev668+g3ee2df303) **and** 2026-08-13 (`8f515d1d`) — both reproduce - Model: `zai-org/GLM-5.2-FP8`, TP=8, `--kv-cache-dtype auto`, `--max-num-batched-tokens 4096`, MoE backend AITER blockscale-FP8 (`fmoe_bf16_a16_bloc
issue
[Bug]: Standalone torch.compile cache uses stale artifact path after relocation
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text Collecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.3 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 Clang version : Could not collect CMake version
pull request
[refactor] Refactor SpeculativeConfig for speculative method extensibility
## Purpose Refactor `SpeculativeConfig:__post_init__` from a monolithic method into a dispatcher to family specific init methods. The current `__post_init__` interleaves the config logic for all speculative methods into a single nested if/elif chain. This makes it hard to trace a single method's setup without reading the entire function, and adding a new spe
pull request
[Misc] Add unit test for write_zeros_to_output MoE kernel
Adds a unit test for `write_zeros_to_output` (`vllm/model_executor/layers/fused_moe/fused_moe.py`), part of RFC #48480. The function zeroes the output tile when a block's expert is not on the current EP rank (`off_experts == -1`). It is a `@triton.jit` device function, so a launcher kernel wraps it and results are compared against a PyTorch reference. 4 test
pull request
[Misc] Add unit test for chunk_local_cumsum kernels
## Purpose Adds unit tests for the `chunk_local_cumsum` GDN/FLA kernels, which currently have none. Tests call the public entry point, so both the scalar (3D) and vector (4D) paths are covered, along with varlen (`cu_seqlens` plus caller-supplied chunk indices, as GDN/KDA pass them), partial tail chunks, reverse mode, and the bf16 downcasting store. ## Test
pull request
[Misc] Add unit tests for the chunked scaled-dot KKT kernels
## Purpose Adds unit tests for both chunked scaled-dot K@K^T kernels in `tests/kernels/test_chunk_scaled_dot_kkt.py`, neither of which had coverage: - `chunk_scaled_dot_kkt_fwd` (GDN/FLA) — scalar gate, also serves grouped K heads - `chunk_kda_scaled_dot_kkt_fwd` (KDA) — per-channel gate, also returns the non-strictly causal `Aqk` ## Test Plan 16 cases (8 pe