vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Bugfix][Build] Restore DeepGEMM SM120 support while preserving SITU
## Purpose Addresses the current-main regression reported in #47436 for block-scaled FP8 checkpoints on SM120. The Kimi K3 integration changed the vendored DeepGEMM revision from the `nv-dev` commit used by vLLM v0.26.0 to `f5a76426`, which provides SITU support but does not contain the SM120 scale-factor layout dispatch. As a result, current nightly builds
pull request
[1/2][Model Runner V2] DBO support, eager mode
DBO for Model Runner V2 (RFC #50738) is two PRs: -> #50945 [1/2][Model Runner V2] DBO support, eager mode (P0–P2) #51700 [2/2][Model Runner V2] FULL CUDA graph capture for microbatched steps (P3–P4) ## Purpose Relate to #50738. This PR finishes stage P0, P1, P2 ## Test Plan Benchmark results are shown in the RFC smoke.sh ```bash #!/usr/bin/env bash # Baselin
pull request
[Bugfix][Model] Pad SigLIP text prompts to the trained sequence length
## Purpose SigLIP is trained with `padding="max_length"` and without an attention mask, so padding tokens are part of the input and the pooled embedding is taken from the last position. Text embeddings computed without padding are not aligned with the image embeddings, which makes image-text similarity unusable. The existing test only passes because it opts
pull request
Bolins/shm tensor arena
<!-- markdownlint-disable --> ## Purpose `MessageQueue.enqueue` (`shm_broadcast.py`) already routes CPU tensors out-of-band via `_reduce_tensor` (protocol-5 `PickleBuffer`, #48442), which removed the dominant cost of the old in-band path — copying tensor bytes into the pickle stream. Two costs remain for a large multimodal `pixel_values` tensor on a TP=N wor
pull request
[BugFix] Reserve the bonus query slot in DFlash scheduling budget
## Purpose The existing generic parallel-drafting calculation only reserves K - 1 slots for DFlash, leaving the scheduling budget short by one slot per request. For example, with: max_num_batched_tokens = 2048 max_num_seqs = 256 num_speculative_tokens = 8 the previous calculation allowed: max_num_scheduled_tokens = 2048 - 7 * 256 = 256 However, a full DFlash
pull request
[Bugfix] Initialize DeepGemmQuantScaleFMT oracle lazily; bound QuantFP8 UE8M0 packed path to group_size 128
## Purpose Fix a crash and a latent kernel-contract violation in `QuantFP8`'s DeepGEMM UE8M0 fast path, found by running the FP8 kernel suite on an RTX PRO 6000 (SM120). `DeepGemmQuantScaleFMT.from_oracle()` asserts its cache is populated, but the cache is only filled as a side effect of `_lazy_init()` running for some other DeepGEMM wrapper (it's called at
pull request
[Frontend] Add reusable TP1 initialized-engine snapshots
<!-- markdownlint-disable MD013 MD041 --> ## Purpose @elizabetht's RFC #34303 and #37921/#37925 split CUDA checkpoint work into in-process primitives and leave CRIU disk persistence for a later phase. #35934 proposes a CRIU-safe engine suspend and resume path, while PRs #42790 and #46089 propose provider-owned pod snapshot hooks. @fergusfinn's data in #34303
pull request
[ROCm] update triton in base docker for gluon compatibility
## Purpose To pick up https://github.com/ROCm/triton/pull/960 for fixing the gluon mla kernel compilation problem related to DistributedLinearLayout was rejected in the previous triton commit. With this, the gluon kernel can compile and serve K3 with accuracy. Context: Initially vllm/vllm-openai-rocm:nightly docker failed at the first MLA decode with a Gluon
pull request
[ROCm] Pad non-aligned AITER MLA heads
## Summary - pad non-16-aligned ROCm AITER MLA query heads to the next supported multiple of 16 and slice padding from the output - size dense and sparse persistent metadata for the padded launch shape - enable Kimi-K3 TP4's 24 heads/rank to use AITER MLA instead of falling back to Triton MLA ## Performance 8x MI355X, Kimi-K3 TP4/DP2/EP8, 100,000 input token
pull request
[Bugfix] Reject negative token ids as out-of-vocabulary
## Purpose `InputProcessor._validate_model_input` validates that caller-supplied token ids are within the vocabulary, but it only checks the upper bound, so a negative token id passes validation. A token id is used as an index downstream, and a negative index is never valid input. This PR adds a symmetric lower-bound check beside the existing upper-bound che
pull request
[Bugfix] Handle HarmonyError in process_chunk to fix gpt-oss streaming 500s
FIX #51977 ## Purpose gpt-oss requests fail with HTTP 500 (`openai_harmony.HarmonyError: unexpected tokens remaining in message header`) when the model samples a malformed Harmony message header. `HarmonyParser.flush()` already catches `HarmonyError` and recovers by returning the raw decoded text, but `process_chunk()` calls `self._harmony_parser.process(tok
issue
[Bug]: KV Cache Quantization not working on v1 (rtx3090) "type fp8e4nv not supported in this architecture"
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```text ============================== System Info ============================== OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version : Could not collect CMake version : version 4.1.0 Libc version : glibc-2
issue
[Feature]: Qwen3 Omni Transcriptions
### 🚀 The feature, motivation and pitch Would love to see this model included in the supported as a Transcription model. According to the docs it looks like only 4 different models are supported as of now. https://docs.vllm.ai/en/latest/models/supported_models/#transcription ### Alternatives _No response_ ### Additional context _No response_ ### Before submi
issue
[Perf] DSD arms pay a large baseline tax vs no-spec under production defaults; PIECEWISE override identified as one factor
## Motivation While benchmarking `num_speculative_tokens_per_batch_size` (Dynamic SD) schedules on H100 NVL 94GB (`prithivMLmods/gemma-4-31B-it-qat-FP8` target + `google/gemma-4-31B-it-qat-q4_0-unquantized-assistant` draft, vLLM `c5d967c23`) I found that **every arm using a `speculative_config` pays a substantial throughput tax vs no-spec under production de
issue
[Bug]: DeepSeekV4 DSpark CUDA graph capture failure on H100
### Your current environment <details> <summary>The output of <code>python collect_env.py</code></summary> ```textCollecting environment information... ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version
issue
[Bug] v0.27.0 engine permanently stalls after ~1 min idle on 4-node TP=4 (GB10/sm_121, aarch64): shm_broadcast writer starves, requests never reach scheduler
## Bug On 4-node TP=4 (GB10/sm_121a, aarch64), a v0.27.0 engine serves its first requests normally, then **permanently stalls after roughly one minute of engine idleness**. Requests after the gap never enter the scheduler (`num_requests_running` stays 0), the API keeps answering `/v1/models` and `/tokenize`, and the head's EngineCore logs repeat: ``` [shm_br
pull request
[ROCm] Add per-call decode budget to sparse-MLA indexer
## Purpose The ROCm AITER sparse-MLA indexer decode path allocates a fresh fp32 logits tensor on every call to `rocm_fp8_paged_mqa_logits`: `(heads, B*next_n, max_model_len)` on the stage1 3D path, or `(B*next_n, max_model_len)` on the gfx942/gfx950 fused 2D path. With long context and a large running batch the per-call working set is multi-GiB; across ~60 l
pull request
[Kernel] Warm up hybrid GDN/Mamba/MRoPE kernels
## Purpose Hybrid Qwen-style GDN/Mamba/MRoPE models can trigger Triton compilation during the first inference request when startup warmup does not cover the exact runtime compile keys. This revision migrates the remaining warmups to the shared kernel-owned warmup contract and removes coverage that is already provided by current upstream `main`. ## Changes -
pull request
[Model][MiniMax-M2] Fix EAGLE-3 aux hidden-state layer off-by-one
## Summary When EAGLE-3 speculative decoding is enabled for `MiniMaxAI/MiniMax-M2.5`, the auxiliary hidden states fed to the draft head are taken from the wrong decoder layer, causing position-0 acceptance to collapse to ~10% (vs ~45% expected) and making MTP a throughput *regression* vs no speculative decoding. The root cause is an off-by-one between the tw
pull request
[CPU][Zen] Route Int8 MoE inference through zentorch on AMD
## Purpose Routes CPU MoE on AMD Zen through a zentorch-backed Int8 W8A8 fused-MoE backend: the per-expert int8 GEMMs are dispatched through torch.ops.zentorch.*, mirroring the existing zentorch W8A8 dense-linear integration. On non-Zen CPUs, or when zentorch is not installed, the int8 MoE oracle falls through to the existing backends, so behaviour elsewhere
pull request
[Kernel] TD operand loads for batched MoE GEMM (moe_mmk) on XPU
### Purpose Add a Tensor-Descriptor (TD) operand-load path to `moe_mmk` / `expert_triton_kernel` (the batched MoE expert GEMM) and enable it on XPU. On XPU, masked `tl.load` feeding `tl.dot` bypasses the Xe XMX 2D-block-read path; loading both operands via `tl.make_tensor_descriptor` restores it. Output is bit-identical to the non-TD path. The batched kernel
pull request
Hardware-agnostic model definition via HF transformer backend (1/N)
## Purpose This PR is an alternative approach to realize hardware-agnostic model definitions based on the HF transformer backend. In particular, the idea is that the modeling code of tail models resides in HF transformers and can be executed in vLLM through the help of the `transformers` backend, i.e., `--model-impl transformers`. The way this backend curren
pull request
[XPU] Enable XPU blockfp8 for DSv3
## Purpose Enable DeepSeek-V3 on Intel XPU through the oneDNN block-scaled FP8 GEMM (XPUFp8BlockScaledMMKernel). Three issues blocked correct execution: 1. oneDNN requires N to be a multiple of block_n. Some DeepSeek-V3 block-FP8 projections have a per-partition N that isn't 128-aligned. We now pad the weight's N up to the next multiple of block_n in `proces
pull request
[Bugfix][CPU] Enable C++ causal_conv1d GDN path and float32 SSM cache on non-AMX AVX-512BF16 CPUs
## Purpose CPU GDN attention (Qwen3.5) selects its causal-conv1d implementation via `torch.cpu._is_amx_tile_supported()`: AMX CPUs (Intel GNR) use the C++ kernels (`causal_conv1d_fwd_cpu` / `causal_conv1d_update_cpu`), everything else falls back to `causal_conv1d_fn_cpu` / `causal_conv1d_update_torch`, a pure-PyTorch per-sequence loop. The C++ conv kernels (
pull request
[ROCm][AMD][Installation] add LMCache kv-connector installation and runtime packages to docker image
## Purpose Ship the LMCache KV connector in the ROCm image so it works out of the box. Note: vLLM's CUDA `docker/Dockerfile` installs LMCache when `INSTALL_KV_CONNECTORS=true`, and the release pipeline sets that for every published `vllm/vllm-openai` tag. docker/Dockerfile.rocm never referenced that arg, so vllm/vllm-openai-rocm ships without LMCache and use
pull request
[Core] Configure custom encoder cache managers from VllmConfig
## Purpose Expose custom encoder cache manager configuration through `VllmConfig` for both online and offline inference. Custom encoder cache managers may require policy-specific parameters in addition to the encoder cache size. This PR provides a generic configuration path while preserving compatibility with existing built-in and constructor-only cache mana
pull request
[1/N] HiSparse: host-resident sparse-MLA decode hot-buffering
Initial implementation based on #46326, with substantial improvements; notably, a heirarchical design which includes a standard device resident KV cache in addition to the hot buffer and host cache. Includes support for DeepSeek v4. Supersedes #46326 Preliminary results on B300: <img width="997" height="601" alt="hisparse_pd_staged_pareto" src="https://githu
pull request
[RFC][Bug Fix][Spec Decode] Require explicit speculative methods
This PR removes speculative-method guessing and makes the method an explicit contract. ## Problem On current `main`, vLLM guesses the speculative method from model path/name substrings such as `eagle3`, `dflash`, and `dspark`. A DFlash or DSpark checkpoint at a neutral training path can therefore be treated as the generic autoregressive `draft_model`, while
pull request
[Bugfix][Model] Fail fast on Inkling's unsupported GPU architectures
<!-- markdownlint-disable --> ## Purpose Fixes the startup crash reported in #51405. Serving Inkling on SM120 (RTX PRO 6000 Blackwell, GeForce Blackwell, DGX Spark) died during startup profiling with ``` AssertionError: Paged KV not supported on SM 12.0 in this PR RuntimeError: Worker failed with error 'Paged KV not supported on SM 12.0' ``` raised from the
pull request
[Attention][MLA] Fuse Kimi-K3 chunked-context K/V packing
## Purpose The K3 MLA layer delegated chunked-context prefill to `impl._compute_prefill_context`, which per chunk casts `kv_nope` to fp8, casts `k_pe`, concatenates `[k_nope | k_pe]`, and re-quantizes a query the fused new-token epilogue had already quantized. This gives the layer its own context loop so that tail collapses into one kernel per chunk: `fused_
pull request
[ROCm][Perf] Kimi-K3 Remove prefill pipeline stall in chunk KDA
## Purpose On ROCm's Kimi-K3 path, each prefill/mixed step has a stall in `prepare_chunk_indices`, caused by two interacting factors: ``` def prepare_chunk_indices(cu_seqlens: torch.Tensor, chunk_size: int) -> torch.Tensor: indices = torch.cat( [ torch.arange(n) for n in triton.cdiv(prepare_lens(cu_seqlens), chunk_size).tolist() ] ) return torch.stack([indic
pull request
Remove NIXL reinstall step
## Purpose Remove the NIXL wheel reinstall step. It is not longer needed (NIXL fixed the dependency issues upstream) and it is not working correctly since it does not use the version pin. Currently it causes nixl-cu13 to float to 1.3.2 while nixl/nixl-cu12 stays on the kv_connectors.txt pin (nixl == 1.3.1). ## Test Plan Verified by simulating the Dockerfile
pull request
[MoE] Refine FlashInfer one-sided All2All integration
## Purpose Refine the FlashInfer NVLink one-sided All2All integration for DeepSeek Blockwise FP8 MoE and sequence parallelism. - Describe one-sided activation payloads explicitly in bytes for BF16, NVFP4, MXFP8, and DeepSeek Blockwise FP8. - Dispatch E4M3 activations with FP32 1x128 scales and feed the received layout directly to the FlashInfer TRT-LLM DeepS
pull request
[CI] Split Quantization job into three directory-based steps
## Purpose Part of the effort to get every CI job under ~30 minutes. The `Quantization` step is one of the 10 longest jobs in main CI: 57–67 min of test execution (timeout 75) in recent nightly full runs ([83094](https://buildkite.com/vllm/ci/builds/83094), [83068](https://buildkite.com/vllm/ci/builds/83068)). This splits `tests/quantization/` into subdirect
pull request
[Bugfix] Preserve Anthropic disable_parallel_tool_use
## Purpose Anthropic’s `disable_parallel_tool_use` was silently discarded, leaving the converted OpenAI request with `parallel_tool_calls=True`. This preserves the field and maps it to the existing inverse OpenAI setting. ## Reproducer ```python request = AnthropicMessagesRequest( model="test", messages=[{"role": "user", "content": "Use a tool"}], max_tokens
pull request
Revert "[Perf][ROCm] Dual-stream decode with hipgraphs"
Reverts vllm-project/vllm#48223
pull request
Support DSpark configs with `architectures=DSparkDraftModel` + `model_type=qwen3`
## Purpose Generic Qwen `DSparkDraftModel` is now normalized to `Qwen3DSparkModel` so models like https://huggingface.co/RadixArk/Qwen3.8-2.4T-A95B-DSpark can now run on vLLM ## Test Plan ## Test Result Tested gsm8k with ``` vllm serve mgoin/Qwen3.8-2.4T-A95B-NVFP4-pruned75 -tp=4 --spec-model RadixArk/Qwen3.8-2.4T-A95B-DSpark --spec-method dspark --spec-toke
issue
[CI Failure]: Gemma3 OOMs with transformers backend
### Test group mi250_1: Multi-Modal Models (Standard) 2: qwen3 + gemma ### Describe the failing test This is not exactly a test failure, but it has been recommended to investigate further the OOMing event of Gemma3, which is a 4B model. The intuition here is that the fake tensor that is used for profiling is large enough that exceeds the 64 GB of MI250 GPUs.
issue
Qwen-3.5 9B often producing repetitive/garbled output with Intel Backend
### Your current environment This was the output from running python collect_env.py (running in docker image) ============================== System Info ============================== OS : Ubuntu 24.04.4 LTS (x86_64) GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version : Could not collect CMake version : version 4.3.1 Libc version : glibc-2.39
issue
[Performance]: Qwen3.5 native MTP can be slower than no-MTP CUDA graph baseline despite good acceptance
## Summary We are benchmarking a Qwen3.5 OCR checkpoint with native MTP support in vLLM. The model has a lightweight native MTP branch: - `mtp_num_hidden_layers=1` - `mtp_use_dedicated_embeddings=false` In our OCR workload, MTP acceptance is not terrible. MTP1 reaches around `82%-88%` acceptance, and MTP2 can reach mean accepted length above `2.3`. However,
issue
[RFC]: StructuredOutputManager x Speculative Decoding Refactor
### Motivation. Simplify the contract between `Scheduler`, grammar backend, and `StructuredOutputManager`, especially when speculative decoding is used. `StructuredOutputManager` exists because constrained decoding often doesn't kick in until a model has finished reasoning. With speculative decoding, grammar-invalid draft tokens need to be rejected, draft to
issue
[Performance]: Dynamic speculative decoding (num_speculative_tokens_per_batch_size) causes catastrophic aggregate-throughput collapse under concurrency at the batch-size threshold (MTP, V1/PIECEWISE)
### Report of performance regression Enabling **dynamic speculative decoding** via `num_speculative_tokens_per_batch_size` produces two effects on a Qwen3.5-122B MTP (k=2) setup: 1. **Expected:** the documented `FULL_AND_PIECEWISE → PIECEWISE` cudagraph downgrade, which costs ~14% single-stream (59.6 → 50.0 tok/s). This is fine and matches the warning. 2. **
issue
[Bug][DSV4-Flash][DSpark] v0.27 weight loader regressed — KeyError routed_experts.w13_weight_scale (v0.26 gracefully skipped)
### Your current environment - vLLM: **0.27.1** (pip, `/data1/vllm027` venv) - Python 3.11 - torch 2.13.0+cu130, triton 3.7.1, flashinfer-python 0.6.16.post3, transformers 5.15.0, CUDA 13.0 - Hardware: 8× NVIDIA H20-3e (141 GB), TP=8 + expert parallel - (Full `collect_env.py` available on request.) ### How to reproduce Serve DeepSeek-V4-Flash-0731 with DSpar
issue
RFC: Kimi K3 MoonEP integration roadmap
## Motivation Track the **MoonEP integration** item under Large-scale Serving in #50001 as a small set of independently reviewable PRs. MoonEP (https://github.com/MoonshotAI/MoonEP) is the balanced EP communication library used by Kimi K3: dynamic redundant experts keep every EP rank at exactly `S × K` tokens regardless of router skew, with static shapes and
issue
[RFC]: Extended online quantization roadmap
### Motivation. Continuation of https://github.com/vllm-project/vllm/issues/32029. Online quantization API is now relatively stable at https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/layers/quantization/online and documented at https://docs.vllm.ai/en/stable/features/quantization/online/. This RFC summarizes and motivates proposed features
pull request
[ModelRunnerV2] Support prompt embeds
## Purpose Support prompt embeds for ModelRunnerV2. ## Test Plan ```bash VLLM_USE_V2_MODEL_RUNNER=1 pytest -sv tests/basic_correctness/test_basic_correctness.py::test_models -k "True-uni or True-mp" ``` Before ```bash E pydantic_core._pydantic_core.ValidationError: 1 validation error for VllmConfig E Value error, VLLM_USE_V2_MODEL_RUNNER does not yet support
pull request
[Model Runner V2][Spec Decode] Support spec decode with draft model
## Purpose Support spec decode with draft models in model runner v2 ## Test Plan ``` python examples/features/speculative_decoding/spec_decode_offline.py \ --method draft_model \ --model-dir /shared/models/modelscope/models/Qwen/Qwen3-32B \ --draft-model /shared/models/modelscope/models/Qwen/Qwen3-1.7B \ --num-spec-tokens 3 \ --dataset-name hf \ --dataset-pa
pull request
[ROCm][MLA] AITER FP8 ASM prefill backend
<!-- markdownlint-disable --> ## Purpose Add AITER persistent ASM kernels as a MLA prefill backend. Only for FP8 on gfx950, where it is now the default. #42509 added the (causal) AITER ASM prefill kernel as part of the AITER MLA backend, but it only works for _non-chunked prefills_. For chunked batches _we still currently fall back to FA_. This PR integrates
pull request
[Frontend] Support count_reasoning_tokens in the Streaming Parser Engine
## Purpose Add token-aware reasoning token counting for the Streaming Parser Engine and surface the count through OpenAI-compatible usage fields. - Adds `completion_tokens_details.reasoning_tokens` to usage responses. - Propagates token counts through the parser engine pipeline: `TokenIDScanner -> IncrementalLexer -> StreamingParserEngine -> SemanticEvent`.
pull request
[Bugfix][MoE] Preserve unquantized weight storage on ROCm
## Summary - Preserve unquantized MoE parameter storage during ROCm post-load padding by copying the padded view into existing storage instead of rebinding `.data`. - Return directly after AITER unquantized MoE `shuffle_weights()` so the AITER runtime layout is not forced through an extra `.contiguous()` conversion. - Add a ROCm MoE weight replay test coveri