vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Core] CUDA Checkpoint/Restore — Phase 2: Engine/Executor/API Integration
## Summary Phase 2 of CUDA checkpoint/restore (RFC #34303). Wires `suspend`/`resume` through the full engine stack and exposes REST endpoints. **Depends on:** #37921 (Phase 1: C extension, Python wrapper, worker methods) ### Changes - **Executor** (`vllm/v1/executor/abstract.py`): `suspend()`/`resume()` via `collective_rpc`, `is_suspended` state tracking - *
pull request
[Build] Add SM121 (DGX Spark / GB10) to published build targets
## Summary - Add compute capability 12.1 (SM121) to `TORCH_CUDA_ARCH_LIST` in all build configurations: release wheel CI, Dockerfile, docker-bake.hcl, versions.json, and FlashInfer AOT build - `CMakeLists.txt` already lists 12.1 in `CUDA_SUPPORTED_ARCHS` for CUDA 12.8+ and 13.0+, and all kernel-specific arch lists (CUTLASS, Marlin FP8, MoE, NVFP4) already in
pull request
fix(structured_output): pass new_token_ids to should_advance() to fix MTP spec-decode off-by-one
Fixes #34650. ## Problem When Multi-Token Prediction (MTP) speculative decoding accepts a draft token, `_update_after_schedule` pre-increments `num_computed_tokens` by `(1 main + N spec)` tokens **before** model execution. After `update_from_output` appends the accepted tokens, the index-based delta in `should_advance()` starts one position past the main tok
pull request
[Attention] Re-enable cross-layer KV cache layout for MLA via stride-aware kernels
## Purpose #37090 disabled the cross-layer (block-major) KV cache layout for all MLA backends after #37032 (GLM-4.7-Flash garbage output with KV offloading), attributing the bug to "MLA kernels requiring contiguous per-layer KV cache views". The actual cause is narrower: a few kernels computed page addresses from `block_size * entry_size` instead of reading
pull request
[Bugfix][Mamba2] Fix assert crash when prefill-reclassified-as-decode occurs with no concurrent spec tokens
## Summary When a Mamba2-attention hybrid model (e.g. NemotronH) is served with both `--enable-prefix-caching` and `--speculative-config`, the server can crash with: ``` AssertionError File "vllm/model_executor/layers/mamba/mamba_mixer2.py", line 963, in conv_ssm_forward assert block_idx_last_scheduled_token_prev_step_d is not None ``` ### Trigger condition
pull request
[Bugfix] Fix UVA offload fallback copies
## Purpose Fixes the non-UVA fallback path reported in #37883. The issue was closed automatically, but received new reports on August 10 and 11. When UVA is unavailable or disabled, `UVAOffloader` copies offloaded module state to the accelerator for each forward call. The previous wrapper had several problems: - `state_dict()` inside `forward()` is not trace
pull request
fix: aggressively zero speculator buffers to prevent CUDA illegal memory access on H100
Fixes #47561. Root Cause: When running vLLM speculative decoding on H100s with quantized models (e.g. w8a8_fp8), the system was crashing with a CUDA illegal memory access exclusively during the speculator's prefill CUDA graph capture. During vLLM's warmup phase prior to graph capture, a "dummy run" eagerly executes the speculator (propose()). This leaves the
pull request
[Feature] Gigachat 3.5 support
## Purpose GigaChat-3.5-432B-A28B is a 432B-total / 28B-active MoE on a DeepSeek-V3-style backbone (MLA + DeepSeek MoE) with a hybrid attention stack: most layers use a Qwen3-Next Gated-Delta-Net (GDN) linear-attention block, and a periodic subset keeps full MLA. It adds gated zero-centered RMSNorm (low-rank gating bottleneck, four-norm sandwich around atten
pull request
[EC Connector] P2P NIXL + CPU EC Connector
## Purpose This PR extends ECCPUConnector to support P2P EC sharing based on NIXL. [The original PR](https://github.com/vllm-project/vllm/pull/42998) was split into two parts; [The first part](https://github.com/vllm-project/vllm/pull/47423) is a standalone CPU-based offloading EC connector, and this PR is an extension of it, allowing CPU-offloaded EC cache
pull request
[Perf] Skip logits and sampling for unfinished prefills
## Purpose Model Runner V2 currently produces one sampling-logit row for every request in the common non-speculative path, including chunked-prefill requests whose prefill does not finish in the current step. Although these sampled tokens are discarded later by setting `num_sampled` to zero, the model has already performed the vocabulary projection and the s
pull request
[Bugfix][V1] Reserve CUDA graph memory in V2 GPU model runner
## Purpose Fixes #49224. Model Runner V2 became the default for **dense** models in 0.25.1, but its `profile_cudagraph_memory()` was a placeholder returning `0`: ```python def profile_cudagraph_memory(self) -> int: # NOTE(woosuk): It is TBD whether we keep this API or not. return 0 ``` `Worker.determine_available_memory()` subtracts this estimate before sizi
pull request
[ROCm][Perf] gfx942: use FlyDSL fp8 MQA logits kernel (ROCm/aiter#3913)
On gfx942, replace the vendored Triton `fp8_mqa_logits` with `aiter.ops.flydsl.flydsl_fp8_mqa_logits` from ROCm/aiter#3913. Drop-in replacement (identical args/semantics), gated behind `_ON_GFX942`. gfx950 and other paths are untouched. ## Results: GLM-5.2-FP8, 8× MI325X TP8 ### ISL=128K, OSL=1K, Conc=8 | Metric | Baseline (Triton) | FlyDSL | Delta | |------
pull request
[Model][NVIDIA] Route DSA models to the SM100 implementation
## What this does Point the DSA architectures at the SM100-optimized implementation in `vllm/models/deepseek_v32/`, which nothing on `main` currently references. - `GlmMoeDsaForCausalLM` now resolves to `vllm.models.deepseek_v32` instead of the generic `deepseek_v2`, and `DeepseekV32MTPModel` is registered so speculative decoding uses the matching MTP module
pull request
[FEAT] Support fast engine recovery through weight cache
## Purpose Author: Siyu Liu @liusy58 Michael Qiu <qiudayu.qdy@antgroup.com>; --- Engine restarts are dominated by weight loading from disk. This PR proposes a persistent per-GPU daemon holds post-quantized, TP-sharded weights in GPU memory; restarting engines map them via CUDA IPC (zero-copy) instead of reloading from disk. ## Test Plan ## Test Result --- <d
pull request
[Core][Spec Decode] Fix KV cache allocation for sliding-window drafters and local-attention pool sizing
## Purpose Three fixes around KV cache allocation for local-attention models and spec-decode drafters. Found via Laguna-S-2.1 + DFlash, but only one commit is model-specific. On current main (GB10, Laguna-S-2.1-NVFP4, DFlash, fp8 KV, max_model_len 131072, max_num_seqs 16, 23.1 GB KV pool) the series takes the KV pool from 415,517 to 871,430 tokens (2.1x), wi
pull request
[ROCm]: Bump torch 2.13, triton 3.8, torchaudio, torchvision
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
pull request
[Bugfix][NIXL] Include transfer mode (push/pull) in the compatibility hash
#### Overview: Include the NIXL transfer mode (push vs pull) in the connector so a push (WRITE) connector and a pull (READ) connector can never be paired, and so an external router can distinguish them. Follow-up to #49230 (now merged), addressing review feedback from @iyastreb ([#49230 thread](https://github.com/vllm-project/vllm/pull/49230#discussion_r3686
pull request
[CI/Release] Add XPU wheel release build
## Summary This PR adds XPU wheel support to the release Buildkite pipeline. Changes included: - Add a new Buildkite release job to build and upload the x86_64 XPU wheel. - Add an `export_vllm_wheel_release` target in `docker/Dockerfile.xpu` for producing the XPU wheel artifact. - Update nightly wheel index parsing so `+xpu` local-version suffixes are recogn
pull request
[Bugfix] Scale KV page size for hidden states extraction with TP
## Summary - When `extract_hidden_states` is combined with tensor parallelism, the target model's KV page size shrinks (`num_kv_heads / TP`) but the hidden-state per-token cost stays at full `hidden_size`. This causes an assertion failure in `KVCacheSpecBase.__post_init__`: `assert self.page_size_padded >= real_page_size` - Scale up all target group block si
pull request
[ROCm] Gate Torch FP8 scaled-MM on architecture support
## Purpose Reject Torch FP8 scaled-MM kernels when the underlying PyTorch operator is unavailable. The existing implementation compares a CUDA-style numeric capability against the SM89 threshold. On ROCm, gfx1100 reports capability `110`, so it incorrectly passes this check even though RDNA3 has no native FP8 matrix support and `torch._scaled_mm` rejects the
pull request
[ROCm][MLA] Use saturated Gluon split bucket for CUDA graphs
## Summary This PR changes the default `AiterMLADecodeMetadata.min_kv_seq_len` from `1` to `16384`. The change is intentionally small, but the reason is specific to the interaction between vLLM FULL CUDA Graph capture and AITER's host-side Gluon split selection: - AITER uses `min_kv_seq_len` to choose the split-KV launch topology in its Python wrapper. - vLL
pull request
[ROCm][CI] Solidify entrypoint LLM lifecycle
- Replace direct `LLM(...)` construction throughout `tests/entrypoints` with the shared `VllmRunner` lifecycle. - Add an `ExitStack`-backed runner factory for tests that need one long-lived runner or several concurrent runners. - Consolidate multimodal, structured-output, offline-mode, collective-RPC, pooling, and weight-transfer cleanup onto the complete ru
pull request
[ROCm][Perf] Skip cleaning sparse prefill MQA logits
## Summary The sparse prefill path creates the MQA-logits buffer with `torch.full(..., -inf)`. This launches an FP32 fill before every logits kernel. This PR passes `clean_logits=False` to AITER, so the buffer uses `torch.empty` and the fill is removed. ## Safety The only consumer is `top_k_per_row_prefill`. It reads each row only inside `[cu_seqlen_ks, cu_s
pull request
Manual act quant fusion llama
Starts the `ActivationQuantFusionPass` manual-fusion migration (RFC #43224, specific tracker https://github.com/vllm-project/vllm/issues/43501) on the producer side of the `QuantizedActivation` contract (#44260). Adds `maybe_fused_act_quant`: given an activation and the linear it feeds, it emits a `QuantizedActivation` via the fused `silu_and_mul_quant` kern
pull request
[CI] Fix and extend PR/issue auto-labeling
## Purpose Auditing the labeling setup against the actual repo turned up rules that had stopped matching the code they were written for, plus issue-side labeling that covered only 6 of 62 labels. ### Rules that were missing real work - **`deepseek` missed the `vllm/models/` tree.** Newer models live in `vllm/models/<model>/` rather than `vllm/model_executor/
pull request
[RL] Add nccl-m2n sharding-aware weight transfer (worker side)
## Purpose Adds `nccl_m2n`, the inference-side half of a sharding-aware weight transfer backend built on [NCCL M2N](https://github.com/NVIDIA/nccl-extensions), per [RFC #46439](https://github.com/vllm-project/vllm/issues/46439). The broadcast NCCL backend assumes both sides hold the same layout, so a trainer that shards its parameters (FSDP, EP) must all-gat
pull request
[Bugfix][NVFP4 MoE] Support CuTeDSL for ModelOpt W4A16 and pad non-gated experts
**Closing since this is not a clean W4A16 as it will still quantize activations, so the comparison between the 2 backends is not apples to apples, and the correctness results below show degradation. I might open a separate PR for the padding fix** ## Purpose Enable the FlashInfer CuTeDSL NVFP4 MoE backend for ModelOpt W4A16 NVFP4 checkpoints on Blackwell, an
pull request
[ROCm][MLA] Add guarded gfx942 FP8 context prefill
## Summary Adds an opt-in gfx942 FP8 context-prefill path for Kimi-K3 MLA using the D192/V128 kernel from ROCm/aiter#4645. It also supports checkpoint-specific static per-head scales through a versioned external artifact, removing dynamic amax reductions while preserving the existing attention boundary. ## Background The existing AITER FP8 ASM kernels do not
pull request
[Hardware][Power] Unqualized MoE Backend for Power (VSX)
This PR adds PowerPC specific unquantized backend support for fused MoE using Power10 VSX MMA instructions. ## Purpose Currently, grouped GEMM is not supported for Power architecture in vLLM. This PR introduces a Power/VSX specific unquantized CPU backend for Fused MoE. Key features include: - Addition of `csrc/cpu/micro_gemm/cpu_micro_gemm_vsx.hpp` to suppo
pull request
[Bugfix] Fix extreme case in pythonic parser argument json safety
# [Bugfix] Pythonic tool parsers: argument JSON safety and per-call salvage ## Purpose Follow-up to #48171. Nine defects in the pythonic tool-call family, all found by running real LFM2 agent traces through the parsers. | Model output | Before | After | |---|---|---| | `calc(x=1e999)` | `{"x": Infinity}` — invalid JSON | call rejected | | `search(**{'query':
pull request
[ROCm] Enable V2 model runner for Kimi-K3 on ROCm
## Purpose Kimi-K3 on ROCm was gated from using V2 model runner. After validation using the up-to-date upstream. V2 model runner is working as expected. ## Test Plan Command to start Kimi-K3 on mi355x ``` export VLLM_ROCM_USE_AITER=1 export SAFETENSORS_FAST_GPU=1 export VLLM_ROCM_USE_AITER_MOE_SITUV2_A8W4=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 # enable/disa
pull request
[MOE] Standardize and abstract fused shared expert optimization selection
## Disclosure AI assistance was used. The changes were reviewed and tested manually. ## Purpose Standardize fused shared-expert (FSE) detection so model construction and checkpoint loading use the same quantization-compatible decision **throughout all models implementing FSE**. This is e.g. useful for `shared_expert` quantization through various quantization
pull request
[Spec Decode] Fix KV cache group explosion for hybrid models
## Purpose When profiling Qwen/Qwen3.6-35B-A3B + z-lab/Qwen3.6-35B-A3B-DFlash, we found the KV cache group_size become 1. This increases build metadata overhead. We found the target model Qwen/Qwen3.6-35B-A3B layer buckets are [30 mamba, 10 full]. The DFlash drafter z-lab/Qwen3.6-35B-A3B-DFlash adds [5 sliding, 1 full], the added 1 full attention layer that
pull request
[Frontend] Add explicit model info cache preparation
## Purpose PR #23558 by @manoelmarques added the source-hash-validated runtime ModelInfo cache and reported model-class inspection at 10.1034 seconds uncached versus 0.6127 seconds cached, n=1 per state. During review, @hmellor objected to mirroring common runtime requirements into the build. The merged design therefore populated the cache at runtime. Withou
pull request
[Frontend] Add prediction token usage details
## Summary - expose per-request speculative decoding acceptance and rejection counts through engine outputs - populate `completion_tokens_details.accepted_prediction_tokens` and `rejected_prediction_tokens` for streaming and non-streaming Chat Completions and Completions responses - preserve per-choice accounting for parallel sampling and add focused schedul
pull request
[CPU] Shard the MoE experts across NUMA nodes
## Purpose On a rank whose OpenMP threads span several NUMA nodes, the CPU MoE experts read their weights from every node. `parallel_2d` splits the work by thread index and has no notion of where a page lives, so each thread ends up pulling roughly `1 - 1/nodes` of its bytes across the interconnect. The kernel is already shaped for the fix. Both of its GEMM
pull request
[KV Offload] Expose data-parallel topology to offloading backends
## Purpose Native KV-offloading backends currently receive the engine's `data_parallel_index`, but not the total number of data-parallel replicas or the process-local DP rank. Consequently, `OffloadingParallelConfig` does not contain enough information to describe the DP topology. Add `data_parallel_size` and `data_parallel_rank_local` to `OffloadingParallel
pull request
[Parser] Migrate Hermes tool parser to the new streaming Parser Engine
# [Parser] Migrate Hermes tool parser to the streaming Parser Engine ## Summary Migrates the Hermes tool parser from the legacy regex/diff implementation (~275 lines) to the declarative streaming Parser Engine, following the pattern already used by Qwen3, MiniMax M2, Gemma4, Kimi K2 and Inkling. - New `vllm/parser/hermes.py`: `HermesParser(ParserEngine)` + a
pull request
[Security] Enforce server-side num_frames ceiling in VideoMediaIO merge
## Summary - **Fixes the incomplete CVE-2026-34755 remediation** reported in [GHSA-vxqj-p4gw-9h4c](https://github.com/vllm-project/vllm/security/advisories/GHSA-vxqj-p4gw-9h4c): request-level `media_io_kwargs.video.num_frames=-1` (or an arbitrarily large positive value) could override the engine frame-count ceiling, restoring the unbounded `video/jpeg` decod
pull request
feat: Add support for profile_prefix payload in HTTP /start_profile endpoint
## Proposal: Upstream API Support for Dynamic Profiling Options Allow users to pass dynamic settings to the profiler over the API (e.g., toggling native options or `experimental_config` flags) without requiring engine server restarts. This will help leverage the various options supported by native [`torch.profiler.profile`](https://docs.pytorch.org/docs/2.13
pull request
[Bugfix] Fix Cosmos3-Edge processor after transformers 5.15 release
## Purpose This PR fixes Cosmos3-Edge processor which is broken when `transformers==5.15` is used, due to refactoring of underlying Qwen3-VL processor. With the fixes preprocessor will work correctly for both transformers==5.14 and 5.15. Also as model was released removed `is_available_online=False` from registry. ## Test Plan ``` python -m pytest tests/mode
pull request
[Bugfix][Model] Fix DiffusionGemma silently freezing attention mask under CUDA graph replay
## Purpose While looking into whether FULL CUDA graphs are worth the added complexity for DiffusionGemma, I found that `DiffusionGemmaModelState._causal_buf` (the per-request flag that switches each request between causal/encoder and bidirectional/denoise attention) stops being updated once `CUDAGraphMode.FULL` graphs are captured. It's bool, but FlashAttent
pull request
chore: Upstream Cohere parser fixes + tests
Adding some local fixes for Cohere parsers alongside corresponding tests
pull request
[V1][CUDA graph] Dispatch uniform-decode batches to a padded FULL graph instead of falling to eager PIECEWISE
**TL;DR:** with speculative decode, some concurrencies silently fall back to running attention *eagerly* every step instead of from a captured CUDA graph, adding a fixed per-step latency bubble. This makes them dispatch to a captured graph instead. **What happens.** When `dispatch()` picks a CUDA graph for a batch, it looks up the smallest captured graph big
pull request
[CI Bug] Fix ci qwen3.5
## Purpose Forward fix for https://github.com/vllm-project/vllm/pull/51908 Fixes https://buildkite.com/vllm/ci/builds/83443#019ff660-8010-40e8-851d-bb6479f24c68 ## Test Covered in CI
pull request
[CI Bug] Fix ci moe test
## Purpose Fixes https://buildkite.com/vllm/ci/builds/83443#019ff2a1-641e-4d2b-bca9-9eda8a060573 There are two errors here: 1. vLLM side, we name it triton test, but actually running the flashinfer path, this PR fixes the issue 2. the root cause of flashinfer is a bug upstream with TRT-LLM BF16 MoE, we may wait for their fix, not related to this PR ## Test C
pull request
[Kernel] Add B12X dense linear backends
<!-- markdownlint-disable --> ## Purpose This PR integrates [B12X](https://github.com/local-inference-lab/b12x) dense linear kernels for NVIDIA SM120 and SM121 GPUs through the existing vLLM linear backend interfaces. B12X is an optional dependency installed with `vllm[b12x]` and pinned to `b12x==1.2.4`; it is a pure-Python CuTe DSL package and requires no a
pull request
[Bugfix] Fix packed GDN decode launch for large batch-head grids
## Purpose Avoid a CUDA launch failure in packed GDN decode when `batch_size * num_value_heads` exceeds the maximum CUDA grid Y/Z dimension of 65,535. The existing launch is preserved for normal sizes. Only overflowing cases use a split `(value_tiles, value_heads, batch)` grid. ## Test Plan ## Test Result - Verified the failing Qwen shape (`B=1024`, `HV=64`,
pull request
[Rust Frontend][gRPC] Advertise LoRA capabilities
## Purpose Advertise the Rust frontend's LoRA capability and capacity through the existing engine ready handshake and gRPC discovery APIs. - Report whether LoRA is enabled and the configured maximum number of active adapters from each Python `EngineCoreProc`. - Decode the capability in the Rust engine-core client and reject internally inconsistent or mixed-c
pull request
[Model] Skip unused Jina V5 output layers
## Purpose Jina Embeddings V5 models are pooling-only, but their vLLM wrappers inherit causal-LM classes. Because these wrappers already declare themselves as pooling models, they bypass the generic pooling adapter that replaces generation-only output layers. The encoder/nano variant therefore retained an unused `ParallelLMHead` with shape `[128256, 768]`. T