vllm-project/vllm
AI InfrastructurevLLM
Publicly indicated next
- No public prerelease or open milestone found.
Prereleases and milestones indicate public plans; they are not delivery commitments.
vllm-project/vllm
Open repository ↗vLLM
599 signals · 707 observation events
A high-throughput and memory-efficient inference and serving engine for LLMs
PROJECT NEWS
All news →Release, roadmap, and discussion
Observation trail
- changedupdatedAt
- discoveredinitial snapshot
- discoveredinitial snapshot
- changedtext, updatedAt, metrics, labels
- discoveredinitial snapshot
- changedupdatedAt, metrics
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt
- changedmetrics
- discoveredinitial snapshot
- changedmetrics
- changedupdatedAt, labels
- changedupdatedAt
- changedupdatedAt, state
- changedtext, updatedAt
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
- changedupdatedAt, state
- changedupdatedAt
- discoveredinitial snapshot
- changedupdatedAt
pull request
[Perf] Use list.extend() over append loops in FlatLogprobs + minor hot-path cleanups
Minor Python-level performance optimizations in hot paths: 1. **FlatLogprobs.append_fast**: Replace per-item append() loop with list.extend() — ~1.37x speedup on the per-token logprobs path. 2. **common_prefix**: Compare elements by index instead of creating slice copies — ~9.4x speedup on long shared prefixes. 3. **FlatLogprobs.__iter__**: Use self[i] inste
pull request
[Doc] Add comprehensive --speculative-config documentation
## Summary - Adds a new `--speculative-config` Reference section to the speculative decoding documentation (`docs/features/speculative_decoding/README.md`) that enumerates all user-facing configuration keys accepted by `--speculative-config` (CLI) and the `speculative_config` Python dict - Documents every key's type, default value, valid ranges/values, and w
pull request
[WIP][Model Runner V2] support spec decode + mamba align prefix caching
## Accuracy Benchmark **Server Command** ``` VLLM_USE_V2_MODEL_RUNNER=1 vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \ -tp 2 -dp 1 \ --enable-prefix-caching \ --mamba-cache-mode align \ --max-num-seqs 64 \ --attention-config '{"use_trtllm_attention": 0}' \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \ --default-chat-template-kwargs '{"enable
pull request
[Bugfix][MooncakeStore] track resumed requests via scheduler's resumed_req_ids
## Summary The MooncakeStore connector tracks block tables per request so it knows whether a step's `new_block_ids` should be **appended** (normal decode) or **replace** the existing table (resumed from preemption). It was deciding this from a connector-local `_preempted_req_ids` set, which drifts from what the scheduler actually does. The scheduler already
pull request
[ROCm][Bugfix] window-correct shuffled fp8 decode for SWA layers
## Purpose `AiterFlashAttentionImpl.forward` routes sliding-window (and sinks / multi-token spec-decode) decode through a branch that calls `flash_attn_with_kvcache` or `unified_attention`, both of which read the plain (non-shuffled) KV layout. That branch opens with `assert not rocm_aiter_ops.is_shuffle_kv_cache_enabled()`, so a sliding-window model with th
pull request
[ROCm][Kimi-K3] Add opt-in gfx942 MXFP4-to-int4 conversion
## Summary Add an explicit `int4_per_group_32` MoE override for Kimi-K3 on gfx942. When selected, the MXFP4 expert weights are requantized once at load time and served through AITER's replacement BF16 x packed-int4 FlyDSL path. The conversion is lossy and never happens by architecture alone. The default MXFP4 path is unchanged. ## Why gfx942 does not support
pull request
[Bugfix] Fix step3p5 tool parser and unnecessary unstreamed tool args in serving.
## Purpose The step3.5 tool parser and test cases have been modified, resulting in a more stable parser. In `vllm/entrypoints/openai/chat_completion/serving.py`, some parsers do not require checking unstreamed tool arguments. For example: **qwen3coder_tool_parser**: does not maintain variables for checking unstreamed tool arguments (`streamed_args_for_tool`)
pull request
[BugFix] Fix MoE g_idx params causing ValueError with actorder=null AWQ models
## Summary - Fix `CompressedTensorsWNA16MarlinMoEMethod.create_weights()` to only register `g_idx` parameters when `self.actorder` is set, preventing `ValueError` from `track_weights_loading()` when loading compressed-tensors AWQ MoE models with `actorder=null` - Fix `CompressedTensorsWNA16MoEMethod.create_weights()` to remove unreachable `g_idx` registratio
pull request
[CI] laguna xs gsm8k test
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
pull request
Fix MRV2 Gumbel sampling for non-finite logits
## Summary Fix MRV2 Gumbel sampling when the candidate logits contain `-inf` entries. The current Gumbel path only clamps the uniform draw away from zero. If the draw reaches the upper endpoint after floating-point rounding, the Gumbel noise becomes `inf`. Adding that noise to a masked/non-candidate `-inf` logit produces `NaN`, and a subsequent Triton reduct
pull request
[Perf][Triton] Reassociate softmax scale onto Q in unified attention
## Purpose Reassociate the softmax scale so ``score_scale * Q`` is loop-invariant, letting the compiler hoist the multiply out of the tile loop instead of scaling the BLOCK_M x TILE_SIZE S matrix on every iteration. Skip the reassociation for the fp8 query descale path, where folding the tiny score_scale into fp8 Q would requantize the product and destroy pr
pull request
[Bugfix] EXAONE 4.5: drop trailing MTP entry from text_config.layer_types
## Purpose Let EXAONE 4.5 (e.g. `LGAI-EXAONE/EXAONE-4.5-33B-FP8`) and its self-MTP speculative-decode path load on mainline `transformers` v5 + vLLM v0.23.0. The LG-AI EXAONE 4.5 release configs publish `text_config.layer_types` of length `text_config.num_hidden_layers + 1`: the trailing entry is the MTP head registered alongside the regular transformer stac
pull request
[Bugfix] EXAONE 4.5: rename Exaone4_5_VisionBlock.forward kwarg seqlens -> sequence_lengths
## Purpose `EXAONE4_5_VisionTransformer` inherits from `Qwen2_5_VisionTransformer`, and the parent class's vision-block loop (`vllm/model_executor/models/qwen2_5_vl.py:1100`) now invokes each block with `sequence_lengths=` rather than the older `seqlens=`: ```python hidden_states = blk( hidden_states, cu_seqlens=cu_seqlens_now, rotary_pos_emb_cos=rotary_pos_
pull request
[Bugfix] test_batch_inference_correctness now uses batch invariance
## Purpose Partially Fixes [48038](https://github.com/vllm-project/vllm/issues/48038). An example PR fixing a test flagged by our audit. The test now enforces batch invariance to prevent accidental numerical changes causes falsely failing tests. The test is no longer coincidentally correct and enforces a strong contract: Spec decode exact matching only when
pull request
[KV Connector][4/N][NIXL] Recover all dedup'd HMA pool members (FA + multi-group SSM) under PP
## Purpose Fixes silent KV-cache corruption for **hybrid (Mamba/SSM) models under pipeline parallelism** in NIXL PD-disaggregated serving. Under PP>1 + the hybrid-memory allocator (HMA), the general-case allocator pools **one layer from every `kv_cache_group`** into a single shared tensor. For a multi-Mamba-group hybrid (e.g. NVIDIA Nemotron-3-Ultra-550B: 4
pull request
[Model][Quant] Fused WNA16 GEMM for tied quantized lm_head logits
## Purpose Follow-up to #45535 (compressed-tensors WNA16 input embeddings + tied `lm_head`). This PR speeds up the **logits matmul** when a quantized embedding is reused as a tied `lm_head`. In #45535, `CompressedTensorsEmbeddingWNA16Int.apply()` dequantizes the entire packed `[vocab, hidden]` table into a dense weight on every call and runs `F.linear`. This
pull request
support rl feature : weight checker
## Purpose This PR extends the POST /weight_checker test suite so the weight-checker flow is exercised under both single-GPU (TP=1) and tensor-parallel (TP=2) configurations)Skip all cases now run for both TP settings via pytest.mark.parametrize. The coverage includes snapshot, compare, checksum, reset, and invalid-request error handling, plus a new assertio
pull request
Revert "[Attention] Add FlashInfer XQA decode support on SM12x" (#49718)
Reverts #49718 ([Attention] Add FlashInfer XQA decode support on SM12x). ## Why `GPQA Eval (GPT-OSS) (DGX Spark)` regressed in nightly [build 83511](https://buildkite.com/vllm/ci/builds/83511) (commit `02ac178`) and is still red on the later build 83539. - Baseline nightly 83443 (same agent `spark-0df9-1`): **passed**, score 0.5619, 9m27s, zero harmony parse
pull request
[Core] Skip broadcasting mm tensor data to workers for prefix-cache-covered items
## Purpose Fixes #52040. ## Background: why this gap exists and who it affects The EngineCore->TP-worker broadcast was designed when request inputs were token IDs (kilobytes), so shipping them unconditionally was both correct and free. Multimodal grafted large tensors onto the same path, but for the classic workload (a fresh image per chat request) the worke
pull request
Revert "Avoid long-blocking H2D copies in ViT" (#51841)
Reverts the changes from #51841 ("Avoid long-blocking H2D copies in ViT"). ## Why `Arm CPU Test` has failed on every build since #51841 merged (nightly [#83608](https://buildkite.com/vllm/ci/builds/83608), plus per-commit postmerge builds 83603 and 83607 on the same commit). It passed on build 83539, the last run before the merge. The failure is at engine st
pull request
[Core] Add shared-disk weight transfer backend
## Purpose Add an inference-side `disk` weight-transfer backend for deployments where a trainer writes a complete safetensors checkpoint to shared storage and an orchestrator reloads it into a running vLLM engine through the existing start/update/finish control plane. The backend: - accepts one absolute local checkpoint directory per update session, visible
pull request
[WIP][XPU][Test]add xpu yaml
## Purpose only for xpu UT, please do not merge ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasti
pull request
[Helion] Route fusion-only kernels to Helion during CUDA-graph capture
vLLM's post-grad fusion passes emit native fused ops (RMSNorm+quant, SiLU+mul+quant, QK-norm+RoPE) that have no eager model call site, so the capture-time call-site router used for per_token_group_fp8_quant (#47799) cannot reach them. Instead, add a single post-fusion FX pass that runs after fix_functionalization and retargets schema-compatible native fused
pull request
[Render][1/n] Paged shared memory storage for mm tensor ipc.
## Purpose Paged shared memory storage for mm tensor ipc. 0. This compares two approaches for IPC of mm tensors: - **Paged shared memory (pshm):** mm tensor → shm → swap_blocks_batch → GPU - **ZMQ RPC:** mm tensor → ZMQ IPC → CPU Buffer → Pin Memory → GPU 1. CPU‑to‑GPU (H2D) bandwidth saturates when transferring 1 MiB blocks, so a page size of 1 MiB is a rea
pull request
[ROCm][Perf] Kimi-K3 latent-MoE: overlap the shared all-reduce with the routed up-projection
## Purpose Kimi-K3's latent MoE combines a **routed** expert output — produced in a compressed latent space, then up-projected back to hidden dim — with a **shared** expert output. Under tensor parallelism both need an all-reduce. Today the ROCm runner (`ROCmLatentMoERunner`) only implements the column-parallel tail (`_shard_up_proj_tail`, "Tier 2"), which f
pull request
[MM][CG][WIP] Encoder CUDA graph support for HunyuanVL
## Summary Working on encoder CUDA graph support for HunYuanVL (`HunYuanVLForConditionalGeneration`), tracked in #38175. Claiming this model in the tracker. This PR starts with a prerequisite refactor: `HunYuanVisionTransformer` currently processes images by splitting the batch into a Python list and running each image through every layer separately (`O(num_
pull request
[MoE] Size batched Triton MoE tiles from per-expert rows
## Purpose `get_default_config` divides `M` by `E` to estimate rows per expert, but the expert-batched layout passes `max_num_tokens`, which is already per-expert — so the fallback over-sizes tiles for the E small GEMMs `batched_triton_kernel` launches. Platform-agnostic as it is a layout mismatch. Only the heuristic fallback changes; override and tuned-file
pull request
[Model] Register sparse DeepSeek V3.2 MTP
## Purpose Register the sparse DeepSeek V3.2 MTP implementation and select it for `glm_moe_dsa` speculative decoding configs. GLM MoE DSA previously inherited the generic `DeepSeekMTPModel` override even though its draft model needs the sparse DeepSeek V3.2 implementation. This PR: - maps the config to `DeepseekV32MTPModel`; - registers that architecture wit
pull request
[XPU][Tests] Make tests device-agnostic
## Purpose Two kernel tests hardcoded a CUDA device, so they gave no coverage on XPU even though the kernels they exercise are available there. This makes both resolve the device via `current_platform` instead. **`test_fused_recurrent_packed_decode.py`** — gated on `torch.cuda.is_available()`, so it skipped on every non-CUDA accelerator. Both `fused_recurren
pull request
Add generic CI trace collectors
## What changed - export CMake/source → target → GNU build-ID artifact → mangled CUDA-kernel provenance from the CUDA image build - capture `ninja -t deps` and emit repo-local header → target edges - scan final nvcc object files and emit exact compiling translation-unit → mangled-kernel edges; ambiguous/generated/external objects are counted and left unmappe
pull request
[Model][NVIDIA] Add PCP support to DeepSeek V3.2 attention
## Summary - add prefill context parallelism (PCP) to the existing DeepSeek V3.2 attention implementation - keep TP and PCP on the same fused attention path instead of maintaining a separate PCP forward - materialize fused MLA and indexer K outputs only when PCP needs to gather them before cache insertion - specialize optional materialization and cache write
pull request
Avoid AITER FP8 BMM for MLA K projection
## Purpose This changes the ROCm AITER FP8 MLA decode path so the K-side up-projection stays dense after the post-load transform, while the V-side up-projection continues to use the AITER FP8 BMM. The K-side MLA projection is a skinny batched matmul. Requantizing the transformed K weight and routing it through AITER FP8 BMM adds overhead for this shape and a
pull request
[ROCm][Perf] Kimi-K3 AMD MLA: fuse the q-a and kv-a RMSNorms
## Purpose On the Kimi-K3 AMD MLA front-end, every token runs **two** separate RMSNorm launches — `q_a_layernorm(q_c)` then `kv_a_layernorm(kv_c)` — once per MLA layer. This PR collapses them into a single `fused_q_kv_rmsnorm` call (`models/common/ops/fused_qk_rmsnorm.py`), a **portable Triton** kernel already in the tree — no CUDA-only gate, and its one PDL
pull request
[Build] Gate cooperative top-k on CUDA 13
## Fixes - Build cooperative top-k kernels only on CUDA 13 or newer. - Let source builds skip bundled FA2/FA3 extensions while retaining Python-only FA4. - Treat an FA4-only installation as a valid FlashAttention installation. ## Verification - The patched source builds as vLLM 0.23.1rc1.dev1407+gdab991476 on CUDA 12.8. - The baked SkyRL image imports FlashA
pull request
[KV Offload] Carry request provenance through stored events
## Purpose Carry request provenance through asynchronous KV-offload completions so locally initiated stores and promotions can emit full `BlockStored` events. The current tracker snapshots full payloads at scheduler lookup/store time and keeps them in a global map. This PR instead keeps a lazy event locator with the request and builds the payload when the ra
pull request
[Bugfix][Model] Preserve Step3-VL zero-patch CUDA graph outputs
## Summary - honor the encoder postprocessor's `clone` contract when a Step3-VL image has zero local patches - keep the no-clone path as an alias while preserving cloned outputs across later CUDA graph replays - add a focused regression test for both ownership modes Without this, the single-part fast path returns the CUDA graph output buffer directly even wh
pull request
Include MM limits predicate in model compute_hash to prevent compile-cache poisoning
Withdrawn by author.
pull request
Demote xgrammar accept_tokens rejection log to DEBUG (spec decode replay spam)
Withdrawn by author.
pull request
Don't use the live processor cache for V1 multimodal profiling
Withdrawn by author.
pull request
[Observability] add model initializing duration log
## Purpose Add model initializing duration log ## Test Plan NA ## Test Result NA --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as
pull request
[Bugfix] Apply attention sinks in the Transformers backend
## Purpose vLLM PR #48270 adds native `GraniteSWAForCausalLM` / `GraniteMoeSWAForCausalLM` because "the transformers backend does NOT handle these models correctly. Sink tokens get dropped silently and so the model gives wrong output." That is accurate, and it is a bug in the Transformers modeling backend rather than something inherent to those models, so th
pull request
[Bugfix] Apply logits scaling in the Transformers backend
## Purpose Transformers scales the logits in the `ForCausalLM` wrapper. This backend only loads the decoder and provides its own head and logits processor, so whatever the wrapper does after the decoder has to be reproduced here. Only `logit_scale` was, which means: - Granite-family and MiniCPM3 models returned logprobs `logits_scaling`x too confident: 9x fo
pull request
[Perf][PCP] Shard decode requests across PCP ranks
## Purpose PCP currently replicates every decode request on every PCP rank. For PCP-only deployments (`DCP == 1`), every rank already has a complete KV-cache replica, so this repeats the same decode computation `pcp_size` times. This PR: - assigns decode-only rows round-robin across PCP ranks on every scheduler step; - keeps the existing DualChunkSwap prefil
pull request
Zen5 image build reuse cpu
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
pull request
Fix Cohere ChatV2 citation and tool handling issues
## Purpose Fix issues ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [X] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison
pull request
[DBO] Add DBO support to the DeepEP V2 backend
## Purpose ## Test Plan ## Test Result --- <details> <summary> Essential Elements of an Effective PR Description Checklist </summary> - [ ] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [ ] The test plan, such as providing test command. - [ ] The test results, such as pasting the results comparison before and
pull request
[Bugfix] Propagate fatal EngineCore shutdown
## Purpose Fixes #48966. When `EngineCore` dies unexpectedly, the watchdog asks Uvicorn to stop, but `serve_http()` currently returns the same no-op shutdown coroutine used for a normal server stop. The caller therefore completes cleanly and the serving process exits with status 0, so supervisors using `Restart=on-failure` do not restart it. This change keep
pull request
# [kimik3][ROCm] Enable torch.compile for so post-grad fusion passes work (aiter::fused_qk_rmsnorm_kernel, aiter::allreduce_fusion_kernel_1stage)
# [kimik3][ROCm] Enable torch.compile for so post-grad fusion passes work (aiter::fused_qk_rmsnorm_kernel, aiter::allreduce_fusion_kernel_1stage) ## Purpose Kimi-K3's model classes carry no `@support_torch_compile` and vLLM logs: ``` `torch.compile` is turned on, but the model /path/to/Kimi-K3 does not support it. Please open an issue on GitHub if you want i
pull request
[Kernel] Add tuned fused_moe config for NVIDIA GB10 (Nemotron-3-Super shape, E=512 N=2688)
**Problem.** No `E=512,N=2688,device_name=NVIDIA_GB10` config exists: every Nemotron-3-Super serve on a DGX Spark falls back to the default heuristic and logs "Using default MoE config. Performance might be sub-optimal!" Second config for this device, first for this shape (#45949 covers E=512,N=512 fp8). **What's in the file.** Decode keys M=1,2,4,8 tuned on