551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
## Summary - Delete the `cutlass_mla` attention backend (kernel + Python integration): SM10.0-only decode kernel, already disabled on GB300 (SM10.3), falls through to `FlashInferMLABackend` for everything except plain decode, no CI coverage. - Delete the non-Marlin GPTQ CUDA kernel (`gptq_gemm`/`gptq_shuffle`) and the `"gptq"` GPU quantization choice: supers
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> DeepSeek-V4's `unified_kv
## The bug An abort or timeout _during a prefill→decode transfer_ frees the request's KV pages back to the allocator, but it **_does not cancel the transfer_**. The next request can then be allocated those same pages and have them overwritten by the earlier, still-in-flight write, causing the next request to hold the first request's KV, with no error raised
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Support streaming session on NPU. ## Modifications Modify streaming_session.py, when pa
## Problem `build_dflash_verify_target_probs()` calls `top_k_renorm_prob()` and `top_p_renorm_prob()` unconditionally, but `sgl_kernel` only exports them under CUDA/MUSA. On ROCm both are `None`, so DSPARK/DFLASH speculative decoding dies with: ``` TypeError: 'NoneType' object is not callable ``` as soon as a request sets `top_p < 1` **or** `top_k > 1`. That
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Improve inference performance by reducing M-RoPE position computation overhead and impr
[by Codex] ## Motivation Llama 4 Scout with FlashAttention 3 crashes during warmup on H100 when the KV cache uses pages larger than one token. The same workload succeeds with `--page-size 1`. This is a long-standing paged-local-attention bug rather than a new 0.5.7 regression: the explicit page-size-16 reproducer fails on 0.5.6.post2, 0.5.7, and current `mai
## Motivation `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1` is meant to be the escape hatch that runs the FP8 MQA-logits indexer without DeepGEMM. Today it only covers the **dsv4** indexer path (`dsv4/metadata.py`, `dsv4/indexer.py`). GLM 5.2 (`GlmMoeDsaForCausalLM`, derived from `DeepseekV2ForCausalLM`) runs the **dsa** indexer instead, which calls `deep_gemm.get_p
## Motivation @humansand - [sgl-project/sglang#32490](https://github.com/sgl-project/sglang/pull/32490) added an SGL-kernel fallback because FlashInfer previously used one `row_starts` tensor for both score and page-table windows. - [flashinfer-ai/flashinfer#4169](https://github.com/flashinfer-ai/flashinfer/pull/4169) adds `page_table_row_starts`, allowing i
## Summary On MI355X (gfx950) the CK bpreshuffle w8a8 blockscale GEMM consumes the per-group activation scale in **column-major** `[num_groups, tokens]` layout. `aiter_w8a8_block_fp8_linear` quantizes the activation row-major and then relays the scale out with `materialize_bpreshuffle_fp8_scale` = `.t().contiguous().t()` — a real relayout **copy per dense w8
It is opened as a draft for partial task of #32607's Integrate MoonEP, not as a final production-ready merge request. ### What is included - Distinct `moonep` MoE A2A backend recognition. - MoonEP dispatch/combine data contracts and process-wide buffer facade. - BF16 contiguous expert layout and correctness-first expert segment runner. - Eager runtime path:
## Motivation Route `topk_transform_512` and `topk_transform_512_v2` to the native SYCL kernels on XPU. The SYCL implementations are added in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) and exposed as `torch.ops.sgl_kernel.topk_transform_512{,_v2}`. Without this dispatch, XPU runs fall through to the JIT CUDA path and fai
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> To use a fused GDN kernel
## Description Reinterpret MXFP4 expert scales as raw `uint8` E8M0 bytes for the Intel XPU fused MoE kernel. <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31554105834](https://github.com/sgl-project/sglang/actions/runs/31554105834)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-test-extr
## Summary Whisper's feature extractor pads or truncates input to its fixed 3000-mel-frame (30-second) encoder window, so `/v1/audio/transcriptions` silently drops everything after 30 seconds when the serving layer sends a long file as one generation. This change: - splits long Whisper audio into contiguous chunks of at most 30 seconds; - places each seam at
## Summary Allow TopK v2 to populate a raw-index output buffer. DSV4 sparse prefill internally allocates `c4_sparse_raw_indices` even when `--enable-return-indexer-topk` is disabled. The previous `raw_indices is None` gate therefore forced the InfX sparse-prefill path to fall back to TopK v1. When `--enable-return-indexer-topk` is enabled, the capture-only t
## Motivation Weight-cache daemons and engines must construct identical model-parallel layouts before CUDA IPC can safely map a cached MoE shard. The existing daemon launcher rejected `dp_size > 1`, preventing static DP/EP deployments from using the daemon-backed weight cache. This change adds static DP/EP layout support to the CUDA IPC weight-cache path. It
## Motivation `process_weights_after_loading` rewrites BF16 MoE expert weights into the flashinfer TRT-LLM BlockMajorK layout whenever `use_flashinfer_trtllm_moe` is set, and that flag covers **both** `flashinfer_trtllm` and `flashinfer_trtllm_routed`. The inverse hook, however, was gated on `is_flashinfer_trtllm_routed()` alone. So with `--moe-runner-backen
## Summary This PR removes redundant Q/K/V materialization from compatible GDN speculative target-verification paths. `causal_conv1d_update` already produces packed QKV. Previously, every GDN layer launched `fused_qkv_split_gdn_prefill_kernel` to copy that output into three contiguous tensors before Triton target verification. Triton accepts explicit token s
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation When deploying deepseek v4 with large-scale EP, cross-node communication latency is non
## Motivation SGL Top-K v1/v2 directly emits sequential indices when a row has no more candidates than `index_topk`, so those logits are never read. The DSV4 eager nonpaged path still computed them with DeepGEMM. ## Modifications - Encode trivial DeepGEMM rows as empty `[ks, ks)` ranges. - Preserve the original lengths used by KV gathering and Top-K. - Apply
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Improve M3 performance on MI350. <!-- Describe the purpose and goals of this pull reque
## Motivation GLM-4 and GLM-4.7 streaming parsers currently emit undeclared tool names even when `SGLANG_FORWARD_UNKNOWN_TOOLS` is disabled, unlike their non-streaming paths. This can expose a call that the request did not advertise. Fixes #33902. ## Modifications - centralize the existing unknown-tool policy in `BaseFormatDetector` - validate each GLM strea
Stacked on top of #33995. Keeps @yhyang201's original commit `d11868d7d9` in history and supersedes it with a rework. ## Why the rework The original fix makes non-stream output content when there are no think markers at all ("still going" was flipped to `content`). That has two leaks: - **Truncated reasoning becomes content**: a `max_tokens` cut also produce
Six reads were left on `self.server_args` outside the per-instance boundary the design reserves for the tokenizer-manager family, and each had a different reason to be there: - `Scheduler.process_input_requests` (`mm_feature_transport`) and `BaseSpecWorker._build_hicache_draft_plan` (`enable_hierarchical_cache`) are plain leaves → `get_mm()` / `get_memory()`
Two small changes that make the configuration tier's shape honest. **A named entry point for resolution.** `__post_init__` was a 200-line dispatcher, which made "where does resolution run?" a question about a dunder. It is now one line — `self._run_resolution_pipeline()` — with the dispatcher and its ordering contract living in the named method. The body is
The previous batch counted `self.server_args.X` and called the runner surface done. It was not: the same read spelled through a local alias — `server_args = model_runner.server_args` (or `sa = kvc.server_args`, `args = ...`) followed by `server_args.leaf` — is the same process-global read wearing a local name, and the AST census counts **57 of them** across
Four batches of conversions, each one a row of the disposition table the previous member pins, and each one driven by that ratchet: after the flip it fails the other way — "gone (delete from `_EXPOSED`)" with the exact entries to remove. 1. **`page_size` in seven speculative-worker constructors.** All run after publish and all keep a copy of a process-level
**The bag contract.** `test_bag_values_match_server_args` asserted `bag == field`. That holds today only because construction resolves in place; once the record stays raw it becomes **false by design** for every field resolution fills in. Rewritten against the resolved projection, which is the half that survives: the bag carries what resolution produced. The
**Not for merge.** This branch carries the whole stack so CI runs it as one unit and review comments have one place to land. The members merge individually, in order: | # | PR | what | |---|---|---| | 0 | #34376 | **the stack's base, and a fix for main**: the linear-attn kernel choice becomes a per-runner stamp, plus a direct draft/target loader-hook parity
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation `Scheduler.max_prefill_bs` previously decayed on every scheduler pass. Because the valu
Implements the SGLang Phase-A KDA integration and supersedes #33647. The integration makes CAKE engagement explicit and fail-closed: - aligned tracking (`return_intermediate_states=True` with no consumed interior checkpoint) stays on the existing final-state path and returns an empty checkpoint stand-in; - packed H12 decode admits the positive, non-overlappi
## Motivation This ports #34016 from `sglang-miles` to `main` following maintainer feedback. AITER pre-shuffles block-FP8 weights on ROCm for optimized inference kernels. The weight checker was treating the shuffled physical layout as the original logical layout during dequantization, causing false weight-update mismatches even though inference was correct.
## Motivation #33312 fixed a per-runner decision that one participant answered differently from the other: the DSV4 DSpark draft entry class did not expose the shared-experts-fusion gate its target family carries, so the loader installed one decision for the target and another for the draft, and the draft's weights were laid out for the wrong one (accept len
## Motivation Kimi-K3 currently takes the `<|media_pad|>` token ID from the checkpoint configuration. When that value is stale or differs from the tokenizer vocabulary, SGLang looks for and expands the wrong token ID, breaking image-placeholder resolution for multimodal requests. The tokenizer is the source of the input IDs used at runtime, so the processor
## Motivation `LoRAManager.init_lora_modules()` decides what to wrap by matching the last one or two components of a module name against `--lora-target-modules`. For multimodal models that is ambiguous: encoder towers name their projections the same way the language model does. `VisionAttention` exposes `qkv_proj` and `proj`, so a target as ordinary as `qkv_
## Motivation The FlashInfer TRT-LLM MoE kernels can return the unweighted expert outputs instead of performing the top-k weighted combine inside the kernel. When the combine is deferred, it is merged with the shared-expert addition into a single kernel, which removes one kernel launch per MoE layer per decode step. This mode was previously enabled only for
## Motivation Qwen3.5 uses grouped-query attention, where multiple query heads share a KV head. For Qwen3.5-397B under TP2, each rank has 16 query heads sharing one TP-local KV head. During EAGLE target verification, the existing split-KV path processes query heads independently and repeatedly scans the same prefix KV cache. This becomes increasingly expensi
## Motivation Qwen3.5 conditional-generation checkpoints store language-model attributes in the nested `text_config`. During MTP draft remapping, SGLang sets `num_nextn_predict_layers = 1` only on the parent Hugging Face config. However, `ModelConfig.num_nextn_predict_layers` is derived from `hf_text_config`, so the draft depth remains `None`. With NEXTN and
## Motivation `monkey_patch_torch_reductions()` installs `_reduce_tensor_modified` as `reductions.reduce_tensor` and calls `init_reductions()`, which rebinds the `ForkingPickler` dispatch for `torch.Tensor` — every tensor in the process, not only the CUDA ones. The replacement rewrites argument 6 unconditionally: ```python output_args = _modify_tuple( output
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> > For testing convenience, this PR cherry-picks #29202 (which enables the draft-extend CUDA graph). P
Consolidates two previously separate typo/grammar PRs (22068 + 22085) into a single submission to reduce review overhead. Changes are limited to comments and docstrings: duplicate words removed, grammar corrected, no behavioral changes. <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: **Missing `run-ci` label**
## Motivation #34458 made DeepSeek-V4 reasoning and tool-call streaming parsing chunk-invariant. Four additional streaming robustness bugs remain on top of that work, causing tool call loss, value truncation, text trapping, and DSML tag leakage in production serving. This PR fixes all four. ## Modifications ### Bug 1: Exception handler clears buffer — tool c
## Summary - make Qwen3.8 PD routing use round-robin explicitly for the main policy and decode policy - document the measured GB300 FP8 and NVFP4 P/D worker layouts and capacity settings - preserve the Qwen-specific `SGLANG_FLASHINFER_MNNVL_CUTEDSL_AR_FUSION` path used by the verified flat-TP recipes Before <img width="1084" height="314" alt="image" src="htt
## Motivation Every KL test in `unified_radix_tree/` gates on a loose threshold today, because none of their models can score a token identically twice: Qwen3-Next's chunkwise prefill scan and its decode recurrence are different algorithms and land an ulp apart, so a tight floor there fails on float noise rather than on bugs. That leaves the exact comparison
## Motivation `FlashInferMLAAttnBackend.forward_extend`'s paged branch up-converts the **whole per-layer KV pool**: ```python k_buf = self.token_to_kv_pool.get_key_buffer(layer.layer_id).to(q.dtype) ``` `O(pool)` per layer per forward. With an fp8 KV cache it OOMs once the pool is large — Kimi-Linear's 16.5M-token pool makes that a **17.76 GiB** bf16 copy pe
## Motivation The Kimi K2.5 NVFP4 GB300 test previously applied the default batch sizes `[1, 8, 16]` to both launch configs. That produced six performance combinations, including batches that are not intended for each config. ## Modifications - Run `TP4+EAGLE3` performance tests only at batch sizes 1 and 8. - Run `TP4+DP4+DPA+EAGLE3` performance tests only a
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This PR is a follow-up to #34148, which introduced SubBlock sparse attention for MiniMa
## Problem `stage-a-test-1-gpu-small-amd-rocm720` fails on every `PR Test ROCm 7.2 (AMD)` run that clears the label gate. Stage-a gates all downstream AMD legs, so no AMD PR run on ROCm 7.2 can complete. ``` Attempt 1/3: docker exec ci_sglang pip install --cache-dir=... sgl-eval @ git+https://github.com/sgl-project/sgl-eval.git@6690895... ERROR: Invalid requ
- RFC: [PD disaggregation: single protocol layer, per-backend transport #33861](https://github.com/sgl-project/sglang/issues/33861) - Staged implementation plan and PR tracking: [PD shared-protocol implementation plan #34510](https://github.com/sgl-project/sglang/issues/34510) ## Background RFC #33861 proposes gradually consolidating the duplicated PD reques