551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
## Describe the bug The SGLang model gateway (Rust router, `sgl-model-gateway` v0.3.2) rejects `/v1/responses` API requests that include tools with `type: "custom"`. This breaks compatibility with OpenAI Codex CLI and other clients that send non-standard tool types. The root cause is a **version mismatch** between the Rust `openai-protocol` crate (v1.0.0) an
## Summary Running DeepSeek-V4-Flash-0731 with `--speculative-algorithm DSPARK` across **2 nodes (TP=2)**, the server intermittently deadlocks minutes to hours into normal OpenAI-compatible traffic. Captured live, the two scheduler ranks sit in a permanent split: - **Rank A (busy)**: spinning in the NCCL host enqueue path (`sched_yield` in `ncclLocalOpAppend
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation The PR introduces changes to enable enhanced profiling of workloads. It introduces deta
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Quark MXFP4 checkpoints leave a handful of large dense projections unquantized in bf16
## Motivation Add ROCm 7.2.4 Docker flavors on Python 3.12 with PyTorch 2.11 and Triton 3.7. PyTorch 2.11 for ROCm 7.2 is available from the PyTorch Foundation index. Its dependency initially installs `triton-rocm==3.6.0`, but this PR replaces it at the end of the build with AITER’s pinned Triton 3.7. Installing Triton last prevents later dependency resoluti
## Motivation DeepSeek-V4 checkpoint metadata declares `scale_fmt: ue8m0` for its FP8 path. The official DSV4 reference applies this policy when quantizing activations before every expert Linear, so both FC1 and FC2 require power-of-two activation scales. The Triton MoE path previously dropped that policy and used continuous `amax / 448` FP32 scales instead.
This PR adds key changes to cover the enabling for Minimax H3 model specific on XPU platforms, including - Mapping to XPU attention backend with optimized kernel from sgl-kernel-xpu - Necessary device dispatching/autocast codes - Runtime XPU memory D2H/H2D fix Tested serving pipelines include: - TP + layer_wise offload for both encoder/dit/vae ( similar as c
## Motivation Extend the native MiniMax-H3 pipeline introduced in [#33275](https://github.com/sgl-project/sglang/pull/33275) to Ascend NPU. MiniMax-H3 packs real multimodal tokens together with sequence-parallel alignment padding. Its attention path is therefore variable-length even when request batch size is one: the real-token segment and the alignment-pad
- Add CLI arg `--eplb-rebalance-async-mode` - Background thread: broadcast `logical_count` → compute `ExpertLocationMetadata` → store in `_rebalance_result` - TP barrier: new gloo `cpu_group` signals (`send_signal_cpu` / `recv_signal_cpu`) ensure all ranks swap plan atomically - Yield-based generator keeps training loop non-blocking; transfer starts after TP
### Checklist - [ ] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Motivation **Problem** The LM head GEMM outputs bf16. We cast the result to fp32 afterwards, in `_copy_logits_to_
# PD Disaggregation Single-Protocol-Layer Unification Plan > Based on the v0.1 draft of [RFC #33861: PD disaggregation: single protocol layer, per-backend transport](https://github.com/sgl-project/sglang/issues/33861) This issue tracks the staged implementation of #33861. ## Step 0 — Behavior and coupling audit - [x] Audit Mooncake/NIXL/Mori protocol behavio
Before: 21.5 us <img width="1652" height="1424" alt="image" src="https://github.com/user-attachments/assets/d9b92a62-dfc1-4593-b86a-7fadbd8b8701" /> After 10.4 us <img width="1994" height="1050" alt="image" src="https://github.com/user-attachments/assets/76f3c5c5-22e9-479f-b0f1-e80c76cc339e" /> This PR is mainly authored by @vincentzed
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> GLM5.1 uses Dynamic Spars
> **Status (2026-08-12): blocked on ROCm runtime fix, not on this bump.** > > Root-cause is complete. Bumping the Mooncake pin here makes the HIP dma-buf path > actually compile in (the old `01d1eb2a` had `USE_HIP_DMABUF` on the wrong CMake > target, so it silently fell back to PeerDirect). Once dma-buf is live, cross-node > KV transfer hits a **ROCm runtime
## Motivation Hybrid GDN models such as Qwen3.5 and Qwen3-Next execute linear-attention prefill between CUDA-graph segments. The established FlashInfer path launches separate kernels for packed-QKV splitting, gating, Q/K normalization, and the final output copy, making short prefills launch-bound across many GDN layers. ## What this PR does This PR adds an i
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> The paged multimodal embe
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Related to: https://github.com/sgl-project/sglang/issues/33289 To fix https://github.co
## Motivation This PR introduces SGLang Simulator, a CPU-based inference simulator for evaluating SGLang scheduling, latency, throughput, and prefix-cache behavior without loading model weights or executing model kernels. It is a current-main rework of the simulator prototype in [#22250](https://github.com/sgl-project/sglang/pull/22250), while retaining SGLa
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> The Triton MLA decode ker
## Motivation `SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS` releases SWA slots that have already left the sliding window while a chunked prefill is being cached, instead of holding them until the request finishes. It has been opt-in since it landed; this makes it the default. **This is a default flip, so it changes behavior for every server running the
## Motivation Currently in Deepseek before every call to DeepGEMM we run a combination of RMSNorm followed by quantization, this merges it into a single kernel improving speed ## Modifications Adding 2 new kernels: - FusedRMSNormQuant - FusedRMSNormAddQuant It also integrates them into the RMSNorm layer, adds tests and integrates it into DeepSeek ## Accuracy
### Describe the bug There are two critical bugs related to the session reaper (`maybe_reap` -> `_close()`) and the OpenAI completion handler (`/v1/completions`), both affecting session tracking: 1. **Session Reaper Race Condition (Mid-decode Tear-down)** Non-streaming sessions with `timeout` are unconditionally torn down by the session reaper, even when a r
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
模型：DeepSeek‑V4‑Flash 关键配置：hidden_size=4096, moe_intermediate_size=2048, n_routed_experts=256, num_experts_per_tok=6 错误：AssertionError: Hidden size mismatch in fused_moe_triton python -m sglang.launch_server --model-path /home/ds4-flash --tp 8 --trust-remote-code --context-length 4096 --cuda-graph-backend-decode disabled --cuda-graph-backend-prefill disabled
### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Motivation Currently, we disable autotune for dense MXFP8 GEMM since it has a IMA issue with the default backend
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
## Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ## Motivation ### Problem The mnnvl allreduce fusion backends sum the ranks in fp32. The trtllm one sums in bf16, roun
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
## Summary - Add nightly AIME25 accuracy test with Think Max reasoning for DSV4-Pro on 8x B200 - Suite: `nightly-8-gpu-b200` - Two variants: - Low-Latency: TP=8, FP4 MoE (flashinfer_mxfp4), EAGLE (3-step) - MegaMoE: TP=8, DP=8, dp-attention, DeepEP + MegaMoE, EAGLE (1-step) <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:s
## Motivation Add an **FID accuracy benchmark** for SGLang diffusion text-to-image generation. It provides a reproducible workflow to (1) build a reference batch from COCO val2014 prompts + images, (2) generate samples with `sglang generate`, and (3) compute FID (Fréchet Inception Distance) between the reference and generated batches. Adapted from the xDiT F
## Motivation SGLang already supports W4A8 MXFP quantization for linear layers on the Ascend NPU, but the corresponding MoE path is not yet supported. This PR adds W4A8 MXFP support for ModelSlim-quantized MoE models. It also adapts the implementation to the latest refactored NPU MoE architecture, avoiding duplicated linear-layer support that is already avai
# Fast Triton Sparse MLA Kernels for DSA (Prefill + Decode) ## Summary This adds `triton` as an explicit DSA prefill/decode backend: ```bash --dsa-prefill-backend triton --dsa-decode-backend triton ``` The new backend provides pure Triton sparse MLA kernels for the fp8 DSA path on ROCm, validated on MI355X (gfx950) and MI300X (gfx942). It replaces the previo
## Motivation `--disaggregation-decode-enable-radix-cache` is unconditionally rejected for **every** hybrid-SWA model in `build_kv_cache`: ``` ValueError: --disaggregation-decode-enable-radix-cache is incompatible with sliding window attention (SWA) models ``` But `disaggregation/decode.py` already implements a full **SWA-tail-prealloc decode-radix path** (`
Fork from https://github.com/djns99/sglang/tree/djns99/mega_moe_flashinfer @djns99 is the main author of this PR. ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Detail the changes made in this pull request. --> ## Accuracy Tests <!-- If this pull request affects model outputs (e.g., changes to the kernel or
### Motivation AITER unified attention already supports FP8 Q + FP8 KV. With BF16 Q, the FP8 KV path cannot select the native FP8-Q matrix-multiply path. Quantizing Q adds one graph node per full-attention layer, but can reduce the much larger `kernel_unified_attention_3d` cost at medium and high decode concurrency. Qwen3.5-397B-A17B-MXFP4 has 15 full-attent
Fixes #32182. In PD decode, target and draft can use different MoE A2A backends. The scheduler kept target-local token counts for the draft worker, so DP ranks disagreed on draft padding and collective shapes. Missing GLM DSA seeds could also make only active ranks fall back to eager. This change synchronizes graph eligibility across DP ranks and keeps draft
## Motivation Five `asyncio.create_task()` calls in `srt/disaggregation` throw the returned task away. The event loop only keeps a **weak** reference to a running task, so one whose sole reference was the `create_task()` expression can be garbage collected before it finishes — [the asyncio docs call this out explicitly](https://docs.python.org/3/library/asyn
## Motivation Prefill breakable CUDA graph (BCG) replay should preserve the same model semantics as eager execution. Qwen3-VL passes intermediate visual features through `input_deepstack_embeds` and adds them at selected language-model layers. However, BCG did not copy this request-specific tensor into a stable graph input slot. As a result, normal BCG execu
## Summary - keep single-node CUDA IPC multimodal feature transport opt-in - preserve automatic CUDA VMM selection for validated multi-node MNNVL deployments - refresh VLM MMMU accuracy and latency baselines for the 1024-token CoT evaluation budget ## Root cause PR #27327 increased the MMMU CoT generation budget from 30 to 1024 tokens, but the latency thresh
## Motivation The nightly precision test can reuse stale tensors from `/tmp/sglang_precision_baselines` on self-hosted runners. `_maybe_hf_fetch()` skipped the remote store whenever local `.pt` files existed, so retries of the same commit compared against runner-specific baselines. In [nightly run 31501558607](https://github.com/sgl-project/sglang/actions/ru
Here is to track new PR CI test cases that are needed. List of PRs: https://github.com/sgl-project/sglang/pull/18182 https://github.com/sgl-project/sglang/pull/22786 - To enable SGLANG_USE_ROCM_FLYDSL=1 in AMD CI end2end test https://github.com/sgl-project/sglang/pull/22985 https://github.com/sgl-project/sglang/pull/23927 https://github.com/sgl-project/sglan
<!-- ci-failure-tracker:start --> <!-- ci-failure-tracker-state: { "python/sglang/multimodal_gen/test/server/test_server_2_gpu.py::TestDiffusionServerTwoGpu.test_diffusion_generation[ideogram4_fp8_tp2_t2i]": { "test_file": "python/sglang/multimodal_gen/test/server/test_server_2_gpu.py", "test_function": "TestDiffusionServerTwoGpu.test_diffusion_generation[id
### Checklist - [ ] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
### Describe the bug `--load-format runai_streamer` with `zai-org/GLM-5.2` (`GlmMoeDsaForCausalLM` / `glm_moe_dsa`) under TP8 **loads without any error but produces fully corrupted generations** — multilingual token soup on plain prompts, and long runs of `!` (token 0) especially when tools are present in the request. Removing `--load-format runai_streamer`
### Checklist - [ ] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
- [x] #32907 - [ ] #32968 - [ ] #32960 - [x] #32855 - [ ] #34604 - [ ] #34260
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version (`a23670ddbf`, verified on 2026-08-13). - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at
## Background The community-wide Quantization 2026 H2 Roadmap (#31783) already covers the common quantization framework, weight lifecycle, capability validation, and related infrastructure. Therefore, #21584 does not propose another Ascend quantization roadmap. It focuses on: > **Ascend A5 MXFP8/MXFP4 model/operator coverage and the consolidation of A5-speci