551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
## Links * Cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 * Day0 PR: https://github.com/sgl-project/sglang/pull/23600 * Rebase PR: https://github.com/sgl-project/sglang/pull/23882 ## Roadmap - [x] Finish W4A16 support on Hopper @Fridge003 - [x] #23686 Marlin implementation - [x] #24816 Flashinfer mxfp4 implementation - [x] [#23
Now we have many kernel backends for moe (e.g., triton, cutlass), attention, and allreduce. For each kernel, we can also tune some configs (e.g., the tile sizes in triton fused moe). Tuning these kernels and choosing dispatching heuristics can be non-trivial. We would like to build a simple command that auto-tune all the kernels and dispatch heuristic for a
As of February 21, 2026 (date of initial creation of this roadmap), SGLang has no support for Apple Silicon. We are excited to announce the initial roadmap for adding Apple Silicon support. We ask for contributors, of all levels of familiarity with SGLang and Apple devices, to pick up tasks on this roadmap and help us run SGLang on Apple Silicon (M-series).
## Links - Cookbook: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3 - Day0 PR: https://github.com/sgl-project/sglang/pull/32541 - Blog: https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support - DSpark: https://huggingface.co/RadixArk/Kimi-K3-DSpark - Bug Tracking: #32970 ## Roadmap ### Kernel Optimization - [ ] Continue optimize perform
# RFC: Programmatic KV Cache for Agentic Workloads Authors: @ishandhanani, @hzh0425 ## Introduction > [!NOTE] > Note: For this RFC, we will define “router” as an orchestrator on top of multiple engine units. Agent workloads make the value of a KV block predictable from above the engine, but that value is invisible to request-local LRU. We propose exposing a
## Motivation CPython 3.13 introduced an experimental free-threaded build (PEP 703) that removes the Global Interpreter Lock (GIL). CPython 3.14t is the first release where this build is stable enough for production use, and the ecosystem of packages with free-threading wheels is growing rapidly. vLLM has demonstrated ([vllm-project/vllm#28762](https://githu
# [Tracking] CI Test Failures and Fixes This issue tracks CI test failures, flaky tests, and infrastructure issues across SGLang's CI pipeline. **Note:** We mainly focus on scheduled CI on the `main` branch. **Last auto-update:** 2026-08-13 17:34 UTC — 3 broken, 11 flaky, 670 recently fixed ## Quick Links - **[CI Coverage Overview](https://github.com/sgl-pro
# Motivation This PR supports a quantized offline model [DeepSeek-R1-AWQ-W4AFP8](https://huggingface.co/TMElyralab/DeepSeek-R1-AWQ-W4AFP8) . We accomplished the W4AFP8 implementation for both MoE and Linear Method，by optimizing and tuning the Machete gemm kernel . The result simultaneously boost sustained throughput and cut time latency on a single H20 or H1
<!-- codex-pr-description:start --> This PR enables device-resident decode-side radix reuse for hybrid SWA models on the experimental unified radix tree. It keeps full-attention KV reusable while transferring a fresh SWA window, reducing repeated P/D KV transfer without introducing HiCache or Mamba support. ### How This Was Implemented - Caps the reused full
## Motivation SGLang's test suite has **600+ test files**, but most are E2E tests that launch a full server. While these cover many code paths, core modules like `managers/`, `mem_cache/`, `entrypoints/`, `sampling/`, `parser/`, `function_call/`, and `utils/` still need better unit test coverage (tests that run without launching a server). **Why this matters
Auto-collected CUDA coredump events from pr-test.yml. One comment per (run, job, attempt). Producer: `.github/actions/upload-cuda-coredumps/action.yml`.
## Motivation When deploying HiCache and PP, multiple PP ranks go diverge and eventually cause sglang crash. The detailed reason was discussed in issue #22607. ## Modifications This patch introduces two kinds of synchronization mechanisms to fix the issue. I will illustrate this idea with an example of pp_size=2 and tp_size=1. 1. **pp_sync** across scheduler
## Motivation Day-0 support for [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) (`BailingMoeV3ForCausalLM`, `model_type=bailing_hybrid`): a hybrid MoE architecture interleaving KDA linear attention (with a safe-gate lower bound) and MLA, with MTP (NEXTN) and DSPARK speculative decoding support. ## Modifications - `BailingMoeV3
### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### **Motivation** SGLang's primary API surface is a FastAPI/Uvicorn HTTP server. The existing `smg-grpc-servicer` pa
### Motivation It would be great to track prometheus metrics for KV cache utilization. ### Related resources vLLM already offers KV cache utilization prometheus metrics, see [here](https://docs.vllm.ai/en/stable/serving/metrics.html), at `vllm:gpu_cache_usage_perc`.
## Motivation On H200, DeepSeek-R1 with MTP/EAGLE (`--speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3`) was **no faster than standard decoding**, which shouldn't happen. nsys showed 3 synchronous device→host copies per attention `plan()` in the target-verify and draft-extend steps. Being blocking, they stall the host threa
### **Summary** RadixAttention reuses KV cache only when a request's prefix is byte-identical and at the same absolute offset as something already cached. In agentic/RAG traffic shared system prompts, tool schemas, retrieved documents, and multi-turn history routinely put identical content at different offsets across requests. This causes a guaranteed cache
## Motivation This follows [DeepSeek V4 Roadmap](https://github.com/sgl-project/sglang/issues/23602), which covers functional enablement; this issue is perf-only. **Scope**: NVIDIA SM90 / SM10X. --- ## High priority - [ ] Integrate TRT-LLM DSv4 attention for SM100/103 #30805 - [x] FlashInfer MNNVL backend for pure (non-fused) allreduce #30700 - [ ] Overlap s
Follow up of #30365. ## Problem The experimental online C128 MTP target-verify path added in #26471 still uses the host prefill planner. #30365 removed the sequence-length D2H dependency from the other DSV4 speculative paths, but intentionally preserved the CPU path for online C128 MTP. That path requires CPU sequence lengths and a pinned plan buffer, introd
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Integrates TRT-LLM attention kernel for DSv4 style attention (CSA, HCA). ## Modificatio
## Summary Adds support in SGLang for loading and inference for ModelOpt (and Quark) NVFP4 checkpoints by dequantizing the supplied NVFP4 weights and requantizing them to MXFP4 at load time to allow efficient inference using supported AMD hardware (e.g. AMD MI355x). Enabled via `--quantization quark_mxfp4`. Weights dequantized then requantized to MXFP4 durin
## Motivation External KV-aware routers need request cache namespaces to remain distinct from SGLang's caller-defined `extra_key`. Concatenating the two values can collide, and the existing KV-event hashes and payloads do not expose a typed cache namespace. ## Modifications - Add an explicit scalar/batched `cache_salt` contract to `Engine.generate`, `Engine.
Optimize fused_q_indexer_rope_hadamard_quant scheduling: 8 warps/block (256 threads) + lane-0-only weights_out write. The math path (RoPE / 128-pt Hadamard / fp8 quant) is unchanged, so output is bitwise-identical for both the V4 (kRopeFirst=false, kHadamard=true) and V3.2 (kRopeFirst=true, kHadamard=false) template configs. Block size / cap default to 8 / 1
## Motivation @humansand - Add FlashInfer CuTe DSL v2 MoE support to `--quantization nvfp4_online`. - Convert eligible BF16, FP16, or FP8 expert weights to NVFP4 at load time. - Compute and forward online per-token FP32 activation scales. - Support no A2A and FlashInfer A2A; both use CuTe DSL v2. - Keep the quantization contract established by merged upstrea
## Motivation On the AITER shared-expert-fusion path, computing the fused shared-expert routing weight launches a standalone GEMV — `self.shared_expert_gate(hidden_states)`, a `[M, hidden] × [hidden, 1]` matrix-vector op (the `Cijk_…MT1x2x512…` kernel, ~8.9 µs in decode) — whose only output feeds the subsequent `_fused_append_shared_experts_with_weights_kern
Do not merge. Opened solely to trigger and observe CI results for `sgl-kernel/tests/test_moe_topk_sigmoid.py` via `/rerun-test`. <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:white_check_mark: [Run #31646179281](https://github.com/sgl-project/sglang/actions/runs/31646179281)<!-- slot:pr-test:end --> Latest PR T
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Adds support for TRT-LLM XQA decode backend for SM90/120 speculative decoding, by build
### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the
## What this does FlashInfer's `AutoTuner.choose_one` runs **independently on every TP rank**, so ranks frequently pick **different kernel tactics** for the same op (sub-millisecond timing noise pushes the per-rank `argmin` around). This cross-rank tactic divergence is undesirable for **any** tensor-parallel run, and under `--enable-symm-mem` it becomes a **
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Fix: https://github.com/sgl-project/sglang/issues/30321 Thanks @cfbdsirlijun-maker 、[Al
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
## Motivation The existing fused AllReduce → RMSNorm → quant path only supports per-group FP8 scales. Checkpoints that use per-token FP8 activation scaling (and MXFP4) therefore fall back to the unfused AllReduce → RMSNorm → quantize sequence — three kernel launches plus intermediate bf16 round-trips — before every tensor-parallel linear. ## Modifications Ad
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation In DSpark compact ragged-verify mode, CUDA graph capture could fail with an illegal mem
## Problem statement With `--tp-size 4 --ep-size 2`, the post-experts reduction requires two all-reduces over orthogonal groups (`_MOE_EP` then `_MOE_TP`). The allreduce fusion gate was skipping *both* once `fuse_mlp_allreduce` was published, then Observed as GSM8k accuracy 0.012 on `nvidia/DeepSeek-V4-Flash-NVFP4` with silently returning under-reduced activ
## Motivation Part of the MoE refactor tracked in #8715. cc @ch-wan `Mxfp4MoEMethod` was the last quant method with a partially migrated path: the SM90 CUTLASS branch went through `MoeRunner` (#26489), but the SM100 trtllm-gen branch still called the kernel inline from `apply`, with `create_moe_runner` falling through to `pass` and a `TODO`. This migrates it
Day-0 support for the Qwen3.8-2.4T-A95B model. #### Nvidia Support Day 0 Cuda 13 image: docker pull lmsysorg/sglang:qwen38 Day 0 Cuda 12 image: docker pull lmsysorg/sglang:qwen38-cu12 #### AMD Support Day 0 MI355 image: docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260812 Day 0 MI300 image: docker pull lmsysorg/sglang-rocm:v0.5.17-rocm700-mi30x-20
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Adds NVIDIA **OmniDreams** — a 2B distilled autoregressive driving-video world model (C
…ol v2) UMBPStore assumed mem_pool_host is a single KV-bearing pool. For the DeepSeek-V4 HiCache stack, mem_pool_host is a HostPoolGroup whose KV anchor is a LogicalHostPool that owns only page indices and holds no physical KV tensor (get_page_buffer_meta() returns None by design). The real KV state lives in page_first side pools (SWA / compressed KV / index
## Summary This PR is adapted from vLLM PR https://github.com/vllm-project/vllm/pull/48726. Credit to [Heisenberg-Yin](https://github.com/Heisenberg-Yin) Adds **[LiteTopk](https://github.com/Heisenberg-Yin/LiteTopK)** as an opt-in DSA indexer top-k path for prefill (ragged extend): fp8 MQA scoring (tcgen05 UMMA) + an online bucketed gate + a compact exact to
# [Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts ## Motivation In [arXiv:2607.21535](https://arxiv.org/abs/2607.21535) we show that a built-in MTP draft head's full-attention KV read grows with the prefix until it dominates the decode step, and that restricting the *draft's* attention to a sink plus recent window cuts the per-decode-s
## Motivation Multi-turn agent workloads (coding agents, tool-calling loops, multi-agent orchestration) resend the **entire growing conversation prefix on every turn**. SGLang's RadixAttention skips prefill for cached prefixes — but **only when consecutive turns of the same session land on the same worker**. Without session affinity, each turn re-prefills th
## Motivation Each GatedDeltaNet (GDN) layer ends a token step with an **output gated-RMSNorm** followed by the `out_proj` linear. On AttnFP8-style Qwen3.5 checkpoints `out_proj` runs as an **a8w8 GEMM** (per-token activation × per-channel weight FP8), so the bf16 gated-RMSNorm output has to be quantized to FP8 per-token *before* the GEMM. Today that is thre
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Three independent fixes for DeepSeek-V4 MTP (EAGLE speculative decoding) on the HIP/ROC
## Motivation On gfx950 (MI355X), GLM-5.2 DSA prefill always ran the triton sparse-MLA path, even at short context where the sparse indexer top-k + gather + mask overhead exceeds the KV it prunes. The dense-MHA prefill fallback — already used on NVIDIA SM90/SM100 and gated by `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` (default = model `index_topk`) — w
## Summary - On GLM-5.2 DSA decode, when a request's `kv_len <= index_topk` the top-k selects **all** valid positions, so the indexer's logits GEMM + `paged_mqa_logits` + top-k selection is wasted work. Add a **k-only** fast path that skips the indexer, stores the K cache, and generates the identity index directly (`[0, 1, ..., kv_len-1, -1, ...]`), feeding
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Roadmap: https://github.com/sgl-project/sglang/issues/30344 Reference PR: #30513 This P
Decode context parallelism (DCP) for DSA models (GLM-5.x, DeepSeek V3.2). Extends the MLA DCP path (#14194) to sparse attention. 1. KV cache is interleaved across DCP ranks (`slot % dcp_size`), same as MLA DCP. KV capacity multiplies by dcp_size: 2,292,288 tokens/rank physical -> 4,252,416 logical at dcp2 on the default fp8 KV stack. 2. Indexer K cache is re
## Motivation AMD gfx1250 is a new GPU architecture that differs substantially from the existing gfx95x and gfx942 targets SGLang already supports. Several components require gfx1250-specific paths due to architectural differences: - **No MFMA/WMMA FP4 scale instructions** (`V_WMMA_SCALE_F32_32X16X128_F4`): the existing CK/ASM `fused_moe` and fp4 dense GEMM
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Adaptation of Minimax M3 for NPU platforms ## Modifications **1、Define the complete att
## Motivation We have spent a lot of time in investigation into PP/TP divergence issues. The most difficult problems is that, when PP/TP divergence happens (e.g. the server hangs), you will have the stacktrace, but it is not the real time-point which divergence happens. Think of a case. A PP/TP divergence may happen silently, forr example, the kv cache avail