{
"schemaVersion": 3,
"dataset": {
"version": 3,
"date": "2026-08-13",
"group": {
"id": "ai-infrastructure",
"name": "AI Infrastructure"
},
"repository": {
"id": "sglang",
"repo": "sgl-project/sglang",
"name": "SGLang",
"keywords": [
"SGLang"
]
},
"context": {
"repository": "sgl-project/sglang",
"url": "https://github.com/sgl-project/sglang",
"description": "SGLang is a high-performance serving framework for large language models and multimodal models.",
"homepage": "https://sglang.io",
"language": "Python",
"topics": [
"attention",
"blackwell",
"cuda",
"deepseek",
"diffusion",
"glm",
"gpt-oss",
"inference",
"llama",
"llm",
"minimax",
"moe",
"qwen",
"qwen-image",
"reinforcement-learning",
"transformer",
"vlm",
"wan"
],
"license": "Apache-2.0",
"defaultBranch": "main",
"stars": 31754,
"forks": 7862,
"openIssues": 4908,
"archived": false,
"collectedAt": "2026-08-13T18:02:08.970665+00:00"
},
"news": {
"repository": "sgl-project/sglang",
"collectedAt": "2026-08-13T18:02:08.970665+00:00",
"latestRelease": {
"repository": "sgl-project/sglang",
"tag": "v0.5.17",
"title": "v0.5.17",
"url": "https://github.com/sgl-project/sglang/releases/tag/v0.5.17",
"publishedAt": "2026-08-08T00:19:16Z",
"notes": "# Highlights\r\n\r\n*582 PRs from 194 contributors.*\r\n\r\n**Kimi K3 day-0 support**: A 2.8T-parameter multimodal LatentMoE (896 experts, top-16, routed in a 3584-dim latent space) with a 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers, and a MoonViT3d vision tower, shipping as a native MXFP4 checkpoint. SGLang serves it from day 0 with DCP, DSpark speculative decoding, chunked-prefill PP with TP decode, KDA-aware prefix caching, HiCache L2 over DCP, LoRA on the quantized weights, and reasoning, tool-call and OpenAI-compatible serving, verified on NVIDIA GB300 and AMD MI35x ([#32541](https://github.com/sgl-project/sglang/pull/32541), [#32828](https://github.com/sgl-project/sglang/pull/32828), [#32890](https://github.com/sgl-project/sglang/pull/32890), [#33025](https://github.com/sgl-project/sglang/pull/33025), [#33112](https://github.com/sgl-project/sglang/pull/33112), [blog](https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support), [cookbook](https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3), [roadmap](https://github.com/sgl-project/sglang/issues/32607)).\r\n\r\n**MiniMax-H3 day-0 support**: MiniMax's video generation model that produces a video and a synchronized stereo audio track in one request, served natively on SGLang-Diffusion across all three public task profiles: text-to-video-and-audio (`t2va`), first/last-frame conditioning (`fl2va`), and image/video/audio reference conditioning (`ref2va`, which also covers video-to-video). Verified on B200 (TP2 + Ulysses4), H100 (TP2 + Ulysses2), AMD MI300X and MI355X (Ulysses1/2/4/8), and 2x RTX 5090 with layerwise offload ([#33275](https://github.com/sgl-project/sglang/pull/33275), [cookbook](https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3)).\r\n\r\n**Other new models added**: [EmbeddingGemma](https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma) and [LFM2.5](https://docs.sglang.io/cookbook/autoregressive/LiquidAI/LFM2.5) embedding models, nvidia/MiniMax-M3-NVFP4, plus cookbook recipes for Poolside's [Laguna-S-2.1](https://docs.sglang.io/cookbook/autoregressive/Poolside/Laguna-S-2.1) family and [Inkling-Small](https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-Small).\r\n\r\n**Initial support for the Rust frontend**: Migrates the front half of the server, everything from network ingress up to the point a tokenized request is handed to the GPU scheduler, from Python to a multi-threaded Rust implementation ([#29799](https://github.com/sgl-project/sglang/pull/29799)).\r\n\r\n**DCP communication backends and q-replicate (Helix)**: The DeepSeek-MLA decode context-parallel path gains pluggable comm backends. `a2a` exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer, with fp8 KV carried as uint8 byte transport; `fi_a2a` delegates the cross-rank exchange to the FlashInfer MNNVL kernel on GB200. `--dcp-replicate-q-proj` projects full-head Q locally and skips the per-layer Q head-dim all-gather. Select with `--dcp-comm-backend {ag_rs, a2a, fi_a2a}` ([#21637](https://github.com/sgl-project/sglang/pull/21637)).\r\n\r\n**DWDP for MoE prefill**: A new prefill parallelism strategy that prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch. On 4x B200 with gpt-oss-120b, prefill-only, DWDP4 reaches **1.92x over DEP4** at MNT 32K / ISL 32K, and **506K vs 329K tok/s (1.54x)** at saturation (CONC=128, ISL=8K). Enable with `--dwdp-size`; the authors mark it early-development ([#29778](https://github.com/sgl-project/sglang/pull/29778)).\r\n\r\n**Session-reference-aware Unified Radix Cache**: For agentic and RL-rollout workloads, requests can carry a stable `session_id` so eviction knows which prefixes an active session still references, instead of evicting purely by cache policy. Release the references with `/close_session`. Opt in with `--enable-session-radix-cache` ([#29173](https://github.com/sgl-project/sglang/pull/29173)).\r\n\r\n**SM90 FP8 MegaMoE for DeepSeek-V4**: Adds the DeepGEMM MegaMoE A2A path on SM90 for DeepSeek-V4-Flash/Pro FP8, including the pre-dispatch JIT kernel and FP8 expert weight preparation. Guarded behind `SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1` ([#29016](https://github.com/sgl-project/sglang/pull/29016)).\r\n\r\n**Faster large-MoE model loading**: Oversized or non-contiguous CPU weight views were driving pathological H2D transfers, with DeepSeek-V4-Pro TP8 spending 27 to 32 minutes in H2D on some ranks. Copying those views into contiguous storage before H2D cuts full model loading from about **35 minutes to 6m20s (5.6x)**, and GPT-OSS-20B BF16 from **545s to 70s (7.8x)**, with Qwen3.5-397B measured at 1.93x to 2.3x. Opt in with `SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D`, off by default ([#32315](https://github.com/sgl-project/sglang/pull/32315)).\r\n\r\n**Lower DeepSeek-V4 memory on AMD**: Removing unnecessary expert padding drops MI355X FP4 MoE model weights from **159.07 GB to 112.36 GB**, and bringing the HIP compress-state pool into the memory_saver KV_CACHE region lets colocated RL reclaim it, cutting the measured training-phase footprint from about **143 GiB to 87 GiB per GPU** ([#31450](https://github.com/sgl-project/sglang/pull/31450), [#31747](https://github.com/sgl-project/sglang/pull/31747)).\r\n\r\n**Faster engine recovery**: Large-model restarts cost 3 to 6+ minutes today, about 6.5 minutes for Qwen3-235B FP8 on 4 GPUs, because weights reload from storage and CUDA graphs recapture. A weight-cache daemon holds weights per GPU so a restarting engine can recover from cache instead ([#27139](https://github.com/sgl-project/sglang/pull/27139)).\r\n\r\n**Lower host overhead in hybrid-linear MTP decode**: Under spec-v2 overlap scheduling each decode step runs draft, verify and extend CUDA graphs, and the eager seams between them become GPU idle time at low concurrency. This trims that host work so the host stays off the critical path ([#32219](https://github.com/sgl-project/sglang/pull/32219)).\r\n\r\n**Dependencies**: flashinfer 0.6.15.post1 ([#31927](https://github.com/sgl-project/sglang/pull/31927)), sgl-deep-gemm 0.1.5.post1 ([#32345](https://github.com/sgl-project/sglang/pull/32345), [#33143](https://github.com/sgl-project/sglang/pull/33143)), helion 1.4 ([#32562](https://github.com/sgl-project/sglang/pull/32562)), mooncake 0.3.12.post1 ([#32302](https://github.com/sgl-project/sglang/pull/32302)), dynamo-tokenizers 1.7.0 ([#32981](https://github.com/sgl-project/sglang/pull/32981)). PyTorch stays at 2.11.0 and the CUDA base image at 13.0.1.\r\n\r\n*Full release notes by category below; breaking changes and known issues are at the end.*\r\n\r\n## New Model Support\r\n\r\n| Model | Type | PRs | Cookbook |\r\n|---|---|---|---|\r\n| Kimi K3 | Autoregressive (Multimodal) | [#32541](https://github.com/sgl-project/sglang/pull/32541), [#32828](https://github.com/sgl-project/sglang/pull/32828), [#32890](https://github.com/sgl-project/sglang/pull/32890), [#33025](https://github.com/sgl-project/sglang/pull/33025) | [link](https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3) |\r\n| MiniMax-H3 | Diffusion | [#33275](https://github.com/sgl-project/sglang/pull/33275) | [link](https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3) |\r\n| MiniMax-M3-NVFP4 | Autoregressive | [#31989](https://github.com/sgl-project/sglang/pull/31989) | |\r\n| EmbeddingGemma | Autoregressive (Embedding) | [#32375](https://github.com/sgl-project/sglang/pull/32375), [#32383](https://github.com/sgl-project/sglang/pull/32383) | [link](https://docs.sglang.io/cookbook/autoregressive/Google/EmbeddingGemma) |\r\n| LFM2.5 | Autoregressive (Embedding) | [#28691](https://github.com/sgl-project/sglang/pull/28691) | [link](https://docs.sglang.io/cookbook/autoregressive/LiquidAI/LFM2.5) |\r\n\r\n## Kimi K3\r\n\r\n* [Kimi] Support kimi-k3: [#32541](https://github.com/sgl-project/sglang/pull/32541) ⭐\r\n* [Kimi] Support DCP + DSpark (ported from kimi-k3 branch): [#32828](https://github.com/sgl-project/sglang/pull/32828)\r\n* [Kimi K3] Add reasoning, tool-call, and OpenAI serving support: [#33025](https://github.com/sgl-project/sglang/pull/33025)\r\n* feat(kernels): port standalone Kimi K3 kernels: [#32890](https://github.com/sgl-project/sglang/pull/32890)\r\n* [Feat] DCP + HiCache L2 Support (ported from kimi-k3): [#33112](https://github.com/sgl-project/sglang/pull/33112)\r\n* Replace Kimi K3 DeepGEMM patch with 0.1.5.post1: [#33143](https://github.com/sgl-project/sglang/pull/33143)\r\n* docker: add Kimi K3 images: [#32760](https://github.com/sgl-project/sglang/pull/32760)\r\n\r\n## Rust Server\r\n\r\nA native Rust serving layer: tokenizer manager, ingress validation and egress, an OpenAI-compatible API server, and PD disaggregation support, shipped as prebuilt release artifacts.\r\n\r\n* support rust sglang server: [#29799](https://github.com/sgl-project/sglang/pull/29799)\r\n* create rust workspace: [#32014](https://github.com/sgl-project/sglang/pull/32014)\r\n* init sglang rust server project: [#32256](https://github.com/sgl-project/sglang/pull/32256)\r\n* add the rust server tokenizer, detokenizer, and egress modules: [#32872](https://github.com/sgl-project/sglang/pull/32872)\r\n* add the rust server ingress request validation and api server common types: [#32873](https://github.com/sgl-project/sglang/pull/32873)\r\n* add the rust server ingress tests, guard, and submit modules: [#32874](https://github.com/sgl-project/sglang/pull/32874)\r\n* add the rust server api frame codec and http server entry: [#32875](https://github.com/sgl-project/sglang/pull/32875)\r\n* add the rust server native api handlers and runtime threads: [#32876](https://github.com/sgl-project/sglang/pull/32876)\r\n* wire the rust server modules into lib, runtime, and tokenizer manager: [#32877](https://github.com/sgl-project/sglang/pull/32877)\r\n* sglang rust server tokenizer manager, ring and runtime: [#32358](https://github.com/sgl-project/sglang/pull/32358)\r\n* feat: rust sglang server openai apis: [#33103](https://github.com/sgl-project/sglang/pull/33103)\r\n* [rust-server] PD disaggregation support: [#33125](https://github.com/sgl-project/sglang/pull/33125)\r\n* [rust-server] Reland: fix TCP-layer TTFT stalls (#33026): [#33269](https://github.com/sgl-project/sglang/pull/33269)\r\n* rust server build release artifacts: [#33096](https://github.com/sgl-project/sglang/pull/33096)\r\n\r\n## Speculative Decoding\r\n\r\n* [MTP] Cut spec-v2 host-seam overhead in hybrid-linear MTP decode: [#32219](https://github.com/sgl-project/sglang/pull/32219) ⭐\r\n* [KDA] Add target_verify support for speculative decoding: [#26888](https://github.com/sgl-project/sglang/pull/26888)\r\n* [DSPARK] Grammar-constrained decoding, incl. tool_choice=auto: [#31753](https://github.com/sgl-project/sglang/pull/31753)\r\n* [DFLASH] Support grammar-constrained decoding in speculative verify: [#30096](https://github.com/sgl-project/sglang/pull/30096)\r\n* [Spec] Enable grammar overlap scheduling for STANDALONE speculative decoding: [#32110](https://github.com/sgl-project/sglang/pull/32110)\r\n* Overlap grammar (constrained decoding) with speculative decode verify: [#31488](https://github.com/sgl-project/sglang/pull/31488)\r\n* [Spec] Support sampling in the DSPARK graph-folded draft proposal: [#33298](https://github.com/sgl-project/sglang/pull/33298)\r\n* [Spec] Add `trtllm_mha` support for Gemma 4 MTP draft attention backend: [#25545](https://github.com/sgl-project/sglang/pull/25545)\r\n* [Perf] Fast-path chain-style draft token organization in multi-layer EAGLE: [#32887](https://github.com/sgl-project/sglang/pull/32887)\r\n* [Perf] Skip the target-verify tree mask fill when the backend never reads it: [#32886](https://github.com/sgl-project/sglang/pull/32886)\r\n* [Spec] Compact the target-verify mask when nothing reads it: [#32920](https://github.com/sgl-project/sglang/pull/32920)\r\n* [Perf] Fold dspark dense draft embedding into the draft graph via forward_embed: [#31985](https://github.com/sgl-project/sglang/pull/31985)\r\n* [Perf] Stack dspark dense draft per-layer ctx KV projection into one GEMM: [#31986](https://github.com/sgl-project/sglang/pull/31986)\r\n* [Mamba] Support speculative decoding with extra_buffer_lazy: [#30437](https://github.com/sgl-project/sglang/pull/30437)\r\n* [gdn] fused replayssm ring write into flashinfer gdn mtp verify kernel: [#33102](https://github.com/sgl-project/sglang/pull/33102)\r\n* [gdn] support replayssm with extra buffer: [#32692](https://github.com/sgl-project/sglang/pull/32692)\r\n* [6/6][kimi-deterministic] Use deterministic seeded coins for EAGLE rejection sampling: [#30822](https://github.com/sgl-project/sglang/pull/30822)\r\n* [Speculative Decoding] Fix GPT-OSS EAGLE3 hidden states: [#32334](https://github.com/sgl-project/sglang/pull/32334)\r\n* [EAGLE] Handle NaNs in fused top-k=1: [#32396](https://github.com/sgl-project/sglang/pull/32396)\r\n* Pack aux hidden states into a preallocated buffer: [#28956](https://github.com/sgl-project/sglang/pull/28956)\r\n* Support SGLANG_SIMULATE_ACC_LEN for DFLASH: [#32595](https://github.com/sgl-project/sglang/pull/32595)\r\n* Fix stop boundaries for grammar-constrained speculative decoding: [#31738](https://github.com/sgl-project/sglang/pull/31738)\r\n* Fix DSpark loading for hybrid DSV4 NVFP4: [#33276](https://github.com/sgl-project/sglang/pull/33276)\r\n* Fix DSpark and DP/EP: [#33098](https://github.com/sgl-project/sglang/pull/33098)\r\n* [Fix] Clamp degenerate all-sentinel draft rows to token 0 in dspark `_online_combine_kernel`: [#32277](https://github.com/sgl-project/sglang/pull/32277)\r\n* [Fix] Bound FULL_MASK verify-mask reuse by the captured max_bs: [#33127](https://github.com/sgl-project/sglang/pull/33127)\r\n\r\n## Piecewise & Breakable CUDA Graph\r\n\r\n* Turn on breakable prefill cuda graph for dp attention by default: [#31682](https://github.com/sgl-project/sglang/pull/31682)\r\n* [FullCG] Support chunked cached-prefix prefill: [#30825](https://github.com/sgl-project/sglang/pull/30825)\r\n* [FullCG] Preserve attention LSE through the custom-op boundary: [#31050](https://github.com/sgl-project/sglang/pull/31050)\r\n* [BCG][3/N] Enable bcg on dsa & deepep a2a backend: [#31987](https://github.com/sgl-project/sglang/pull/31987)\r\n* [BCG][4/N] Enable bcg on megamoe & flashinfer a2a backend: [#33150](https://github.com/sgl-project/sglang/pull/33150)\r\n* [CUDA Graph] Allow custom decode graph runners: [#33553](https://github.com/sgl-project/sglang/pull/33553)\r\n* [cuda_graph] Gate breakable-CG capture_inputs retention to DP-gather paths: [#32678](https://github.com/sgl-project/sglang/pull/32678)\r\n* Enable multimodal prefill BCG for VL and audio models: [#30872](https://github.com/sgl-project/sglang/pull/30872)\r\n* Enable post-capture KV sizing with DP attention: [#33427](https://github.com/sgl-project/sglang/pull/33427)\r\n* fix(server): capture legal multi-request prefill CUDA graph batches: [#30206](https://github.com/sgl-project/sglang/pull/30206)\r\n* Fix BCG circular import during server startup: [#33371](https://github.com/sgl-project/sglang/pull/33371)\r\n\r\n## Attention Backends\r\n\r\n* [Attention Backend] Add HPC-Ops attention backend: [#30540](https://github.com/sgl-project/sglang/pull/30540) ([blog](https://www.lmsys.org/blog/2026-08-07-hpc-ops-sglang))\r\n* [Attention Backend] Extend hpc_ops dynamic-scheduled decode to bf16: [#32304](https://github.com/sgl-project/sglang/pull/32304)\r\n* [DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.2 & DeepSeek-V3.2: Q8-Path & Shared-Path Optimizations: [#31888](https://github.com/sgl-project/sglang/pull/31888)\r\n* [unified-memory] Support MLA-hybrid-Mamba (Kimi-Linear) on the Triton backend: [#32971](https://github.com/sgl-project/sglang/pull/32971)\r\n* [unified-memory] Let Kimi-Linear use the paged MLA attention backends: [#32972](https://github.com/sgl-project/sglang/pull/32972)\r\n* [unified-memory] Support fa3, the default MLA backend on pre-Blackwell hosts: [#33046](https://github.com/sgl-project/sglang/pull/33046)\r\n* feat(inkling): migrate short convs onto the ShortConv attention backend: [#33023](https://github.com/sgl-project/sglang/pull/33023)\r\n* [GDN] Support FlashInfer GDN prefill with extra-buffer radix cache: [#29735](https://github.com/sgl-project/sglang/pull/29735)\r\n* fix(gdn): skip the -1 padding sentinel in the chunked extend kernel: [#33810](https://github.com/sgl-project/sglang/pull/33810)\r\n* [Kernel] Fuse KV-cache writes for asymmetric K/V (head_dim != v_head_dim): [#32813](https://github.com/sgl-project/sglang/pull/32813)\r\n* [Perf][DSA] Pass topk_length to flash_mla_sparse_fwd in the sparse attention path: [#31128](https://github.com/sgl-project/sglang/pull/31128)\r\n* [Perf] Skip page-table columns past kv length in DSA draft-extend metadata kernel: [#31981](https://github.com/sgl-project/sglang/pull/31981)\r\n* [Perf] Skip blocks past per-request live length in full-width Triton kernels: [#32109](https://github.com/sgl-project/sglang/pull/32109)\r\n* Support a same-size mixed q dtype in the fused RoPE kernels: [#31834](https://github.com/sgl-project/sglang/pull/31834)\r\n* Avoid TRTLLM prefill output copy: [#33306](https://github.com/sgl-project/sglang/pull/33306)\r\n* Fix attention backends for models with per-layer head counts (num_attention_heads_per_layer): [#32625](https://github.com/sgl-project/sglang/pull/32625)\r\n* [Fix] Route asymmetric-KV models to fa4 on SM100 and pin MiMoV2 FP8 MoE to flashinfer_trtllm: [#32818](https://github.com/sgl-project/sglang/pull/32818)\r\n* [Fix] Fix trtllm_mla backend + fp8 kv cache without rope: [#32181](https://github.com/sgl-project/sglang/pull/32181)\r\n* [Fix] Support ENCODER_ONLY target-verify in the trtllm_mha backend: [#32178](https://github.com/sgl-project/sglang/pull/32178)\r\n* [Fix] missing max_context_len on HybridAttnBackend: [#32690](https://github.com/sgl-project/sglang/pull/32690)\r\n* fix(dsa): correct packed FlashInfer top-k and backend selection semantics: [#32490](https://github.com/sgl-project/sglang/pull/32490)\r\n* fix(dsa): fail fast on fp8_e4m3 KV with tilelang DSA backend on CUDA: [#31346](https://github.com/sgl-project/sglang/pull/31346)\r\n* [KDA] Fix mixed exponent bases in Triton chunk prefill: [#31904](https://github.com/sgl-project/sglang/pull/31904)\r\n* [FIX] Prevent Lightning Attention extra-buffer mamba state corruption: [#29973](https://github.com/sgl-project/sglang/pull/29973)\r\n* Add configurable FlashInfer autotune skips: [#31389](https://github.com/sgl-project/sglang/pull/31389)\r\n* Fix FlashInfer MNNVL workspace size check: [#32318](https://github.com/sgl-project/sglang/pull/32318)\r\n\r\n## MoE & Expert Parallelism\r\n\r\n* [Feature] Add DWDP (Distributed Weight Data Parallelism) for MoE prefill: [#29778](https://github.com/sgl-project/sglang/pull/29778) ⭐\r\n* Add SM90 FP8 MegaMoE support for DeepSeek-V4: [#29016](https://github.com/sgl-project/sglang/pull/29016) ⭐\r\n* [MoE Backend] Add HPC-Ops FP8 MoE runner backend: [#30541](https://github.com/sgl-project/sglang/pull/30541) ([blog](https://www.lmsys.org/blog/2026-08-07-hpc-ops-sglang))\r\n* Integrate pplx a2a backend: [#30756](https://github.com/sgl-project/sglang/pull/30756)\r\n* Support DeepGEMM for standard MoE dispatch: [#33128](https://github.com/sgl-project/sglang/pull/33128)\r\n* [DSV4] Support megamoe for CP: [#29569](https://github.com/sgl-project/sglang/pull/29569)\r\n* [MoE] Make DeepEP auto serve flashinfer_cutedsl FP4 (coerce to low_latency) + guard: [#29523](https://github.com/sgl-project/sglang/pull/29523)\r\n* [2/N] elastic-ep: Enable EPLB after scale-up: [#30553](https://github.com/sgl-project/sglang/pull/30553)\r\n* [Elastic EP] Centralize Mooncake PG configuration: [#31708](https://github.com/sgl-project/sglang/pull/31708)\r\n* [Elastic EP] Fix recovery lifecycle and add manual coverage: [#31744](https://github.com/sgl-project/sglang/pull/31744)\r\n* perf: speed up marlin moe with occupancy-aware launch specialization: [#31552](https://github.com/sgl-project/sglang/pull/31552)\r\n* Migrate CompressedTensorsW4A4Nvfp4MoE TRT-LLM path onto MoeRunner: [#32248](https://github.com/sgl-project/sglang/pull/32248)\r\n* Load initial expert location metadata on CPU: [#32435](https://github.com/sgl-project/sglang/pull/32435)\r\n* Fix silently wrong EPLB output with --moe-a2a-backend none (rank-invariant dispatch): [#32962](https://github.com/sgl-project/sglang/pull/32962)\r\n* Fix MoE reduce-scatterv eligibility check: [#32663](https://github.com/sgl-project/sglang/pull/32663)\r\n* Fix pad-row top-k masking with custom_routing_function under DP attention: [#31838](https://github.com/sgl-project/sglang/pull/31838)\r\n* Add DeepSeek-reference 1e-20 epsilon to top-k renormalization to prevent 0/0 NaN: [#31017](https://github.com/sgl-project/sglang/pull/31017)\r\n* [Fix] Clear stale FlashInfer BF16 MoE index cache: [#33016](https://github.com/sgl-project/sglang/pull/33016)\r\n* Optimize LongCat-Flash router GEMM with the HPC-Ops bf16xfp32 kernel: [#30247](https://github.com/sgl-project/sglang/pull/30247)\r\n* [Fix] Reject online weight updates while the HPC-Ops router GEMM split cache is active: [#31943](https://github.com/sgl-project/sglang/pull/31943)\r\n\r\n## Quantization\r\n\r\n* Support CuteDSL GEMM BF16 on SM100 on by default when allowed by heuristic: [#30567](https://github.com/sgl-project/sglang/pull/30567)\r\n* [Quant] Support NVFP4_AWQ checkpoints in ModelOpt FP4 path: [#31825](https://github.com/sgl-project/sglang/pull/31825)\r\n* [Quant] Keep the flashinfer_deepgemm FP8 GEMM to 1 <= M < 32: [#32843](https://github.com/sgl-project/sglang/pull/32843)\r\n* Qwen3.5-MoE: support modelopt_fp4 checkpoints that quantize attention (+ load baked FP8 KV scales): [#31220](https://github.com/sgl-project/sglang/pull/31220)\r\n* [Qwen3.5][MTP] Support FlashInfer CuTe DSL for online NVFP4 draft MoE: [#31382](https://github.com/sgl-project/sglang/pull/31382)\r\n* Fixing MXFP8 online quantization pipeline: [#31510](https://github.com/sgl-project/sglang/pull/31510)\r\n* [Fix] Restore online MXFP8 quantization for linear layers: [#32953](https://github.com/sgl-project/sglang/pull/32953)\r\n* [Bugfix] compressed-tensors: mixed-precision checkpoints silently load unquantized: [#32736](https://github.com/sgl-project/sglang/pull/32736)\r\n* Fix compressed-tensors NVFP4 MoE W13 layout: [#32430](https://github.com/sgl-project/sglang/pull/32430)\r\n* fix(marlin_nvfp4): only apply routed_scaling_factor in moe_sum_reduce: [#31762](https://github.com/sgl-project/sglang/pull/31762)\r\n* [Perf] Halve the non-finite sanitization overhead in per_token_group_quant: [#32296](https://github.com/sgl-project/sglang/pull/32296)\r\n* [JIT] Trait-driven per_token_group_quant: unify the quant kernel family (flat + masked): [#30924](https://github.com/sgl-project/sglang/pull/30924)\r\n* [JIT] Restore the previous division behavior in per-token group quantization: [#32616](https://github.com/sgl-project/sglang/pull/32616)\r\n* Fix FP8 Triton dtype selection on A100: [#31340](https://github.com/sgl-project/sglang/pull/31340)\r\n\r\n## Parallelism & Disaggregation\r\n\r\n* [Feature] DCP: A2A + FlashInfer-MNNVL comm backends and q-replicate (Helix): [#21637](https://github.com/sgl-project/sglang/pull/21637) ⭐\r\n* [DeepSeek V4] CP decode opt: slice repeat attention weights to local TP partition: [#27657](https://github.com/sgl-project/sglang/pull/27657) ⭐\r\n* [4/N][CP] Support interleave strategy for cp v2: [#30482](https://github.com/sgl-project/sglang/pull/30482)\r\n* [CP] Fuse zigzag attention into a single call: [#33137](https://github.com/sgl-project/sglang/pull/33137)\r\n* [CP] Support breakable CUDA graphs for zigzag strategy: [#33136](https://github.com/sgl-project/sglang/pull/33136)\r\n* [CP]: FIx some issue for glm5.2 cp v2: [#33100](https://github.com/sgl-project/sglang/pull/33100)\r\n* Support DCP for Kimi Linear model: [#32612](https://github.com/sgl-project/sglang/pull/32612)\r\n* feat: support Kimi Linear PD disaggregation with DCP: [#32837](https://github.com/sgl-project/sglang/pull/32837)\r\n* [Bugfix] Fix Kimi-Linear state transfer across heterogeneous TP: [#32262](https://github.com/sgl-project/sglang/pull/32262)\r\n* [DCP] Match the replicated draft KV pool's page granularity to its allocator: [#33348](https://github.com/sgl-project/sglang/pull/33348)\r\n* [DCP] Bound a request by the aggregate KV pool, not one rank's share: [#33448](https://github.com/sgl-project/sglang/pull/33448)\r\n* [Fix] Honor FlashMLA natural-log LSE in DCP reduction: [#33065](https://github.com/sgl-project/sglang/pull/33065)\r\n* fix(mem_cache): state the MLA KV bound in the DCP index space: [#33432](https://github.com/sgl-project/sglang/pull/33432)\r\n* [comm] Enable multi-node custom-AR v2 on a single NVLink clique: [#32339](https://github.com/sgl-project/sglang/pull/32339)\r\n* [Fix] Enable graph capture and MSCCL++ for attention TP groups: [#31629](https://github.com/sgl-project/sglang/pull/31629)\r\n* Disable extra NCCL CUDA event synchronization with symm mem: [#27089](https://github.com/sgl-project/sglang/pull/27089)\r\n* [PD] NIXL connector: shard by destination: [#32025](https://github.com/sgl-project/sglang/pull/32025)\r\n* [PD] pool decode bootstrap HTTP sessions: [#31543](https://github.com/sgl-project/sglang/pull/31543)\r\n* [PD] Handle abort requests in PP mode: [#32797](https://github.com/sgl-project/sglang/pull/32797)\r\n* [PD+PP] Honor PP consensus for bootstrap and prealloc: [#31869](https://github.com/sgl-project/sglang/pull/31869)\r\n* [PD] Prevent decode scheduler from blocking on ZMQ sends to a stalled prefill peer: [#31144](https://github.com/sgl-project/sglang/pull/31144)\r\n* [PD] Fix false health-503 during decode retraction re-admission: [#33118](https://github.com/sgl-project/sglang/pull/33118)\r\n* [PD] Drain NIXL completion notifications before enforcing the WaitingForInput timeout: [#32267](https://github.com/sgl-project/sglang/pull/32267)\r\n* [Disagg][NIXL] Fix heterogeneous attn-TP KV transfer for replicated GQA heads (NIXL_ERR_NOT_FOUND): [#31968](https://github.com/sgl-project/sglang/pull/31968)\r\n* [Disagg][StagingBuffer][1/2] Robustness and failure handling: [#31217](https://github.com/sgl-project/sglang/pull/31217)\r\n* fix(disagg): support pipeline-parallel hybrid-linear transfer: [#32270](https://github.com/sgl-project/sglang/pull/32270)\r\n* [BugFix][EPD] Harden zmq_to_scheduler receiver failures; sync error info across TP: [#31592](https://github.com/sgl-project/sglang/pull/31592)\r\n* [EPD] Fix HTTP dispatch lock blocking cross-request encoder batching: [#31275](https://github.com/sgl-project/sglang/pull/31275)\r\n* [EPD] Make encoder register/unregister health-check robust: [#31576](https://github.com/sgl-project/sglang/pull/31576)\r\n* [BugFix][EPD] Fix Mooncake source-MR lifecycle for multi-TP /send: [#32071](https://github.com/sgl-project/sglang/pull/32071)\r\n* [BugFix][EPD] Early-release mooncake GPU embeddings; fix gpu_id via scheduler.ps: [#31591](https://github.com/sgl-project/sglang/pull/31591)\r\n* [HiSparse]Fix DeepSeek V4 HiSparse PD Transfers with Separate Host and Device KV Indices: [#31901](https://github.com/sgl-project/sglang/pull/31901)\r\n* support dp attn with client lb: [#33105](https://github.com/sgl-project/sglang/pull/33105)\r\n* Fix token count localization for replicated attention-TP forwards: [#32411](https://github.com/sgl-project/sglang/pull/32411)\r\n\r\n## Scheduler & Runtime\r\n\r\n* [FEAT] Support fast engine recovery through weight cache: [#27139](https://github.com/sgl-project/sglang/pull/27139) ⭐\r\n* feat(grpc): add generation request semantics: [#32588](https://github.com/sgl-project/sglang/pull/32588)\r\n* [Scheduler] Enable decode retraction ordering under speculative decoding: [#32023](https://github.com/sgl-project/sglang/pull/32023)\r\n* Allow optimistic prefill with L2 hierarchical cache and write-back policy: [#33545](https://github.com/sgl-project/sglang/pull/33545)\r\n* Bound prefill delayer all-branch delay and decay the max_prefill_bs high-watermark: [#32880](https://github.com/sgl-project/sglang/pull/32880)\r\n* Negotiate PrefillDelayer only after KV-budget admission checks: [#31835](https://github.com/sgl-project/sglang/pull/31835)\r\n* [Bugfix] Fix prefill suspension caused by delayed negotiate_should_allow_prefill invocation: [#32389](https://github.com/sgl-project/sglang/pull/32389)\r\n* Fix SWA admission livelock on cached-prefix resumes: [#32379](https://github.com/sgl-project/sglang/pull/32379)\r\n* [core/loader] Add presharded load format: [#24256](https://github.com/sgl-project/sglang/pull/24256)\r\n* Support fastsafetensors no-GDS loading and page-cache release: [#31859](https://github.com/sgl-project/sglang/pull/31859)\r\n* Fix async loading of RunAI-streamed tensors: [#32896](https://github.com/sgl-project/sglang/pull/32896)\r\n* Fix DeepSeek V4 loading with RunAI Model Streamer.: [#30240](https://github.com/sgl-project/sglang/pull/30240)\r\n* [Fix] Fail fast when a safetensors index references missing shard files: [#32279](https://github.com/sgl-project/sglang/pull/32279)\r\n* AutoWeightLoader support Sglang native models 1: demo: [#28671](https://github.com/sgl-project/sglang/pull/28671)\r\n* Fix reward/classification models broken by `load_weights` v2 dispatch (#28671): [#31988](https://github.com/sgl-project/sglang/pull/31988)\r\n* perf(startup): skip unused PyTorch headers for KV VMM allocator stub: [#33126](https://github.com/sgl-project/sglang/pull/33126)\r\n* Reduce startup log noise and fix Dynamo / CUDA-graph edge cases: [#33428](https://github.com/sgl-project/sglang/pull/33428)\r\n* [Feature] Add leveled invariant-check primitive for nan/inf/oob validity checks: [#32308](https://github.com/sgl-project/sglang/pull/32308)\r\n* [Observability] Add startup, memory, and hybrid SWA diagnostics: [#33375](https://github.com/sgl-project/sglang/pull/33375)\r\n* [metrics] Split tokenizer request metrics by stream: [#32734](https://github.com/sgl-project/sglang/pull/32734)\r\n* Add stream label to TTFT metrics: [#32363](https://github.com/sgl-project/sglang/pull/32363)\r\n* fix(metrics): clear forward occupancy on idle: [#33562](https://github.com/sgl-project/sglang/pull/33562)\r\n* Report accelerator type in /v1/loads: [#32348](https://github.com/sgl-project/sglang/pull/32348)\r\n* [Router] Report accelerator count in /v1/loads: [#33548](https://github.com/sgl-project/sglang/pull/33548)\r\n* Add prefill and decode load counters to LoadSnapshot: [#32245](https://github.com/sgl-project/sglang/pull/32245)\r\n* [PD] Add a queues.prealloc_ready counter to the load snapshot: [#33133](https://github.com/sgl-project/sglang/pull/33133)\r\n* [Fix] Include disagg prefill waiting queue in FPM: [#32122](https://github.com/sgl-project/sglang/pull/32122)\r\n* fix: prevent ReqTimeStats from being dropped during IPC serialization: [#31339](https://github.com/sgl-project/sglang/pull/31339)\r\n* config: make ServerArgs read-only with a single audited mutation entry: [#31811](https://github.com/sgl-project/sglang/pull/31811)\r\n* runtime_context: per-role namespace enforcement behind SGLANG_ROLE_NAMESPACES: [#33172](https://github.com/sgl-project/sglang/pull/33172)\r\n* [misc] Deep-merge nested config overrides and parse request bodies with orjson: [#33351](https://github.com/sgl-project/sglang/pull/33351)\r\n* [Perf] Free KV pages by segment in the paged allocator without a device sync: [#32701](https://github.com/sgl-project/sglang/pull/32701)\r\n* [mem_cache] Build empty-prefix last_loc sentinel on-device to avoid per-call H2D sync: [#32575](https://github.com/sgl-project/sglang/pull/32575)\r\n* [Kernel] Skip KV writes to reserved padding slots: [#32477](https://github.com/sgl-project/sglang/pull/32477)\r\n* Skip dist_init/nccl port prechecks when the dist init method is overridden: [#31410](https://github.com/sgl-project/sglang/pull/31410)\r\n* [BugFix] Prevent TBO crash when return_logprob is enabled: [#32180](https://github.com/sgl-project/sglang/pull/32180)\r\n* [BugFix] Fix DS/Kimi crash on non-first PP ranks when resolving input length: [#31752](https://github.com/sgl-project/sglang/pull/31752)\r\n* [Fix] Two root causes of the H100 deepep TBO CI break: scale-tensor use-after-free + missing non-finite quant sanitization: [#32188](https://github.com/sgl-project/sglang/pull/32188)\r\n\r\n## HiCache & Radix Cache\r\n\r\n* feat: Session-reference-aware Unified Radix Cache for agentic multi-turn workloads: [#29173](https://github.com/sgl-project/sglang/pull/29173) ⭐\r\n* Radix Cache Split: Spin off TreeCore: [#29901](https://github.com/sgl-project/sglang/pull/29901)\r\n* [Hicache][1/2]Support Mamba branching in Unified Radix Cache with HiCache: [#31181](https://github.com/sgl-project/sglang/pull/31181)\r\n* feat(hicache): Add shared memory allocator for host KV cache: [#29326](https://github.com/sgl-project/sglang/pull/29326)\r\n* [HiCache] Add model-aware key isolation to Mooncake Store: [#31920](https://github.com/sgl-project/sglang/pull/31920)\r\n* Add Mooncake tenant id support: [#30256](https://github.com/sgl-project/sglang/pull/30256)\r\n* Eliminate redundant DSA state transfers (Mooncake): [#32620](https://github.com/sgl-project/sglang/pull/32620)\r\n* [HiCache] Merge HiCache event checks to reduce decode overhead: [#30511](https://github.com/sgl-project/sglang/pull/30511)\r\n* [HiCache]: Optimize hybrid/DSA L3 prefetch result sync and usable-prefix clamping: [#31443](https://github.com/sgl-project/sglang/pull/31443)\r\n* fix(hicache): support staged write-back for asymmetric MHA: [#30981](https://github.com/sgl-project/sglang/pull/30981)\r\n* Write-back policy fix for unified tree: [#31845](https://github.com/sgl-project/sglang/pull/31845)\r\n* Fix --hicache-size allocating ~2x host memory on hybrid SWA: [#32373](https://github.com/sgl-project/sglang/pull/32373)\r\n* Fix --hicache-size allocating ~2x host memory on hybrid Mamba: [#32915](https://github.com/sgl-project/sglang/pull/32915)\r\n* [UnifiedTree] fix: drop prefetched host refill under an un-backed-up parent: [#31902](https://github.com/sgl-project/sglang/pull/31902)\r\n* [UnifiedRadixCache][mamba] Fix mamba state corruption and slot leak when load_back aborts: [#30986](https://github.com/sgl-project/sglang/pull/30986)\r\n* [Fix] Evict only the KV shortfall in evict_from_tree_cache: [#32016](https://github.com/sgl-project/sglang/pull/32016)\r\n* [Fix] Unify pinned host pool release on graceful shutdown: [#32029](https://github.com/sgl-project/sglang/pull/32029)\r\n* fix(hisparse): correct DSA KV memory budget: [#31992](https://github.com/sgl-project/sglang/pull/31992)\r\n* perf(hisparse): eliminate redundant swap output fill: [#32483](https://github.com/sgl-project/sglang/pull/32483)\r\n* [Mamba] Add a per-path cap for cached states: [#31230](https://github.com/sgl-project/sglang/pull/31230)\r\n* Skip mamba lock during decoding: [#32228](https://github.com/sgl-project/sglang/pull/32228)\r\n* [Fix] Reserve the mamba pool's +1 padding slot in the memory budget solve: [#32184](https://github.com/sgl-project/sglang/pull/32184)\r\n* Fix unnecessary gather/scatter on CPU for non-contiguous Mamba statepool: [#31754](https://github.com/sgl-project/sglang/pull/31754)\r\n* [XPU] Add XPU device support for LMCache radix cache integration: [#23534](https://github.com/sgl-project/sglang/pull/23534)\r\n\r\n## LoRA\r\n\r\n* [LoRA] Support LoRA under the breakable/full prefill CUDA graph: [#30988](https://github.com/sgl-project/sglang/pull/30988)\r\n* Split #32584 into 2/2: [LoRA] Shard attention LoRA by attn-TP and allow dynamic LoRA with dp attention: [#32708](https://github.com/sgl-project/sglang/pull/32708)\r\n* Split #32584 into 1/2: [LoRA] Guard DP-attention idle forwards against stale LoRA batch state: [#32707](https://github.com/sgl-project/sglang/pull/32707)\r\n* [LoRA] 1/n Per-rank tensor serialization for load_lora_adapter_from_tensors under dp_size > 1: [#32580](https://github.com/sgl-project/sglang/pull/32580)\r\n* [LoRA] Guard TMA down path for LoRA hooks: [#31608](https://github.com/sgl-project/sglang/pull/31608)\r\n* [lora] Fix WAR race: never write MoE runner output into hidden_states in place: [#31870](https://github.com/sgl-project/sglang/pull/31870)\r\n* [LoRA] Fix Marlin MoE kernel import: [#32884](https://github.com/sgl-project/sglang/pull/32884)\r\n* LoRA: Ascend: Update ascend LoRA backend to support new kernels: [#15912](https://github.com/sgl-project/sglang/pull/15912)\r\n\r\n## Multimodal\r\n\r\n* [mm] sglang-mm: server vision pipeline core (fetch/driver/pipeline) + Qwen VL: [#32364](https://github.com/sgl-project/sglang/pull/32364)\r\n* vlm: parallelize multimodal preprocessing with customized worker num: [#31438](https://github.com/sgl-project/sglang/pull/31438)\r\n* [Perf] Broadcast single-image DP vision embedding instead of pad-to-max all-gather: [#33307](https://github.com/sgl-project/sglang/pull/33307)\r\n* [Perf] Speed up the Kimi-K2.5 vision path and match PIL bicubic in the GPU resize: [#33349](https://github.com/sgl-project/sglang/pull/33349)\r\n* [EPD][VLM] Fix Kimi-VL 2D encoder grids: [#32104](https://github.com/sgl-project/sglang/pull/32104)\r\n* [mm] Accept per-item embedding lists from DataEmbeddingFunc: [#31826](https://github.com/sgl-project/sglang/pull/31826)\r\n* [mm] Handle per-item embeddings in cache misses: [#32498](https://github.com/sgl-project/sglang/pull/32498)\r\n* Decode input_audio media containers with PyAV & Update memory profiler: [#31832](https://github.com/sgl-project/sglang/pull/31832)\r\n* fix(vlm): materialize Qwen3-VL features on the vision device: [#31596](https://github.com/sgl-project/sglang/pull/31596)\r\n* vlm: reject moss vision metadata mismatches: [#31957](https://github.com/sgl-project/sglang/pull/31957)\r\n* [Bugfix] Place empty Qwen encoder-DP embeddings on the communication device: [#31663](https://github.com/sgl-project/sglang/pull/31663)\r\n* [Fix] --mm-process-config crash when video config contains: [#30260](https://github.com/sgl-project/sglang/pull/30260)\r\n* [Fix] Drop deprecated multimodal processor residency state: [#33308](https://github.com/sgl-project/sglang/pull/33308)\r\n\r\n## Model Support & Optimizations\r\n\r\n* [Feature] Support return_hidden_states=\"last\": [#30177](https://github.com/sgl-project/sglang/pull/30177)\r\n* Add return_token_ids support to completions and chat completions APIs: [#30917](https://github.com/sgl-project/sglang/pull/30917)\r\n* embedding: centralize capabilities and complete OpenAI compatibility: [#32481](https://github.com/sgl-project/sglang/pull/32481)\r\n* [Model] Support standalone text-only Qwen3.5 checkpoints: [#32401](https://github.com/sgl-project/sglang/pull/32401)\r\n* model: serve bare Qwen3Model backbone natively as an embedding model: [#32457](https://github.com/sgl-project/sglang/pull/32457)\r\n* [feat] Opt-in flat response format for prompt top logprobs: [#32078](https://github.com/sgl-project/sglang/pull/32078)\r\n* [feat] Optional base64 encoding for the flat prompt top logprob arrays: [#31960](https://github.com/sgl-project/sglang/pull/31960)\r\n* [perf] Assemble flat prompt top logprobs scheduler-side as numpy arrays: [#32223](https://github.com/sgl-project/sglang/pull/32223)\r\n* Use native batched llguidance mask generation: [#32412](https://github.com/sgl-project/sglang/pull/32412)\r\n* [Spec] Hold the grammar bitmask in one `GrammarMask` type across all decode paths: [#32409](https://github.com/sgl-project/sglang/pull/32409)\r\n* [Spec] Consolidate the grammar sync decision into ScheduleBatch.grammar_needs_sync: [#32353](https://github.com/sgl-project/sglang/pull/32353)\r\n* [Spec] Share the grammar mask build and verify-tree staging across spec workers: [#32393](https://github.com/sgl-project/sglang/pull/32393)\r\n* [Spec] Derive NGRAM grammar tree links on the host instead of reading back `retrive_next_token`: [#32380](https://github.com/sgl-project/sglang/pull/32380)\r\n* support regex that compatible with python re lib however apply more l...: [#32676](https://github.com/sgl-project/sglang/pull/32676)\r\n* fix(sampling): reject conflicting structural tag constraints: [#32525](https://github.com/sgl-project/sglang/pull/32525)\r\n* [Fix] Treat an empty grammar constraint as unset in SamplingParams: [#33328](https://github.com/sgl-project/sglang/pull/33328)\r\n* Add 'anyOf' schema support for qwen3_coder tool call parser: [#30832](https://github.com/sgl-project/sglang/pull/30832)\r\n* Fix dropped tool calls when a stream delta carries several: [#31860](https://github.com/sgl-project/sglang/pull/31860)\r\n* Fix Inkling tool-call parsing recovery, content handling, and streaming: [#32861](https://github.com/sgl-project/sglang/pull/32861)\r\n* Fix LFM 2 tool parser.: [#27614](https://github.com/sgl-project/sglang/pull/27614)\r\n* [Fix]: render tool_reference schema regardless of tool_result part order: [#32522](https://github.com/sgl-project/sglang/pull/32522)\r\n* Treat partial_json_parser AssertionError as incomplete JSON: [#31975](https://github.com/sgl-project/sglang/pull/31975)\r\n* Flush dropped reasoning at stream end when stream_reasoning=False: [#32225](https://github.com/sgl-project/sglang/pull/32225)\r\n* fix(reasoning): let --enable-strict-thinking works for DeepSeek-V4: [#32400](https://github.com/sgl-project/sglang/pull/32400)\r\n* fix(reasoning): honor Poolside template thinking defaults: [#32540](https://github.com/sgl-project/sglang/pull/32540)\r\n* Guard min_new_tokens penalizer against None eos_token_id: [#31973](https://github.com/sgl-project/sglang/pull/31973)\r\n* [Inkling] Add minimal DFLASH support: [#31840](https://github.com/sgl-project/sglang/pull/31840)\r\n* [Inkling] Hold the short-conv per-step state on one metadata struct: [#33116](https://github.com/sgl-project/sglang/pull/33116)\r\n* Fix broken Nemotron DP attention: [#33123](https://github.com/sgl-project/sglang/pull/33123)\r\n* [Nemotron] Fix decode track-save reading the stale tail of the CUDA-graph track buffer: [#32555](https://github.com/sgl-project/sglang/pull/32555)\r\n* [Nemotron] Hoist mamba track-mask host syncs out of the per-layer prefill path: [#32589](https://github.com/sgl-project/sglang/pull/32589)\r\n* [LFM2] Wire Lfm2MoeForCausalLM into the LFM2 serving override tables: [#30780](https://github.com/sgl-project/sglang/pull/30780)\r\n* Fix LongCat n-gram token-table crashes on padded batches: [#31312](https://github.com/sgl-project/sglang/pull/31312)\r\n* fix(qwen3.5): restrict MoE weights to local PP layers: [#32022](https://github.com/sgl-project/sglang/pull/32022)\r\n* [minimax-m3] fp8 attention GEMMs on SM100 (fp8_e4m3 KV + trtllm_mha): [#30971](https://github.com/sgl-project/sglang/pull/30971)\r\n* Prewarm DSV4 MHC post kernel at model load: [#30741](https://github.com/sgl-project/sglang/pull/30741)\r\n* fix mqa preshuffle layout issue for deepseek v4: [#31563](https://github.com/sgl-project/sglang/pull/31563)\r\n* [DeepSeek-V4] Fix nvcc 13 crash building the topk_v2 kernel: [#32910](https://github.com/sgl-project/sglang/pull/32910)\r\n* [RL] DSV4: dispatch indexer topk_transform_512 through DSATopKBackend: [#31087](https://github.com/sgl-project/sglang/pull/31087)\r\n* [RL] DSV4: add env to quantize SWA KV cache from bf16-rounded values: [#31086](https://github.com/sgl-project/sglang/pull/31086)\r\n* [RL] Support FlashInfer TRT-LLM NVFP4 MoE in the RL weight checker: [#31085](https://github.com/sgl-project/sglang/pull/31085)\r\n* [DLLM] vectorized joint/low-confidence decoding and skip redundant attn init: [#21094](https://github.com/sgl-project/sglang/pull/21094)\r\n* [Bugfix] Fix Cohere2MoeConfig import crash from huggingface_hub @strict: [#31769](https://github.com/sgl-project/sglang/pull/31769)\r\n\r\n## Kernel Library (`sglang.kernels`, RFC #29630)\r\n\r\n* [Kernel] RFC #29630 finale: retire sglang.jit_kernel into sglang.kernels: [#32072](https://github.com/sgl-project/sglang/pull/32072)\r\n* [Kernel] Move sgl-kernel under sglang.kernels.aot: [#32648](https://github.com/sgl-project/sglang/pull/32648)\r\n* [Kernel] Phase 3+4: move JIT infra + operator groups into sglang.kernels (RFC #29630): [#31666](https://github.com/sgl-project/sglang/pull/31666)\r\n* [Kernel] Phase 4 batch-2: migrate JIT operator groups into kernels.ops (no shims) (RFC #29630): [#32015](https://github.com/sgl-project/sglang/pull/32015)\r\n* [Kernel] Phase 4 batch-3: migrate tangled JIT subsystems + new groups into kernels.ops (RFC #29630): [#32045](https://github.com/sgl-project/sglang/pull/32045)\r\n* [Kernel] Sweep missed dedicated kernels into kernels.ops (moe/quant siblings + dspark) (RFC #29630): [#32160](https://github.com/sgl-project/sglang/pull/32160)\r\n* [Kernel] Classification cleanup: unify _jit_ naming, drop empty/model groups, add elementwise (RFC #29630): [#32148](https://github.com/sgl-project/sglang/pull/32148)\r\n* [Kernel] Reclassify kernel tests by ops group + move helpers out of the package (RFC #29630): [#32128](https://github.com/sgl-project/sglang/pull/32128)\r\n* Delete sgl-kernel AOT `bmm_fp8`, use `flashinfer.bmm_fp8`: [#31202](https://github.com/sgl-project/sglang/pull/31202)\r\n* Delete sgl-kernel AOT router GEMM and fused A GEMM: [#30280](https://github.com/sgl-project/sglang/pull/30280)\r\n\r\n## SGLang-Diffusion\r\n\r\n* [diffusion] model: support minimax-h3: [#33275](https://github.com/sgl-project/sglang/pull/33275)\r\n* [diffusion] FLUX.2 VAE decoder fast path behind quality=high (H200: 1024^2 97.6->29.2 ms, 2048^2 437.2->168.5 ms): [#33451](https://github.com/sgl-project/sglang/pull/33451)\r\n* [diffusion] Fuse DiT FFN tanh-GELU into up-proj GEMM (cublasLt epilogue) behind quality=high (Qwen-Image 1024^2 denoise 12.36 -> 12.05 s on H200): [#33536](https://github.com/sgl-project/sglang/pull/33536)\r\n* [diffusion] feat: add dynamic cuDNN SDPA attention backend: [#30090](https://github.com/sgl-project/sglang/pull/30090)\r\n* [diffusion] feat: support resident layers for DiT: [#31538](https://github.com/sgl-project/sglang/pull/31538)\r\n* [diffusion] feat: unify encoder folding and batch data-parallel encoding: [#30211](https://github.com/sgl-project/sglang/pull/30211)\r\n* feat(diffusion): add regional torch compile: [#32696](https://github.com/sgl-project/sglang/pull/32696)\r\n* perf(diffusion): decode Wan VAE in BF16: [#32697](https://github.com/sgl-project/sglang/pull/32697)\r\n* [diffusion] optimization: accelerate CUDA video output finalization: [#32784](https://github.com/sgl-project/sglang/pull/32784)\r\n* [diffusion] optimization: support cuda-ipc zero-staging all-to-all for 2-rank Ulysses: [#31854](https://github.com/sgl-project/sglang/pull/31854)\r\n* [Diffusion] offload rollout weights to pinned host memory: [#32032](https://github.com/sgl-project/sglang/pull/32032)\r\n* [diffusion] fix: preserve tensor stride when offloading rollout weights to pinned host memory: [#32420](https://github.com/sgl-project/sglang/pull/32420)\r\n* [Diffusion] Return scheduler sigmas snapshot in rollout dit_trajectory: [#32683](https://github.com/sgl-project/sglang/pull/32683)\r\n* [diffusion] fix: reject ring parallelism where it would silently miscompute: [#33353](https://github.com/sgl-project/sglang/pull/33353)\r\n* fix(diffusion): size VSA top-k from padded blocks: [#32695](https://github.com/sgl-project/sglang/pull/32695)\r\n* [diffusion] fix: keep fused qk-norm-rope out of dynamo tracing: [#31849](https://github.com/sgl-project/sglang/pull/31849)\r\n* [diffusion] fix: fix dual-DiT models crash with (1,)-placeholder weights after compile-time offload: [#32743](https://github.com/sgl-project/sglang/pull/32743)\r\n* [diffusion] fix: per-shard FP8 scale shape for single-GPU fused linears: [#32157](https://github.com/sgl-project/sglang/pull/32157)\r\n* [diffusion] fix: fix local-path detection for MiniMax-H3 and other non-diffusers models: [#33365](https://github.com/sgl-project/sglang/pull/33365)\r\n* [diffusion] fix: don't self-kill diffusion worker when PID 1 is the real parent: [#31361](https://github.com/sgl-project/sglang/pull/31361)\r\n* [Diffusion] Fix component accuracy topology reuse: [#33317](https://github.com/sgl-project/sglang/pull/33317)\r\n* [vla] fix: pi05 models does not apply scale factor for language embeddings: [#33367](https://github.com/sgl-project/sglang/pull/33367)\r\n\r\n## Local & Desktop AI\r\n\r\n* feat: SM120 (Blackwell Desktop) support for GLM-5.1 inference: [#26928](https://github.com/sgl-project/sglang/pull/26928)\r\n* Add FP4 Indexer for DeepSeek V4 on SM120: [#27059](https://github.com/sgl-project/sglang/pull/27059)\r\n* Enable GPT-OSS FlashInfer MXFP4 on SM120: [#32668](https://github.com/sgl-project/sglang/pull/32668)\r\n* Change the FP8 per-tensor GEMM backend on SM120 to cuBLAS: [#31961](https://github.com/sgl-project/sglang/pull/31961)\r\n* [SM120] Only split touched SWA pages in FlashMLA page-split kernel: [#32320](https://github.com/sgl-project/sglang/pull/32320)\r\n* [DSV4/SM120] Allow fused MHC opt-in with standalone TileLang pre disabled: [#30954](https://github.com/sgl-project/sglang/pull/30954)\r\n* Sm120 scatter fallback: [#31669](https://github.com/sgl-project/sglang/pull/31669)\r\n* [MLX] Fix overlap-loop request bookkeeping and graceful shutdown: [#32447](https://github.com/sgl-project/sglang/pull/32447)\r\n* [MLX] Size request capacity by attention DP: [#32115](https://github.com/sgl-project/sglang/pull/32115)\r\n* [MLX] Handle configs without quant_method in Humming: [#31460](https://github.com/sgl-project/sglang/pull/31460)\r\n* [diffusion] fix: fix diffusion output stability on mps: [#30017](https://github.com/sgl-project/sglang/pull/30017)\r\n\r\nNew recipes: [RTX 5090 DeepSeek-V4](https://github.com/sgl-project/sglang/pull/33157), [DGX Spark for Inkling-Small](https://github.com/sgl-project/sglang/pull/33131), [MiniMax-H3 on 2x RTX 5090](https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3) (TP2 + layerwise offload).\r\n\r\n## AMD / ROCm\r\n\r\n* [AMD] Add fused all-reduce RMSNorm per-group quant for Qwen3.5 FP8: [#24651](https://github.com/sgl-project/sglang/pull/24651)\r\n* [AMD][MXFP4] Reland \"Online MXFP4 quantization 2/N - FP8 to MXFP4 requantization on AMD GPUs\": [#28291](https://github.com/sgl-project/sglang/pull/28291)\r\n* [AMD] Cache AITER expert mask across decode: [#31889](https://github.com/sgl-project/sglang/pull/31889)\r\n* [AMD] Derive AITER verify tokens-per-req from input shape: [#31221](https://github.com/sgl-project/sglang/pull/31221)\r\n* [AMD] Speed up DSV4 MoE weight loading from mmap views: [#32315](https://github.com/sgl-project/sglang/pull/32315)\r\n* [AMD] Fix DeepSeek-V4 FP4 MoE expert memory bloat: [#31450](https://github.com/sgl-project/sglang/pull/31450)\r\n* [AMD] DSv4: bring HIP compress-state pool into the memory_saver KV_CACHE region: [#31747](https://github.com/sgl-project/sglang/pull/31747)\r\n* [AMD] Fix DeepSeek-V4 fused-RMS FP8 scale metadata on gfx950: [#31727](https://github.com/sgl-project/sglang/pull/31727)\r\n* [AMD] [Fix] Enable aiter hd256 FP8 prefill FMHA on gfx950: [#33399](https://github.com/sgl-project/sglang/pull/33399)\r\n* [AMD]Qwen3.5 integration gfx950 fmha fp8 hd256: [#32046](https://github.com/sgl-project/sglang/pull/32046)\r\n* [Fix][AMD] Qwen3.5 MoE: disable global-slot shared-expert fusion under per-rank EP backends (MoRI + dp-attention init crash): [#31793](https://github.com/sgl-project/sglang/pull/31793)\r\n* [AMD] MiniMax-M3: opt-in custom/quick all-reduce on ROCm: [#32230](https://github.com/sgl-project/sglang/pull/32230)\r\n* [AMD] Minimax-M3 : unblock mxfp8 block convert on gfx950: [#32036](https://github.com/sgl-project/sglang/pull/32036)\r\n* [AMD] Enable mamba JIT transfer kernel on ROCm (fix transfer_kv_mamba NameError): [#31741](https://github.com/sgl-project/sglang/pull/31741)\r\n* [AMD] add Gemma3RMSNorm.forward_hip to unbreak ROCm: [#32613](https://github.com/sgl-project/sglang/pull/32613)\r\n* [AMD][Fix] Restore aiter-padded MoE weight dims for serialized checkpoints: [#33090](https://github.com/sgl-project/sglang/pull/33090)\r\n* [AMD] Fix JIT compile failure in sgl_kernel/warp.cuh: [#33195](https://github.com/sgl-project/sglang/pull/33195)\r\n\r\n## NPU / Ascend\r\n\r\n* [NPU] Optimize DeepSeek-V4 performance: [#31931](https://github.com/sgl-project/sglang/pull/31931)\r\n* :sparkles: [llm][npu][quant] Add W8A8 MXFP8 quantization for Qwen3 MoE on Ascend NPU: [#30768](https://github.com/sgl-project/sglang/pull/30768)\r\n* [NPU] adapt dflash v2 on npu: [#31739](https://github.com/sgl-project/sglang/pull/31739)\r\n* [NPU] Adapt MiMo-V2.5-W8A8: [#29131](https://github.com/sgl-project/sglang/pull/29131)\r\n* [NPU] Enable automatic ascend_attn selection for vision attention and graph runners: [#31948](https://github.com/sgl-project/sglang/pull/31948)\r\n* [NPU] ascend fuseep use moe ep group: [#32040](https://github.com/sgl-project/sglang/pull/32040)\r\n* [NPU] Fix MTP IndexShare warm-up for attention DP and prefill CP: [#32210](https://github.com/sgl-project/sglang/pull/32210)\r\n* [NPU] fix dsv4 mtp condition on NPU graph: [#32711](https://github.com/sgl-project/sglang/pull/32711)\r\n* [NPU] [FIX] Fix performance degradation of Qwen3.5-397B-A17B: [#32130](https://github.com/sgl-project/sglang/pull/32130)\r\n* [Bugfix] [NPU] Fix w4a8 MoE performance degradation: [#32113](https://github.com/sgl-project/sglang/pull/32113)\r\n* [NPU] Fix LLaDA2 MoE OOM after the FRACTAL_NZ cast, re-enabling the NZ speedup: [#31772](https://github.com/sgl-project/sglang/pull/31772)\r\n* [NPU] Determine the topk norm_type through scoring_func: [#31393](https://github.com/sgl-project/sglang/pull/31393)\r\n* [NPU] Acc fix for afmoe model introduced by topk refactor.: [#31280](https://github.com/sgl-project/sglang/pull/31280)\r\n* [NPU] bugfix for extra device memory on Ascend: [#30112](https://github.com/sgl-project/sglang/pull/30112)\r\n* [NPU][Fix Issue]: Send expert weights contiguous tensor across cards during EPLB rebalance: [#32001](https://github.com/sgl-project/sglang/pull/32001)\r\n* [NPU]Add Ascend transfer version compatibility.: [#31189](https://github.com/sgl-project/sglang/pull/31189)\r\n* [NPU] Update non-vit vision part for cumulative seqlen: [#31867](https://github.com/sgl-project/sglang/pull/31867)\r\n* :bug: [llm][npu][quant] Fix ModelSlim MXFP4 packed weight loading: [#32013](https://github.com/sgl-project/sglang/pull/32013)\r\n* add fill_draft_extend_prepare_buffers_native for NPU: [#32427](https://github.com/sgl-project/sglang/pull/32427)\r\n* sgl-kernel-npu tag update to 2026.7.27: [#32596](https://github.com/sgl-project/sglang/pull/32596)\r\n\r\n## CPU / Intel / XPU\r\n\r\n* [Intel GPU] Add XPU Platform support: [#31949](https://github.com/sgl-project/sglang/pull/31949)\r\n* [XPU] Enable breakable prefill CUDA graph on XPU: [#30273](https://github.com/sgl-project/sglang/pull/30273)\r\n* [XPU][GDN] add XPU path for causal_conv1d_fn and causal_conv1d_update: [#31250](https://github.com/sgl-project/sglang/pull/31250)\r\n* [Intel GPU] calculate free memory based on allocated memory for XPU: [#32044](https://github.com/sgl-project/sglang/pull/32044)\r\n* [sgl-kernel][CPU] add kernel for shm_allgather_into_tensor and shm_reduce_scatter_tensor: [#13397](https://github.com/sgl-project/sglang/pull/13397)\r\n* [CPU] refactor rope kernels: [#31897](https://github.com/sgl-project/sglang/pull/31897)\r\n* [CPU] Fix mxfp4 padding size: [#31334](https://github.com/sgl-project/sglang/pull/31334)\r\n* [Bugfix] Fix CUDA import on non-CUDA platforms: [#31919](https://github.com/sgl-project/sglang/pull/31919)\r\n\r\n## Dependencies\r\n\r\n* Bump FlashInfer to 0.6.15.post1: [#31927](https://github.com/sgl-project/sglang/pull/31927) ⭐\r\n* Bump sgl-deep-gemm to 0.1.5: [#32345](https://github.com/sgl-project/sglang/pull/32345)\r\n* Replace Kimi K3 DeepGEMM patch with 0.1.5.post1: [#33143](https://github.com/sgl-project/sglang/pull/33143)\r\n* Bump helion version to 1.4: [#32562](https://github.com/sgl-project/sglang/pull/32562)\r\n* chore: bump mooncake version to 0.3.12.post1: [#32302](https://github.com/sgl-project/sglang/pull/32302)\r\n* bump dynamo-tokenizers to 1.7.0: [#32981](https://github.com/sgl-project/sglang/pull/32981)\r\n* [AMD] ROCm AITER pin reverted to 9127c94, then updated to d9e5ef7: [#32879](https://github.com/sgl-project/sglang/pull/32879), [#32939](https://github.com/sgl-project/sglang/pull/32939)\r\n* Fix PyPI release: drop the git-only sgl-eval dep from packaged metadata: [#32354](https://github.com/sgl-project/sglang/pull/32354)\r\n\r\n## Breaking Changes & Upgrade Notes\r\n\r\n* **`helion` jumps 0.2.6 to 1.4**, a major-version move for anyone depending on helion-backed kernels: [#32562](https://github.com/sgl-project/sglang/pull/32562)\r\n* **`sglang.jit_kernel` is retired into `sglang.kernels`**, completing RFC [#29630](https://github.com/sgl-project/sglang/issues/29630). Imports from the old module path must move: [#32072](https://github.com/sgl-project/sglang/pull/32072), [#31666](https://github.com/sgl-project/sglang/pull/31666), [#32015](https://github.com/sgl-project/sglang/pull/32015), [#32045](https://github.com/sgl-project/sglang/pull/32045)\r\n* **Breakable prefill CUDA graph is now on by default for DP attention**: [#31682](https://github.com/sgl-project/sglang/pull/31682)\r\n* **CuteDSL BF16 GEMM on SM100 is on by default** when the heuristic allows it: [#30567](https://github.com/sgl-project/sglang/pull/30567)\r\n* **sgl-kernel AOT `bmm_fp8` is deleted in favor of `flashinfer.bmm_fp8`; the AOT router GEMM and fused A GEMM are also removed**: [#31202](https://github.com/sgl-project/sglang/pull/31202), [#30280](https://github.com/sgl-project/sglang/pull/30280)\r\n* **The FP8 per-tensor GEMM backend on SM120 switches to cuBLAS**: [#31961](https://github.com/sgl-project/sglang/pull/31961)\r\n* **Unfetchable or unparseable multimodal inputs now return HTTP 400 instead of 500**, and text-only models reject media input outright: [#31417](https://github.com/sgl-project/sglang/pull/31417), [#32914](https://github.com/sgl-project/sglang/pull/32914)\r\n* **Diffusion request-level quality is restricted to two validated tiers**, lossless (default) and high: [#33453](https://github.com/sgl-project/sglang/pull/33453)\r\n* **`ServerArgs` is being made read-only** behind namespace config bags. Code that mutated `ServerArgs` at runtime must route through the new accessors: [#31809](https://github.com/sgl-project/sglang/pull/31809), [#31811](https://github.com/sgl-project/sglang/pull/31811), [#33334](https://github.com/sgl-project/sglang/pull/33334), [#33338](https://github.com/sgl-project/sglang/pull/33338)\r\n\r\n## Known Issues\r\n\r\n* **The FlashInfer rmsnorm + quant fusion for SM90/SM100/SM120 was landed and reverted** this cycle ([#32994](https://github.com/sgl-project/sglang/pull/32994), [#33455](https://github.com/sgl-project/sglang/pull/33455)). It is not in this release.\r\n* **Breakable CUDA graph is disabled for NemotronH** pending a fix: [#33130](https://github.com/sgl-project/sglang/pull/33130)\r\n* **`test_update_weights_from_disk` is skipped on ROCm** pending a reload fix ([#31924](https://github.com/sgl-project/sglang/issues/31924)): [#31925](https://github.com/sgl-project/sglang/pull/31925)\r\n* **The SM120 DeepGEMM release-pipeline test is temporarily skipped**: [#32193](https://github.com/sgl-project/sglang/pull/32193)\r\n\r\n**Full Changelog**: [`v0.5.16...v0.5.17`](https://github.com/sgl-project/sglang/compare/v0.5.16...v0.5.17)\r\n\r\n## New Contributors\r\n* @windscope made their first contribution in https://github.com/sgl-project/sglang/pull/31738\r\n* @kfhfar made their first contribution in https://github.com/sgl-project/sglang/pull/31825\r\n* @fkuner made their first contribution in https://github.com/sgl-project/sglang/pull/29973\r\n* @ilyasher-harmonic made their first contribution in https://github.com/sgl-project/sglang/pull/30832\r\n* @tedzhouhk made their first contribution in https://github.com/sgl-project/sglang/pull/32122\r\n* @calvin0327 made their first contribution in https://github.com/sgl-project/sglang/pull/31460\r\n* @libinta made their first contribution in https://github.com/sgl-project/sglang/pull/23534\r\n* @houseroad made their first contribution in https://github.com/sgl-project/sglang/pull/31826\r\n* @mosya415 made their first contribution in https://github.com/sgl-project/sglang/pull/31346\r\n* @yangspirit made their first contribution in https://github.com/sgl-project/sglang/pull/29326\r\n* @SovietPower made their first contribution in https://github.com/sgl-project/sglang/pull/31144\r\n* @hsthe29 made their first contribution in https://github.com/sgl-project/sglang/pull/30096\r\n* @yyq0210 made their first contribution in https://github.com/sgl-project/sglang/pull/26888\r\n* @shanemort1982 made their first contribution in https://github.com/sgl-project/sglang/pull/31753\r\n* @ormandj made their first contribution in https://github.com/sgl-project/sglang/pull/30954\r\n* @daii-0818 made their first contribution in https://github.com/sgl-project/sglang/pull/31992\r\n* @gongy made their first contribution in https://github.com/sgl-project/sglang/pull/31417\r\n* @yilin-void made their first contribution in https://github.com/sgl-project/sglang/pull/26928\r\n* @xiaolin2004 made their first contribution in https://github.com/sgl-project/sglang/pull/32448\r\n* @xiaojun-zhang made their first contribution in https://github.com/sgl-project/sglang/pull/32104\r\n* @waynel96 made their first contribution in https://github.com/sgl-project/sglang/pull/31968\r\n* @andrew52522 made their first contribution in https://github.com/sgl-project/sglang/pull/30614\r\n* @Lin-z-w made their first contribution in https://github.com/sgl-project/sglang/pull/30256\r\n* @13524182838 made their first contribution in https://github.com/sgl-project/sglang/pull/31563\r\n* @Hert4 made their first contribution in https://github.com/sgl-project/sglang/pull/32736\r\n* @Oxygen56 made their first contribution in https://github.com/sgl-project/sglang/pull/32540\r\n* @wirybeaver made their first contribution in https://github.com/sgl-project/sglang/pull/32115\r\n* @JinuJeong made their first contribution in https://github.com/sgl-project/sglang/pull/31339\r\n* @qiushixiaoyu made their first contribution in https://github.com/sgl-project/sglang/pull/29016\r\n* @Broduker made their first contribution in https://github.com/sgl-project/sglang/pull/30240\r\n* @vroomfondel made their first contribution in https://github.com/sgl-project/sglang/pull/31220\r\n* @zkyue made their first contribution in https://github.com/sgl-project/sglang/pull/31128\r\n* @ramm made their first contribution in https://github.com/sgl-project/sglang/pull/32896\r\n* @ajtulloch made their first contribution in https://github.com/sgl-project/sglang/pull/33126\r\n* @alumkal made their first contribution in https://github.com/sgl-project/sglang/pull/30971\r\n* @jinzhenfan made their first contribution in https://github.com/sgl-project/sglang/pull/31221\r\n* @Zhangmj0621 made their first contribution in https://github.com/sgl-project/sglang/pull/29173\r\n* @ltaodream made their first contribution in https://github.com/sgl-project/sglang/pull/30177\r\n* @guptaishaan made their first contribution in https://github.com/sgl-project/sglang/pull/32910\r\n* @gitover22 made their first contribution in https://github.com/sgl-project/sglang/pull/32525\r\n* @mrain made their first contribution in https://github.com/sgl-project/sglang/pull/33103\r\n* @amd-oshkarav made their first contribution in https://github.com/sgl-project/sglang/pull/32046\r\n* @CyberSecurityErial made their first contribution in https://github.com/sgl-project/sglang/pull/33065\r\n* @zealoct made their first contribution in https://github.com/sgl-project/sglang/pull/33367\r\n* @TobyMint made their first contribution in https://github.com/sgl-project/sglang/pull/33365\r\n\r\n**Full Changelog**: https://github.com/sgl-project/sglang/compare/v0.5.16...v0.5.17",
"highlights": [
"New Model Support",
"Kimi K3",
"[Kimi] Support kimi-k3: #32541 ⭐",
"[Kimi] Support DCP + DSpark (ported from kimi-k3 branch): #32828",
"[Kimi K3] Add reasoning, tool-call, and OpenAI serving support: #33025",
"feat(kernels): port standalone Kimi K3 kernels: #32890"
],
"prerelease": false
},
"upcoming": [
{
"repository": "sgl-project/sglang",
"kind": "milestone",
"title": "#13290, #13240",
"url": "https://github.com/sgl-project/sglang/milestone/1",
"progress": 0,
"openIssues": 1,
"closedIssues": 0
}
],
"communityDiscussions": []
},
"runs": [
{
"collectedAt": "2026-08-13T12:26:38.318Z",
"since": "2026-08-12T12:26:38.318Z",
"observedCount": 500,
"changedCount": 500
},
{
"collectedAt": "2026-08-13T13:48:00.446149Z",
"since": "2026-08-12T13:48:00.446149Z",
"observedCount": 500,
"changedCount": 500
},
{
"collectedAt": "2026-08-13T16:19:22.035158Z",
"since": "2026-08-12T16:19:22.035158Z",
"observedCount": 500,
"changedCount": 73
},
{
"collectedAt": "2026-08-13T17:43:20.785491Z",
"since": "2026-08-12T17:43:20.785491Z",
"observedCount": 500,
"changedCount": 49
},
{
"collectedAt": "2026-08-13T17:47:07.884300Z",
"since": "2026-08-12T17:47:07.884300Z",
"observedCount": 500,
"changedCount": 4
},
{
"collectedAt": "2026-08-13T18:01:55.420671Z",
"since": "2026-08-12T18:01:55.420671Z",
"observedCount": 500,
"changedCount": 10
}
],
"signals": [
{
"id": "github:sgl-project/sglang:issue:13363",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Roadmap] sglang auto tuner",
"text": "Now we have many kernel backends for moe (e.g., triton, cutlass), attention, and allreduce. For each kernel, we can also tune some configs (e.g., the tile sizes in triton fused moe). Tuning these kernels and choosing dispatching heuristics can be non-trivial. We would like to build a simple command that auto-tune all the kernels and dispatch heuristic for a model. ## Todo 1. Implement a script sglang.auto_tune to tune the kernels and hyperparameters for a specific model. It should dump the optimal tile sizes/dispatching heuristics for all kernels used in this model. Example usage: - `python3 -m sglang.auto_tune --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --tp 8` - `python3 -m sglang.auto_tune --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --tp 4` 2. Start from tuning the triton fused moe https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton 3. Implement a github action workflow that tunes 20 popular models on 5 common platforms. - The workflow should take in two arguments: a list of model names, and a list of runner names - Our CI has H100, H20, H200, B200, GB200 4. Auto choose allreduce algorithms (custom allreduce, nccl, nccl symmetric memory, torch symmetric memory). 5. Auto choose attention kernel backend and moe runner backend. 6. Auto tune the cutlass gemm kernels (with cutlass profiler) 7. Auto choose speculative decoding parameters for different batch sizes.",
"url": "https://github.com/sgl-project/sglang/issues/13363",
"createdAt": "2025-11-16T04:31:44Z",
"updatedAt": "2026-08-13T07:20:24Z",
"timestamp": "2026-08-13T07:20:24Z",
"metrics": {
"reactions": 29,
"comments": 13
},
"labels": [
"good first issue"
],
"author": "sglang-bot",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:17050",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Tracking] CI Test Failures and Fixes",
"text": "# [Tracking] CI Test Failures and Fixes This issue tracks CI test failures, flaky tests, and infrastructure issues across SGLang's CI pipeline. **Note:** We mainly focus on scheduled CI on the `main` branch. **Last auto-update:** 2026-08-13 17:34 UTC — 3 broken, 11 flaky, 670 recently fixed ## Quick Links - **[CI Coverage Overview](https://github.com/sgl-project/sglang/actions/workflows/ci-coverage-overview.yml)** - Check which tests are covered, by which backend/suite, and whether they are disabled - **[CI Failure Monitor](https://github.com/sgl-project/sglang/actions/workflows/ci-failure-monitor.yml)** - Automated failure analysis --- ## Ongoing Issues ### NV CI | Date | Test | Backend | Error | Notes | CI Status | Assignee | Related | |------|------|---------|-------|-------|-----------|----------|---------| | 2026-08-07 | `test_component_accuracy_1_gpu.py` | call-multimodal-gen-tests / multimodal-gen-component-accuracy | Job timed out while running this test | timeout (1% fail, 30/2579) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31148342046/job/92772628663) | | 2026-05-13 | `test_deepseek_v4_pd_disagg_b200.py` | stage-c-test-dsv4-8-gpu-b200 | RuntimeError: Server process exited with code -9 | server (100% fail, 4/4) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/25814445163/job/75839184349) | | 2026-08-07 | `test_disaggregation_decode_radix_cache.py` | base-c-test-8-gpu-h20 / base-c-test-8-gpu-h20 (0) | AssertionError: 0.8 not greater than 0.8 | accuracy (2% fail, 111/4694) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31145822964/job/92765060024) | | 2026-05-22 | `test_dsa_models_mtp.py` | stage-c-test-8-gpu-h200 (0) | AssertionError: 179.9824287273711 not greater than 180 | performance (9% fail, 35/375) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/26282270975/job/77394696209) | | 2026-08-07 | `test_dsv4_nonpaged_indexer.py` | base-a-test-cpu (4) | AssertionError: False is not true | assertion (0% fail, 2/4331) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31153774995/job/92788895381) | | 2026-08-06 | `test_gpt_oss_4gpu_mxfp4_cp.py` | call-pr-test-extra / extra-b-test-4-gpu-b200 / extra-b-test-4-gpu-b200 (0) | AssertionError: np.float64(0.0) not greater than or equal to 0.58 | accuracy (1% fail, 1/170) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31098531699/job/92606388948) | | 2026-08-07 | `test_inkling_attn_prologue_tau.py` | call-jit-kernel-tests / jit-kernel-b200-test | ValueError: config namespace 'exec' not published | unknown (0% fail, 3/1961) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31150586351/job/92779317014) | | 2026-08-07 | `test_kda_prefill.py` | base-c-test-4-gpu-gb300 / base-c-test-4-gpu-gb300 (1) | Job timed out while running this test | timeout (0% fail, 4/777) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31145822964/job/92765060117) | | 2026-08-07 | `test_minimax_fused_qkv_index_gemm.py` | call-jit-kernel-tests / jit-kernel-b200-test | TypeError: _deepgemm_w8a8_mxfp8_linear_with_fallback() got an unexpected keyw... | environment (0% fail, 2/1429) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31140998760/job/92750824496) | | 2026-08-07 | `test_ngram_mamba_verify_update.py` | base-a-test-cpu (0) | AssertionError: Expected 'update_mamba_state_after_mtp_verify' to have been c... | assertion (0% fail, 12/8795) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31152212236/job/92784171994) | | 2026-08-07 | `test_phase_checker.py` | base-b-test-1-gpu-small / base-b-test-1-gpu-small (2) | AssertionError: 2 not found in (0, -6) : unexpected returncode 2; stdout: pid... | assertion (0% fail, 13/6023) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31145822964/job/92765042429) | | 2026-08-07 | `test_self_e2e_bench_speed.py` | call-pr-test-extra / extra-a-test-1-gpu-large / extra-a-test-1-gpu-large (0) | AssertionError: 68.509327374142 not less than 3.0 : [canary self-bench] qwen3... | performance (3% fail, 5/185) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31135821597/job/92734882237) | | 2026-08-07 | `test_spec_eagle_fa3.py` | base-b-test-1-gpu-large / base-b-test-1-gpu-large (3) | ConnectionRefusedError: [Errno 111] Connection refused | connection (0% fail, 73/16282) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31145822964/job/92765042431) | | 2026-08-07 | `test_vision_openai_server_a.py` | base-b-test-1-gpu-large / base-b-test-1-gpu-large (1) | | unknown (2% fail, 207/12602) | Enabled | - | [Failure](https://github.com/sgl-project/sglang/actions/runs/31148872118/job/92779252396) | ---
Recently Fixed (click to expand)
| Date | Issue | Fix | |------|-------|-----| | 2026-08-13 | `test_basic_sanity_eagle3.py` — AssertionError: np.float64(0.11491628614916286) not great... | - | | 2026-08-13 | `test_bench_serving_2gpu.py` — | - | | 2026-08-13 | `test_component_accuracy_2_gpu.py` — Job timed out while running this test | - | | 2026-08-13 | `test_disaggregation_basic.py` — TimeoutError: Endpoint http://127.0.0.1:11200/health did ... | - | | 2026-08-13 | `test_disaggregation_different_tp.py` — Job timed out while running this test | - | | 2026-08-13 | `test_disaggregation_nixl.py` — RuntimeError: Server process exited with code -9 | - | | 2026-08-13 | `test_disaggregation_pp.py` — RuntimeError: Server process exited with code -9 | - | | 2026-08-13 | `test_dsa_glm52_nvfp4_tp_mtp.py` — AssertionError: 1.5867223620114779 not greater than 3.7 | - | | 2026-08-13 | `test_full_cuda_graph_prefill.py` — | - | | 2026-08-13 | `test_fused_store_index_cache.py` — AssertionError: FP8 code difference exceeds 1-ULP: max re... | - | | 2026-08-13 | `test_hadamard_jit.py` — Job timed out while running this test | - | | 2026-08-13 | `test_hicache_variants.py` — AssertionError: 0.0 not greater than or equal to 0.65 | - | | 2026-08-13 | `test_modelopt_fp8.py` — AttributeError: 'GenerateSampler' object has no attribute... | - | | 2026-08-13 | `test_nvfp4_gemm_sm120.py` — | - | | 2026-08-13 | `test_openai_server.py` — AssertionError: 'prompt_tokens' not found in {'input_toke... | - | | 2026-08-13 | `test_plan_fuzz.py` — Job timed out while running this test | - | | 2026-08-13 | `test_server_args.py` — Job timed out while running this test | - | | 2026-08-13 | `test_spec_eagle.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-08-13 | `test_srt_endpoint.py` — | - | | 2026-08-13 | `test_streaming_session_swa.py` — | - | | 2026-08-13 | `test_swa_radix_cache_kl.py` — AssertionError: avg_kl_div=0.021777368345877342 > thresho... | - | | 2026-08-13 | `test_unified_radix_cache_kl_full.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-13 | `test_unified_radix_cache_kl_swa.py` — | - | | 2026-08-13 | `test_unified_radix_cache_unittest.py` — AttributeError: 'SWAKVPool' object has no attribute 'k_bu... | - | | 2026-08-12 | `test_decode_queue_cleanup.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-08-12 | `test_disaggregation_optimistic_prefill.py` — RuntimeError: Server process exited with code -9 | - | | 2026-08-12 | `test_dp_attention.py` — | - | | 2026-08-12 | `test_dsv32_indexer_fusion.py` — AssertionError: Tensor-likes are not close! | - | | 2026-08-12 | `test_extend_init_contract.py` — AttributeError: 'TinyModelConfig' object has no attribute... | - | | 2026-08-12 | `test_flash_attention_4.py` — AssertionError: np.float64(0.01141552511415525) not great... | - | | 2026-08-12 | `test_flashinfer.py` — AttributeError: 'TritonGDNKernel' object has no attribute... | - | | 2026-08-12 | `test_fused_add_rmsnorm.py` — AssertionError: Tensor-likes are not close! | - | | 2026-08-12 | `test_gpt_oss_4gpu_mxfp4.py` — AssertionError: 0.5707070707070707 not greater than or eq... | - | | 2026-08-12 | `test_hicache_storage_mooncake_backend.py` — | - | | 2026-08-12 | `test_moe_lora_align_block_size.py` — Job timed out while running this test | - | | 2026-08-12 | `test_pooled_hidden_states.py` — Job timed out while running this test | - | | 2026-08-12 | `test_qwen3_next_models.py` — AssertionError: avg_kl_div=0.0024535020873649323 > thresh... | - | | 2026-08-12 | `test_spec_eagle_topk.py` — timeout after 1200s | - | | 2026-08-11 | `test_awq.py` — AssertionError: 0.625 not greater than 0.64 | - | | 2026-08-11 | `test_deepseek_v32_indexcache.py` — | - | | 2026-08-11 | `test_disaggregation_unified_memory.py` — | - | | 2026-08-11 | `test_dsa_glm52_hisparse.py` — | - | | 2026-08-11 | `test_dsa_glm52_nvfp4_dp_mtp.py` — AssertionError: 1.109921757082632 not greater than 3.7 | - | | 2026-08-11 | `test_dsa_glm52_tp_mtp.py` — AssertionError: 1.972693097165525 not greater than 3.7 | - | | 2026-08-11 | `test_eagle_reject_sampling.py` — | - | | 2026-08-11 | `test_fa4.py` — KeyError: 'page_table_draft_decode' | - | | 2026-08-11 | `test_flashinfer_a2a_cutedsl_v2.py` — | - | | 2026-08-11 | `test_fused_norm_scale_shift.py` — Job timed out while running this test | - | | 2026-08-11 | `test_hicache_storage.py` — | - | | 2026-08-11 | `test_hicache_storage_3fs_backend.py` — | - | | 2026-08-11 | `test_llama8b_nvfp4_kv_cache_sm120.py` — AssertionError: np.float64(0.6118721461187214) not greate... | - | | 2026-08-11 | `test_lora_qwen3_30b_a3b_instruct_2507_logprob_diff.py` — AssertionError: 0.005007561296224594 not less than or equ... | - | | 2026-08-11 | `test_penalty.py` — AssertionError: 0.20533333333333328 not less than 0.05555... | - | | 2026-08-11 | `test_qknorm.py` — TypeError: __init__(): incompatible function arguments. T... | - | | 2026-08-11 | `test_qwen35_fp4_mtp.py` — | - | | 2026-08-11 | `test_return_indexer_topk.py` — | - | | 2026-08-11 | `test_spec_eagle_stress.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-10 | `test_deepseek_v3_fp4.py` — Job timed out while running this test | - | | 2026-08-10 | `test_dsa_litetopk.py` — RuntimeError: CUDA driver error (/deepgemm/csrc/apis/../j... | - | | 2026-08-10 | `test_hidden_states.py` — AssertionError: 1 not greater than 1 | - | | 2026-08-10 | `test_hisparse_allocator.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-08-10 | `test_kimi_k3_vision.py` — assert False | - | | 2026-08-10 | `test_nvidia_nemotron_3_nano.py` — | - | | 2026-08-10 | `test_qwen3_next_models_mtp.py` — | - | | 2026-08-10 | `test_tracing.py` — exit code -9 | - | | 2026-08-10 | `test_vlm_tp4.py` — AssertionError: 0.3440625 not greater than or equal to 0.... | - | | 2026-08-09 | `test_basic_sanity_dflash.py` — | - | | 2026-08-09 | `test_openai_completion_rust.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-08-09 | `test_spec_eagle_triton.py` — | - | | 2026-08-09 | `test_token_id_retokenize_e2e.py` — | - | | 2026-08-09 | `test_transformers_models.py` — | - | | 2026-08-09 | `test_unified_radix_cache_kl_mamba.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-08-09 | `test_weight_checker_comparator.py` — AssertionError: Compa[36 chars]00732421875, mean_abs_err=... | - | | 2026-08-08 | `test_diffusion_modelopt_fp8_scaled_mm.py` — RuntimeError: size of scales_a is not matched | - | | 2026-08-08 | `test_disaggregation_decode_offload.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-08-08 | `test_disaggregation_dp_attention.py` — AssertionError: 34.73591541443441 not less than 20 | - | | 2026-08-08 | `test_hf_transformers_fastokens.py` — ValueError: model error: byte 0x48 has no token in vocabu... | - | | 2026-08-08 | `test_server_1_gpu.py` — Job timed out while running this test | - | | 2026-08-08 | `test_server_b200.py` — Job timed out while running this test | - | | 2026-08-08 | `test_unified_radix_cache_kl_dsv4.py` — | - | | 2026-08-07 | `test_basic_sanity_dspark.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-07 | `test_deepep_small.py` — AssertionError: 0.6 not greater than 0.6 | - | | 2026-08-07 | `test_deepseek_v4_flash_fp4_b200.py` — AssertionError: 0.0 not greater than 0.85 : output looks ... | - | | 2026-08-07 | `test_disaggregation_dsv4.py` — RuntimeError: Server process exited with code 1 | - | | 2026-08-07 | `test_multi_item_scoring.py` — ImportError: cannot import name 'PrefillCPBCGInput' from ... | - | | 2026-08-07 | `test_mxfp4_sm100_trtllm_gen.py` — AttributeError: 'Mxfp4MoEMethod' object has no attribute ... | - | | 2026-08-07 | `test_openai_embedding.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-08-07 | `test_openai_function_calling.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-07 | `test_reward_models.py` — exit code -9 | - | | 2026-08-07 | `test_spec_eagle_topk_page.py` — AssertionError: np.float64(0.01) not greater than 0.7 | - | | 2026-08-07 | `test_spec_standalone.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-07 | `test_unlimited_ocr_server.py` — | - | | 2026-08-06 | `test_anthropic_tool_use.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-06 | `test_attention_backend_setup.py` — AttributeError: '_FakeBackend' object has no attribute 'e... | - | | 2026-08-06 | `test_dflash.py` — | - | | 2026-08-06 | `test_dp_attention_bcg_kl.py` — AssertionError: 2.093433481397076e-13 != 0.0 | - | | 2026-08-06 | `test_dsa.py` — AttributeError: 'DeepseekSparseAttnBackend' object has no... | - | | 2026-08-06 | `test_dsa_glm52_dp_mtp.py` — AssertionError: 0.046 not greater than or equal to 0.935 | - | | 2026-08-06 | `test_flashinfer_a2a.py` — AssertionError: np.float64(0.06) not greater than 0.9 | - | | 2026-08-06 | `test_function_call_parser.py` — | - | | 2026-08-06 | `test_laguna_config.py` — ModuleNotFoundError: No module named 'vllm' | - | | 2026-08-06 | `test_model_overrides.py` — AssertionError: Items in the first set but not the second: | - | | 2026-08-06 | `test_mooncake_ep_small.py` — | - | | 2026-08-06 | `test_multi_tokenizer.py` — RuntimeError: GPU(s) still not idle after waiting 30s bef... | - | | 2026-08-06 | `test_nixl_backend_basic.py` — AttributeError: 'KVArgsRegisterInfo' object has no attrib... | - | | 2026-08-06 | `test_post_capture_kv_sizing.py` — AssertionError: unexpectedly None : Post-capture KV sizin... | - | | 2026-08-06 | `test_qwen3_vl_feature_materialization.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-08-06 | `test_server_2_gpu.py` — Job timed out while running this test | - | | 2026-08-06 | `test_spec_eagle_page.py` — AssertionError: np.float64(0.015) not greater than 0.7 | - | | 2026-08-06 | `test_spec_ngram.py` — AssertionError: 0.005 not greater than or equal to 0.79 | - | | 2026-08-06 | `test_streaming_session.py` — ConnectionRefusedError: [Errno 111] Connection refused | - | | 2026-08-06 | `test_tekken_tokenizer_routing.py` — | - | | 2026-08-06 | `test_verify_mask.py` — AttributeError: '_FakeAttnBackend' object has no attribut... | - | | 2026-08-05 | `test_dcp_lse_combine.py` — ImportError: cannot import name '_is_mla_dcp_lse_base_on_... | - | | 2026-08-05 | `test_disaggregation_rust_server.py` — AttributeError: 'GenerateSampler' object has no attribute... | - | | 2026-08-05 | `test_encoder_scheduler.py` — AttributeError: 'FakeEncoder' object has no attribute 'se... | - | | 2026-08-05 | `test_flashkda_strided_state_access.py` — AttributeError: 'tuple' object has no attribute 'shape' | - | | 2026-08-05 | `test_gdn_mtp_cache_mode.py` — AssertionError: ValueError not raised | - | | 2026-08-05 | `test_gdn_replayssm_spec_fold.py` — AssertionError: False is not true : dtype=torch.float32 s... | - | | 2026-08-05 | `test_grammar_manager.py` — ValueError: config namespace 'serving' not published | - | | 2026-08-05 | `test_inkling.py` — | - | | 2026-08-05 | `test_kda_mtp_cutedsl_replayssm_ring.py` — cutlass.base_dsl.compiler.CompilerDiagnosticError: error:... | - | | 2026-08-05 | `test_legacy_global_ratchet.py` — AssertionError: get_global_server_args call-sites grew: 3... | - | | 2026-08-05 | `test_minimax_h3_time_shift_sigmas.py` — exit code 1 | - | | 2026-08-05 | `test_schedule_batch_out_of_place.py` — ValueError: Global server args is not set yet! | - | | 2026-08-05 | `test_server_args_namespaces.py` — AssertionError: ['gdn_mtp_cache_mode'] is not false : Ser... | - | | 2026-08-05 | `test_torch_compile_moe.py` — AssertionError: 280.07555297720876 not greater than or eq... | - | | 2026-08-05 | `test_triton.py` — AttributeError: Mock object has no attribute '_recover_ssm' | - | | 2026-08-05 | `test_trtllm_mha_graph_metadata.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-08-04 | `test_benchmark_datasets_api.py` — AssertionError: expected call not found. | - | | 2026-08-04 | `test_dflash_overlap_hostsync.py` — AssertionError: False is not true | - | | 2026-08-04 | `test_dspark_dp_tier.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-08-04 | `test_encoder_server_metrics.py` — Job timed out while running this test | - | | 2026-08-04 | `test_epd_disaggregation.py` — timeout after 1200s | - | | 2026-08-04 | `test_flux2_vae_fastpath.py` — AttributeError: module 'torch' has no attribute 'ptp' | - | | 2026-08-04 | `test_fp8_utils.py` — RuntimeError: size of scales_a is not matched | - | | 2026-08-04 | `test_gpt_oss_4gpu_bf16.py` — AssertionError: 0.5606060606060606 not greater than or eq... | - | | 2026-08-04 | `test_gptq_marlin.py` — Job timed out while running this test | - | | 2026-08-04 | `test_ipc_a2a_timeout.py` — exit code 1 | - | | 2026-08-04 | `test_kernels_namespace.py` — AssertionError: assert is [stride... | - | | 2026-08-04 | `test_mla_fp8.py` — | - | | 2026-08-04 | `test_mxfp4_sm90_cutlass.py` — AttributeError: '_MockLayer' object has no attribute 'moe... | - | | 2026-08-04 | `test_parallel_adoption_ratchet.py` — AssertionError: ['layers/layernorm.py:397', 'layers/layer... | - | | 2026-08-04 | `test_priority_scheduling_disaggregation.py` — AttributeError: 'DecodePreallocQueue' object has no attri... | - | | 2026-08-04 | `test_ray_wrappers.py` — | - | | 2026-08-04 | `test_rope_cache_invalidation.py` — ModuleNotFoundError: No module named 'vllm' | - | | 2026-08-04 | `test_scheduler_init_req_max_new_tokens.py` — AttributeError: 'Scheduler' object has no attribute 'serv... | - | | 2026-08-04 | `test_server_args_writer_ratchet.py` — AssertionError: ServerArgs.override call-sites shrank: 17... | - | | 2026-08-04 | `test_template_manager.py` — | - | | 2026-08-04 | `test_trtllm_mha.py` — RuntimeError: Error in function 'TllmGenFmhaRunner' at /w... | - | | 2026-08-03 | `test_basic_sanity.py` — | - | | 2026-08-03 | `test_decode_bookkeeping_ownership.py` — AssertionError: [\"New bookkeeping mutation(s) beyond the ... | - | | 2026-08-03 | `test_deepseek_v3_cutedsl_4gpu.py` — | - | | 2026-08-03 | `test_deepseek_v3_fp4_mtp_small.py` — | - | | 2026-08-03 | `test_deepseek_v4_flash_fp4_h200.py` — | - | | 2026-08-03 | `test_deterministic.py` — | - | | 2026-08-03 | `test_dllm_fdfo_kv_reuse.py` — AssertionError: Lists differ: [1, 8] != [1, 2] | - | | 2026-08-03 | `test_driver.py` — AssertionError: ValueError not raised | - | | 2026-08-03 | `test_gemma4_fp8_per_expert_loading.py` — | - | | 2026-08-03 | `test_hicache_dcp_host_pool.py` — AttributeError: 'types.SimpleNamespace' object has no att... | - | | 2026-08-03 | `test_hicache_storage_file_backend.py` — | - | | 2026-08-03 | `test_kimi_linear_models.py` — | - | | 2026-08-03 | `test_linear_attn_config.py` — AttributeError: module 'sglang.srt.layers.attention.linea... | - | | 2026-08-03 | `test_lora_update.py` — | - | | 2026-08-03 | `test_mimo_v2.py` — | - | | 2026-08-03 | `test_minimax_m25_basic.py` — | - | | 2026-08-03 | `test_mla_int8_deepseek_v3.py` — | - | | 2026-08-03 | `test_mooncake_store_config.py` — exit code 1 | - | | 2026-08-03 | `test_moss_vl_processor.py` — Job timed out while running this test | - | | 2026-08-03 | `test_multi_detokenizer.py` — | - | | 2026-08-03 | `test_openai_server_hidden_states.py` — | - | | 2026-08-03 | `test_pp_single_node.py` — | - | | 2026-08-03 | `test_prefill_cuda_graph_runner.py` — AssertionError: GraphCapture(runner=",
"url": "https://github.com/sgl-project/sglang/issues/17050",
"createdAt": "2026-01-14T01:49:31Z",
"updatedAt": "2026-08-13T17:34:27Z",
"timestamp": "2026-08-13T17:34:27Z",
"metrics": {
"reactions": 10,
"comments": 13
},
"labels": [],
"author": "alisonshao",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:19137",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Roadmap] Apple Device Support (2026 Q2)",
"text": "As of February 21, 2026 (date of initial creation of this roadmap), SGLang has no support for Apple Silicon. We are excited to announce the initial roadmap for adding Apple Silicon support. We ask for contributors, of all levels of familiarity with SGLang and Apple devices, to pick up tasks on this roadmap and help us run SGLang on Apple Silicon (M-series). Please comment below to pick up tasks that interest you, and we will assign them to you. Please join the [mlx-backend](https://sgl-fru7574.slack.com/archives/C0AG519A6CW) Slack channel as well (via the SGLang Slack workspace). ## Installation Instructions Here are some quick instructions to set up SGLang on your Mac. Currently, SGLang must be built from source to work on your Mac. **Beware: the Mac version of SGLang is currently verified to work only with Python 3.11 and is known to have issues with other Python versions.** **Beware 2: make sure the `mlx` and `mlx-lm` packages are installed with the LATEST version after you run `uv pip install -e \"python[all_mps]\"`.** They were only added as dependencies in this PR: https://github.com/sgl-project/sglang/pull/22162 ```bash # Install ffmpeg brew install ffmpeg # Install uv brew install uv # Clone the repository git clone https://github.com/sgl-project/sglang.git cd sglang # Create and activate a virtual environment uv venv -p 3.11 my-venv source my-venv/bin/activate # Install the Python packages uv pip install --upgrade pip rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml uv pip install -e \"python[all_mps]\" ``` ## Basic Server Launch Instructions Once you have built SGLang from source using the above instructions, you can launch a basic server using this command: ```bash SGLANG_USE_MLX=1 python -m sglang.launch_server \\ --model-path Qwen/Qwen3-0.6B \\ --port 43440 ``` Now, send a request to that server via this curl command from another shell: ```bash curl http://localhost:43440/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"Qwen/Qwen3-0.6B\", \"messages\": [ {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, {\"role\": \"user\", \"content\": \"Tell me a short joke.\"} ], \"max_tokens\": 10 }' ``` ## Profiling Instructions When contributing PRs, it is important to profile your changes to make sure they work as intended. There are 4 different ways to profile SGLang at 4 different levels of the stack. ```markdown | Tool | HTTP Server | Scheduler | Use Case | | -------------------------- | --------------------------------------------- | --------------------------------------- | -------------------------------------------------------------------------- | | `bench_serving` | Yes (async HTTP client to a running server) | Yes (indirectly, via server) | Realistic online serving benchmarks with latency metrics (TTFT, TPOT, ITL) | | `bench_one_batch_server` | Yes (sends HTTP requests to a running server) | Yes (indirectly, via server) | End-to-end single-batch latency including HTTP and scheduler overhead | | `bench_offline_throughput` | No | Yes (directly uses `Engine` in-process) | Maximum throughput measurement without HTTP overhead | | `bench_one_batch` | No | No (directly calls `ModelRunner`) | Kernel-level latency profiling of a single static batch ``` (taken from benchmark_and_profiling.md, read that for more comprehensive information) **Note: MLX and MLX-LM must be updated to the LATEST version to correctly profile the SGLang Apple Silicon backend. If MLX and MLX-LM are not updated, you may end up with very noisy traces.** There are two types of traces: Metal-only and system-wide. ### Metal-only profiling `bench_one_batch.py` directly runs the prefill and decode phases: ```python # this is a summary of the bench_one_batch.py file start_profile model_runner.extend(...) stop_profile start_profile model_runner.decode(...) stop_profile ``` Because `bench_one_batch.py` directly calls the `extend` and `decode` functions, it cannot be used to profile the server or scheduler. To use `bench_one_batch.py` to profile on Apple Silicon and get a **Metal-only trace**, use this command: ```bash SGLANG_TORCH_PROFILER_DIR=./profiler_output_baseline MTL_CAPTURE_ENABLED=1 SGLANG_USE_MLX=1 python -m sglang.bench_one_batch \\ --model-path Qwen/Qwen3-0.6B \\ --trust-remote-code \\ --disable-radix-cache \\ --disable-cuda-graph \\ --tp-size 1 \\ --batch-size 1 \\ --input-len 60 \\ --output-len 10 \\ --port 43440 \\ --profile \\ --profile-start-step 0 \\ --profile-steps 10 ``` This saves a .gputrace file for prefill and a .gputrace file for decode. Double-click either .gputrace file, and it will automatically open in the XCode app. To use `bench_offline_throughput.py` to profile the GPU operations on Apple Silicon and get a Metal-only trace, use this command: ```bash SGLANG_TORCH_PROFILER_DIR=. SGLANG_USE_MLX=1 MTL_CAPTURE_ENABLED=1 python3 -m sglang.bench_offline_throughput \\ --model-path Qwen/Qwen2.5-0.5B-Instruct \\ --num-prompts 1 \\ --profile ``` To use `bench_one_batch_server.py` to profile the GPU operations on Apple Silicon and get a Metal-only trace, use this command: ```bash SGLANG_TORCH_PROFILER_DIR=. SGLANG_USE_MLX=1 MTL_CAPTURE_ENABLED=1 python3 -m sglang.test.bench_one_batch_server_internal \\ --model Qwen/Qwen2.5-0.5B-Instruct \\ --batch-size 2 \\ --input-len 128 \\ --output-len 16 \\ --profile ``` To use `bench_server.py` to profile the GPU operations on Apple Silicon and get a Metal-only trace, use these TWO commands in different terminals: ```bash SGLANG_TORCH_PROFILER_DIR=. SGLANG_USE_MLX=1 MTL_CAPTURE_ENABLED=1 python3 -m sglang.launch_server \\ --model-path Qwen/Qwen2.5-0.5B-Instruct \\ --port 30000 ``` ```bash python3 -m sglang.bench_serving \\ --backend sglang \\ --port 30000 \\ --num-prompts 1 \\ --profile ``` ### System-wide profiling To profile the entire system (CPUs, GPU, drivers, etc.), any Python process can be recorded by simply wrapping the process start command in `xctrace record --template \"Metal System Trace\" --launch -- /usr/bin/env`. Thus, for example, to profile the entire system during the execution of `bench_one_batch.py`, this command can be used: ```bash xctrace record --template \"Metal System Trace\" --launch -- /usr/bin/env SGLANG_USE_MLX=1 $(which python) -m sglang.bench_one_batch \\ --model-path Qwen/Qwen3-0.6B \\ --trust-remote-code \\ --disable-radix-cache \\ --disable-cuda-graph \\ --tp-size 1 \\ --batch-size 1 \\ --input-len 60 \\ --output-len 10 \\ --port 43440 \\ ``` This saves a single .trace file that includes prefill and decode. Double-click the .trace file, and it will open in the Instruments app. ## Initial Support (Proof-Of-Concept) - [x] Add initial `mps` support so that models can run on MacOS. @yeahdongcn https://github.com/sgl-project/sglang/pull/19549 - [x] Add check-env script for Apple devices. @yeahdongcn https://github.com/sgl-project/sglang/pull/20753 ## Implement Full MLX Backend - [x] Switch over completely from PyTorch to MLX via skeleton MLX backend @yeahdongcn https://github.com/sgl-project/sglang/pull/20342 ### MLX runner-stub / framework contract redesign The MLX backend integrates with the framework through `MlxModelRunnerStub`, which re-implements the framework's implicit assumptions about `ModelRunner`. As SGLang features and server arguments evolve, the stub silently drifts from framework behavior — recent instances: #30181, #30389, #30547. Rather than continuing to extend the stub per feature, re-evaluate the integration design for a more fundamental solution. - [ ] Audit the framework↔stub contract: catalog assumptions the stub satisfies, misses, or silently diverges on — @noob-se7en - [ ] Design note with options + trade-offs for a long-term backend contract — @noob-se7en - [ ] Implementation + migration (following design review) — @noob-se7en ## KV Cache - [x] Support radix cache across all batch sizes @yeahdongcn https://github.com/sgl-project/sglang/pull/21509 - [x] Adds env to control allowed number of tokens before clearing the MLX cache @changminbark https://github.com/sgl-project/sglang/pull/28160 ## UMA Memory Management - [ ] https://github.com/sgl-project/sglang/issues/21443 -> https://github.com/sgl-project/sglang/pull/21539 @karanb192 ## Scheduler - [x] #22114 -> https://github.com/sgl-project/sglang/pull/22416 @changminbark ## Metal Kernels To enable blazing fast inference, we need to write custom Metal kernels that are designed for inference instead of relying on MLX's general-purpose \"fast\" kernels. - [x] Enable sgl-kernel to build custom Metal kernels @yeahdongcn https://github.com/sgl-project/sglang/pull/23449 - [ ] Add custom Metal PagedAttention implementation - [ ] Add custom Metal PagedAttention prefill kernel - [ ] Add custom Metal PagedAttention decode kernel @adityavaid https://github.com/sgl-project/sglang/pull/28335/ - [ ] Add custom Metal FusedMoE implementation @jlee5814 - [ ] Add custom Metal FusedMoe kernel + MetalRunnerCore - [ ] Add custom RMSNorm kernel @SasankYadati - [x] Add custom RoPE kernel with fused KV cache store operation @adityavaid https://github.com/sgl-project/sglang/pull/22868 ## Mixture-of-Experts - [ ] #22283 @jlee5814 - https://github.com/sgl-project/sglang/pull/24712 ## Model Support We must rewrite all of the model architecture files in SGLang's main sglang/python/sglang/srt/models folder using the MLX library instead of PyTorch. - [ ] qwen3.py - [x] qwen3.5/3.6 - ~~@JINO-ROHIT https://github.com/sgl-project/sglang/pull/25253~~ - @yeahdongcn https://github.com/sgl-project/sglang/pull/25754 - [ ] qwen3_moe.py - [ ] qwen2.py - [ ] qwen2_moe.py - [ ] gpt_oss.py - [ ] hunyuan.py **Please comment below if there is another model that you want to add support for and I will add it here with your name and PR.** ## Quantization - [x] On-the-fly quant (mlx_q4/mlx_q8) @damahua https://github.com/sgl-project/sglang/pull/24907 - [x] Auto-detect quant method @jlee5814 https://github.com/sgl-project/sglang/pull/25191 ## Speculative Decoding - [ ] Add EAGLE support @Yatogaii ## Multi-LoRA Serving We need to enable LoRA inference that is aligned with the method proposed in [Punica: Multi-Tenant LoRA Serving](https://arxiv.org/abs/2310.18547). Our goal is to utilize as much of the current SGLang LoRA as possible. - [ ] TBD ## Profiling - [x] Add MLX profiling hooks to profile using Apple-native profiling tools with `bench_one_batch.py` https://github.com/sgl-project/sglang/pull/22159 @Jonahcb - [ ] Add MLX profiling hooks to profile using Apple-native profiling tools with `bench_serving.py`, `bench_offline_throughput.py`, and `bench_one_batch_server.py` @LijuanTang94 https://github.com/sgl-project/sglang/pull/28122 https://github.com/sgl-project/sglang/pull/29217 ## CI - [ ] Set up self-hosted MacOS GitHub runners @yeahdongcn @Kangyan-Zhou - [x] Add unit test workflow @jlee5814 https://github.com/sgl-project/sglang/pull/29691 - [ ] Add accuracy-based correctness tests for popular models https://github.com/sgl-project/sglang/pull/21770 @linzhonghong ## Documentation * https://github.com/sgl-project/sglang/pull/25178 @JINO-ROHIT ## Bug Fixes & Bug Tracker * [Python 3.12+ Support](https://github.com/sgl-project/sglang/issues/21548): https://github.com/sgl-project/sglang/pull/21551 @karanb192 * Dependency updates: https://github.com/sgl-project/sglang/pull/22162 @changminbark * Server crash: https://github.com/sgl-project/sglang/issues/22466 * Quant config: https://github.com/sgl-project/sglang/issues/25119 * Qwen3.5/3.6: https://github.com/sgl-project/sglang/issues/25017 * `canary_manager` fix: https://github.com/sgl-project/sglang/pull/26882 * Orphan process handling: https://github.com/sgl-project/sglang/pull/27190 * Server startup fix: https://github.com/sgl-project/sglang/pull/28660 https://github.com/sgl-project/sglang/pull/28770 https://github.com/sgl-project/sglang/pull/29681 * `write_token` fix: https://github.com/sgl-project/sglang/pull/29271 * Unit test fix: https://github.com/sgl-project/sglang/pull/29311 *This roadmap will change rapidly as we work on the above tasks and encounter obstacles that require new tasks.*",
"url": "https://github.com/sgl-project/sglang/issues/19137",
"createdAt": "2026-02-22T03:21:15Z",
"updatedAt": "2026-08-12T21:16:02Z",
"timestamp": "2026-08-12T21:16:02Z",
"metrics": {
"reactions": 16,
"comments": 27
},
"labels": [
"apple-silicon"
],
"author": "jonahbernard",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:20865",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Improve Unit Test Coverage",
"text": "## Motivation SGLang's test suite has **600+ test files**, but most are E2E tests that launch a full server. While these cover many code paths, core modules like `managers/`, `mem_cache/`, `entrypoints/`, `sampling/`, `parser/`, `function_call/`, and `utils/` still need better unit test coverage (tests that run without launching a server). **Why this matters:** - E2E tests launch a server and load model weights, taking minutes. Unit tests run in seconds with mocked dependencies. - When an E2E test fails, the root cause could be anywhere. Unit tests pinpoint the exact broken function. ## What's Already Done - **Unit test directory** [`test/registered/unit/`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit) with READMEs ([`test/registered/README.md`](https://github.com/sgl-project/sglang/blob/main/test/registered/README.md), [`test/registered/unit/README.md`](https://github.com/sgl-project/sglang/blob/main/test/registered/unit/README.md)) documenting conventions - **32 unit tests migrated** across 12 subdirectories - **Coverage tooling** configured ([`.coveragerc`](https://github.com/sgl-project/sglang/blob/main/.coveragerc), `pytest-cov`) - **CI coverage reporting** in [`ci-coverage-overview.yml`](https://github.com/sgl-project/sglang/blob/main/.github/workflows/ci-coverage-overview.yml) (daily + manual trigger) — [example run](https://github.com/sgl-project/sglang/actions/runs/24032286387/attempts/1#summary-70083527423) - **Contribution guide** updated with [unit test section](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/contribution_guide.md#run-and-add-unit-tests) ## Directory Structure ``` test/registered/unit/ # mirrors python/sglang/srt/ ├── mem_cache/ (6 files) ├── function_call/ (5 files) ├── utils/ (4 files) ├── parser/ (3 files) ├── managers/ (3 files) ├── entrypoints/ (2 files) ├── layers/ (2 files) ├── observability/ (2 files) ├── model_loader/ (2 files) ├── server_args/ (1 file) ├── model_executor/ (1 file) ├── batch_invariant_ops/ (1 file) └── ... (new subdirs added as tests are written) ``` **Principle:** Source file at `srt/mem_cache/radix_cache.py` → test at `unit/mem_cache/test_radix_cache.py`. ## Open Tasks Below are some suggested starting points. Comment on this issue to claim one. You can check the latest UT coverage report run ([example](https://github.com/sgl-project/sglang/actions/runs/24032286387/attempts/1#summary-70083527423)) or run coverage locally to find your own opportunities — modules with low coverage under core directories (`managers/`, `sampling/`, `parser/`, etc.) are good candidates: ```bash pytest test/registered/unit/ --cov --cov-config=.coveragerc --cov-report=term-missing -v ``` ### Easy (good first issue) - `srt/parser/` — conversation templates, reasoning parsers, code completion parser - `srt/sampling/` — parameter validation, normalization, logit processor serialization - `srt/entrypoints/openai/` — encoding/decoding, tool call formatting - `srt/function_call/` — extend existing detector tests, add new detector coverage ### Medium - `srt/utils/` — common util helpers - `srt/multimodal/` — media utils, processor base logic - `srt/constrained/` — grammar dispatch logic - `srt/managers/` — template manager, tokenizer utils, batch metadata helpers ## How to Contribute 1. **Pick a task** from above (or find your own via coverage), comment to claim it 2. **Look at existing examples:** - `test/registered/unit/mem_cache/test_evict_policy.py` — pure logic, zero deps - `test/registered/unit/managers/test_prefill_adder.py` — mock factories for scheduler tests - `test/registered/unit/parser/test_reasoning_parser.py` — streaming parser tests 3. **Follow the conventions** in [`test/registered/unit/README.md`](https://github.com/sgl-project/sglang/tree/main/test/registered/unit/README.md) and the [contribution guide](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/contribution_guide.md#run-and-add-unit-tests) 4. **Run locally:** ```bash pytest test/registered/unit/ -v pytest test/registered/unit/ --cov --cov-config=.coveragerc -v # with coverage ``` 5. **Submit a PR** titled `[Test] Add unit tests for `, referencing this issue ### PR Quality Bar **Note:** AI-assisted code is acceptable, but you are responsible for the quality — **understand every line you submit.** We will reject PRs that look auto-generated without understanding — e.g., tests that only check trivial cases, assert on mock return values, or don't actually exercise the real code logic. Read the source code first, then write tests that would catch real bugs. ### PR Requirements - [ ] Test is in `test/registered/unit//` (mirroring `srt/`) - [ ] Does NOT launch a server or load real model weights - [ ] Includes edge cases, not just happy paths - [ ] Use `CustomTestCase` instead of `unittest.TestCase`. - [ ] Registered with `register_cpu_ci()` or `register_cuda_ci()` - [ ] **Locally tested and passing.** Paste both the command and its output in your PR description: ```bash # command pytest test/registered/unit//test_xxx.py -v # output (copy-paste the full result) ```",
"url": "https://github.com/sgl-project/sglang/issues/20865",
"createdAt": "2026-03-18T14:28:20Z",
"updatedAt": "2026-08-13T02:46:05Z",
"timestamp": "2026-08-13T02:46:05Z",
"metrics": {
"reactions": 0,
"comments": 82
},
"labels": [
"enhancement",
"good first issue",
"ci"
],
"author": "ispobock",
"state": "open",
"assignees": [
"Kangyan-Zhou",
"ispobock",
"alisonshao"
]
},
{
"id": "github:sgl-project/sglang:issue:22558",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[RFC] Native gRPC Server for SGLang in Rust",
"text": "### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### **Motivation** SGLang's primary API surface is a FastAPI/Uvicorn HTTP server. The existing `smg-grpc-servicer` package provides an alternative standalone gRPC server (via `--grpc-mode`), but it replaces the HTTP server entirely and still performs all request parsing, and response serialization in Python under the GIL. Neither path addresses the fundamental bottleneck: * **Serialization overhead.** Every request/response traverses JSON serialization, Python dict construction, and Pydantic validation — all under the GIL. * **No native streaming contract.** HTTP SSE is a text-based protocol bolted onto HTTP/1.1. gRPC server-streaming over HTTP/2 provides multiplexed, binary-framed, flow-controlled streams with typed messages. * **GIL contention at the API boundary.** HTTP path acquires the GIL for request parsing, tokenization, argument normalization, and response serialization. SMG gRPC avoids server-side tokenization for Generate/Embed (expects tokenized input) but still uses Python for request conversion, sampling param normalization, and scheduler IPC serialization. Under high concurrency these become the bottleneck — not the GPU. * **Dual-protocol gap.** With smg-grpc-servicer, operators must choose HTTP *or* gRPC — they cannot serve both simultaneously from the same process. Production deployments (Kubernetes, Triton-adjacent stacks, microservice meshes) often need both. ### **Why Rust \\+ In-Process?** An in-process Rust gRPC server (via PyO3/Maturin) serves both protocols simultaneously while progressively moving GIL-bound work out of Python: | Concern | HTTP (current) | smg-grpc-servicer | Native Rust gRPC (this RFC) | | :---- | :---- | :---- | :---- | | Protocol | HTTP/1.1 \\+ JSON | gRPC (standalone) | gRPC (alongside HTTP) | | Process model | In-process | In-process (replaces HTTP) | In-process (additive) | | Dual-protocol | HTTP only | gRPC only | **HTTP \\+ gRPC simultaneously** | | GIL for tokenize | Yes | No | **No** (Rust `tokenizers` crate) | | GIL for request parse | Yes | Yes | **No** (proto → dict in Rust) | | GIL for response delivery | Yes (SSE serialization) | Yes | **Minimal** (brief callback only) | --- ## **Proposed Change** ### **Architecture Overview** A new `sglang-grpc` Rust crate (built with Maturin as a Python extension module) embeds a Tonic gRPC server that runs in a background thread with its own Tokio runtime. It communicates with the existing Python `TokenizerManager` through a thin `RuntimeHandle` bridge, progressively reducing GIL acquisition across five phases. ``` ┌──────────────────────────────────────────────────┐ │ SGLang Process │ │ │ gRPC clients ───►│ ┌─────────────────┐ ┌─────────────────────┐ │ │ │ Rust gRPC │ │ Python │ │ │ │ (Tonic/Tokio) │───►│ RuntimeHandle │ │ │ │ │ │ (grpc_bridge.py) │ │ │ │ • Proto decode │ │ │ │ │ │ • Rust tokenize │ │ TokenizerManager │ │ │ │ • Dict build │ │ Scheduler │ │ │ │ • Stream via │ │ DetokenizerManager │ │ │ │ crossbeam ch. │◄───│ │ │ │ └─────────────────┘ └─────────────────────┘ │ │ │ HTTP clients ───►│ FastAPI / Uvicorn (unchanged) │ └──────────────────────────────────────────────────┘ ``` ### **Proto Definition** A new `proto/sglang/runtime/v1/sglang.proto` defines the full service contract: **SGLang-native RPCs** (typed proto messages): * `TextGenerate` / `Generate` — server-streaming text/token generation * `TextEmbed` / `Embed` — unary embedding * `Classify` — unary classification * `Tokenize` / `Detokenize` — local tokenization (no inference) * `HealthCheck`, `GetModelInfo`, `GetServerInfo`, `ListModels`, `GetLoad` * `Abort`, `FlushCache`, `PauseGeneration`, `ContinueGeneration` **OpenAI-compatible RPCs** (JSON pass-through): * `ChatComplete` / `Complete` — server-streaming (SSE → gRPC stream) * `OpenAIEmbed`, `OpenAIClassify`, `Score`, `Rerank` — unary **Admin RPCs:** * `StartProfile` / `StopProfile`, `UpdateWeightsFromDisk` ### **Directory Layout** ``` proto/sglang/runtime/v1/ sglang.proto # Service + message definitions rust/sglang-grpc/ Cargo.toml # Rust dependencies (tonic, pyo3, tokenizers, etc.) build.rs # tonic-build proto compilation pyproject.toml # Maturin build config src/ lib.rs # PyO3 module: start_server(), GrpcServerHandle server.rs # Tonic service impl (all RPCs) bridge.rs # PyBridge: channels, callbacks, PyO3 ↔ Python tokenizer.rs # Rust-native tokenizer (HF tokenizers crate) sglang_grpc/__init__.py # Python package re-export python/sglang/srt/entrypoints/ grpc_bridge.py # RuntimeHandle: async bridge to TokenizerManager http_server.py # Modified: starts gRPC alongside HTTP test/registered/core/ test_grpc_server.py # Integration tests (gRPC + HTTP coexistence) ``` ### **Key Design Decisions** 1. **Dual-protocol by default.** When the `sglang-grpc` package is installed, the gRPC server starts automatically alongside HTTP (on `--port + 10000`). No flag required. Opt out with `--disable-grpc`. If the package is not installed, the server runs HTTP-only with an info-level log message. 2. **Callback-based response channel.** Rust creates a per-request `crossbeam::bounded(64)` channel, passes a PyO3 callback object to Python. Python's async generators invoke the callback to push chunks into the channel. Rust's Tokio tasks drain chunks via `spawn_blocking(|| receiver.recv())`. The GIL is held by the Python thread only during the brief callback invocation — the Rust/Tokio side never acquires the GIL to read responses. 3. **Consolidated dict submission.** Instead of per-field PyO3 kwargs, Rust builds a `HashMap` in Rust, converts it to a Python dict in a single `Python::with_gil` block, and passes it to `GenerateReqInput(**dict)`. Dict construction, callback creation, and the `submit_request` call all happen within one GIL acquisition rather than requiring separate GIL round-trips per field. 4. **Rust-native tokenization with Python fallback.** The `tokenizers` crate (same Rust library underlying Python's `tokenizers` package) is loaded at startup from `tokenizer.json`. Tokenize/Detokenize RPCs execute entirely in Rust with zero GIL. Falls back to Python if `tokenizer.json` is unavailable. 5. **OpenAI pass-through.** OpenAI-compat RPCs send raw JSON bytes to Python, which handles Pydantic parsing and template application. This preserves full compatibility without duplicating the complex OpenAI serving logic in Rust. 6. **ZMQ as Phase 4 transport with shared memory as a future option**. Phase 4 uses ZMQ IPC for Rust→Scheduler communication. ZMQ adds ~5-10μs per request (kernel syscall + msgpack ser/de) vs ~200ns for shared memory ring buffers. This overhead is negligible against inference latency (10-100ms) and ZMQ is already in the codebase with native Python bindings. The transport layer is abstracted behind a serialization boundary — both sides produce/consume [u8] byte slices. Swapping to shared memory later requires only replacing the transport implementation, not the serialization format or request lifecycle. We start with ZMQ because: (a) Scheduler already speaks it, (b) it has built-in backpressure (HWM), observability, and error recovery, (c) the Python side needs zero new dependencies. Shared memory would require a custom ring buffer, a Python polling loop or ctypes wrapper, manual backpressure, and crash-safety logic for corrupted shared regions. This is a two-way door — if benchmarks show ZMQ syscall overhead matters at scale, we swap the transport without touching the protocol. --- ## **Plan (5 Phases)** ### **Phase 1: Rust-Native Tokenization \\+ Consolidated Python Bridge** **Goal:** Ship a working gRPC server that runs alongside HTTP. Eliminate GIL for Tokenize/Detokenize RPCs. Consolidate per-field PyO3 calls into single-dict submission. **Risk: Low.** **What's implemented (https://github.com/sgl-project/sglang/compare/main...alexnails:sglang:alexnails/grpc):** | Task | Status (PR) | Files | | :---- | :---- | :---- | | Define proto service contract (`sglang.proto`) | Done | `proto/sglang/runtime/v1/sglang.proto` | | Rust crate scaffold (Cargo.toml, build.rs, pyproject.toml, Maturin config) | Done | `rust/sglang-grpc/` | | Tonic gRPC server with all RPCs implemented | Done | `rust/sglang-grpc/src/server.rs` | | PyBridge with crossbeam channels \\+ callback pattern | Done | `rust/sglang-grpc/src/bridge.rs` | | Rust-native tokenizer with Python fallback | Done | `rust/sglang-grpc/src/tokenizer.rs` | | PyO3 module entry point (`start_server`, `GrpcServerHandle`) | Done | `rust/sglang-grpc/src/lib.rs` | | Python `RuntimeHandle` (async bridge to TokenizerManager) | Done | `python/sglang/srt/entrypoints/grpc_bridge.py` | | HTTP server integration (auto-start gRPC alongside HTTP) | Done | `python/sglang/srt/entrypoints/http_server.py` | | Server args (`--grpc-port`, `--disable-grpc`, `--grpc-mode`deprecation) | Done | `python/sglang/srt/server_args.py` | | Integration tests (25 test methods across 2 test classes, gRPC \\+ HTTP coexistence) | Done | `test/registered/core/test_grpc_server.py` | | Optional dependency in pyproject.toml | Done | `python/pyproject.toml` | **Remaining Phase 1 tasks:** | Task | Status (PR) | Description | | :---- | :---- | :---- | | CI wheel build | | Add `sglang-grpc` to the CI build matrix (maturin build \\+ publish) | | Proto lint / buf.yaml | | Add buf linting config for proto style enforcement | | gRPC reflection | | Enable gRPC server reflection for `grpcurl` / `grpc_cli` discoverability | | TLS / mTLS support | | Add `--grpc-tls-cert` / `--grpc-tls-key` flags for encrypted transport | | Connection keepalive tuning | | Expose Tonic keepalive settings via server args | | Python stub generation | | Generate and package `sglang_pb2.py` / `sglang_pb2_grpc.py` from the proto, or automate via build step | | Benchmarks | | Comparative latency/throughput benchmarks vs HTTP (tokenize, generate, streaming) | --- ### **Phase 2: Rust-Side Normalization and Validation** **Goal:** Move `normalize_batch_and_arguments()`, RID generation, field validation, and `SamplingParams` validation into Rust. Invalid requests never touch Python. **Risk: Medium.** | Task | Status (PR) | Description | | :---- | :---- | :---- | | Port `SamplingParams` validation to Rust | | Validate ranges (temperature \\>= 0, top\\_p in \\[0,1\\], max\\_new\\_tokens \\> 0, etc.) in Rust before converting to dict. Return `Status::INVALID_ARGUMENT` for bad params. | | Port RID generation to Rust | | Already partially done (uuid::Uuid::new\\_v4 in server.rs). Formalize as the canonical RID source for all gRPC requests. | | Port `normalize_batch_and_arguments()`to Rust | | Handle batch/single normalization for text\\[\\], input\\_ids\\[\\], and mixed inputs. Rust returns a Vec of individual request dicts. | | Field validation | | Validate required fields (text or input\\_ids present, not both empty), string length limits, token count limits against `context_len`. | | Error mapping | | Map Rust validation errors to gRPC status codes: `INVALID_ARGUMENT` for bad params, `RESOURCE_EXHAUSTED` for context overflow, `FAILED_PRECONDITION` for server not ready. | | Unit tests | | Rust-side unit tests for validation logic (no Python needed). | **GIL impact:** After Phase 2, the GIL is only acquired for: (1) the single `Python::with_gil` block that builds the PyDict, creates `GenerateReqInput(**dict)`, and calls into `tokenizer_manager.generate_request()`, and (2) Python-side tokenization and async dispatch within that coroutine. Invalid requests are rejected entirely in Rust before any GIL acquisition. --- ### **Phase 3: Rust Tokenization for Text-Input Generate** **Goal:** For the hot path (text-only, non-multimodal, non-LoRA generate requests), tokenize in Rust and call a Python fast-path that skips `_tokenize_one_request()`. GIL held only for `ReqState` \\+ ZMQ send. **Risk: Medium.** | Task | Status (PR) | Description | | :---- | :---- | :---- | | Detect fast-path eligibility | | In Rust, check: text input (not input\\_ids), no image/video/audio data, no LoRA adapter, no custom chat template. | | Rust tokenization for generate | | Use the Rust `tokenizers` crate to tokenize the prompt text, producing `input_ids` and `input_text` in Rust. | | Python fast-path entry point | | Add `TokenizerManager.submit_pretokenized_request(input_ids, input_text, sampling_params_dict, rid, ...)` that skips `_tokenize_one_request()` and goes directly to `_send_one_request()`. | | Pad/truncate handling | | Handle `context_len` truncation in Rust (matching Python's behavior). | | Special token handling | | Match Python's `add_special_tokens` behavior for the specific tokenizer (chat template BOS handling). | | Equivalence tests | | Exhaustive tests comparing Rust fast-path output vs Python path for diverse inputs (unicode, long sequences, edge cases). | **GIL impact:** After Phase 3, text-only generate requests acquire the GIL only for: (1) `ReqState` registration, (2) ZMQ send to scheduler. --- ### **Phase 4: Direct ZMQ to Scheduler** **Goal:** For pre-tokenized input_ids requests, Rust sends directly to the Scheduler via ZMQ using msgpack serialization. GIL held only for ReqState registration. Transport is abstracted so ZMQ can be swapped for shared memory without protocol changes. **Risk: High.** | Task | Status (PR) | Description | | :---- | :---- | :---- | | ZMQ socket in Rust | | Open a ZMQ PUSH socket in Rust connecting to the Scheduler's IPC endpoint. Use `zmq.rs` or `zeromq` Rust crate. | | Msgpack serialization | | Serialize `TokenizedGenerateReqInput` equivalent in Rust using `rmp-serde`, matching the exact format Python's Scheduler expects. | | Transport Abstraction | |Define SchedulerTransport trait in Rust (send/recv over [u8]). ZMQ as first implementation. Shared memory ring buffer as future drop-in replacement if benchmarks justify it. | | Dual-format recv loop in Scheduler | | Modify Scheduler's `recv_requests()` (currently uses `recv_pyobj` / pickle) to also accept msgpack-serialized requests from Rust. Discriminate via a 1-byte format tag prefix. | | ReqState registration | | Still requires GIL: create `ReqState` object in Python, register in `tokenizer_manager.rid_to_state`. Explore moving to a Rust-side registry with Python callback for cleanup. | | Backpressure | | Implement ZMQ HWM (high-water mark) and flow control to prevent Rust from overwhelming the Scheduler. | | Fallback | | If ZMQ send fails, fall back to the Python path transparently. | | Integration tests | | Test mixed workloads: some requests via Rust ZMQ, some via Python path, verify ordering and correctness. | **GIL impact:** After Phase 4, pre-tokenized requests acquire the GIL only for `ReqState` registration (\\~1 us). --- ### **Phase 5: Rust Response Loop (Full Python Bypass)** **Goal:** Rust receives responses directly from `DetokenizerManager` via a second ZMQ PULL socket. Zero GIL acquisition for the full request-response path. **Risk: Very high.** | Task | Status (PR) | Description | | :---- | :---- | :---- | | New IPC endpoint in DetokenizerManager | | Add a second ZMQ PUSH socket in `DetokenizerManager` that sends responses to a Rust-side PULL socket (in addition to the existing Python response path). | | Response routing | | DetokenizerManager checks whether a RID was registered by Rust or Python and routes accordingly. Requires a shared RID registry or a routing tag in the request. | | Rust-side response deserialization | | Deserialize DetokenizerManager's response format (msgpack or custom) in Rust. Extract text, meta\\_info, finish\\_reason. | | Rust-side ReqState management | | Move ReqState tracking entirely to Rust. Python only needs to be notified for cleanup (LoRA adapter release, etc.) via a batched cleanup callback. | | Streaming response assembly | | For streaming requests, Rust assembles incremental text deltas and pushes them directly to the gRPC stream via the crossbeam channel — no Python involved. | | Metrics / observability | | Expose Rust-side request latency, queue depth, and throughput metrics via Prometheus endpoint or gRPC health service. | | Graceful degradation | | If the Rust response loop encounters an error, fall back to the Python callback path for that request. | | Full end-to-end tests | | Test the complete Rust-only path: gRPC request → Rust tokenize → Rust ZMQ to Scheduler → Scheduler → DetokenizerManager → Rust ZMQ response → gRPC stream. | **GIL impact:** After Phase 5, the full request-response hot path acquires zero GIL. Python is only involved for cold-path operations (model loading, LoRA management, OpenAI template application, multimodal preprocessing). --- ## **Phase Summary** \"GIL-bound segments\" counts the distinct stages of a request's lifecycle where Python holds the GIL, limiting concurrency with other requests. Rust-side `Python::with_gil` blocks count as one segment each; Python-internal work (tokenization, async dispatch) counts when it contends with other requests. | Phase | GIL-Bound Segments (text generate hot path) | What Moves to Rust | Risk | | :---- | :---- | :---- | :---- | | **1** (current) | \\~3 (1 Rust→Python submit with dict build \\+ callback setup, 1 Python-side tokenize \\+ async dispatch, 1 per response callback) | Tokenize/Detokenize RPCs, proto decode, dict construction | Low | | **2** | \\~3 (same as Phase 1, but invalid requests rejected before any GIL) | Validation, normalization, RID gen | Medium | | **3** | \\~2 (1 Rust→Python submit with pre-tokenized input, 1 per response callback) | Text tokenization for generate hot path | Medium | | **4** | \\~1 (ReqState registration only) | ZMQ send to Scheduler | High | | **5** | **0** (hot path) | Response loop, ReqState management | Very High | --- # **Usage** ## **gRPC starts automatically alongside HTTP (default: port \\+ 10000\\)** ``` python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct ``` ## **Custom gRPC port** ``` python -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --grpc-port 50051 ``` ## **Disable gRPC (HTTP only)** ``` python -m \\ sglang.launch_server \\ --model-path meta-llama/Llama-3.1-8B-Instruct \\ --disable-grpc ``` ## **Client Example (Python \\+ grpcio)** Generate Python stubs from the proto definition first: ``` python -m grpc_tools.protoc \\ -I proto \\ --python_out=. --grpc_python_out=. \\ proto/sglang/runtime/v1/sglang.protop ``` Then use the generated stubs: ``` import grpc from sglang.runtime.v1 import sglang_pb2, sglang_pb2_grpc channel = grpc.insecure_channel(\"localhost:40000\") stub = sglang_pb2_grpc.SglangServiceStub(channel) ``` ## **TextGenerate (server-streaming)** ``` request = sglang_pb2.TextGenerateRequest( text=\"Explain quantum computing in one sentence.\", sampling_params=sglang_pb2.SamplingParams( temperature=0.7, max_new_tokens=64, ), stream=True, ) for response in stub.TextGenerate(request): print(response.text, end=\"\", flush=True) if response.finished: break ``` ## **Tokenize (unary, Rust-native, zero GIL)** ``` tok_resp = stub.Tokenize(sglang_pb2.TokenizeRequest(text=\"Hello, world!\")) print(f\"\\nTokens: {tok_resp.tokens}, Count: {tok_resp.count}\") ``` **Note**: Python stub generation (\\`sglang\\_pb2.py\\`, \\`sglang\\_pb2\\_grpc.py\\`) is not yet automated. The integration tests use raw protobuf wire encoding via \\`grpcio\\` directly. Adding pre-generated or build-time-generated stubs is a remaining Phase 1 task. ``` tok_resp = stub.Tokenize(sglang_pb2.TokenizeRequest(text=\"Hello, world!\")) print(f\"\\nTokens: {tok_resp.tokens}, Count: {tok_resp.count}\") ``` ## **Backward Compatibility** ``` tok_resp = stub.Tokenize(sglang_pb2.TokenizeRequest(text=\"Hello, world!\")) print(f\"\\nTokens: {tok_resp.tokens}, Count: {tok_resp.count}\") ``` * **HTTP API is unchanged.** All existing HTTP endpoints continue to work identically. * **`--grpc-mode` is deprecated with a \\`DeprecationWarning\\`**. It now sets \\`smg\\_grpc \\= True\\` internally, preserving the existing behavior of launching the standalone smg-grpc-servicer server instead of HTTP. * **Port allocation.** The gRPC port defaults to `--port + 10000` (e.g., 30000 → 40000), avoiding conflicts with existing deployments. ## **Related Work** * **smg-grpc-servicer** — Existing standalone gRPC server package (triggered via `--grpc-mode` / `--smg-grpc`). Runs in-process but *replaces* the HTTP server — operators must choose one protocol or the other. Using this does not allow us to have tight coupling for future performance wins across the stack. This RFC's native server complements it by running gRPC *alongside* HTTP and progressively moving GIL-bound work to Rust inline with great rust migration. Backward compat preserved via `--smg-grpc`. * **sglang-router** (Rust) — The existing Rust router crate demonstrates the PyO3/Maturin pattern in the SGLang ecosystem. This RFC follows the same build/packaging conventions. * **vLLM gRPC** — vLLM offers a gRPC server via `grpc_server.py` (Python grpcio). This RFC's Rust implementation provides lower per-request overhead by avoiding GIL acquisition for serialization and tokenization. * **Transport alternatives considered** Shared memory ring buffers offer lower per-request latency (~200ns vs ~5-10μs for ZMQ) by eliminating kernel syscalls, but require custom infrastructure on the Python side and provide no built-in backpressure, observability, or crash recovery. Apache Arrow Flight was considered for batch-oriented zero-copy data exchange but is unnecessarily complex for single-request submission. The ZMQ→shared memory path is preserved as a two-way door via a transport abstraction layer. ### Related resources _No response_",
"url": "https://github.com/sgl-project/sglang/issues/22558",
"createdAt": "2026-04-10T23:38:33Z",
"updatedAt": "2026-08-13T13:32:28Z",
"timestamp": "2026-08-13T13:32:28Z",
"metrics": {
"reactions": 7,
"comments": 8
},
"labels": [
"high priority",
"RFC"
],
"author": "alexnails",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:22889",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Free-Threaded Python (3.14t / nogil) Support for SGLang",
"text": "## Motivation CPython 3.13 introduced an experimental free-threaded build (PEP 703) that removes the Global Interpreter Lock (GIL). CPython 3.14t is the first release where this build is stable enough for production use, and the ecosystem of packages with free-threading wheels is growing rapidly. vLLM has demonstrated ([vllm-project/vllm#28762](https://github.com/vllm-project/vllm/issues/28762)) that a large-scale CUDA-accelerated serving framework can run under a free-threaded interpreter. SGLang shares a similar dependency footprint and architecture. Enabling free-threaded support would unlock true CPU-level parallelism in the Python layer — benefiting the tokenizer manager, scheduler, HTTP server, disaggregation controller, and other asyncio + thread-pool components that today serialize behind the GIL. This RFC tracks the dependency readiness, identifies the internal code changes needed, and proposes a phased plan to make `uv pip install sglang` work out of the box in a clean Python 3.14t environment on Linux x86-64 (CPU and CUDA). ## Background | Term | Meaning | |---|---| | **3.14t** | The free-threaded (no-GIL) CPython 3.14 build, identified by the `t` suffix in the ABI tag (`cp314t`). | | **cp314** | The regular (with-GIL) CPython 3.14 build. | | **Stable ABI** | A subset of the CPython C API that is ABI-compatible across Python versions (PEP 384). Extensions built against the stable ABI only need one wheel per platform, not one per Python minor version. | | **`Py_GIL_DISABLED`** | The compile-time macro set in free-threaded builds. C extensions must check this to use thread-safe reference counting and data structures. | ### Why 3.14t and not 3.13t? - CPython 3.14t itself is significantly more stable than 3.13t. - Key packages (cffi, aiohttp, etc.) support 3.14t but will not back-port to 3.13t. - PyTorch 2.10.0 (January 2026) will ship full cp314t wheels; PyTorch 2.9.0 already has \"preview\" wheels. ### Prerequisites Free-threaded Python support depends on SGLang first supporting regular Python 3.14 (with-GIL). This RFC assumes that prerequisite is met or being tracked separately. ## Dependency Audit The following tables categorize every SGLang dependency with compiled (C/C++/Rust/CUDA) code by its current free-threading readiness. Pure-Python packages are omitted — they work automatically. ### Core Dependencies (from `pyproject.toml` `dependencies`) | Package | Current Version | cp314t Wheels on PyPI | Builds from Source | Tracking Issue | Notes | |---|---|---|---|---|---| | `torch` | 2.9.1 | Preview (2.9.0); full in 2.10.0 | Yes | — | Core dependency. 2.10.0 (Jan 2026) is the target. | | `torchaudio` | 2.9.1 | Follows torch | Yes | — | | | `torchvision` | latest | Follows torch | Yes | — | | | `torchao` | 0.9.0 | Follows torch | TBD | — | | | `sglang-kernel` | 0.4.1 | ❌ No | ❌ Needs work | — | **SGLang-owned**. Uses stable ABI (cp310). Needs `Py_GIL_DISABLED` audit. See below. | | `flash-attn-4` | ≥4.0.0b4 | ❌ No | TBD | — | CUDA extension. | | `flashinfer_python` | 0.6.7.post2 | ❌ No | Resolved |https://github.com/flashinfer-ai/flashinfer/pull/1687| Uses stable ABI; fix was to not use limited API with free-threaded Python. | | `flashinfer_cubin` | 0.6.7.post2 | ❌ No | TBD | — | Binary CUDA kernels, may need rebuild only. | | `cuda-python` | 12.9 | TBD | TBD | — | NVIDIA-maintained. | | `xgrammar` | 0.1.32 | ✅ Yes (≥0.1.31) | ✅ Yes | [xgrammar#500](https://github.com/mlc-ai/xgrammar/issues/500) | Full support. | | `llguidance` | ≥0.7.11 | ✅ Yes (≥1.6.0) | ✅ Yes | [llguidance#256](https://github.com/guidance-ai/llguidance/issues/256) | Full support. | | `msgspec` | latest | ✅ Yes (≥0.20.0) | ✅ Yes | — | Full support. | | `outlines` | 0.1.11 | TBD (outlines-core) | ✅ Yes | [outlines-core#248](https://github.com/dottxt-ai/outlines-core/issues/248) | Depends on outlines-core. | | `sentencepiece` | latest | TBD | TBD | — | C++ extension. | | `tiktoken` | latest | TBD | TBD | — | Rust extension. | | `orjson` | latest | ✅ Yes | ✅ Yes | — | Rust extension, typically early adopter. | | `pyzmq` | ≥25.1.2 | ✅ Yes | ✅ Yes | — | | | `numpy` | latest | ✅ Yes | ✅ Yes | — | | | `scipy` | latest | ✅ Yes | ✅ Yes | — | | | `aiohttp` | latest | ✅ Yes | ✅ Yes | — | Supports 3.14t. | | `pybase64` | latest | TBD | TBD | — | C extension. | | `setproctitle` | latest | TBD | TBD | — | C extension. | | `nvidia-ml-py` | latest | Pure Python | N/A | — | | | `pillow` | latest | ✅ Yes | ✅ Yes | — | | | `pydantic` | latest | ✅ Yes | ✅ Yes | — | Rust-compiled core. | | `uvloop` | latest | TBD | TBD | — | C extension (libuv). Critical for asyncio perf. | | `soundfile` | 0.13.1 | TBD | TBD | — | CFFI-based. | | `compressed-tensors` | latest | TBD | TBD | — | | | `quack-kernels` | ≥0.3.0 | ❌ No | TBD | — | CUDA extension. | | `kernels` | latest | TBD | TBD | — | | | `smg-grpc-servicer` | ≥0.5.0 | TBD | TBD | — | gRPC bindings. | ### Optional Dependencies (diffusion, ray, tracing) | Package | cp314t Status | Notes | |---|---|---| | `ray` | ❌ No 3.14 support at all | Made optional in vLLM. SGLang already has it as optional (`[ray]`). Not blocking. | | `opencv-python-headless` | ❌ No | Diffusion extra only. Tracking: [opencv/opencv#27933](https://github.com/opencv/opencv/issues/27933), build fix PR: [opencv-python#1051](https://github.com/opencv/opencv-python/pull/1051). | | `xformers` | ✅ Yes (≥0.0.35) | Resolved by removing CPython C API dependency. | ### SGLang-Owned Compiled Components | Component | Language | Build System | Free-Threading Status | |---|---|---|---| | `sgl-kernel` | C++17 / CUDA | scikit-build-core + CMake | ❌ Uses stable ABI (`cp310`). Needs `Py_GIL_DISABLED` audit and potentially per-version builds. | | `cpp_radix_tree` (radix cache) | C++ | pybind11 (torch extension) | ❌ Needs thread-safety audit. | | `ngram_corpus_ffi` | C++ | pybind11 (torch extension) | ❌ Needs thread-safety audit. | | `hf3fs_utils` | C++ | pybind11 (torch extension) | ⚠️ Uses `py::gil_scoped_release`. Needs `Py_GIL_DISABLED` review. | | `sgl-model-gateway` (router) | Rust (PyO3) | maturin | ✅ PyO3 has free-threading support. | | `multimodal_gen` CUDA kernels | C++ / CUDA | setuptools | ❌ Needs audit. | ## Internal Code Audit ### GIL-Dependent Patterns The following patterns in SGLang's Python code rely on the GIL for correctness and must be reviewed: 1. **Explicit GIL assumptions.** `staging_handler.py` documents: *\"CPython GIL guarantees ordering\"* for flag visibility between threads. Under free-threading, this needs an explicit memory barrier or `threading.Event`. 2. **`py::gil_scoped_release` in C++ extensions.** `hf3fs_utils.cpp` uses `py::gil_scoped_release` to release the GIL during memcpy. Under free-threading, pybind11's `gil_scoped_release` is a no-op, but the surrounding data structures must be thread-safe without the GIL. 3. **Shared mutable state across threads.** The following areas use thread pools or daemon threads that share mutable Python objects: - `TokenizerManager`: `ThreadPoolExecutor` for tokenization with shared request queues - `CacheController`: background threads for cache management - `KVEvents`: daemon threads for ZMQ event distribution - `weight_utils.py`: multi-threaded safetensors loading with shared iterators 4. **Module-level mutable globals.** Several modules use module-level dicts/lists as registries (model registry, format registry, etc.). Under free-threading, concurrent imports or first-access initialization can race. 5. **asyncio + ThreadPoolExecutor interaction.** `run_coroutine_threadsafe` and `call_soon_threadsafe` are used throughout the disaggregation layer. These are asyncio-safe but the callbacks they schedule may access shared state without locks. ### Thread-Safety Classification | Risk Level | Pattern | Example Locations | Mitigation | |---|---|---|---| | **High** | GIL-ordering assumptions | `staging_handler.py:192` | Replace with `threading.Event` or `atomics` | | **High** | Shared mutable containers across threads | `weight_utils.py` (iterator sharing) | Add `threading.Lock` or use `queue.Queue` | | **Medium** | Module-level mutable registries | Model/format registries | Use `threading.Lock` for lazy init | | **Medium** | `concurrent.futures` with shared closures | `async_dynamic_batch_tokenizer.py` | Audit closure captures | | **Low** | `py::gil_scoped_release` in C++ | `hf3fs_utils.cpp` | Verify data is not shared with Python threads | | **Low** | Pure asyncio code (single-threaded event loop) | HTTP server, tokenizer manager main loop | Safe by design (single-threaded) | ## Proposed Plan ### Phase 0: Python 3.14 (with-GIL) Support **Goal:** `pip install sglang` works on regular Python 3.14. ### Phase 1: Dependency Readiness Tracking **Goal:** All dependencies are installable in a 3.14t environment. ### Phase 2: sgl-kernel Free-Threading Support **Goal:** `sglang-kernel` builds and passes tests under 3.14t. ### Phase 3: Internal C++ Extension Audit **Goal:** All SGLang-owned C++ extensions are thread-safe without the GIL. ### Phase 4: Python-Level Thread-Safety Fixes **Goal:** SGLang's Python code does not rely on GIL for correctness. ### Phase 5: CI and Release **Goal:** Free-threaded builds are tested and released. ### Phase 6: Performance Validation and Optimization **Goal:** Quantify the benefit of free-threading for SGLang workloads. ## References - [vLLM free-threaded Python tracking issue (vllm-project/vllm#18937)](https://github.com/vllm-project/vllm/issues/18937) - [PEP 703 – Making the Global Interpreter Lock Optional in CPython](https://peps.python.org/pep-0703/) - [CPython 3.14 free-threaded howto](https://docs.python.org/3.14/howto/free-threading-python.html) - [PyTorch free-threading support](https://github.com/pytorch/pytorch/issues/130249) ### Related resources _No response_",
"url": "https://github.com/sgl-project/sglang/issues/22889",
"createdAt": "2026-04-15T15:27:31Z",
"updatedAt": "2026-08-13T11:08:40Z",
"timestamp": "2026-08-13T11:08:40Z",
"metrics": {
"reactions": 12,
"comments": 10
},
"labels": [
"inactive"
],
"author": "liusy58",
"state": "open",
"assignees": [
"liusy58"
]
},
{
"id": "github:sgl-project/sglang:issue:23579",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "Session Reaper Mid-Decode Race Condition & /v1/completions Parameter Drop",
"text": "### Describe the bug There are two critical bugs related to the session reaper (`maybe_reap` -> `_close()`) and the OpenAI completion handler (`/v1/completions`), both affecting session tracking: 1. **Session Reaper Race Condition (Mid-decode Tear-down)** Non-streaming sessions with `timeout` are unconditionally torn down by the session reaper, even when a request is still actively decoding. The `_close()` method in `session_controller.py` only checks for unfinished requests when `session.streaming == True`, causing premature session deletion for all non-streaming sessions. If `--enable-streaming-session` is used, this deletes the active KV pool mid-flight and causes corruption/OOB memory accesses. 2. **`/v1/completions` Silently Drops `session_params`** The OpenAI-compatible handler (`serving_completions.py`) accepts `session_params` in `CompletionRequest`, but fails to pass it to `GenerateReqInput`. As a result, the session is silently dropped, and session-based contexts are ignored, forcing users to use the native `/generate` API instead. ### Reproduction The following script targets the mid-decode race condition using `/generate`. It opens a session with a 2.0s timeout and dispatches a generation task that takes ~3.8s to decode. The reaper fires mid-generation and wipes the session tracking. repro_mid_decode.py ```python import argparse, asyncio, time, httpx, sys async def open_session(client, base_url, session_id, timeout_s): resp = await client.post( f\"{base_url}/open_session\", json={\"session_id\": session_id, \"capacity_of_str_len\": 32768, \"timeout\": timeout_s}, timeout=10, ) if resp.status_code != 200: return False data = resp.json() if isinstance(data, str): return True if isinstance(data, dict): return data.get(\"success\", False) return False async def main(args): base_url = args.base_url.rstrip(\"/\") session_id = f\"repro_mid_decode_{int(time.time())}\" async with httpx.AsyncClient(timeout=120) as client: if not await open_session(client, base_url, session_id, timeout_s=2.0): sys.exit(1) print(f\"Session {session_id} opened (timeout=2.0s)\") print(\"Sending slow request (800 tokens) via /generate...\") start_t = time.time() payload = { \"text\": \"Write a very long essay about the history of artificial intelligence. Make it at least 800 words.\", \"sampling_params\": {\"temperature\": 0.0, \"max_new_tokens\": 800}, \"stream\": False, \"session_params\": {\"id\": session_id}, } resp = await client.post(f\"{base_url}/generate\", json=payload, timeout=120) print(f\"Request finished in {time.time() - start_t:.1f}s\") if resp.status_code == 200: print(\"SUCCESS: Generated tokens\") print(\"Sending quick follow-up to check if session survived...\") payload[\"text\"] = \"Continue.\" payload[\"sampling_params\"][\"max_new_tokens\"] = 10 resp = await client.post(f\"{base_url}/generate\", json=payload, timeout=10) if resp.status_code == 200: print(\"RESULT: Session SURVIVED\") else: print(f\"RESULT: Session DEAD (Bug present): {resp.text[:100]}\") if __name__ == \"__main__\": parser = argparse.ArgumentParser() parser.add_argument(\"--base-url\", default=\"http://localhost:30000\") asyncio.run(main(parser.parse_args())) ``` **Before Logs (Bug present)** ``` Session repro_mid_decode_1776969519 opened (timeout=2.0s) Sending slow request (800 tokens) via /generate... Request finished in 3.8s SUCCESS: Generated tokens Sending quick follow-up to check if session survived... RESULT: Session DEAD (Bug present): {\"error\":{\"message\":\"Invalid request: session id repro_mid_decode_1776969519 does not exist\"}} ``` ### Environment setup Tested natively via empirical logs on a GCP `g2-standard-32` using `Qwen/Qwen2.5-0.5B-Instruct`. Bug affects latest main branch and previous releases (`0.5.10` etc).",
"url": "https://github.com/sgl-project/sglang/issues/23579",
"createdAt": "2026-04-23T18:43:56Z",
"updatedAt": "2026-08-13T00:23:33Z",
"timestamp": "2026-08-13T00:23:33Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"inactive"
],
"author": "glaziermag",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:23602",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "DeepSeek V4 Roadmap",
"text": "## Links * Cookbook: https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4 * Day0 PR: https://github.com/sgl-project/sglang/pull/23600 * Rebase PR: https://github.com/sgl-project/sglang/pull/23882 ## Roadmap - [x] Finish W4A16 support on Hopper @Fridge003 - [x] #23686 Marlin implementation - [x] #24816 Flashinfer mxfp4 implementation - [x] [#23980](https://github.com/sgl-project/sglang/pull/23980) Add Cookbook instructions - [x] Support DeepGemm warmup #23756 - [x] Support non-standard extra chat template feature @JustinTong0323 https://github.com/sgl-project/sglang/pull/23915 - [x] Support pipeline parallel @ShangmingCai @whybeyoung - #24704 - [x] Optimize MegaMoE (implemented, to be tested) @DarkSharpness - [x] HiCache Support DeepSeekV4 https://github.com/sgl-project/sglang/pull/24691 - [ ] Further integrate various kernels - [x] Integrate FP4 indexer @liz-badada #26209 - [x] Integrate prefill kernel of FlashMLA & integrate faster prefill attention kernels #25418 - [x] Integrate ragged (no prefix hit) indexer @YAMY1234 #29619 - [x] Apply latest DeepGemm #24268 - [ ] Check whether DeepEP v2 helps #29402 #29525 - [ ] Check whether TileKernel helps https://github.com/deepseek-ai/TileKernels - [x] MegaMoE on Sm90 https://github.com/sgl-project/DeepGEMM/pull/36 #29016 - [ ] Further optimize & aggressively fuse various kernels - [x] w4a4 MegaMoE #25052 - [x] Breakable Cuda Graph #25195 - [ ] Bf16 Kv cache support #26915 - [ ] Unify HiSparse pool internals — extract a shared mixin between `HiSparseC4DevicePool` (DSv4 c4) and `HiSparseNSATokenToKVPool` (NSA) so both share the `full_to_hisparse_device_index_mapping` + translate_loc API and only override the compression step (names aligned in #23882). - [x] EPLB @xutizhou #25285 - [x] NVFP4 checkpoint @trevor-m #25598 - [x] Support DSpark Spec Decoding #29488 Depdendencies - [x] Upgrade tilelang's version. #29554 - [x] Unpin tvm-ffi's version. #29554 - [x] Fix sgl_kernel.flash_mla for V4 usage @zcnrex #26132 #26499 Code cleanup - [ ] Clarify NSA / DSA / V3.2 / V4 boundaries across modules — many components named `deepseek_v4_*` are actually generic NSA / sparse-attention infrastructure that V3.2 (and future sparse-attention models) could reuse. Rename and generalize where appropriate, e.g. memory pool indexer, index buffer accessors, chat encoding spec, and reasoning-parser variants. - [ ] `mem_cache/` pool / allocator module restructure — see #24335. Untangles DSv4 / HiSparse cross-pollination at the pool/allocator level, separately from the NSA/V4 naming clarification above. - [ ] `layers/attention/` subpackage restructure — current family-bundled layout (`nsa/`, `compression/`, ...) doesn't fit cross-family sharing between NSA and DSA, and some DSv4-only files (`_v4` suffix) currently live under `nsa/`. Needs a topology that accommodates shared primitives and correctly homes family-specific files. - [ ] Replace `DeepseekV4AttnBackend` `*BackendMixin` inheritance with module composition (`backend.compressor_ops` / `backend.c4_indexer_ops`) for consistency with sibling backends. Other SM versions (community may implement them if needed) - [x] SM120 #24692 - [x] SM121 - [ ] SM89 #28620 - [ ] SM80 ## Slack Join [#deepseek-v4] channel at https://slack.sglang.io",
"url": "https://github.com/sgl-project/sglang/issues/23602",
"createdAt": "2026-04-24T03:22:31Z",
"updatedAt": "2026-08-13T10:38:39Z",
"timestamp": "2026-08-13T10:38:39Z",
"metrics": {
"reactions": 29,
"comments": 74
},
"labels": [
"high priority",
"deepseek",
"roadmap"
],
"author": "fzyzcjy",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:26340",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "CUDA Coredump Tracker",
"text": "Auto-collected CUDA coredump events from pr-test.yml. One comment per (run, job, attempt). Producer: `.github/actions/upload-cuda-coredumps/action.yml`.",
"url": "https://github.com/sgl-project/sglang/issues/26340",
"createdAt": "2026-05-26T03:03:57Z",
"updatedAt": "2026-08-13T15:21:17Z",
"timestamp": "2026-08-13T15:21:17Z",
"metrics": {
"reactions": 0,
"comments": 233
},
"labels": [],
"author": "hnyls2002",
"state": "open",
"assignees": [
"hnyls2002"
]
},
{
"id": "github:sgl-project/sglang:issue:27521",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[AMD] PR CI new test cases to cover",
"text": "Here is to track new PR CI test cases that are needed. List of PRs: https://github.com/sgl-project/sglang/pull/18182 https://github.com/sgl-project/sglang/pull/22786 - To enable SGLANG_USE_ROCM_FLYDSL=1 in AMD CI end2end test https://github.com/sgl-project/sglang/pull/22985 https://github.com/sgl-project/sglang/pull/23927 https://github.com/sgl-project/sglang/pull/24651 https://github.com/sgl-project/sglang/pull/26922 - `TestMoriTransferEngineHybridMambaE2E.test_generate_smoke_hybrid_mamba` in CI `stage stage-b-test-large-8-gpu-mi35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)` is caused by the model `Qwen/Qwen3-Next-80B-A3B-Instruct` attempting to register approximately 2.47 GB of RDMA memory, which exceeds the 2 GB limitation imposed by the older AINIC firmware. This issue has already been resolved in the newer AINIC firmware release (`a-77`) and does not reproduce there. As a temporary workaround, we disable this test. https://github.com/sgl-project/sglang/pull/27063 https://github.com/sgl-project/sglang/pull/27163 - SGLANG_USE_AITER_AG was disabled in this PR, for [test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py](https://github.com/sgl-project/sglang/pull/27163/changes#diff-c99872f08e7ecf62aac2051b008928c13156da3c867f72b22c73d30c0749b3d2) , should be re-enabled once the issue is resolved. https://github.com/sgl-project/sglang/pull/25094 - To enable SGLANG_MORI_USE_XGMI=1 in AMD CI end2end test https://github.com/sgl-project/sglang/pull/27380 https://github.com/sgl-project/sglang/pull/27529 https://github.com/sgl-project/sglang/pull/27656 https://github.com/sgl-project/sglang/pull/27935",
"url": "https://github.com/sgl-project/sglang/issues/27521",
"createdAt": "2026-06-08T02:44:07Z",
"updatedAt": "2026-08-13T00:23:37Z",
"timestamp": "2026-08-13T00:23:37Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"inactive",
"amd"
],
"author": "HaiShaw",
"state": "closed",
"assignees": [
"bingxche",
"yctseng0211"
]
},
{
"id": "github:sgl-project/sglang:issue:27574",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Agentic Inference] Programmatic KV Cache for Agentic Workloads",
"text": "# RFC: Programmatic KV Cache for Agentic Workloads Authors: @ishandhanani, @hzh0425 ## Introduction > [!NOTE] > Note: For this RFC, we will define “router” as an orchestrator on top of multiple engine units. Agent workloads make the value of a KV block predictable from above the engine, but that value is invisible to request-local LRU. We propose exposing a narrow, router-initiated hint surface so an external router can pass cache intent to SGLang without making invasive changes to the engine scheduler and cache manager. In this way, SGLang keeps ownership of scheduling and memory and is free to clip, defer, or reject any hint. Today the engine can identify cached content and local residency, but it does not know why a trajectory will reuse a prefix or how long an external tool/subagent gap will last. Session membership is useful metadata for lifecycle-scoped policies, but it is not the only way to address KV: request-scoped hints can name an exact token prefix after tokenization. The session work in [#27058](https://github.com/sgl-project/sglang/pull/27058) remains complementary. It lets the engine associate radix ownership with a session when the policy is genuinely session-scoped. The first L3 Pin POC described below is deliberately token-relative and does not require session identity. With the work that is being done in PRs like 1. https://github.com/sgl-project/sglang/pull/20535 - allow for HiCache to become buffer mode and allow L3 cache to take more space on nodes. Buffer-mode isn't strictly required but the idea of giving L3 more memory is useful here 2. https://github.com/kvcache-ai/Mooncake/pull/2214 - allow for MC to start sending KV events which indexers can use We can leverage L3 cache as the shared retention and movement substrate. The router expresses intent; SGLang resolves that intent against exact radix content and remains responsible for allocation, transfer completion, and visibility; the storage backend owns stored-object lifecycle. ```mermaid flowchart LR R[\"Router / orchestrator policy\"] --> K[\"Provider-neutral KV hint\"] K --> S[\"SGLang scheduler + HiCache\"] S <--> G[\"L1: GPU KV\"] S <--> H[\"L2: host KV\"] S <--> M[\"L3: shared storage / Mooncake\"] M -. retain now; share, prefetch, demote later .-> S ``` The hint surface is broader than any one backend operation. The concrete work so far validates one mapping—bounded Pin intent to a Mooncake L3 lease—without making that mechanism the whole RFC. ## High level roadmap (subject to change) #### Phase 1 - session based kv cache 1. https://github.com/sgl-project/sglang/pull/29436 - enable a top level `session_id` so that we do not have to hijack `session_params.session_id` which is used for the `StreamingSession` cache 2. https://github.com/sgl-project/sglang/pull/27058 - enable `session_id` to be tagged into kv blocks allowing us to refer to kv at the session level 3. TODO - enable `SessionRadixCache` to be implemented at the HiCache layer #### Phase 2 - design router initiated hint API 1. POC implemented / PRs in progress - carry a provider-neutral `KvHintEnvelope` from an external orchestrator through request preprocessing and into the SGLang engine 2. POC implemented / PRs in progress - map bounded Pin intent onto exact committed HiCache page groups and a Mooncake L3 TTL lease 3. TODO - productionize admission, telemetry, expiry, namespace, and version-skew behavior for L3 Pin 4. TODO - measure request-time L3 restoration before adding proactive Prefetch or Demote execution APIs ## Problem Statement The orchestrator knows the structure request-local policy cannot: which sessions are live, which token ranges are shared prefixes vs unique tails, which tool gap is 10ms vs 10min, when a subagent opened and closed. The engine sees a block hash and a refcount. Current shape: ```text request arrives -> router chooses target worker via KV overlap/load -> target worker checks local cache hit: reuse miss: recompute or apply local offload policy ``` This is insufficient when: - another worker already holds the prefix; - the workload knows a request will resume soon; - a session has ended and its KV should be demoted/freed; - the orchestrator wants to protect high-value KV; - or local policy cannot tell a short tool gap from a long one. The missing abstraction is a precise, observable surface where the orchestrator biases the cache manager without owning scheduler or memory internals. Letting an external system manipulate cache internals directly is the wrong design: it is brittle, it duplicates scheduler policy outside the engine, and it forces a refactor of the scheduler/cache-manager boundary. Hints keep ownership inside SGLang and let the orchestrator soft-influence behavior at request and lifecycle boundaries. --- ## Design Principles 1. **Orchestrator owns policy; engine executes.** The agent-graph / workflow intelligence lives outside. The engine understands priority, TTL, session membership, tier - nothing about why. Keeps things simple 2. **Zero overhead when unused.** Un-hinted workloads behave exactly like today. 3. **Hints are soft, bounded, and safe to reject.** The engine may accept, clip, defer, or ignore. Every hint is observable. Nothing a client says can pin memory unboundedly or deadlock the scheduler. 4. **Router-initiated by default.** Workloads can still emit intent, but the router is where workload context merges with global KV placement, worker load, health, and admission. In production environments, the router has: a global KV index from events, built-in HA/fault-tolerance, existing overlap/load routing + admission control, and (with the harness<->orchestrator work) trajectory awareness, not just request awareness. --- ## Hint Taxonomy (conceptual) ### Share Reuse a prefix that already lives on another worker or a shared tier: route a continuation to a less-loaded worker but pull the prefix from the old one; share a common prefix across sibling subagents; warm a scale-up worker. It is the existence proof for the whole model: the machinery to move KV natively between workers is what every other hint also needs. We can leverage L3 cache for this. ### Prefetch Move KV into a hotter tier *before* it is needed: warm GPU for a likely next-turn prefix; pull shared KV into a freshly selected worker; reload main-agent KV during a subagent's close. This is easily enabled after we get the APIs that the `Share` hint gives us. ```mermaid sequenceDiagram participant R as Router participant W as Worker participant L as Mooncake / L3 participant H as HiCache host / L2 participant G as GPU / L1 R->>W: prefetch hint (exact prefix likely needed soon) W->>L: locate committed KV in shared L3 L-->>H: stage KV into worker host cache H-->>G: promote only after transfer completion Note over W,G: SGLang owns allocation and radix visibility R->>W: continuation request arrives W->>G: cache hit, skip prefill W-->>R: tokens ``` ### Demote Move KV to a colder tier instead of dropping it: long external tool call, paused trajectory, low-priority-but-reusable subagent state, memory pressure where recompute is expensive. Demote moves together with Prefetch/Onboard - the KV offloaded during a pause is the KV warmed back before the continuation resumes. This is easily enabled after we get the APIs that the `Share` hint gives us. ```mermaid sequenceDiagram participant R as Router participant W as Worker participant G as GPU / L1 participant H as HiCache host / L2 participant L as Mooncake / L3 R->>W: demote hint (long tool gap, pause) W->>H: ensure the exact prefix is staged H->>L: ensure the prefix is published to shared L3 Note over W,L: Publication and local reclamation are separate operations W->>G: allow local pages to become normally evictable R->>W: continuation resumes (paired prefetch) L-->>H: restore retained KV H-->>G: promote after completion W-->>R: tokens ``` ### Pin Keep a high-value prefix protected from ordinary eviction for a **bounded** TTL. Examples include expensive retrieved context, shared planner state, and tool/subagent gaps where recomputation dominates latency. A Pin hint should not imply permanent HBM residency: the engine may realize it at a colder tier while keeping L1/L2 normally evictable. The first implementation POC realizes Pin at Mooncake L3: ```mermaid sequenceDiagram participant C as Client / agent harness participant R as Dynamo router participant S as SGLang + HiCache participant M as Mooncake L3 C->>R: OpenAI prompt_cache_retention or Anthropic cache_control R->>S: Request + normalized prefix_tokens / ttl_seconds S->>S: Resolve exact committed, page-aligned prefix S->>M: Queue retain_groups(group_ids, ttl_ms) M->>M: Extend current members and future-member deadline Note over S,M: GPU and host copies remain normally evictable C->>R: Later continuation R->>S: Route normally S->>M: Existing request-time HiCache L3 lookup M-->>S: Retained KV pages ``` - [Mooncake#2835](https://github.com/kvcache-ai/Mooncake/pull/2835) adds TTL-bounded `retain_groups`, automatic expiry, future-member inheritance, and bounded group admission. - Dynamo normalizes provider-native controls into a small `KvHintEnvelope`; SGLang resolves token offsets to exact HiCache page groups and queues the Mooncake metadata operation on its existing storage worker. - A fresh MiniMax M2.7 H100 A/B evicted 24.18 GB / 95,232 keys under pressure. The unretained cold-worker probe recomputed 10,032 tokens; the one-hour retained probe restored 10,016 tokens from L3 and computed 16. - This is \"low hanging fruit\" and can be used to implement Anthropic's `cache_control` API and OpenAI's `prompt_cache` APIs The implementation uses “retention” because that is the provider API language. In this taxonomy it is a bounded L3 Pin. The Retain concept below remains priority-biased eviction rather than guaranteed lease protection. ### Retain Bias eviction order rather than protect a prefix with a lease: some token ranges are worth more than others when eviction is unavoidable. The orchestrator attaches a relative priority (optionally with a duration) to a token range, and under memory pressure the engine evicts low-priority KV first. We have existing mechanisms to do this with the `priority` radix-cache strategy and can augment them with duration semantics similar to TensorRT-LLM's `TokenRetentionConfig`. --- ## Non-Goals - Not a public user-facing cache API; the producer is the router/orchestrator. - Not direct orchestrator manipulation of cache-manager internals; hints only bias. - Not a replacement for local prefix matching or for sglang's LRU - it augments and reorders them. - Not a guarantee the engine obeys any hint; accept/clip/defer/reject is always allowed. - Not a commitment to freeze the current POC schema. `KvHintEnvelope { retention: [{prefix_tokens, ttl_seconds}] }` is the first exercised shape, not the final taxonomy. --- ## Relationship to existing SGLang work This RFC is complementary to two in-flight SGLang efforts; it attacks the same agentic-KV problem from a different angle (a router above many engines) rather than from inside a single engine. - **[#24656](https://github.com/sgl-project/sglang/issues/24656) - Agent-Aware KV Cache (Phase 1)** is the *in-engine, client-driven* angle: an optional `agent_hints` field on the OpenAI request flows into one engine, annotates radix nodes, and feeds an experimental `agent_aware` eviction policy. This RFC is the *router-initiated, multi-engine* angle on the same intent, and the two compose: #24656's `agent_hints` is the natural request-scoped envelope, and its `cache_ttl_ms` / `reuse_hint` are exactly our Pin / Retain hints. This RFC then adds the out-of-band control path and the cross-worker KV *movement* (Share / Prefetch / Demote) that #24656 explicitly defers (HiCache/storage metadata inheritance, cross-process coordination). - **[#21846](https://github.com/sgl-project/sglang/issues/21846) - Distributed KVCache System for Agentic Workload** is the *mechanism / substrate*: HiCache tiering, PD incremental transfer, a storage prefetch interface, and hybrid-model support. This RFC is the *policy layer* that rides those rails - Share uses the worker-to-worker HiCache movement, Prefetch maps onto the roadmap's storage-prefetch interface, and Demote onto multi-tier offload. #21846 builds the plumbing; the router decides when to use it. Its own \"Agent/Rollout KVCache Management\" item already points back to #24656, so all three are one arc. --- ## References External RFCs / APIs, research, and our prior work External RFCs / APIs: - vLLM #37003 - Context-Aware KV-Cache Retention API (Prioritized Evictions): (impl PR #38514) - vLLM #37168 - Active Coordination and Two-Zone Scheduling for Long-Running Agents: (impl vllm-ascend#6722) - vLLM agentic-api # 18 - Session-aware KV cache management - vLLM #39305 - Selective KV Cache offload: (impl PR #39983) - vLLM #38260 - Multi-tier KV offloading via the offloading connector - TensorRT-LLM `KvCacheRetentionConfig` / `TokenRangeRetentionConfig` (token_start/token_end/priority 0-100/duration_ms; default 35; decode_retention_priority; secondaryOffloadMinPriority) Research: - KVCache in the Wild (Alibaba traces): https://arxiv.org/abs/2506.02634 - Continuum (KV cache TTL for multi-turn agents): https://arxiv.org/abs/2511.02230 - Tail-Optimized Caching for LLM Inference: https://arxiv.org/abs/2510.15152 - KVFlow (workflow-aware prefix caching): https://arxiv.org/abs/2507.07400 - MARCONI (prefix caching for hybrid LLMs): https://arxiv.org/abs/2411.19379 Our prior work (sglang / Mooncake): - #24656 (agent-aware KV phase 1 / API feedback), #21846 (distributed KV roadmap), #27058 (radix-native sessions), #27024 / #27025 (streaming-session deadlock + bound), #22273 / #21875 (streaming-session leak fixes), #18941 (TTL prefix pinning), #21045 (priority retention duration), [Mooncake#2214](https://github.com/kvcache-ai/Mooncake/pull/2214) (group semantics), [Mooncake#2835](https://github.com/kvcache-ai/Mooncake/pull/2835) (TTL-bounded group retention POC). Our prior work (dynamo): - [#7665](https://github.com/ai-dynamo/dynamo/pull/7665) / [#7377](https://github.com/ai-dynamo/dynamo/pull/7377) / [#7384](https://github.com/ai-dynamo/dynamo/pull/7384) (session_control + ephemeral KV routing), pi-dynamo-provider#4 (per-subagent sessions), [#6213](https://github.com/ai-dynamo/dynamo/pull/6213) / [#6571](https://github.com/ai-dynamo/dynamo/pull/6571) (Anthropic-style cache_control), [#8789](https://github.com/ai-dynamo/dynamo/pull/8789) / [#9140](https://github.com/ai-dynamo/dynamo/pull/9140) (agent_context / ATIF), [#9448](https://github.com/ai-dynamo/dynamo/pull/9448) (thunderagent_router program scheduler). ",
"url": "https://github.com/sgl-project/sglang/issues/27574",
"createdAt": "2026-06-08T14:37:15Z",
"updatedAt": "2026-08-13T17:22:00Z",
"timestamp": "2026-08-13T17:22:00Z",
"metrics": {
"reactions": 15,
"comments": 19
},
"labels": [
"high priority"
],
"author": "ishandhanani",
"state": "open",
"assignees": [
"xiezhq-hermann",
"hzh0425",
"ishandhanani"
]
},
{
"id": "github:sgl-project/sglang:issue:27937",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Failure Tracker] PR Test (AMD)",
"text": " # PR Test (AMD) — Failure Tracker _Maintained by amd-bot · workflow `pr-test-amd.yml` · last updated 2026-08-13 11:25 UTC · 0 failing in latest scan · 214 tracked total_ This is a **long-lived failure tracker** for `pr-test-amd.yml`: once a test failure is recorded its **Broken since** date is preserved indefinitely (even months), so you can always see how long something has been red. It complements the bot's [daily report](https://github.com/bingxche/sglang-ci-bot/issues/165) (a rolling recent-days view) — click **Detail** on any row to jump to the full analysis (stack trace, suspected commits, in-flight fixes) in that day's daily report. - **Duration** = observed broken span (`Last seen − Broken since`), not time-since-first-seen. - **State** 🔴 = seen failing in the latest scan · ⚪ = not seen in the latest scan (may be fixed, or simply did not re-run — check **Last seen**). ## 🔴 Currently failing (latest scan) _No `pr-test-amd.yml` failures observed in the latest scan._ ⚪ Quiet / no longer observed failing (214) · click to expand | Broken since | Last seen | Duration | State | Test File | Test Function | Cluster | Error | Job | Detail | |---|---|---|---|---|---|---|---|---|---| | 2026-06-26 | 2026-06-26 | 0d | ⚪ quiet 48d | `python/sglang/multimodal_gen/test/server/test_server_1_gpu.py` | `TestDiffusionServerOneGpu.test_diffusion_generation[ltx_2_3_hq_pipeline]` | Diffusion server startup: cosmos timeout -> strict-ports port-5555 cascade | RuntimeError: Server exited early (code 1) -> Scheduler port 5555 unavailable (strict-ports) | [job](https://github.com/sgl-project/sglang/actions/runs/28192951457/job/83512127256) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/116#job-83512127256) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_token_oracle.py` | `TestTokenOracleManager.test_fill_expected_inputs_expands_draft_extend_generalized_req_ids_per_token` | `AssertionError: token_oracle DRAFT_EXTEND_V2 tokens != input_ids` (unit-test l… | `AssertionError: False is not true` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81441818801) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81441818801) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/spec/eagle/test_eagle_dp_attention.py` | `TestEAGLE3EngineDPAttention.setUpClass` | EAGLE draft-extend `spec_info` missing `kv_indptr` on triton/DP-attention warmup | `AttributeError: 'EagleDraftExtendInput' object has no attribute 'kv_indptr'` (server crashed → exit -9 / set… | [job](https://github.com/sgl-project/sglang/actions/runs/27517929362/job/81330028885) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81330028885) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/model_loading/test_load_weights_from_remote_instance.py` | `TestLoadWeightsFromRemoteInstance.test_load_weights_from_remote_instance` | Remote-instance NCCL weight-load hang — EADDRINUSE on port 60000 (job-level 45-… | Hang → `##[error]The action 'Run test' has timed out after 45 minutes` | [job](https://github.com/sgl-project/sglang/actions/runs/27517929362/job/81330028726) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81330028726) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/run_suite.py` | `N/A` | ImportError: shuffle_scale missing from aiter (stale CI image / aiter version s… | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' (import-time crash, 0 tests ran) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417479) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417479) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/perf/test_bench_serving_2gpu.py` | `test_moe_offline_throughput_default` | ImportError: `shuffle_scale` missing from `aiter.ops.shuffle` (stale CI Docker… | `Exception: Server process exited with code 1` (server crashed on `ImportError: cannot import name 'shuffle_s… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417748) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417748) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/quant/test_int4fp8_moe.py` | `setUpClass` | ImportError: `shuffle_scale` missing from `aiter.ops.shuffle` (stale CI Docker… | `Exception: Server process exited with code 1` (same `ImportError: shuffle_scale`) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417748) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417748) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_moriep_small.py` | `TestEPLBMoriStat.setUpClass` | ImportError: aiter `shuffle_scale` missing (stale aiter in CI image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` → server exits code 1 → `Exception… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418248) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418248) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/moe/test_torch_compile_moe.py` | `TestTorchCompileMoe.setUpClass` | ImportError: aiter `shuffle_scale` missing (stale CI image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server process exited code 1) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418033) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418033) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_per_token_group_quant_8bit.py` | `test_per_token_group_quant_with_column_major` (ALL 747 non-skipped param cases)` | JIT per-token-group fp8 quant — ROCm kernel never ported (all `column_major` ca… | Numerical mismatch — sglang JIT fp8 kernel output ≠ Triton reference (custom runner prints `FAILED` per case;… | [job](https://github.com/sgl-project/sglang/actions/runs/27517929362/job/81330028461) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81330028461) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `(step-level)` | `Run JIT kernel unit tests` | JIT per-token-group fp8 quant — ROCm kernel never ported (all `column_major` ca… | `##[error]The action 'Run JIT kernel unit tests' has timed out after 10 minutes` | [job](https://github.com/sgl-project/sglang/actions/runs/27517929362/job/81330028461) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81330028461) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/server/test_server_2_gpu.py` | `TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]` (setup error)` | GatedRepoError 403 — gated HF model `ideogram-ai/ideogram-4-fp8` not accessible… | `huggingface_hub.errors.GatedRepoError: 403 Client Error — Access to model ideogram-ai/ideogram-4-fp8 is rest… | [job](https://github.com/sgl-project/sglang/actions/runs/27517929362/job/81330028501) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81330028501) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `N/A — suite runner `run_suite.py` (`--suite 2-gpu`)` | `N/A (import-time crash, no test collected)` | `ImportError: shuffle_scale missing from aiter (stale CI image)` | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417479) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417479) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/run_suite.py` | `N/A (import-time failure, no test collected)` | ImportError: `shuffle_scale` missing from stale CI-image aiter | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417399) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417399) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `N/A — collection-time import error (no test ran)` | `run_suite.py` module import` | ImportError: `shuffle_scale` missing from CI-image aiter (stale aiter version s… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417419) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417419) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/mock_model/test_self_unit_install.py` | `(import-time, no function reached)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/mock_model/test_self_unit_canary_mock_wiring.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/mock_model/test_self_unit_sampler_hookpoint.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_plan_input.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_endpoint.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_runner_per_forward.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_perturb.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_runner_swa_divergence.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_runner_sweep.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_sweep_plan_builder.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_runner_health.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_token_oracle.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_req_to_expected_token_ids_manager.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_pool_patcher.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/kv_canary/test_self_unit_radix_walker.py` | `(import-time)` | ImportError: aiter `shuffle_scale` missing (stale CI image aiter) | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/run_suite.py` | `N/A (module-import failure during suite collection)` | ImportError: `shuffle_scale` missing from aiter (stale container image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417394) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417394) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_activation.py` | `test_activation_correctness`, `test_activation_out_param`, `test_activation_filter_expert*`, `test_relu2_*` (all params, fp16/bf16/fp32)` | 1. `JIT ROCm build failure: activation.cuh kReLU2 return-type deduction` | `RuntimeError: ninja exited with status 1` → `activation.cuh:213` \"cannot initialize return object of type `d… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_per_token_group_quant_8bit.py` | `(collection error at import)` | 1. `JIT ROCm build failure: activation.cuh kReLU2 return-type deduction` | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_fused_qk_gemma_rmsnorm_gate.py` | `(collection error at import)` | 1. `JIT ROCm build failure: activation.cuh kReLU2 return-type deduction` | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/diffusion/test_flydsl_fused_norm.py` | `test_fused_residual_norm_scale_shift`, `test_norm_scale_shift` (8 params)` | 1. `JIT ROCm build failure: activation.cuh kReLU2 return-type deduction` | `ModuleNotFoundError: No module named 'flydsl.expr'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/run_suite.py` | `N/A — import-time crash (no test collected)` | ImportError: `shuffle_scale` missing from aiter (stale aiter in CI image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417406) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417406) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/run_suite.py` | `N/A (import-time collection error, no test executed)` | ImportError: `shuffle_scale` missing from aiter.ops.shuffle (stale aiter dev im… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417423) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417423) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `N/A — suite import error (no test executed)` | `run_suite.py` module import` | ImportError: aiter `shuffle_scale` missing (stale CI image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417460) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417460) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/core/test_basic_sanity_eagle3.py` | `TestBasicSanityEagle3.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale aiter in CI i… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server exits code 1) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417487) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417487) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/core/test_basic_sanity.py` | `TestBasicSanity.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale aiter in CI i… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server exits code 1) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417487) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417487) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/quant/test_awq_dequant.py` | `module import (collection)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale aiter in CI i… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417487) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417487) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/perf/test_bench_one_batch_2gpu.py` | `TestBenchOneBatch2GPU.test_moe_tp2_bs1` | Stale aiter image — `ImportError: shuffle_scale` at fp8.py import | `AssertionError: -1 not greater than 85` (bench subprocess crashed with ImportError → throughput -1) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417752) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417752) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/hicache/test_hicache_storage_file_backend.py` | `TestHiCacheStorageAccuracy.setUpClass` | Stale aiter image — `ImportError: shuffle_scale` at fp8.py import | `Exception: Server process exited with code 1` (ImportError on launch) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417752) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417752) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/rl/test_update_weights_from_distributed.py` | `TestUpdateWeightsFromDistributed.test_update_weights_from_distributed` | Stale aiter image — `ImportError: shuffle_scale` at fp8.py import | Server failed to launch (ImportError); job then **cancelled** mid-test at 08:40 | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417752) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417752) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/sessions/test_streaming_session.py` | `TestStreamingSession.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale dev-patched a… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` → server exit 1 | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417818) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417818) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/sessions/test_session_control.py` | `TestSessionControl.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale dev-patched a… | same `ImportError` → server exit 1 | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417818) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417818) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/lightning/test_triton.py` | `module import (collection)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale dev-patched a… | same `ImportError` at import time | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417818) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417818) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/perf/test_bench_serving_1gpu_part1.py` | `collection (`test_offline_throughput_default`)` | ImportError: `shuffle_scale` missing from stale aiter in CI image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/quant/test_awq.py` | `collection` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same ImportError | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/dense/test_triton.py` | `collection` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same ImportError | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/dense/test_torch_native.py` | `collection` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same ImportError | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/gdn/test_torch_native.py` | `collection` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same ImportError | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/gdn/test_triton.py` | `collection` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same ImportError | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/swa/test_torch_native.py` | `collection` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same ImportError | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417836) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417836) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/tokenizer/test_multi_detokenizer.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | `ImportError: shuffle_scale` → server exit 1 | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/lora/test_lora_hf_sgl_logprob_diff.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/scheduler/test_mixed_chunked_prefill.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/tokenizer/test_skip_tokenizer_init.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/core/test_request_queue_validation.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/profiling/test_start_profile.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/features/test_openai_server_ebnf.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/model_loader/test_modelopt_export.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_priority_scheduling_disaggregation.py` | `server launch (setUp)` | ImportError: `shuffle_scale` missing from stale aiter in CI image (server crash… | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418031) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418031) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/perf/test_bench_serving_1gpu_part2.py` | `(import / collection)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417839) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417839) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/perf/test_bench_serving_1gpu_large.py` | `(import / collection)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417839) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417839) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/spec/eagle/test_eagle_constrained_decoding.py` | `(import / collection)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417839) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417839) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/kda/test_triton.py` | `(import / collection)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417839) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417839) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/unittests/swa/test_triton.py` | `(import / collection)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417839) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417839) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/spec/dflash/test_dflash.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/test_deterministic.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/rl/test_update_weights_from_tensor.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/observability/test_tracing.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/core/test_engine_child_pids.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/prefill_only/test_serving_rerank.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/distributed/test_parallel_state.py` | `test_parallel_group_construction_tp8_attn_cp2` (module-level)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_mm_hashes.py` | `(import)` | `ImportError: shuffle_scale from aiter.ops.shuffle (stale CI image aiter)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418041) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418041) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/scheduler/test_retract_decode.py` | `setUpClass` | ImportError: `shuffle_scale` missing from stale aiter in CI image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/constrained_decoding/test_constrained_decoding.py` | `setUpClass` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/models/test_generation_models.py` | `setUpClass` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/rl/test_lora_load_from_tensor.py` | `setUpClass` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/model_loading/test_external_models.py` | `setUpClass` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/rl/test_fp32_lm_head.py` | `setUpClass` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_mamba_unittest.py` | `(import)` | ImportError: `shuffle_scale` missing from stale aiter in CI image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418053) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418053) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/hicache/test_hicache_variants.py` | `TestHiCacheMLA.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/basic/test_openai_server.py` | `setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/basic/test_anthropic_server.py` | `TestAnthropicServer.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/radix_cache/test_radix_attention.py` | `TestRadixCacheFCFS.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/sampling/test_original_logprobs.py` | `TestOriginalLogprob.test_logprob_match` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_unified_radix_cache_bench.py` | `(import)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/vlm/test_evs.py` | `test_resolve_evs_config`, `test_replace_offsets_with_tokens_per_frame` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/models/test_llava.py` | `(import)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_radix_cache_unit.py` | `(import)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale docker image… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418060) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418060) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/dllm/test_llada2_mini.py` | `TestLLaDA2Mini.setUpClass` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/core/test_srt_engine.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/features/test_json_mode.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/constrained/test_e2e_constrained_reasoning.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/function_call/test_openai_function_calling.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/observability/test_metrics.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_swa_unittest.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/disaggregation/test_specv2_kvcache_offloading.py` | `server warmup (setUpClass)` | ImportError: aiter.ops.shuffle missing `shuffle_scale` (stale aiter image) | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418071) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418071) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/dllm/test_llada2_mini_amd.py` | `import-time (collection)` | ImportError: `shuffle_scale` missing from stale aiter (CI image / aiter version… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418081) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418081) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/input_embedding/test_input_embeds_chunked.py` | `import-time (collection)` | ImportError: `shuffle_scale` missing from stale aiter (CI image / aiter version… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418081) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418081) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/quant/test_block_int8.py` | `import-time (collection)` | ImportError: `shuffle_scale` missing from stale aiter (CI image / aiter version… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418081) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418081) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_dsa_pool_host_unit.py` | `import-time (collection)` | ImportError: `shuffle_scale` missing from stale aiter (CI image / aiter version… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418081) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418081) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_mm_process_config.py` | `test_configs_extracted` | ImportError: `shuffle_scale` missing from stale aiter (CI image / aiter version… | `ImportError: cannot import name 'shuffle_scale'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418081) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418081) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/backends/test_torch_compile.py` | `TestTorchCompile.setUpClass` | ImportError: `shuffle_scale` missing from stale aiter build (server startup cra… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server process exited code 1) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418102) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418102) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/hicache/test_hicache_storage.py` | `TestHiCache.setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` → server exit 1 | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/scheduler/test_scheduler_control.py` | `setUpClass` / `TestAbort.test_memory_leak` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/test_torch_native_attention_backend.py` | `TestTorchNativeAttnBackend.test_mmlu` / `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/function_call/test_anthropic_tool_use.py` | `TestAnthropicToolUse.setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/lora/test_lora_overlap_loading.py` | `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/models/test_transformers_backend_eval.py` | `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/validation/test_large_max_new_tokens.py` | `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_swa_eviction_boundary.py` | `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_profile_merger_http_api.py` | `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_prefill_adder.py` | `setUpClass` | `ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI aiter image)` | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418083) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418083) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/prefill_only/test_reward_models.py` | `module import (collection)` | ImportError: `shuffle_scale` missing from stale CI-image aiter | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418110) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418110) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/models/test_vlm_models.py` | `test_vlm_mmmu_benchmark` | ImportError: `shuffle_scale` missing from stale CI-image aiter | `AssertionError: Server process exited with code 1` (server crashed on same `shuffle_scale` ImportError) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418110) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418110) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/perf/test_vlm_perf_5090.py` | `TestVLMPerf5090.test_vlm_offline_throughput` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` (server failed to start) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/test_triton_sliding_window.py` | `TestSlidingWindowAttentionTriton.test_cuda_graph` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/prefill_only/test_openai_embedding.py` | `TestMatryoshkaEmbeddingModel.setUpClass` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/prefill_only/test_pooled_hidden_states.py` | `setUpClass` (server start)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/validation/test_matched_stop.py` | `TestMatchedStop.setUpClass` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/models/test_compressed_tensors_models.py` | `TestCompressedTensorsLlama3FP8.setUpClass` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/lora/test_mem_pool_ep_unit.py` | `(module import)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_radix_cache_slru_accuracy.py` | `(module import)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418147) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418147) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/attention/test_triton_attention_backend.py` | `test_latency` (`TestTritonAttnBackend`)` | ImportError: `shuffle_scale` missing from stale aiter image | `AssertionError: -1 not greater than 153` (subprocess crashed with `ImportError: cannot import name 'shuffle_… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418112) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418112) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/models/test_transformers_models.py` | `(import)` | ImportError: shuffle_scale missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/vlm/test_vision_chunked_prefill.py` | `TestVisionChunkedPrefill.test_chunked_prefill` (import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/scheduler/test_priority_scheduling.py` | `TestPriorityScheduling.setUpClass` (import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/lora/test_lora_drainer.py` | `(import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/sampling/test_penalty.py` | `TestPenalty.setUpClass` (import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/validation/test_openai_server_ignore_eos.py` | `TestOpenAIServerIgnoreEOS.setUpClass` (import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/input_embedding/test_input_embeddings.py` | `TestInputEmbeds.setUpClass` (import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_decode_radix_lock_ref.py` | `(import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/utils/test_type_based_dispatcher.py` | `(import)` | ImportError: shuffle_scale missing from stale aiter image | same | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418192) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418192) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/tokenizer/test_multi_tokenizer.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/lora/test_lora_eviction.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/reasoning/test_reasoning.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/core/test_srt_endpoint.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/sampling/test_pytorch_sampling_backend.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/radix_cache/test_radix_cache_hit.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/moe/test_fused_moe.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/quant/test_triton_scaled_mm.py` | `(collection/import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/utils/test_profile_merger.py` | `(import)` | `ImportError: shuffle_scale missing from stale aiter in CI image` | `ImportError: shuffle_scale` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418214) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418214) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/pp/test_pp_single_node.py` | `import-time (collection)` | ImportError: `shuffle_scale` missing from stale aiter image | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418207) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418207) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/pp/test_pp_single_node_extra.py` | `test_pp_consistency` → `run_gsm8k_test` (server launch)` | ImportError: `shuffle_scale` missing from stale aiter image | `Server process exited with code 1` (root: same `ImportError: shuffle_scale`) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418207) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418207) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/spec/eagle/test_eagle_dp_attention.py` | `setUpClass` (server launch)` | ImportError: `shuffle_scale` missing from stale aiter image | `Server process exited with code 1` (root: same `ImportError: shuffle_scale`) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418207) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418207) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/mla/test_mla_fp8.py` | `setUpClass` (server exited code 1)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/lora/test_multi_lora_backend.py` | `module import` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/observability/test_priority_metrics.py` | `TestPriorityMetrics.setUpClass` (server exited code 1)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/ops/test_aiter_greedy_sample_amd.py` | `TestAiterGreedyIntegration.test_aiter_greedy_with_return_logprob` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/openai_server/validation/test_request_length_validation.py` | `TestRequestLengthValidation.setUpClass` (server exited code 1)` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_swa_lock_release_lifecycle.py` | `module import` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/prefill_only/test_embed_overrides.py` | `module import` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_io_struct.py` | `module import` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale CI image)` | same `ImportError` | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_deepseek_v32_basic.py` | `TestDeepseekV32TP.setUpClass` | ImportError: `shuffle_scale` missing from aiter (stale CI image) | `Exception: Server process exited with code 1` (root: `ImportError: cannot import name 'shuffle_scale' from '… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418264) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418264) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_kimi_k2_instruct.py` | `TestKimiK2Instruct0905.setUpClass` | ImportError: aiter `shuffle_scale` missing (stale dev aiter in CI image) | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server exits code 1 → `Exception:… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418269) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418269) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_deepseek_v3_mtp.py` | `TestDeepseekV3MTP.setUpClass` | \"ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI image)\" — plus… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server failed to launch) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418287) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418287) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_deepseek_v3_basic.py` | `TestDeepseekV3Basic.setUpClass` | \"ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI image)\" — plus… | `ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle'` (server failed to launch) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418287) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418287) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/ops/test_aiter_allreduce_fusion_amd.py` | `test_fused_ar_rms_benchmark` | \"ImportError: aiter.ops.shuffle.shuffle_scale missing (stale CI image)\" — plus… | `AssertionError: Benchmark command failed.` (subprocess hit same `shuffle_scale` ImportError) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418287) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418287) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py` | `test_a_gsm8k` (`TestDeepseekR1MXFP4MTP`)` | EAGLE/MTP speculative accept-length below threshold on DeepSeek-R1-MXFP4 (8-GPU… | `AssertionError: 1.4461491018425718 not greater than 2.04` (retry → `1.430396407556519 not greater than 2.04`) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374419346) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374419346) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/diffusion/test_flydsl_fused_norm.py` | `test_fused_residual_norm_scale_shift` | ModuleNotFoundError: flydsl.expr missing | ModuleNotFoundError: No module named 'flydsl.expr' | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py` | `TestDeepseekR1MXFP4MTP.test_a_gsm8k` | EAGLE/MTP speculative accept-length collapse on DeepSeek-R1-MXFP4 (accuracy OK,… | `AssertionError: 1.4497769573911705 not greater than 2.04` (`avg_spec_accept_length`) | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442043808) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442043808) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/srt/layers/quantization/fp8.py` | `N/A` | shuffle_scale ImportError (stale aiter CI image, run-wide) | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' — wiped out ~95 test files across ru… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/server/test_server_2_gpu.py` | `TestDiffusionServerTwoGpu::test_diffusion_generation[ideogram4_fp8_tp2_t2i]` | GatedRepoError 403 — gated HF model `ideogram-ai/ideogram-4-fp8` not accessible… | `huggingface_hub.errors.GatedRepoError: 403 Client Error` → server scheduler dead → `EOFError` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042653) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042653) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_activation.py` | `test_activation_correctness / test_activation_out_param / test_activation_filter_expert* / test_relu2_*` | JIT ROCm build failure: activation.cuh kReLU2 return-type deduction | RuntimeError: ninja exited with status 1 (activation.cuh:213 decltype) | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/diffusion/test_flydsl_fused_norm.py` | `test_fused_residual_norm_scale_shift / test_norm_scale_shift` | ModuleNotFoundError: flydsl.expr (JIT diffusion) | ModuleNotFoundError: No module named 'flydsl.expr' | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417349) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417349) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `aiter.ops.shuffle.shuffle_scale ImportError — ~120 tests across the 08:07 run (stale CI image)` | `N/A (mass import/server-launch failure across stage-a/b/c, jit, multimodal, amd shards)` | shuffle_scale ImportError (stale CI image / wrong pinned aiter on run 275326873… | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/spec/eagle/test_eagle_dp_attention.py` | `TestEAGLE3EngineDPAttention.test_a_gsm8k` | EAGLE3 draft-extend idle-batch crash: `EagleDraftExtendInput` missing `kv_indpt… | Server crash → `ConnectionRefusedError` (root cause: `AttributeError: 'EagleDraftExtendInput' object has no a… | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442043889) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442043889) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_activation.py` | `test_activation_correctness`, `test_activation_out_param`, `test_activation_filter_expert*`, `test_relu2_*` (all 73 collected)` | JIT kernel C++ compile failure on ROCm clang — `decltype`/deduced-return-type r… | `RuntimeError: ninja exited with status 1` → `activation.cuh:213: cannot initialize return object of type 'de… | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042619) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042619) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/jit/test_per_token_group_quant_8bit.py` | `test_per_token_group_quant_with_column_major` (all params)` | JIT kernel C++ compile failure on ROCm clang — `decltype`/deduced-return-type r… | `RuntimeError: ninja exited with status 1` → `utils.cuh:301: function 'forward' with deduced re… | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042619) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042619) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/mem_cache/test_swa_lock_release_lifecycle.py` | `N/A (module import)` | shuffle_scale ImportError (stale CI image) | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/prefill_only/test_embed_overrides.py` | `N/A (module import)` | shuffle_scale ImportError (stale CI image) | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/unit/managers/test_io_struct.py` | `N/A (module import)` | shuffle_scale ImportError (stale CI image) | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374418228) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374418228) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `test/registered/quant/test_block_int8.py` | `test_w8a8_block_int8_fused_moe` | `ValueError: Global server args not set in MoE unit test (HIP eplb path)` | `ValueError: Global server args is not set yet!` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442043400) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442043400) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `#` | `Test File` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | Test Function | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `1` | `sglang/multimodal_gen/test/server/test_server_1_gpu.py` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[qwen_image_t2i_cache_dit_enabled]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `2` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[zimage_image_t2i]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `3` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[qwen_image_edit_ti2i]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `4` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[qwen_image_edit_2511_ti2i]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `5` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[qwen_image_layered_i2i]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `6` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[wan2_1_t2v_1.3b_frame_interp_2x]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `7` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[wan2_1_t2v_1_3b_lora_1gpu]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `8` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[fast_hunyuan_video]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `9` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[ltx_2_3_hq_pipeline]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `10` | `same` | Diffusion server scheduler port 5555 cascade after HF model-download timeout | `test_diffusion_generation[lingbot_world_realtime_plastic_beach]` | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/server/test_server_1_gpu.py` | `TestDiffusionServerOneGpu.test_diffusion_generation[qwen_image_t2i_cache_dit_enabled]` | Diffusion server scheduler port 5555 cascade after HF download timeout | RuntimeError: Server exited early (HF download stall) -> port 5555 cascade (10 errors) | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/server/test_server_1_gpu.py` | `TestDiffusionServerOneGpu.test_diffusion_generation (9-case port-5555 cascade)` | Diffusion server port 5555 cascade after HF model-download timeout | RuntimeError: Scheduler port 5555 is unavailable and --strict-ports is enabled (cascade from trigger above) | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042667) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042667) | | 2026-06-15 | 2026-06-15 | 0d | ⚪ quiet 59d | `multiple (~110 registered tests across ~30 jobs, run 27532687319 only)` | `N/A` | ImportError: shuffle_scale missing from aiter (stale miles-*-test CI image) | ImportError: cannot import name 'shuffle_scale' from 'aiter.ops.shuffle' (stale aiter g417de6df4; image-wide,… | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374410654) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374410654) | | 2026-06-11 | 2026-06-15 | 4d | ⚪ quiet 59d | `python/sglang/multimodal_gen/test/server/test_server_2_gpu.py` | `TestDiffusionServerTwoGpu.test_diffusion_generation[ideogram4_fp8_tp2_t2i]` | GatedRepoError 403 - gated HF model ideogram-ai/ideogram-4-fp8 | huggingface_hub.errors.GatedRepoError: 403 Client Error | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442042653) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442042653) | | 2026-06-11 | 2026-06-15 | 4d | ⚪ quiet 59d | `test/registered/jit/test_per_token_group_quant_8bit.py` | `test_per_token_group_quant_with_column_major` | JIT kernel C++ compile failure on ROCm clang (deduced return type) | RuntimeError: ninja exited with status 1 (utils.cuh:301 deduced return type) | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442342619) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442342619) | | 2026-06-11 | 2026-06-15 | 4d | ⚪ quiet 59d | `test/registered/jit/test_activation.py` | `test_activation_correctness` | JIT kernel C++ compile failure on ROCm clang (activation.cuh:213 decltype) | RuntimeError: ninja exited with status 1 (activation.cuh:213 kReLU2 decltype) | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442342619) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442342619) | | 2026-06-11 | 2026-06-15 | 4d | ⚪ quiet 59d | `test/registered/ops/test_aiter_allreduce_fusion_amd.py` | `test_fused_ar_rms_residual_accuracy` | aiter collective-op CI failures: (a) fused all-reduce+RMSNorm 1-ULP residual mi… | `AssertionError: Residual accuracy check failed` — fused residual diverges, `max_diff=6.25e-2`, `AR_exact=yes… | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442043916) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442043916) | | 2026-06-11 | 2026-06-15 | 4d | ⚪ quiet 59d | `test/registered/ops/test_aiter_allgather_amd.py` | `test_aiter_allgather_matches_rccl` | aiter collective-op CI failures: (a) fused all-reduce+RMSNorm 1-ULP residual mi… | `RuntimeError: custom allreduce only supports float32, float16 and bfloat16` (raised in aiter `all_gather_unr… | [job](https://github.com/sgl-project/sglang/actions/runs/27552331342/job/81442043916) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81442043916) | | 2026-06-11 | 2026-06-15 | 4d | ⚪ quiet 59d | `test/registered/cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py` | `TestPiecewiseCudaGraphQwen25VL.setUpClass` | `ImportError: shuffle_scale missing from aiter.ops.shuffle (stale dev-patched a… | same `ImportError` → server exit 1 | [job](https://github.com/sgl-project/sglang/actions/runs/27532687319/job/81374417818) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/105#job-81374417818) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `test/registered/amd/test_kimi_k25_mxfp4.py` | `TestKimiK25MXFP4.test_a_gsm8k` | Kimi-K2.5-MXFP4 GSM8K accuracy just below 0.92 threshold (borderline variance) | AssertionError: 0.9166034874905231 not greater than 0.92 | [job](https://github.com/sgl-project/sglang/actions/runs/27260759828/job/80505660602) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80505660602) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `test/registered/core/test_basic_sanity.py` | `TestBasicSanity.test_fwd_occupancy` | fwd_occupancy median below 99.0 threshold on AMD MI325 (single-batch decode occ… | AssertionError: 98.24 not greater than 99.0 : sglang:fwd_occupancy median did not exceed threshold | [job](https://github.com/sgl-project/sglang/actions/runs/27370702558/job/80881934442) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80881934442) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `test/registered/hicache/test_hicache_variants.py` | `TestHiCachePage.test_mmlu` | MMLU eval dataset download fails — Network is unreachable on AMD runner | urllib.error.URLError: [Errno 101] Network is unreachable -> retry() exceed maximum number of retries | [job](https://github.com/sgl-project/sglang/actions/runs/27370702558/job/80881935418) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80881935418) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `Start CI container (linux-mi325-8gpu-sglang runner)` | `N/A` | 8-GPU MI325 container-start hang (~600s, runner lost communication) | Runner lost communication during Start CI container; job killed at ~600s, no log | [job](https://github.com/sgl-project/sglang/actions/runs/27331990696/job/80746491500) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80746491500) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `Install dependencies (mori build)` | `N/A` | mori build failure - corrupt libabsl_time_zone.so (invalid ELF header) | libabsl_time_zone.so.20210324: invalid ELF header (exit 127) -> ninja build stopped | [job](https://github.com/sgl-project/sglang/actions/runs/27331990696/job/80746491258) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80746491258) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `stage-c-test-large-8-gpu-amd: Start CI container (no test ran)` | `N/A` | 8-GPU MI325 container-start hang (job killed at ~600s in Start CI container; ru… | Self-hosted runner lost communication during 'Start CI container'; job killed at ~600s, BlobNotFound | [job](https://github.com/sgl-project/sglang/actions/runs/27331990696/job/80746491500) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80746491500) | | 2026-06-12 | 2026-06-12 | 0d | ⚪ quiet 62d | `stage-b-test-1-gpu-large-amd: Install dependencies (mori build, not a test)` | `N/A` | mori build failure — corrupt libabsl_time_zone.so (invalid ELF header) during g… | test_router_dedup: error loading libabsl_time_zone.so.20210324: invalid ELF header -> ninja exit 1 | [job](https://github.com/sgl-project/sglang/actions/runs/27331990696/job/80746491258) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80746491258) | | 2026-06-11 | 2026-06-12 | 1d | ⚪ quiet 62d | `python/sglang/multimodal_gen/test/server/test_server_2_gpu.py` | `TestDiffusionServerTwoGpu.test_diffusion_generation[wan2_2_t2v_a14b_lora_2gpu]` | Diffusion server generation timeout -> leaked process holds scheduler port 5555… | openai.APITimeoutError / httpx.ReadTimeout on generation; orphaned server then cascades 8x 'Scheduler port 55… | [job](https://github.com/sgl-project/sglang/actions/runs/27316483669/job/80698092259) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80698092259) | | 2026-06-11 | 2026-06-12 | 1d | ⚪ quiet 62d | `test/registered/amd/test_deepseek_r1_mxfp4_8gpu.py` | `TestDeepseekR1MXFP4.setUpClass` | AMD prefill piecewise CUDA graph shape mismatch in MLA attention (view([4,16,51… | RuntimeError: shape '[4, 16, 512]' is invalid for input of size 8192 -> server exits -9 | [job](https://github.com/sgl-project/sglang/actions/runs/27299038645/job/80639481499) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80639481499) | | 2026-06-11 | 2026-06-12 | 1d | ⚪ quiet 62d | `python/sglang/multimodal_gen/test/server/test_disagg_server.py` | `TestDisaggZImage1Rank.test_generates_image` | Disaggregated diffusion warmup hang — DiffusionServer timeout: request not comp… | RuntimeError: Warmup request failed for zimage_1rank: 500 Server Error -> DiffusionServer timeout 120.0s | [job](https://github.com/sgl-project/sglang/actions/runs/27370702558/job/80881934479) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80881934479) | | 2026-06-11 | 2026-06-12 | 1d | ⚪ quiet 62d | `python/sglang/multimodal_gen/test/server/test_disagg_server.py` | `TestDisaggZImage2RankDenoiser.test_generates_image_with_sp2_denoiser` | Disaggregated diffusion warmup hang — DiffusionServer timeout: request not comp… | DiffusionServer timeout: request not completed within 120.0s during warmup | [job](https://github.com/sgl-project/sglang/actions/runs/27299038645/job/80639480374) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80639480374) | | 2026-06-11 | 2026-06-12 | 1d | ⚪ quiet 62d | `python/sglang/multimodal_gen/test/server/test_disagg_server.py` | `TestDisaggZImageTracing.test_disagg_spans_share_trace_id` | Disaggregated diffusion warmup hang — DiffusionServer timeout: request not comp… | DiffusionServer timeout: request not completed within 120.0s during warmup | [job](https://github.com/sgl-project/sglang/actions/runs/27299038645/job/80639480374) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80639480374) | | 2026-06-11 | 2026-06-12 | 1d | ⚪ quiet 62d | `python/sglang/multimodal_gen/test/server/test_server_1_gpu.py` | `TestDiffusionServerOneGpu.test_diffusion_generation[qwen_image_edit_2509_ti2i]` | External image download failure (Alibaba OSS unreachable from AMD runner) | openai.BadRequestError: 400 - Failed to download image from qianwen-res.oss-cn-beijing.aliyuncs.com | [job](https://github.com/sgl-project/sglang/actions/runs/27299038645/job/80639480303) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80639480303) | | 2026-06-11 | 2026-06-11 | 0d | ⚪ quiet 63d | `test/registered/core/test_basic_sanity.py` | `test_fwd_occupancy` | fwd_occupancy median below threshold on AMD | AssertionError: 98.25 not greater than 99.0 (sglang:fwd_occupancy median) | [job](https://github.com/sgl-project/sglang/actions/runs/27350011251/job/80809006515) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80809006515) | | 2026-06-11 | 2026-06-11 | 0d | ⚪ quiet 63d | `N/A — Start CI container (stage-c-test-large-8-gpu-amd, linux-mi325-8gpu-sglang)` | `N/A` | Self-hosted runner container-start hang on linux-mi325-8gpu-sglang pool | Job hung at 'Start CI container' step ~10 min; runner lost communication; no test ran (BlobNotFound) | [job](https://github.com/sgl-project/sglang/actions/runs/27316483669/job/80698092451) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80698092451) | | 2026-06-11 | 2026-06-11 | 0d | ⚪ quiet 63d | `N/A — Install dependencies / mori build (stage-b-test-1-gpu-large-amd, linux-mi325-1gpu-sglang)` | `N/A` | mori build failure — corrupt libabsl_time_zone.so (invalid ELF header) | libabsl_time_zone.so.20210324: invalid ELF header (exit 127 in gtest discovery) → ninja build stopped | [job](https://github.com/sgl-project/sglang/actions/runs/27331990696/job/80746491258) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80746491258) | | 2026-06-11 | 2026-06-11 | 0d | ⚪ quiet 63d | `test/registered/amd/test_kimi_k25_mxfp4.py` | `test_a_gsm8k` | Kimi-K2.5-MXFP4 GSM8K accuracy just below 0.92 threshold | AssertionError: 0.9166034874905231 not greater than 0.92 | [job](https://github.com/sgl-project/sglang/actions/runs/27260759828/job/80505660602) | [详情](https://github.com/bingxche/sglang-ci-bot/issues/100#job-80505660602) | --- *Auto-generated by amd-bot · 2026-08-13 11:25 UTC* ",
"url": "https://github.com/sgl-project/sglang/issues/27937",
"createdAt": "2026-06-11T15:46:32Z",
"updatedAt": "2026-08-13T11:25:40Z",
"timestamp": "2026-08-13T11:25:40Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "amd-bot",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:27987",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Systematic CUDA illegal memory access in DP-attention forward_idle on NVFP4 trtllm MoE (GLM-5.1, GB300, PD-prefill)",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug We see a **systematic** (not hardware-random) `CUDA error: an illegal memory access was encountered` on **PD-prefill** nodes running **GLM-5.1-NVFP4** with **DP-attention**. It always surfaces on a **DP-attention idle rank** (`forward_idle`), with the async error landing in the NVFP4 trtllm MoE path. **Crash stack (byte-for-byte identical across 3 independent pods / 3 nodes / 3 different DP ranks):** ``` disaggregation/prefill.py:447 event_loop_overlap_disagg_prefill → scheduler.py:3039 run_batch → tp_worker.py:472 forward_batch_generation → model_runner.py:3323 forward → :3457 _forward_raw → :3259 forward_idle ← DP-attention idle rank (bs=0, unpadded) → deepseek_v2.py:2485 forward → :2284 layer → :1973 mlp → :737 forward → :906 forward_normal → maybe_fuse_routed_scale_and_shared_add → mxfp4_flashinfer_trtllm_moe.py:468 routed += shared ← async surface point, not the fault site torch.AcceleratorError: CUDA error: an illegal memory access was encountered ``` The `routed += shared` line is just the next sync point that surfaces a delayed async error — the log itself prints \"stacktrace might be incorrect\". It lands here consistently only because GLM-5.1 has `n_shared_experts=1` + `disable_shared_experts_fusion=True`, so every MoE layer takes this non-fused branch (the densest sync point in an idle forward). **Key facts (verified against the full per-rank logs of 3 crashed pods):** 1. **3 pods / 3 nodes / 3 different DP ranks, byte-identical Python stack** within a ~1 h window. A hardware Xid cannot produce byte-identical Python stacks across nodes → this is a **software bug**, not bad silicon. All three crash on a DP-attention idle rank, in `forward_idle → ... → mxfp4_flashinfer_trtllm_moe.py:468`: | pod | UTC | crashing DP rank | last real prefill on that rank | Δ to crash | KV transfer at that batch | |---|---|---|---|---|---| | prefill-1 | 09:54:39 | DP3 | 09:54:36 | 3 s | `#inflight-req 1` | | prefill-5 | 10:28:25 | DP2 | 10:28:12 | 13 s | `#inflight-req 2` | | prefill-4 | 10:47:39 | DP1 | 10:47:29 | 10 s | `#inflight-req 1` | 2. **Only the idle rank crashes in each pod.** `Scheduler hit an exception` appears exactly once per pod (always the idle rank); the other 3 ranks are SIGQUIT'd ~1 s later with no CUDA error of their own. 3. **In every case the crashing rank had just finished a real prefill 3–13 s earlier** (with an in-flight KV transfer), then transitioned into `forward_idle` (see table above). The just-finished-prefill rank is exactly the one that has just freed a large block of prefill activations. 4. Suspected trigger: **long-context traffic.** All three pods were serving long cached-prefix requests in the minutes around each crash (per-batch `#cached-token` up to ~130k on prefill-1, ~192k on prefill-5, and batches summing to ~230k on prefill-4; `max_position=202752`). The long-context request is not necessarily on the crashing rank itself — e.g. on prefill-5 the 192k-cached batch was on DP0 while DP2 crashed — consistent with a pod-wide load wave rather than a single poisoned request. 5. **`exit code 0` blind spot:** the scheduler subprocess crash (`exit code -3`) triggers a SIGQUIT cleanup that ends in `sys.exit(0)`, so the container exits **0** and k8s marks the pod `Completed` (not `Error`, no `CrashLoopBackOff`). Alerting based on non-zero exit / crashloop misses this entirely — we only caught it via log scraping for `illegal memory access`. **What we ruled out (with evidence):** - **0-token / shape-degenerate MoE input.** The idle batch is `batch_size=0`, unpadded (`DpPaddingMode.SUM_LEN` on the PD-prefill path). The MoE input is the DP-gathered global buffer, which is `fill_(0)` before all-reduce, so **all 4 ranks feed a byte-identical buffer** — a data-dependent GEMM bug would kill all 4 ranks, but only the idle rank dies. - **Stale `forward_metadata` / PR #26313.** `forward_idle` only re-inits metadata when `batch_size>0`, leaving stale metadata on the unpadded idle path — but #26313's causal chain does not assemble on GLM: its consumer `_maybe_upgrade_forward_metadata` exists only in the dsv4 backend (GLM-5.1 is `GlmMoeDsaForCausalLM(DeepseekV2ForCausalLM)`, runs deepseek_v2/nsa code — 0 grep hits), and its UAF target (SWA full→swa mapping) doesn't exist on GLM (no `sliding_window` in config, no `translate_loc_from_full_to_swa` in the nsa backend). (The crashes above are all on **stock 0.5.12 without #26313**. We are carrying the cherry-pick in a future build as zero-risk insurance only — it is not expected to fix this.) - **In-flight mooncake transfer at crash instant.** On prefill-1 (the one pod that emitted a full faulthandler thread dump), all 4 `transfer_worker` threads were idle in `conn.py:1207 queue.get()` at crash time; transfer is synchronous with no failure logs. - **Periodic `empty_cache` / VA unmap.** `SGLANG_EMPTY_CACHE_INTERVAL=-1` (default), `sleep_on_idle=False`, no `expandable_segments`. - **flashinfer upgrade.** Audited every fused_moe commit from 0.6.11.post1 → main: no IMA / zero-token / NaN-routing fix. Upgrading flashinfer does not address this. **Current leading hypothesis (unproven — closed-source cubin):** the only open-source UB channel in the trtllm NVFP4 MoE path is `permuted_idx_to_token_idx`, which is allocated per call and **never memset** (`flashinfer .../trtllm/fused_moe/launcher.cu` FP4 `prepare_routing`); routing writes only the real slots, leaving padding slots holding **residue of whatever tensor previously occupied that allocation**. The upstream header even self-documents this: `RoutingKernel.h` — *\"Any out-of-bounds values are undefined\"*. This routeMap is gathered (read) by the closed PermuteGemm1 cubin **before** the predicate discard. Reading mapped garbage is silently harmless; reading an **unmapped VA** is an IMA (a **read** fault — matching the Xid 31 VIRT_READ in #27520). The **rank-local asymmetry** explains the idle-rank bias: the MoE *input* is identical across ranks, but this workspace *residue* comes from each rank's own allocator history. In steady state a rank reuses last round's MoE workspace (residue = old valid small indices, harmless). A rank that **just freed 8k–16k tokens of prefill activations** perturbs its free list, so the next (idle) MoE may get a former activation block — bf16 bits reinterpreted as int32 are ~1e9, × row stride → TB-scale offset → guaranteed unmapped → IMA. This is consistent with every observation: only just-finished idle ranks crash, sporadic in steady state, correlated with long-context waves. We'd appreciate maintainers who own the trtllm-gen routing/permute path confirming whether the FP4 `permuted_idx_to_token_idx` padding slots are supposed to be initialized, and whether a `cudaMemsetAsync(ptr, 0, bytes, stream)` after allocation is the right fix (filling 0 = a valid row index, discarded by the predicate after gather). ### Reproduction This reproduces **sporadically** under production long-context traffic — we don't have a deterministic minimal script (matching #27520's \"hard to reproduce\"). Configuration: - **Topology:** PD disaggregation, **prefill** role, mooncake transfer backend (MNNVL/NVLINK custom mem pool), `page_size=64`. - **Model:** GLM-5.1-NVFP4 (`GlmMoeDsaForCausalLM`, 256 routed experts / top-8 / 1 shared, NVFP4 group_size 16, max_position 202752). - **Engine args (prefill):** ``` --quantization modelopt_fp4 --moe-runner-backend flashinfer_trtllm --disable-shared-experts-fusion --attention-backend nsa --enable-dp-attention --dp-size 4 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --chunked-prefill-size 8192 --max-prefill-tokens 16384 --page-size 64 ``` - **Trigger condition:** long-context requests (cached/extend length approaching the 202752 max position) hitting a DP rank that then goes idle. Crashes cluster within a ~1h window during such a wave. - **Note on the model class in logs:** the MoE-backend selection line prints `Use flashinfer_trtllm as MoE runner backend on sm100 for DeepseekV3ForCausalLM` — GLM-5.1's `GlmMoeDsaForCausalLM` subclasses `DeepseekV2ForCausalLM` and runs the deepseek_v2 MoE/attention code path, so the routing/permute code under discussion is the shared trtllm NVFP4 MoE path (`sm100` is the kernel-family bucket; autotune compiles for `sm103`/GB300). ### Environment Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3: NVIDIA GB300 GPU 0,1,2,3 Compute Capability: 10.3 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 13.0, V13.0.88 CUDA Driver Version: 580.105.08 PyTorch: 2.11.0+cu130 sglang: 0.5.12 sglang-kernel: 0.4.2.post2 flashinfer_python: 0.6.11.post1 flashinfer_cubin: 0.6.11.post1 flashinfer_jit_cache: 0.6.11.post1+cu130 triton: 3.6.0 transformers: 5.6.0 torchao: 0.17.0 numpy: 2.3.5 aiohttp: 3.13.5 fastapi: 0.136.1 huggingface_hub: 1.15.0 interegular: 0.3.3 modelscope: 1.37.0 orjson: 3.11.9 outlines: 0.1.11 packaging: 26.2 psutil: 7.2.2 pydantic: 2.13.4 python-multipart: 0.0.28 pyzmq: 27.1.0 uvicorn: 0.47.0 uvloop: 0.22.1 vllm: Module Not Found xgrammar: 0.2.0 openai: 2.6.1 tiktoken: 0.13.0 anthropic: 0.102.0 litellm: Module Not Found torchcodec: Module Not Found NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 NIC0 NIC1 CPU Affinity NUMA A GPU0 X NV18 NV18 NV18 SYS SYS 0-71 0 2 GPU1 NV18 X NV18 NV18 SYS SYS 0-71 0 10 GPU2 NV18 NV18 X NV18 NODE NODE 72-143 1 18 GPU3 NV18 NV18 NV18 X NODE NODE 72-143 1 NIC0 SYS SYS NODE NODE X PIX NIC1 SYS SYS NODE NODE PIX X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_4 NIC1: mlx5_5 ulimit soft: 1048576",
"url": "https://github.com/sgl-project/sglang/issues/27987",
"createdAt": "2026-06-12T02:05:30Z",
"updatedAt": "2026-08-13T00:23:35Z",
"timestamp": "2026-08-13T00:23:35Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"inactive"
],
"author": "ryanx-sir",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:28111",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] function call parse error, python 字符串 10220_3939392 转成了 102203939392",
"text": "### Checklist - [ ] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [ ] Please use English. Otherwise, it will be closed. ### Describe the bug function call 有一个严重的bug，非流式解析tool 参数，会把参数值：\"102020_499494994\", 解析成 \"102020499494994\", 下划线是python 的特性，数字之间可以用_分割。tool 解析参数的代码： ```python def parse_arguments( json_value: str, arg_type: Optional[str] = None ) -> Tuple[Any, bool]: \"\"\"Parse argument value with multiple fallback strategies. Args: json_value: Raw string value to parse arg_type: Expected type hint ('string', 'number', 'object', etc.) Returns: Tuple of (parsed_value, is_valid_json) \"\"\" # Strategy 1: Direct JSON parsing try: parsed_value = json.loads(json_value) # Type coercion for number type if arg_type == \"number\" and isinstance(parsed_value, str): parsed_value = _convert_to_number(parsed_value) return parsed_value, True except (json.JSONDecodeError, ValueError): pass # Strategy 2: Unescape and parse try: wrapped = json.loads('{\"tmp\": \"' + json_value + '\"}') parsed_value = json.loads(wrapped[\"tmp\"]) if arg_type == \"number\" and isinstance(parsed_value, str): parsed_value = _convert_to_number(parsed_value) return parsed_value, True except (json.JSONDecodeError, ValueError, KeyError): pass # Strategy 3: ast.literal_eval try: parsed_value = ast.literal_eval(json_value) # 会把字符串\"28383_99494\"， 的下划线去掉 return parsed_value, True except (ValueError, SyntaxError): pass # Strategy 4: Treat as string try: quoted_value = json.dumps(str(json_value)) return json.loads(quoted_value), True except (json.JSONDecodeError, ValueError): return json_value, False ``` ### Reproduction [glm4_moe_detector.py](https://github.com/user-attachments/files/28905400/glm4_moe_detector.py) ### Environment Any Environment Example:python3 -m sglang",
"url": "https://github.com/sgl-project/sglang/issues/28111",
"createdAt": "2026-06-13T01:03:28Z",
"updatedAt": "2026-08-13T00:23:32Z",
"timestamp": "2026-08-13T00:23:32Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"inactive"
],
"author": "phychaos",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:29942",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] DeepEP low_latency buffer lazy init fails during CUDA graph capture with PP=2, TP=8, DP-attention, EP=8 on Kimi K2.6 W4A8",
"text": "### Checklist - [ ] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [ ] Please use English. Otherwise, it will be closed. ### Describe the bug ### Describe the bug When launching Kimi K2.6 W4A8 with PP=2, TP=8, DP=8, EP=8, DP-attention and DeepEP low_latency mode, scheduler initialization fails during CUDA graph capture. The failure happens when CUDA graph capture reaches the first MoE/DeepEP dispatch path. DeepEP buffer appears to be lazily initialized inside capture, then `deep_ep.cpp:230` returns `invalid argument`. This does not look like a memory size issue: reducing `NVSHMEM_SYMMETRIC_SIZE`, reducing `--cuda-graph-max-bs`, and changing `--chunked-prefill-size` did not change the failure mode. The same stack appears on different ranks. ### Environment - Hardware: 2 nodes × 8×H800 - Model: Kimi K2.6 W4A8 - Parallelism: - `--pp-size 2` - `--tp-size 8` - `--dp-size 8` - `--enable-dp-attention` - `--ep-size 8` - MoE: - `--moe-a2a-backend deepep` - `--deepep-mode low_latency` - Quantization: - `--quantization w4afp8` - `--kv-cache-dtype fp8_e4m3` - CUDA graph enabled - Tried `--cuda-graph-max-bs 16` and lower values ### Error ```text Exception: Capture cuda graph failed: Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:230 'invalid argument' File \"/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py\", line 791, in initialize self.init_device_graphs() File \"/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py\", line 2965, in init_device_graphs self.graph_runner = graph_runners[self.device](self) File \"/sgl-workspace/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 738, in __init__ raise Exception( Exception: Capture cuda graph failed: Failed: CUDA error /build/DeepEP/csrc/deep_ep.cpp:230 'invalid argument' Question Is PP=2 + DeepEP low_latency + CUDA graph expected to be supported for Kimi/DeepSeek-style MoE models? If yes, could SGLang add an eager DeepEP buffer initialization or model warmup path before CUDA graph capture ### Reproduction exec python3 -m sglang.launch_server \\ --model-path \"$MODEL\" \\ --served-model-name \"$SERVED_NAME\" \\ --trust-remote-code \\ --tp-size 8 \\ --pp-size 2 \\ --dp-size 8 \\ --enable-dp-attention \\ --moe-a2a-backend deepep \\ --deepep-mode low_latency \\ --ep-size 8 \\ --moe-dense-tp-size 1 \\ --quantization w4afp8 \\ --kv-cache-dtype fp8_e4m3 \\ --mem-fraction-static 0.7 \\ --max-running-requests 32 \\ --cuda-graph-max-bs 16 \\ --context-length 105536 \\ --chunked-prefill-size 4096 \\ --disable-custom-all-reduce \\ --dist-init-addr \"${MASTER_IP}:${DIST_PORT}\" \\ --nnodes \"$NNODES\" \\ --node-rank \"$NODE_RANK\" \\ --host 0.0.0.0 \\ --port \"$API_PORT\" \\ --enable-metrics \\ 2>&1 | tee \"$LOG\" ### Environment exec python3 -m sglang.launch_server \\ --model-path \"$MODEL\" \\ --served-model-name \"$SERVED_NAME\" \\ --trust-remote-code \\ --tp-size 8 \\ --pp-size 2 \\ --dp-size 8 \\ --enable-dp-attention \\ --moe-a2a-backend deepep \\ --deepep-mode low_latency \\ --ep-size 8 \\ --moe-dense-tp-size 1 \\ --quantization w4afp8 \\ --kv-cache-dtype fp8_e4m3 \\ --mem-fraction-static 0.7 \\ --max-running-requests 32 \\ --cuda-graph-max-bs 16 \\ --context-length 105536 \\ --chunked-prefill-size 4096 \\ --disable-custom-all-reduce \\ --dist-init-addr \"${MASTER_IP}:${DIST_PORT}\" \\ --nnodes \"$NNODES\" \\ --node-rank \"$NODE_RANK\" \\ --host 0.0.0.0 \\ --port \"$API_PORT\" \\ --enable-metrics \\ 2>&1 | tee \"$LOG\"",
"url": "https://github.com/sgl-project/sglang/issues/29942",
"timestamp": "2026-08-12T16:29:29Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [],
"author": "zmtttt",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:issue:29998",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] runai_streamer load-format silently corrupts GLM-5.2 (glm_moe_dsa) weights under TP8 — loads clean, generates token-0 garbage",
"text": "### Describe the bug `--load-format runai_streamer` with `zai-org/GLM-5.2` (`GlmMoeDsaForCausalLM` / `glm_moe_dsa`) under TP8 **loads without any error but produces fully corrupted generations** — multilingual token soup on plain prompts, and long runs of `!` (token 0) especially when tools are present in the request. Removing `--load-format runai_streamer` (default loader), with every other flag identical, produces correct output. This looks like the same class as #22701 / #22522 / #22715 (Run:AI streamer reuses staging buffers across iterator batches; weight-load code holding tensor references across batches gets silently overwritten — the `!`/token-0 spam signature matches #22701 exactly). The image I tested **already contains the `clone()` fix in `deepseek_common/deepseek_weight_loader.py`**, so I suspect the newer GLM-5.2 DSA-indexer / MTP (`nextn`) weight-load paths repeat the stale-reference pattern and are not covered by the existing fixes. ### Reproduction ```bash # corrupted (loads clean, generates garbage): python3 -m sglang.launch_server \\ --model-path zai-org/GLM-5.2 --tp 8 --trust-remote-code \\ --load-format runai_streamer \\ --tool-call-parser glm47 --reasoning-parser glm45 \\ --context-length 131072 --mem-fraction-static 0.85 # correct (only delta = drop --load-format): python3 -m sglang.launch_server \\ --model-path zai-org/GLM-5.2 --tp 8 --trust-remote-code \\ --tool-call-parser glm47 --reasoning-parser glm45 \\ --context-length 131072 --mem-fraction-static 0.85 ``` Weights are a local HF-cache snapshot (no object storage involved); load completes normally in both cases (`Load weight end ... type=GlmMoeDsaForCausalLM, mem usage=177.36 GB` per rank). With the streamer, `\"What is 2+2? Answer briefly.\"` yields e.g.: ``` ' wherein hubby большинстве yelled autoridades cách Crafting Half resultMap的自己_bulk:\",\\n倪ॐicanoouce ascending...' ``` and tool-bearing requests degenerate to `'!!!!!!!!!!...'` (token 0) with the tool parser then logging `Tool call parsing error: Input is a zero-length, empty document`. Without the streamer, the same requests produce correct reasoning, answers, and parsed `tool_calls`. ### Environment - SGLang: `lmsysorg/sglang:latest` pulled 2026-07-02 (reports `sglang 0.5.14`) - Model: `zai-org/GLM-5.2` (BF16, `glm_moe_dsa`), local HF cache - Hardware: 8× NVIDIA B300 (Blackwell Ultra, sm_103), single node, TP8 - CUDA driver 595 / CUDA 13.2 - Auto-selected at runtime: `attention_backend=dsa` (`dsa_prefill_backend=trtllm`, `dsa_decode_backend=trtllm`), `kv_cache_dtype=fp8_e4m3`, `moe_runner_backend=flashinfer_trtllm` Happy to run patched wheels / additional repros on this hardware if useful.",
"url": "https://github.com/sgl-project/sglang/issues/29998",
"createdAt": "2026-07-03T02:47:41Z",
"updatedAt": "2026-08-13T09:16:16Z",
"timestamp": "2026-08-13T09:16:16Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "vshah1016",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:30595",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "SGLang not support hidden_size=4096, moe_intermediate_size=2048 MoE",
"text": "模型：DeepSeek‑V4‑Flash 关键配置：hidden_size=4096, moe_intermediate_size=2048, n_routed_experts=256, num_experts_per_tok=6 错误：AssertionError: Hidden size mismatch in fused_moe_triton python -m sglang.launch_server --model-path /home/ds4-flash --tp 8 --trust-remote-code --context-length 4096 --cuda-graph-backend-decode disabled --cuda-graph-backend-prefill disabled --moe-runner-backend triton [2026-07-09 03:41:08 TP0] Entering DeepGEMM JIT Pre-Compile session. It may take a long time (typically 10-20 mins) if you have not run `sglang.compile_deep_gemm`. It is recommended to run `sglang.compile_deep_gemm` with same args as `sglang.launch_server` for pre-compilation to reduce the overhead if you have not run it before. For example: `python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code` [2026-07-09 03:41:08 TP0] Try DeepGEMM JIT Compiling for N=1536, K=4096, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run `python3 -m sglang.compile_deep_gemm`. [2026-07-09 03:41:10 TP0] Required memory for warmup: 0.115234375GB, Available memory: 13.732421875GB DeepGEMM warmup: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16384/16384 [00:07<00:00, 2212.99it/s] [2026-07-09 03:41:18 TP0] Entering DeepGEMM JIT Pre-Compile session. It may take a long time (typically 10-20 mins) if you have not run `sglang.compile_deep_gemm`. It is recommended to run `sglang.compile_deep_gemm` with same args as `sglang.launch_server` for pre-compilation to reduce the overhead if you have not run it before. For example: `python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code` [2026-07-09 03:41:18 TP0] Try DeepGEMM JIT Compiling for N=4096, K=1024, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run `python3 -m sglang.compile_deep_gemm`. [2026-07-09 03:41:18 TP0] Required memory for warmup: 0.14453125GB, Available memory: 13.73046875GB DeepGEMM warmup: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16384/16384 [00:05<00:00, 3043.28it/s] [2026-07-09 03:41:24 TP0] Entering DeepGEMM JIT Pre-Compile session. It may take a long time (typically 10-20 mins) if you have not run `sglang.compile_deep_gemm`. It is recommended to run `sglang.compile_deep_gemm` with same args as `sglang.launch_server` for pre-compilation to reduce the overhead if you have not run it before. For example: `python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --trust-remote-code` [2026-07-09 03:41:24 TP0] Try DeepGEMM JIT Compiling for N=512, K=4096, num_groups=1 with all Ms. It only takes a little time (typically 1 sec) if you have run `python3 -m sglang.compile_deep_gemm`. [2026-07-09 03:41:24 TP0] Required memory for warmup: 0.080078125GB, Available memory: 13.69140625GB DeepGEMM warmup: 0%| | 0/16384 [00:00` in `server.rs:221`, which uses this enum for validation. ### Python side (sglang protocol.py) The Python `RESPONSE_TOOL_TYPES` has **12 variants**: ```python # sglang/srt/entrypoints/openai/protocol.py:1315 RESPONSE_TOOL_TYPES = Literal[ \"function\", \"web_search\", \"web_search_preview\", \"code_interpreter\", \"file_search\", \"image_generation\", \"computer_use_preview\", \"local_shell\", \"mcp\", \"custom\", # ← exists in Python, missing in Rust \"namespace\", # ← exists in Python, missing in Rust \"tool_search\", # ← exists in Python, missing in Rust ] ``` ### Mismatch | Variant | Python | Rust (openai-protocol v1.0.0) | |----------|--------|-------------------------------| | function | ✅ | ✅ | | web_search_preview | ✅ | ✅ | | code_interpreter | ✅ | ✅ | | mcp | ✅ | ✅ | | custom | ✅ | ❌ | | namespace | ✅ | ❌ | | local_shell | ✅ | ❌ | | file_search | ✅ | ❌ | | image_generation | ✅ | ❌ | | computer_use_preview | ✅ | ❌ | | web_search | ✅ | ❌ | | tool_search | ✅ | ❌ | ## Impact - **OpenAI Codex CLI** cannot use SGLang through the router — Codex sends built-in tools with `type: \"custom\"` (e.g., shell, apply_patch, unified_exec) - Any client that sends non-standard tool types through the Responses API will fail when using the router - Direct connection to SGLang server works fine (no router in the path) ## Expected behavior The router should either: 1. **Accept all tool types** that the Python SGLang server accepts (sync the Rust enum with Python `RESPONSE_TOOL_TYPES`) 2. **Transparently proxy** the request body without deserializing/validating tool types (since the backend server does its own validation) ## Suggested fix ### Option A: Update openai-protocol crate Add missing variants to the `ResponseToolType` enum in the `openai-protocol` crate and publish a new version: ```rust enum ResponseToolType { Function, WebSearch, WebSearchPreview, CodeInterpreter, FileSearch, ImageGeneration, ComputerUsePreview, LocalShell, Mcp, Custom, // ← add Namespace, // ← add ToolSearch, // ← add } ``` ### Option B: Use `#[serde(other)]` for unknown variants Add a catch-all variant to gracefully handle unknown tool types: ```rust #[serde(other)] Unknown, ``` ### Option C: Skip tool validation in router Since the router is a proxy, it should forward the request body as-is without validating tool types. The backend SGLang server already validates tools. ## Environment details - SGLang base image: `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi30x-20260708` - Router config: `--policy cache_aware --worker-urls http://worker1:30000 http://worker2:30000 ...` - Codex CLI version: 0.143.0 - Model: GLM-5.2-FP8 ## Workaround Bypass the router and connect directly to a SGLang server instance: ```toml # ~/.codex/config.toml [model_providers.custom] base_url = \"http://:30000/v1\" # direct, not via router ``` This loses the router's cache_aware prefix routing and load balancing benefits.",
"url": "https://github.com/sgl-project/sglang/issues/30781",
"createdAt": "2026-07-10T13:41:49Z",
"updatedAt": "2026-08-13T11:06:52Z",
"timestamp": "2026-08-13T11:06:52Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [],
"author": "tanguofu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:30928",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[RFC] Position-Independent KV Cache Reuse for Agentic/RAG Workloads",
"text": "### **Summary** RadixAttention reuses KV cache only when a request's prefix is byte-identical and at the same absolute offset as something already cached. In agentic/RAG traffic shared system prompts, tool schemas, retrieved documents, and multi-turn history routinely put identical content at different offsets across requests. This causes a guaranteed cache miss even though the content itself hasn't changed. _Three recent approaches tackle this differently. Wanted to check if this is on anyone's radar before digging further._ - **Irminsul** ([arXiv:2605.05696](https://arxiv.org/abs/2605.05696)) (sglang-targeted but no code). Exploits MLA's factorization into a position-free latent ($$c_{KV}$$) and a small rotatable slice ($$k_r$$). The idea is that reusing a cached span at a new offset only requires a closed-form \"delta-rotation\" on the small rotatable slice. it is MLA-specific, though the same split exists in any partial-RoPE model (rotary_dim < head_dim), not just MLA, so the mechanism plausibly generalizes further than the paper states explicitly. - **MiniPIC** ([arXiv:2606.13126](https://arxiv.org/abs/2606.13126)) ( vLLM). Stores unrotated K, defers RoPE to attention-read-time. More general ( RoPE models) but touches every attention backend, conflicts with fused write-path kernels, and needs new per-token position plumbing through CUDA-graph-captured decode. Code exists ([IBM/vllm](https://github.com/IBM/vllm), traced to the actual authors' commits). It is worth knowing that it went through two implementation strategies, and the first one ([PR #88](https://github.com/IBM/vllm/pull/88)) shipped with an acknowledged accuracy bug from concurrent requests reusing the same span at different positions, serious enough that they abandoned it for a different strategy rather than patch it. Real, useful precedent for anyone attempting this class of feature. In principle this can be applied on top of sglang, but requires substantial engineering. I wonder if it is worthwhile. - **COMB** ([arXiv:2602.01519](https://arxiv.org/abs/2602.01519)) — a third, different category: trains a dedicated encoder component into the model rather than changing only the inference engine. Claims 51-94% TTFT reduction, 3x throughput, tested on DeepSeek-V2-Lite-Chat. Code released: [github.com/shijuzhao/Comb](https://github.com/shijuzhao/Comb). ### **What's been looked into so far** Confirmed against current sglang code that RoPE is baked into K at write time and the radix-cache hash chain is explicitly position-coupled. today's architecture can't do any of the above without new work. All three are recent (Feb-June 2026) with limited independent validation, so there are potential bugs, I searched existing issues/PRs and didn't find prior discussion of this specific angle. Closest related work is #27574 / #24656 (agent-aware/session KV cache hints), but that's a different problem targeting cache locality/lifecycle, not reuse across a shifted position with identical content. **Questions for the community** Has anyone evaluated any of these for sglang already, even informally? Is there a reason this hasn't come up that isn't obvious from the papers? Would a narrow, opt-in delta-rotation implementation (gated to MLA/partial-RoPE models, feature-flagged behind a --radix-cache-backend variant) be a welcome contribution, or better suited as an external/experimental extension? Happy to share more detailed technical notes if there's interest.",
"url": "https://github.com/sgl-project/sglang/issues/30928",
"createdAt": "2026-07-12T10:09:58Z",
"updatedAt": "2026-08-13T15:11:24Z",
"timestamp": "2026-08-13T15:11:24Z",
"metrics": {
"reactions": 4,
"comments": 12
},
"labels": [],
"author": "krakhit",
"state": "open",
"assignees": [
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:issue:31310",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] fa3 backend slow with mla page-size 64 for H20",
"text": "### Checklist - [ ] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [ ] Please use English. Otherwise, it will be closed. ### Describe the bug I tested some our inner model, and found performance drop with page64 in some situation, and I reproduce this problem by a minimal test with fa3. ### Reproduction reproduce script:bench_fa3_paged_mla.py ``` import argparse import gc import json import math import re import statistics from pathlib import Path import torch from sgl_kernel.flash_attn import flash_attn_with_kvcache DTYPE = torch.bfloat16 QK_ROPE_HEAD_DIM = 64 V_HEAD_DIM = 512 def parse_args(): parser = argparse.ArgumentParser( description=\"Reproduce FA3 paged-KV TMA dispatch for absorbed MLA.\" ) parser.add_argument(\"--page-sizes\", nargs=\"+\", type=int, default=[1, 16, 32, 64]) parser.add_argument(\"--seqlen-q\", type=int, default=8192) parser.add_argument(\"--seqlen-k\", type=int, default=8270) parser.add_argument(\"--num-q-heads\", type=int, default=8) parser.add_argument( \"--pool-factor\", type=int, default=128, help=\"Physical KV pages per active page. 128 approximates the serving trace.\", ) parser.add_argument(\"--warmup\", type=int, default=3) parser.add_argument(\"--iters\", type=int, default=20) parser.add_argument(\"--seed\", type=int, default=0) parser.add_argument( \"--contiguous-pages\", action=\"store_true\", help=\"Use the first physical pages instead of a randomized page table.\", ) parser.add_argument( \"--force-non-tma-page-sizes\", nargs=\"*\", type=int, default=[], help=( \"Also benchmark these page sizes with a zero cache_leftpad tensor. \" \"The pointer disables paged-KV TMA without changing attention semantics.\" ), ) parser.add_argument( \"--trace-dir\", type=Path, help=\"Export one torch-profiler trace per page size and print the selected mode.\", ) return parser.parse_args() def build_paged_cache(k_tokens, v_tokens, page_size, pool_factor, random_pages): seqlen_k = k_tokens.shape[0] num_active_pages = math.ceil(seqlen_k / page_size) num_physical_pages = num_active_pages * pool_factor padded_tokens = num_active_pages * page_size k_padded = torch.zeros( padded_tokens, 1, QK_ROPE_HEAD_DIM, dtype=DTYPE, device=\"cuda\", ) v_padded = torch.zeros( padded_tokens, 1, V_HEAD_DIM, dtype=DTYPE, device=\"cuda\", ) k_padded[:seqlen_k].copy_(k_tokens) v_padded[:seqlen_k].copy_(v_tokens) k_cache = torch.empty( num_physical_pages, page_size, 1, QK_ROPE_HEAD_DIM, dtype=DTYPE, device=\"cuda\", ) v_cache = torch.empty( num_physical_pages, page_size, 1, V_HEAD_DIM, dtype=DTYPE, device=\"cuda\", ) if random_pages: page_ids = torch.randperm(num_physical_pages, device=\"cuda\")[ :num_active_pages ] else: page_ids = torch.arange(num_active_pages, device=\"cuda\") k_cache.index_copy_( 0, page_ids, k_padded.view(num_active_pages, page_size, 1, QK_ROPE_HEAD_DIM), ) v_cache.index_copy_( 0, page_ids, v_padded.view(num_active_pages, page_size, 1, V_HEAD_DIM), ) page_table = page_ids.to(torch.int32).unsqueeze(0) return k_cache, v_cache, page_table def find_fa3_mode(trace_path): with trace_path.open() as f: trace = json.load(f) kernels = [ event for event in trace[\"traceEvents\"] if event.get(\"cat\") == \"kernel\" and \"CollectiveMainloopFwdSm90\" in event.get(\"name\", \"\") and \"cute::C<64>, cute::C<64>, cute::C<64> >, 512\" in event[\"name\"] ] if not kernels: return \"unknown\", 0.0 kernel = max(kernels, key=lambda event: event.get(\"dur\", 0.0)) flags_match = re.search( r\"cutlass::arch::Sm90,\\s*((?:true|false)(?:,\\s*(?:true|false)){12})\", kernel[\"name\"], ) if flags_match is None: return \"unknown\", kernel[\"dur\"] / 1000.0 flags = [flag.strip() for flag in flags_match.group(1).split(\",\")] return (\"NonTMA\" if flags[4] == \"true\" else \"TMA\"), kernel[\"dur\"] / 1000.0 def profile_once(fn, trace_path): with torch.profiler.profile( activities=[ torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA, ], record_shapes=True, ) as prof: fn() torch.cuda.synchronize() prof.export_chrome_trace(str(trace_path)) return find_fa3_mode(trace_path) def benchmark(fn, warmup, iters): for _ in range(warmup): fn() torch.cuda.synchronize() starts = [torch.cuda.Event(enable_timing=True) for _ in range(iters)] ends = [torch.cuda.Event(enable_timing=True) for _ in range(iters)] for start, end in zip(starts, ends): start.record() fn() end.record() torch.cuda.synchronize() times = sorted(start.elapsed_time(end) for start, end in zip(starts, ends)) return { \"mean_ms\": statistics.fmean(times), \"p50_ms\": times[len(times) // 2], \"p90_ms\": times[min(len(times) - 1, int(len(times) * 0.9))], \"min_ms\": times[0], \"max_ms\": times[-1], } def main(): args = parse_args() if not torch.cuda.is_available() or torch.cuda.get_device_capability()[0] != 9: raise RuntimeError(\"This benchmark requires an SM90 Hopper GPU and FA3.\") if args.seqlen_k < args.seqlen_q: raise ValueError(\"seqlen-k must be greater than or equal to seqlen-q\") if args.pool_factor < 1: raise ValueError(\"pool-factor must be at least 1\") torch.manual_seed(args.seed) torch.cuda.manual_seed_all(args.seed) q_rope = torch.randn( args.seqlen_q, args.num_q_heads, QK_ROPE_HEAD_DIM, dtype=DTYPE, device=\"cuda\", ) q_nope = torch.randn( args.seqlen_q, args.num_q_heads, V_HEAD_DIM, dtype=DTYPE, device=\"cuda\", ) k_tokens = torch.randn( args.seqlen_k, 1, QK_ROPE_HEAD_DIM, dtype=DTYPE, device=\"cuda\" ) v_tokens = torch.randn(args.seqlen_k, 1, V_HEAD_DIM, dtype=DTYPE, device=\"cuda\") cache_seqlens = torch.tensor([args.seqlen_k], dtype=torch.int32, device=\"cuda\") cu_seqlens_q = torch.tensor( [0, args.seqlen_q], dtype=torch.int32, device=\"cuda\" ) out_buffer = torch.empty_like(q_nope) if args.trace_dir is not None: args.trace_dir.mkdir(parents=True, exist_ok=True) print( f\"device={torch.cuda.get_device_name()} q={tuple(q_rope.shape)} \" f\"qv={tuple(q_nope.shape)} seqlen_k={args.seqlen_k} \" f\"pool_factor={args.pool_factor}\" ) print(\"case mode mean_ms p50_ms p90_ms min_ms max_ms max_diff\") reference = None for page_size in args.page_sizes: k_cache, v_cache, page_table = build_paged_cache( k_tokens, v_tokens, page_size, args.pool_factor, random_pages=not args.contiguous_pages, ) variants = [False] if page_size in args.force_non_tma_page_sizes: variants.append(True) for force_non_tma in variants: cache_leftpad = ( torch.zeros(1, dtype=torch.int32, device=\"cuda\") if force_non_tma else None ) def run_attention(): return flash_attn_with_kvcache( q=q_rope, qv=q_nope, k_cache=k_cache, v_cache=v_cache, page_table=page_table, cache_seqlens=cache_seqlens, cache_leftpad=cache_leftpad, cu_seqlens_q=cu_seqlens_q, max_seqlen_q=args.seqlen_q, softmax_scale=1.0 / math.sqrt(192.0), causal=True, num_splits=1, ver=3, out=out_buffer, ) stats = benchmark(run_attention, args.warmup, args.iters) mode = \"not-profiled\" case = f\"page{page_size}\" + (\"-forced\" if force_non_tma else \"\") if args.trace_dir is not None: trace_path = args.trace_dir / f\"fa3_{case}.trace.json\" mode, profiled_ms = profile_once(run_attention, trace_path) mode = f\"{mode}:{profiled_ms:.2f}ms\" output = run_attention().clone() torch.cuda.synchronize() if reference is None: reference = output max_diff = 0.0 else: max_diff = (output - reference).abs().max().item() print( f\"{case:<13} {mode:<12} \" f\"{stats['mean_ms']:>8.3f} {stats['p50_ms']:>8.3f} \" f\"{stats['p90_ms']:>8.3f} {stats['min_ms']:>8.3f} \" f\"{stats['max_ms']:>8.3f} {max_diff:>10.6f}\" ) del output del k_cache, v_cache, page_table gc.collect() torch.cuda.empty_cache() if __name__ == \"__main__\": main() ``` Case | Kernel | rt | divide page1 -- | -- | -- | -- page1 | NonTMA | 4.533 ms | 1.00x page16 | NonTMA | 4.535 ms | 1.00x page32 | NonTMA | 4.530 ms | 1.00x page64 | TMA | 35.005 ms | 7.72x I think this is caused by fa3 tma kvcache load by analyzing selected kernel name ### Environment cuda13 and torch2.11.0+cu130, with latest sglang-kernel==0.4.4 ``` $ python3 -m sglang.check_env Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] CUDA available: True GPU 0: NVIDIA H20 GPU 0 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 13.0, V13.0.48 CUDA Driver Version: 535.183.06 PyTorch: 2.11.0+cu130 sglang: 0.5.16.dev6+g3397f79e7 sglang-kernel: 0.4.4 flashinfer_python: 0.6.12 flashinfer_cubin: 0.6.12 flashinfer_jit_cache: Module Not Found triton: 3.6.0 transformers: 5.6.0 torchao: 0.17.0+cu130 numpy: 1.26.4 aiohttp: 3.13.5 fastapi: 0.136.1 huggingface_hub: 1.14.0 interegular: 0.3.3 modelscope: 1.36.3 orjson: 3.11.9 outlines: 0.1.11 packaging: 26.2 psutil: 7.2.2 pydantic: 2.13.4 python-multipart: 0.0.28 pyzmq: 27.1.0 uvicorn: 0.46.0 uvloop: 0.22.1 vllm: Module Not Found xgrammar: 0.2.0 openai: 2.6.1 tiktoken: 0.12.0 anthropic: 0.101.0 litellm: Module Not Found decord2: 2.1.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE SYS SYS 0-47,96-143 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE SYS SYS 0-47,96-143 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE SYS SYS 0-47,96-143 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE PIX SYS SYS 0-47,96-143 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS PIX NODE 48-95,144-191 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS NODE NODE 48-95,144-191 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS NODE PIX 48-95,144-191 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS NODE NODE 48-95,144-191 1 N/A NIC0 NODE PIX NODE NODE SYS SYS SYS SYS X NODE SYS SYS NIC1 NODE NODE NODE PIX SYS SYS SYS SYS NODE X SYS SYS NIC2 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS X NODE NIC3 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 ulimit soft: 1048576 ```",
"url": "https://github.com/sgl-project/sglang/issues/31310",
"createdAt": "2026-07-15T09:20:46Z",
"updatedAt": "2026-08-13T14:10:21Z",
"timestamp": "2026-08-13T14:10:21Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "strgrb",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:31766",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] fd exhaustion on the prefill side",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug fd exhaustion on the prefill side. The crash happens in `CommonKVManager._connect()` (common/conn.py:505) when creating the ZMQ monitor socket via `sock.get_monitor_socket()`. Each cached endpoint holds TWO fds (PUSH socket + monitor PAIR socket). Under traffic spikes, the number of cached endpoints grows and fd count rises until the process hits the fd limit, at which point `context.socket(zmq.PAIR)` raises `ZMQError: Too many open files`. The transfer_worker thread then dies (conn.py:1616 raises RuntimeError), leaving all in-flight transfers stuck after which subsequent requests also fail. ## Traceback (innermost failure: monitor socket creation) Exception in thread Thread-14 (transfer_worker): Traceback (most recent call last): File \"/local-ssd/pv0/sglang/python/sglang/srt/disaggregation/mooncake/conn.py\", line 1561, in transfer_worker self.sync_status_to_decode_endpoint( File \"/local-ssd/pv0/sglang/python/sglang/srt/disaggregation/mooncake/conn.py\", line 1354, in sync_status_to_decode_endpoint self._connect(na.to_tcp(), is_ipv6=na.is_ipv6).send_multipart( File \"/local-ssd/pv0/sglang/python/sglang/srt/disaggregation/common/conn.py\", line 505, in _connect self._monitor_cache[endpoint] = sock.get_monitor_socket( File \"/usr/local/lib/python3.12/dist-packages/zmq/sugar/socket.py\", line 1109, in get_monitor_socket self._monitor_socket = self.context.socket(zmq.PAIR) File \"/usr/local/lib/python3.12/dist-packages/zmq/sugar/context.py\", line 352, in socket socket_class( # set PYTHONTRACEMALLOC=2 to get the calling frame File \"/usr/local/lib/python3.12/dist-packages/zmq/sugar/socket.py\", line 162, in __init__ super().__init__( File \"zmq/backend/cython/_zmq.py\", line 740, in zmq.backend.cython._zmq.Socket.__init__ raise ZMQError() zmq.error.ZMQError: Too many open files During handling of the above exception, another exception occurred: Traceback (most recent call last): File \"/usr/lib/python3.12/threading.py\", line 1073, in _bootstrap_inner self.run() File \"/usr/lib/python3.12/threading.py\", line 1010, in run self._target(*self._args, **self._kwargs) File \"/local-ssd/pv0/sglang/python/sglang/srt/disaggregation/mooncake/conn.py\", line 1616, in transfer_worker raise RuntimeError( RuntimeError: Transfer thread failed because of Too many open files. Prefill instance with bootstrap_port=8998 is dead. ### Reproduction --disable-overlap-schedule --tp-size 16 --disaggregation-mode prefill --tool-call-parser glm47 --reasoning-parser glm45 --grammar-backend xgrammar --speculative-algo EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 --enable-hierarchical-cache --hicache-ratio 6 --hicache-io-backend kernel --tokenizer-worker-num 8 --enable-dp-lm-head --enable-cache-report --moe-a2a-backend deepep --deepep-mode normal --enable-dp-attention --dp-size 4 ### Environment Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H800 GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 13.0, V13.0.88 CUDA Driver Version: 535.261.03 PyTorch: 2.11.0+cu130 sglang: 0.5.14 sglang-kernel: 0.4.4 flashinfer_python: 0.6.12 flashinfer_cubin: 0.6.12 flashinfer_jit_cache: 0.6.12+cu130 triton: 3.6.0 transformers: 5.9.0 torchao: 0.17.0+cu130 numpy: 2.3.5 aiohttp: 3.14.1 fastapi: 0.138.1 huggingface_hub: 1.21.0 interegular: 0.3.3 modelscope: 1.37.1 orjson: 3.11.9 outlines: 0.1.11 packaging: 26.2 psutil: 7.2.2 pydantic: 2.13.4 python-multipart: 0.0.32 pyzmq: 27.1.0 uvicorn: 0.49.0 uvloop: 0.22.1 vllm: Module Not Found xgrammar: 0.2.1 openai: 2.6.1 tiktoken: 0.13.0 anthropic: 0.112.0 litellm: Module Not Found torchcodec: 0.11.1+cu130 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 SYS PIX PHB PHB PHB SYS SYS SYS SYS 0-89 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 SYS PHB PIX PHB PHB SYS SYS SYS SYS 0-89 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 SYS PHB PHB PIX PHB SYS SYS SYS SYS 0-89 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 SYS PHB PHB PHB PIX SYS SYS SYS SYS 0-89 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS PIX PHB PHB PHB 90-179 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS PHB PIX PHB PHB 90-179 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS PHB PHB PIX PHB 90-179 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS PHB PHB PHB PIX 90-179 1 N/A NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS NIC1 PIX PHB PHB PHB SYS SYS SYS SYS SYS X PHB PHB PHB SYS SYS SYS SYS NIC2 PHB PIX PHB PHB SYS SYS SYS SYS SYS PHB X PHB PHB SYS SYS SYS SYS NIC3 PHB PHB PIX PHB SYS SYS SYS SYS SYS PHB PHB X PHB SYS SYS SYS SYS NIC4 PHB PHB PHB PIX SYS SYS SYS SYS SYS PHB PHB PHB X SYS SYS SYS SYS NIC5 SYS SYS SYS SYS PIX PHB PHB PHB SYS SYS SYS SYS SYS X PHB PHB PHB NIC6 SYS SYS SYS SYS PHB PIX PHB PHB SYS SYS SYS SYS SYS PHB X PHB PHB NIC7 SYS SYS SYS SYS PHB PHB PIX PHB SYS SYS SYS SYS SYS PHB PHB X PHB NIC8 SYS SYS SYS SYS PHB PHB PHB PIX SYS SYS SYS SYS SYS PHB PHB PHB X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 Hypervisor vendor:: KVM ulimit soft: 1048576",
"url": "https://github.com/sgl-project/sglang/issues/31766",
"createdAt": "2026-07-20T08:19:19Z",
"updatedAt": "2026-08-13T10:52:39Z",
"timestamp": "2026-08-13T10:52:39Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "dongyibo",
"state": "open",
"assignees": [
"whybeyoung"
]
},
{
"id": "github:sgl-project/sglang:issue:31833",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] NemotronH --mamba-scheduler-strategy extra_buffer accuracy drop on AIME26 (Nemotron-3-Super-120B)",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Bug Feature Tracking for NemotronH `--mamba-scheduler-strategy extra_buffer`: | mamba-scheduler-strategy | attention-backend | AIME26 pass@1 (avg-of-16, n=30x16) | |---|---|---| | `no_buffer` (default) | (auto) | 84.38% ± 3.59% | | `extra_buffer` | `trtllm_mha` | 85.21% ± 3.44% — **overlaps baseline, no regression observed (see update below)** | ### Describe the bug Serving `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16` (NemotronH, hybrid Mamba2 + attention) with `--mamba-scheduler-strategy extra_buffer` produces a visible accuracy drop on AIME26 vs. the default `no_buffer` strategy, with everything else (model, TP/EP layout, parsers, eval config) held identical. `extra_buffer` is meant to be a pure scheduling/caching optimization (overlap schedule + mamba-state radix-cache branching) with no expected effect on generation quality. Root cause is suspected to be in the mamba-state tracking/checkpointing math in `hybrid_linear_attn_backend.py::_init_track_ssm_indices`, used by `Mamba2AttnBackend` for this model's SSM layers: `_init_track_ssm_indices` computes a per-request \"how many packed intermediate `h` states did this request produce\" count as `extend_seq_lens // mamba_cache_chunk_size` and cumulative-sums it across requests to find each request's offset into the packed `intermediate_states` tensor returned by `mamba_chunk_scan_combined`. That tensor is actually indexed by a **global physical chunk grid over the whole flattened extend batch** (`ssd_state_passing.py`'s `_state_passing_fwd_kernel` loops `for c in range(nchunks)` over the *shared* grid, writing one state per physical chunk regardless of request boundaries), not by a per-request-local contiguous count. The cumulative local-count offset is only correct if every earlier request in the same tracked prefill batch has a chunk-aligned length — not guaranteed in general. For this model (`chunk_size=128` in `config.json`, i.e. native `mamba_chunk_size=128`) served at `--page-size 64` (the effective default when radix cache + `extra_buffer` + `trtllm_mha` are combined), half of all page-aligned tracked prefill lengths are *not* chunk-128-aligned, so the affected `_init_track_ssm_indices` \"unaligned\" branch fires routinely — this is not an edge case. The net effect: a wrong (but same-shaped) SSM state gets read from `intermediate_states` and written into a request's mamba radix-cache slot whenever this happens, silently corrupting later prefix-cache-hit continuations for that node — no crash, no assertion, just wrong logits from that point on. This matches the observed AIME26 score degradation: `extra_buffer` should be numerically a no-op vs. `no_buffer`, and any drop attributable to it (beyond noise/seed variance) is evidence of this corruption path. _(Score for the `extra_buffer` run is pending — will update this issue once the run completes.)_ ### Reproduction Server (baseline — accuracy OK, `no_buffer` is the resolved default): ``` python3 -m sglang.launch_server \\ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \\ --trust-remote-code \\ --tp 4 \\ --ep 4 \\ --reasoning-parser nemotron_3 \\ --tool-call-parser qwen3_coder ``` Server (suspected buggy — `extra_buffer` mamba radix-cache strategy): ``` python3 -m sglang.launch_server \\ --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 \\ --trust-remote-code \\ --tp 4 \\ --ep 4 \\ --reasoning-parser nemotron_3 \\ --tool-call-parser qwen3_coder \\ --mamba-scheduler-strategy extra_buffer \\ --attention-backend trtllm_mha ``` Eval (identical for both): ``` sgl-eval run aime26 --base-url http://localhost:30000/v1 --n-repeats 16 --max-tokens 64000 --temperature 1.0 --top-p 0.95 --num-threads 416 ``` Baseline (`no_buffer`) result: ``` == aime26 == 30 examples x 16 repeats | 1384.2s | 7824 tok/s | 10.8M tokens * pass@1[avg-of-16] = 84.38% +/- 3.59% (SEM 0.90%) pass@16 = 93.33% majority@16 = 90.00% no_answer = 11.88% [warn: consider --max-tokens] stop_rate = 88.12% truncated_rate = 11.88% [warn: hitting max_tokens] error_rate = 0.00% ``` `extra_buffer` result: **TBD** ### Environment ``` Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA B300 SXM6 AC GPU 0,1,2,3,4,5,6,7 Compute Capability: 10.3 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 13.0, V13.0.88 CUDA Driver Version: 610.43.02 PyTorch: 2.11.0+cu130 sglang: 0.5.15.dev1093+ge856eae92 sglang-kernel: 0.4.5 flashinfer_python: 0.6.14 flashinfer_cubin: 0.6.14 flashinfer_jit_cache: 0.6.14+cu130 triton: 3.6.0 transformers: 5.12.1 ```",
"url": "https://github.com/sgl-project/sglang/issues/31833",
"createdAt": "2026-07-20T22:12:15Z",
"updatedAt": "2026-08-12T17:31:07Z",
"timestamp": "2026-08-12T17:31:07Z",
"metrics": {
"reactions": 0,
"comments": 10
},
"labels": [
"bug"
],
"author": "b8zhong",
"state": "closed",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:issue:32607",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Kimi K3 Roadmap",
"text": "## Links - Cookbook: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3 - Day0 PR: https://github.com/sgl-project/sglang/pull/32541 - Blog: https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support - DSpark: https://huggingface.co/RadixArk/Kimi-K3-DSpark - Bug Tracking: #32970 ## Roadmap ### Kernel Optimization - [ ] Continue optimize performance in bs=1/64 and TP8(GB300) - [ ] Integrate custom KDA Prefill kernel and make it compatible with RadixCache - [ ] Refactor to generalize kernel supporting different parallelism. - [ ] Support GEMM Allreduce fusion - [ ] KDA MTP decode - [ ] BF16 gemm / gemm fusion with SiTU - [ ] Optimize kernels in TP16xhopper ### Parallelism - Decode Context Parallelism (DCP) Enhancement - [ ] Support more optimized A2A backends for DCP on Blackwells (e.g how to enable fi_a2a on Bs) - [ ] Kernel fusion / optimization in DCP - [x] Investigate the scalability of DCP x DP Attention in WideEP settings - [ ] Implement DCP between multi nodes connected by RDMA - [ ] Support DCP + Hicache L3 - [ ] Support DCP + DSpark x {Hicache, PD Disagg} - Chunked Pipeline Parallelism Enhancement - [ ] Support PP + Spec + PD Disagg - [ ] Integrate [MoonEP](https://github.com/MoonshotAI/MoonEP) for better all-to-all performance #33249 ### Extending Unified Memory Compatibility - [ ] Attention backends - [ ] Speculative Decoding - [ ] PD disaggregation - [ ] Hi-cache ### DSpark - [x] Enhance its capability on longer context https://huggingface.co/RadixArk/Kimi-K3-DSpark ### Code Cleaning - [x] Merge Day0 PR #32541 - [x] Merge DeepGemm patch into [sglang fork][(https://github.com/sgl-project/DeepGEMM/tree/dev) and upgrade sgl-deep-gemm wheel https://github.com/sgl-project/DeepGEMM/pull/67 - [x] Upgrade Flashinfer after the cutedsl+dcp patch (https://github.com/flashinfer-ai/flashinfer/pull/4178) and trtllm moe patch (https://github.com/flashinfer-ai/flashinfer/pull/4180) has been included #33997 - [ ] Build a custom wheel for DeepEP, and include the patch in https://github.com/sgl-project/sglang/blob/bd51cab0c01075c9b73cc1dfae9d27b7b0c95619/docker/kimi_k3/apply_deepep_k3_patch.sh ## Slack Join [#kimi-k3] channel at https://slack.sglang.io/",
"url": "https://github.com/sgl-project/sglang/issues/32607",
"createdAt": "2026-07-28T04:04:41Z",
"updatedAt": "2026-08-13T09:56:36Z",
"timestamp": "2026-08-13T09:56:36Z",
"metrics": {
"reactions": 22,
"comments": 8
},
"labels": [],
"author": "Fridge003",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:32950",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Integrate `cute-dsl` backend of `mm_mxfp8` and other improvements",
"text": "### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Motivation Currently, we disable autotune for dense MXFP8 GEMM since it has a IMA issue with the default backend (CUTLASS). However, that currently leaves perf on the table for dense cases. With the SplitK feature, it should be quite a bit better even than autotuned. It should also be controlled by `--fp8-gemm-backend=cute-dsl`, similar to FP4. Benefits: - Perf. It could be even better than trtllm-gen backend for small problem sizes. - Extensibility: supporting future archs, like Rubin. Potentially it should be the default for SM100/SM103. We should **also consider not autotuning** it still, as the compilation for different tactics might take a lot of time. We can compare the perf under no autotune, vs autotuned. If it has some gaps in some edge cases, we can also make a PR to Flashinfer with improvements ### Further cleaning - Also we might delete the Triton implementation. As on SM120, we can use Flashinfer CUTLASS, which doesn't have the hang. ### Related resources SplitK MXFP8 from FI: https://github.com/flashinfer-ai/flashinfer/pull/3847",
"url": "https://github.com/sgl-project/sglang/issues/32950",
"createdAt": "2026-07-30T17:52:44Z",
"updatedAt": "2026-08-13T00:50:02Z",
"timestamp": "2026-08-13T00:50:02Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"blackwell"
],
"author": "b8zhong",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:32970",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "Kimi-K3 bug tracking",
"text": "- [x] #32907 - [ ] #32968 - [ ] #32960 - [x] #32855 - [ ] #34604 - [ ] #34260",
"url": "https://github.com/sgl-project/sglang/issues/32970",
"createdAt": "2026-07-30T22:36:30Z",
"updatedAt": "2026-08-12T18:43:11Z",
"timestamp": "2026-08-12T18:43:11Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"bug",
"kimi"
],
"author": "Fridge003",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:33289",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Multi-node TP rank-divergence deadlock: one rank wedges in NCCL proxy append (logits all-gather), peer idles at request broadcast — DeepSeek-V4 + DSpark on 2× DGX Spark (GB10)",
"text": "## Summary Running DeepSeek-V4-Flash-0731 with `--speculative-algorithm DSPARK` across **2 nodes (TP=2)**, the server intermittently deadlocks minutes to hours into normal OpenAI-compatible traffic. Captured live, the two scheduler ranks sit in a permanent split: - **Rank A (busy)**: spinning in the NCCL host enqueue path (`sched_yield` in `ncclLocalOpAppend` / `SaveProxy`) while posting an **all-gather of logits** — the proxy thread no longer consumes ops. - **Rank B (peer)**: idle, parked at `_broadcast_reqs_across_ranks` — it has exited the batch/step loop and will never join that collective. The unmatched collective wedges the NCCL proxy, the GPU stream never completes, and the scheduler watchdog eventually fires and kills the server. The failure shape matches the family described in #27221 (per-rank divergence in batch composition → mismatched collectives → NCCL deadlock), but with request timeouts **disabled** (both `SGLANG_REQ_WAITING_TIMEOUT` and `SGLANG_REQ_RUNNING_TIMEOUT` at default `-1`), so the divergence trigger here is a different, still-unidentified per-rank decision — all captured incidents have the DSpark verify path in the wedged rank's stack. ## Environment - **SGLang**: v0.5.16 (`lmsysorg/sglang:latest`, pulled 2026-08-02, digest `sha256:7b6a35df…`) - **Model**: `deepseek-ai/DeepSeek-V4-Flash-0731` (FP4 MoE + FP8 dense, bundled DSpark draft head) - **Hardware**: 2× NVIDIA DGX Spark (GB10, SM121, aarch64, 128 GB unified memory each), TP=2 across nodes - **Interconnect**: ConnectX-7 RoCEv2 between the nodes (12.9 GB/s measured NCCL all-reduce, zero error counters) - **NCCL**: 2.28.9 (same in the reference vLLM stack, which is stable on identical hardware) - **Key flags**: `--speculative-algorithm DSPARK --moe-runner-backend flashinfer_mxfp4 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.85 --cuda-graph-backend-decode=disabled --watchdog-timeout 120` - Local patches (bind-mounted, needed for DSpark on SM120/121): topk-192→512 index padding in `flash_mla_sm120.py` (the SM120 sparse-MLA kernel lacks a topk=192 instantiation — see flashinfer#3828) and a referenced-page bitmap for the pbs=256→64 page-split copy. Both wedges below also reproduce **without** these patches. ## Observed behavior 1. Server runs fine for minutes–hours, then a request stops making progress mid-decode. 2. No decode batches in the log; no error; one rank's scheduler pegged, the other idle. 3. Watchdog fires (`Scheduler watchdog timeout`), `kill_process_tree`, containers exit(0). 4. Repeatable across **five independent incidents**, under decode CUDA graphs both **on and off**. ### Incident 1 — worker (TP1) wedged, head (TP0) idle TP1 py-spy (native, active): ``` sched_yield ncclLocalOpAppend (proxy.cc:499) SaveProxy (proxy.cc:580) ncclProxySaveOp (proxy.cc:597) uploadProxyOps (enqueue.cc:1293) ncclLaunchKernelAfter_NoCuda (enqueue.cc:1670) doLaunches / groupLaunch / ncclGroupEndInternal (group.cc) ncclAllGather (collectives.cc:91) c10d::ProcessGroupNCCL::_allgather_base all_gather_into_tensor (distributed_c10d.py:4186) tensor_model_parallel_all_gather (distributed/communication_op.py:47) __call__ (triton_symm_mem_ag.py:515) _get_logits (logits_processor.py:710) forward (logits_processor.py:450) forward (deepseek_v4.py:2530) _execute_extend (runner/eager_runner.py:332) ... _forward_prepared_verify (dspark_components/dspark_verify.py:271) _forward_decode (dspark_components/dspark_worker_v2.py:590) run_batch (scheduler.py:3351) ``` TP0 py-spy (idle): ``` broadcast (torch/distributed/distributed_c10d.py:2974) broadcast_pyobj (utils/common.py:2175) _broadcast_reqs_across_ranks (scheduler_components/request_receiver.py:198) recv_requests (scheduler_components/request_receiver.py:90) event_loop_overlap (scheduler.py:1570) ``` ### Incident 2 — mirrored: head (TP0) wedged, worker (TP1) idle Identical frames, roles swapped: TP0 active in `ncclLocalOpAppend` during `ncclAllGather` via `_get_logits` → `_execute_extend` → DSpark verify; TP1 idle at `_broadcast_reqs_across_ranks`. Earlier incidents (before decode graphs were disabled) showed the downstream projection of the same wedge: the scheduler parked in `process_batch_result_decode → torch.cuda.synchronize` waiting for a stream that never completes. ## What we ruled out experimentally - **NCCL fabric / config**: all-reduce microbenchmarks clean (12.9 GB/s, 14 µs at 2 KB, no stalls); zero RoCE error/retransmit counters on both nodes; per-node RoCEv2 GID auto-resolved correctly; env matched to a known-good vLLM stack on the same pair (`CROSS_NIC=1`, RoCEv2, no plugins). Same NCCL 2.28.9 in both engines. `NCCL_PROTO=Simple` and `NCCL_LAUNCH_MODE=PARALLEL` — wedge still recurs. - **CUDA graphs**: recurs with `--cuda-graph-backend-decode=disabled` (prefill graphs are auto-disabled for this model anyway). - **Attention backend / kernels**: recurs on `dsv4` (flashinfer SM120 sparse-MLA) and on the triton SM120 fallback. - **DSpark ragged/confidence scheduling**: `SGLANG_RAGGED_VERIFY_MODE` at default `static` (budget planner not created); `SGLANG_SIMULATE_ACC_LEN` unset. - **Request timeout aborts (#27221's trigger)**: both env timeouts at default `-1`. - **Client aborts**: 35 streaming requests cancelled mid-prefill and mid-decode in a loop — no wedge. Normal completed requests also pass for hours under synthetic load. - **Host memory**: recurs at `mem-fraction-static 0.85` with tens of GB free on both nodes. The remaining common factor in all live captures: **multi-node TP=2 + DSpark verify path** in the wedged rank's stack, and the peer rank having already moved on to the inter-step request broadcast — i.e., a one-step divergence in batch composition between the per-rank schedulers. ## Expected behavior Both scheduler ranks agree on batch composition every step; no unmatched collectives; no proxy wedge. ## Notes - A reference vLLM stack (same two machines, same checkpoint, DSpark) is stable under the same client traffic for days, consistent with vLLM's driver-broadcast step table not having independent per-rank scheduler loops. - Happy to provide full container logs (NCCL `PROXY,NET,INIT` INFO level from both nodes), the watchdog scheduler-debug dumps, and to test patches. If there is a debug flag that makes the two ranks log their batch composition per step (`cur_batch_for_debug`-style), a run with that enabled should catch the divergent step in the act.",
"url": "https://github.com/sgl-project/sglang/issues/33289",
"createdAt": "2026-08-02T16:29:08Z",
"updatedAt": "2026-08-13T02:12:52Z",
"timestamp": "2026-08-13T02:12:52Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [],
"author": "MiaAI-Lab",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:33627",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Should we make the LM head GEMM output fp32 instead of bf16?",
"text": "### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Motivation **Problem** The LM head GEMM outputs bf16. We cast the result to fp32 afterwards, in `_copy_logits_to_buffer`. The cast happens after the rounding, so the fp32 precision the GEMM already computed internally is thrown away. It should strictly increase accuracy. ```python # logits_processor.py::_compute_lm_head, default path logits = torch.matmul(hidden_states.to(lm_head.weight.dtype), lm_head.weight.T) # bf16 out ``` **Proposal** Ask the GEMM for fp32 output instead: ```python logits = torch.mm(hidden_states, lm_head.weight.T, out_dtype=torch.float32) ``` The weight stays bf16. Only the output dtype changes. **Facts** Measured on B300. DeepSeek-V4-Flash-0731 (vocab 129280, hidden 4096), TP4, so N=32320 per rank and K=4096. `bench_gpu_time_with_cupti` with CUDA graph. 1. Accuracy, against an fp32 reference. Current bf16 output changes the greedy token on 141/5120 positions (2.75%). `out_dtype=torch.float32` changes 0/5120. 2. Time. fp32 output makes the GEMM faster (it removes a separate cast kernel over the whole vocab) and the vocab all-gather slower (2x the bytes). | M | GEMM now | GEMM fp32 | all-gather now | all-gather fp32 | net | |---|---|---|---|---|---| | 1 | 62.2us | 58.8us | 6.5us | 6.9us | -3.0us | | 8 | 61.6us | 57.9us | 9.8us | 11.2us | -2.3us | | 32 | 62.8us | 58.5us | 17.4us | 28.8us | +7.1us | | 128 | 73.8us | 64.8us | 52.4us | 98.6us | +37.2us | | 512 | 137.7us | 100.4us | 198.9us | 369.8us | +133.6us | (all-gather column uses whichever of multimem / NCCL is faster at that size) 3. TP invariance. The LM head is column-parallel, so K is never sharded and every logit is the same dot product at any TP size. TP1 and TP4 agree exactly today, and would still agree if fp32 is applied everywhere. They would stop agreeing if fp32 is applied only when there is no all-gather. | | greedy tokens that differ | |---|---| | now: TP1 bf16 vs TP4 bf16 | 0.00% | | fp32 everywhere: TP1 vs TP4 | 0.00% | | fp32 only when no all-gather: TP1 fp32 vs TP4 bf16 | 2.75% | So this should be applied to all TP sizes or none. **Open question** Either flip the default to fp32 output, or leave the default and keep `--enable-fp32-lm-head` as the opt-in. The overhead is likely negligible at large batch. The LM head runs once per forward, while this model has 43 transformer layers, and at large batch the decode step is dominated by those layers. So +133.6us at M=512 is probably a small fraction of the step. But we should benchmark it for TP case for some common models, like GLM-5.2, Deepseek V4, etc. **Two unrelated things found while measuring** - The multimem all-gather is slower than NCCL above M ~= 300 in bf16: 257.8us vs 198.9us at M=512, 1018us vs 705us at M=2048. `MultimemAllGatherer` guards on `recommended_max_tokens`, which is a token-count cap from `max_running_requests`, not a performance crossover. - The multimem all-gather only supports bf16, so `--enable-fp32-lm-head` on TP>1 falls back to NCCL for the logits gather. ### Related resources - #32370 uses `torch.mm(out_dtype=torch.float32)` for the same reason, on the `--enable-fp32-lm-head` path. - `torch.mm` `out_dtype`: https://docs.pytorch.org/docs/2.13/generated/torch.mm.html",
"url": "https://github.com/sgl-project/sglang/issues/33627",
"createdAt": "2026-08-05T02:25:16Z",
"updatedAt": "2026-08-13T11:55:38Z",
"timestamp": "2026-08-13T11:55:38Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"reinforcement-learning"
],
"author": "b8zhong",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:33636",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[NVIDIA] DeepSeek V4 Perf Tracking",
"text": "## Motivation This follows [DeepSeek V4 Roadmap](https://github.com/sgl-project/sglang/issues/23602), which covers functional enablement; this issue is perf-only. **Scope**: NVIDIA SM90 / SM10X. --- ## High priority - [ ] Integrate TRT-LLM DSv4 attention for SM100/103 #30805 - [x] FlashInfer MNNVL backend for pure (non-fused) allreduce #30700 - [ ] Overlap scheduling for online C128 MTP #30497 - [ ] Split mixed-chunk attention: route decode tokens to the fp8 paged MLA kernel #30338 - [ ] Emit TMA-aligned UE8M0 scales for the FP8 einsum #34277 - [ ] Keep fp32 routing weights in the MXFP4 trtllm MoE #33608 — drops the `PackTopkIds` launch ## Attention & compression kernels - [ ] Fused norm + RoPE + uniform fp8 store for TRT-LLM DSv4 sparse attention #32975 - [x] Optimize the c128 epilogue #26671 ## Indexer & top-k - [x] Occupancy tuning for the DSA indexer fp8-quant Q kernel #32755 - [ ] Rewrite `paged_mqa_metadata` #25855 - [ ] Skip trivial nonpaged indexer logits #33857 - [ ] FlashInfer GVR top-k backend for decode #33914 ## mHC - [x] FlashInfer mHC fusion #33616 — TileLang fusion is still faster today; this is the path when TileLang is unavailable (for future architectures), and FlashInfer should gain full pre / post+pre fusion soon ## MoE & quantization - [ ] FlashInfer MegaMoE #31470, plus the zero-copy adapter path #33571 - [ ] Fused SwiGLU + quant for shared experts and EP-normal #27561 - [ ] Fused SiLU + clamp + mul + FP8-quant AOT kernel for the EP MoE path #32058 ## Speculative decoding (MTP / DSpark / EAGLE3) - [ ] DSpark C128 online compressor #32035 - [ ] EAGLE3 for DeepSeek-V4-Flash-0731 #33344 ## Communication - [ ] Fix non-EP TBO for attention TP > 1 #33250 (blocked-path guard in #33217) - [ ] Decode two-batch overlap with the DeepEP backend #33834 - [x] Fix DP-attention gather semantics #31700 — `dp_gather_partial` treats attention-TP replicas as partial contributions ## CUDA graph & scheduling - [x] Alt stream during BCG prefill #29070 - [ ] Breakable CUDA graph for mixed-chunk prefill #30420 ## Memory & KV capacity - [ ] SWA recompute #31713 — keep the FULL prefix hit and rebuild only the trailing SWA window - [ ] Decode radix cache with MTP under P/D disagg #31097 - [ ] BF16 C4/C128 compressed-state storage #24041 — for HBM-constrained parts - [ ] Size the SWA state pool by storage page size, not the model window #30371 — recovers wasted c4-state capacity ## Context Parallel - [ ] Batched and round-robin CP prefill in the non-paged indexer #33570 - [ ] Remove prefill CP KV and compressor materialization #33236 - [ ] `ag_gemm` + `moe_rs` symmetric-memory overlap kernels for CP prefill #28639 - [ ] CP cache LayerSplit: common infrastructure #33382, full implementation #29187 - [ ] Decode context parallelism for DeepSeek V4 #30416 - [ ] Shared KV cache for prefill CP via VMM #32059 — each CP rank owns a shard of the physical pages ## Docs & recipes - [ ] GB300 FP4 multi-node PD recipes for DeepSeek-V4-Pro #31377 - [ ] Slurm cluster deployment in the cookbook #30996 ## CI & bug tracking - [ ] Add DSV4 NVFP4 tests #28966 - [ ] Nightly AIME25 for DeepSeek-V4-Pro on B200 #24891 --- If an open PR belongs here and I missed it, comment and I'll add it.",
"url": "https://github.com/sgl-project/sglang/issues/33636",
"createdAt": "2026-08-05T03:38:42Z",
"updatedAt": "2026-08-13T09:33:36Z",
"timestamp": "2026-08-13T09:33:36Z",
"metrics": {
"reactions": 5,
"comments": 8
},
"labels": [
"nvidia"
],
"author": "b8zhong",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:issue:34384",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] DSpark compact ragged CUDA Graph uses incompatible request-slot geometry for the same token tier",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug Related to #31023, but this issue tracks a deterministic request-slot geometry mismatch rather than the timing-sensitive overlap/lifetime failure. In DSpark compact ragged target verification, CUDA Graphs are keyed by the rounded verify-token tier. However, the same token tier can represent different request-slot geometries. With a target verify width of 6 and capture sizes `[1, 32, 192]`, the 192-token tier is captured as: ```text 192 request slots x 1 verify token ``` A runtime batch with 32 requests and `cap_len=6` selects the same 192-token graph, but its layout is: ```text 32 request slots x 6 verify tokens 160 padded request slots x 0 verify tokens ``` The target and draft CUDA Graphs finish capture successfully. The first replay of the `bs=32, cap_len=6` target graph then deterministically fails with a CUDA illegal memory access. The relevant behavior is: ```python def _ragged_capture_slots(self, num_tokens: int) -> int: return min(num_tokens, self.max_bs) ``` For the 192-token tier, this returns 192 capture slots. The admission check only verifies that the runtime batch fits within that slot capacity; it does not verify capture/replay request geometry compatibility. As a diagnostic, I changed the 192-token tier to use 32 capture slots while preserving the ragged capture layout: ```text capture: 32 slots x 6 tokens replay: 32 slots x 6 tokens ``` The same workload then replayed successfully for repeated decode steps. Using `SGLANG_TEST_RAGGED_VERIFY_FORCE_UNIFORM_CAPTURE=1` directly is not a valid workaround because it also disables the capture-time ragged layout. This makes the runner use token-keyed graphs while the DSV4 attention metadata falls back to request-keyed metadata, causing a separate overlapping-copy error. Possible production fixes appear to be: 1. Keep token-keyed graphs and make the DSV4 metadata/attention/compressor/indexer path safely support zero-length padded request slots. 2. Key and select graphs using both `(token_tier, slot_tier)`. Expected behavior: every admitted compact ragged layout should either replay safely on a compatible CUDA Graph or fall back to eager execution. ### Reproduction The failure reproduces on the unmodified commit: ```text ee464fedc63e9cd3e8b5a56c6bbbe4272cd00580 ``` Configuration: - Model: DeepSeek-V4-Flash DSpark-compatible checkpoint - Hardware: 4 x NVIDIA H20-3e - Tensor parallelism: TP4 - Ragged verify mode: compact - Target verify width: 6 tokens per request Start the profiling server: ```bash export CUDA_VISIBLE_DEVICES=4,5,6,7 export SGLANG_RAGGED_VERIFY_MODE=compact export SGLANG_DSPARK_ENABLE_SPS_RECORD=1 export SGLANG_SIMULATE_ACC_LEN=1.0 export SGLANG_DSV4_FP4_DEQUANT=1 python3 -m sglang.launch_server \\ --model-path \\ --speculative-algorithm DSPARK \\ --tp 4 \\ --swa-full-tokens-ratio 0.2 \\ --chunked-prefill-size 8192 \\ --mem-fraction-static 0.85 \\ --cuda-graph-max-bs-decode 192 \\ --cuda-graph-bs-decode 1 32 192 \\ --max-running-requests 256 \\ --disable-radix-cache \\ --trust-remote-code \\ --host 0.0.0.0 \\ --port 8006 ``` Run the deterministic workload: ```bash python3 -m sglang.benchmark.dspark_sps_profiler run \\ --base-url http://127.0.0.1:8006 \\ --batch-size 32 \\ --fracs 1.0 \\ --input-len 512 \\ --round-timeout 600 \\ --out debug_bs32_frac1.json ``` Observed result: - Target CUDA Graph capture succeeds. - Draft CUDA Graph capture succeeds. - The first `bs=32, cap_len=6` target replay fails with `CUDA error: an illegal memory access was encountered`. Control matrix: ```text Graph list [1, 8, 16, 32], max_running_requests=48: pass Graph list [1, 8, 16, 32], max_running_requests=256: pass Graph list [1, 8, 16, 32, 128]: pass Graph list [1, 32, 192]: fail Graph list [1, 32, 256]: fail Graph list [1, 192], native bs=192: pass Graph list [1, 192], runtime bs=32 replaying the capture-size-192 / 1152-token graph: pass ``` This isolates the failure to the smaller graph captured after a sufficiently large graph, with the same token tier but incompatible request-slot geometry. Diagnostic validation: - Changing the 192-token tier to capture 32 slots while preserving `RaggedVerifyLayout` makes the `bs=32, frac=1.0` workload stable. - This is diagnostic evidence, not a proposed production fix: uniform slot capture can cause lower-fraction, high-request-count batches to fall back to eager execution. ### Environment Output of `python3 -m sglang.check_env`: ```text Python: 3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0] CUDA available: True GPU 0,1,2,3,4,5,6,7: NVIDIA H20-3e GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 13.0, V13.0.88 CUDA Driver Version: 550.163.01 PyTorch: 2.11.0+cu130 sglang: 0.0.0.dev1+gee464fedc sglang-kernel: 0.4.4 flashinfer_python: 0.6.14 flashinfer_cubin: 0.6.14 flashinfer_jit_cache: 0.6.14+cu130 triton: 3.6.0 transformers: 5.12.1 torchao: 0.17.0+cu130 numpy: 2.3.5 aiohttp: 3.14.1 fastapi: 0.139.0 huggingface_hub: 1.23.0 interegular: 0.3.3 modelscope: 1.38.1 orjson: 3.11.9 outlines: 0.1.11 packaging: 26.2 psutil: 7.2.2 pydantic: 2.13.0 python-multipart: 0.0.32 pyzmq: 27.1.0 uvicorn: 0.51.0 uvloop: 0.22.1 vllm: Module Not Found xgrammar: 0.2.1 openai: 2.6.1 tiktoken: 0.13.0 anthropic: 0.116.0 litellm: Module Not Found torchcodec: 0.15.0 NVIDIA Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE NODE NODE SYS SYS SYS SYS 0-51,104-155 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE PIX NODE NODE SYS SYS SYS SYS 0-51,104-155 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX NODE SYS SYS SYS SYS 0-51,104-155 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE PIX SYS SYS SYS SYS 0-51,104-155 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PIX NODE NODE NODE 52-103,156-207 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS NODE PIX NODE NODE 52-103,156-207 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS NODE NODE PIX NODE 52-103,156-207 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE NODE NODE PIX 52-103,156-207 1 N/A NIC0 NODE NODE NODE NODE SYS SYS SYS SYS X PIX NODE NODE NODE NODE SYS SYS SYS SYS NIC1 NODE NODE NODE NODE SYS SYS SYS SYS PIX X NODE NODE NODE NODE SYS SYS SYS SYS NIC2 PIX NODE NODE NODE SYS SYS SYS SYS NODE NODE X NODE NODE NODE SYS SYS SYS SYS NIC3 NODE PIX NODE NODE SYS SYS SYS SYS NODE NODE NODE X NODE NODE SYS SYS SYS SYS NIC4 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE NODE NODE X NODE SYS SYS SYS SYS NIC5 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE NODE X SYS SYS SYS SYS NIC6 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS X NODE NODE NODE NIC7 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS NODE X NODE NODE NIC8 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS SYS NODE NODE X NODE NIC9 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge PXB = Connection traversing multiple PCIe bridges without traversing the PCIe Host Bridge PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NIC8: mlx5_8 NIC9: mlx5_9 ulimit soft: 120000 ```",
"url": "https://github.com/sgl-project/sglang/issues/34384",
"createdAt": "2026-08-11T07:55:04Z",
"updatedAt": "2026-08-13T03:43:49Z",
"timestamp": "2026-08-13T03:43:49Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "LeeZZY1",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34399",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Paged KV allocator launches allocation kernels before checking OOM",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version (`a23670ddbf`, verified on 2026-08-13). - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug On the paged CUDA KV-cache path, `PagedTokenToKVPoolAllocator.alloc_extend` and `alloc_decode` launch Triton allocation kernels before checking whether `free_pages` contains enough pages. The kernels index `free_pages` using computed page offsets, but the accesses are not masked by the logical free-list length. When the KV cache is exhausted, the kernel can read beyond the valid logical free-page list before the caller returns the expected out-of-memory error. The visible result is normally a controlled scheduler OOM, but the preceding GPU read is invalid with respect to the allocator's logical free-page list and may cause undefined behavior or poison the CUDA worker. This report does not claim that every allocator layout produces a deterministic CUDA illegal-address exception. ### Reproduction The latest upstream `main` at commit `a23670ddbf89678a53230ceb2cc5ed75236c4d85` still reproduces the issue. #### Minimal CPU reproduction With `page_size=4` and an allocator containing two free pages, the request below requires three new pages. The kernel is replaced with a mock, so this checks the Python-to-kernel ordering without requiring a model: ```python from unittest.mock import patch import torch from sglang.srt.mem_cache.allocator.paged import PagedTokenToKVPoolAllocator allocator = PagedTokenToKVPoolAllocator( size=8, page_size=4, dtype=torch.float16, device=\"cpu\", kvcache=None, need_sort=False, ) prefix_lens = torch.tensor([0], dtype=torch.int64) seq_lens = torch.tensor([12], dtype=torch.int64) last_loc = torch.tensor([-1], dtype=torch.int64) with patch(\"sglang.srt.mem_cache.allocator.paged.alloc_extend_kernel\") as kernel: result = allocator.alloc_extend( prefix_lens, prefix_lens, seq_lens, seq_lens, last_loc, extend_num_tokens=12, ) print(result is None, kernel.__getitem__.call_count) ``` On the latest unmodified `main` this prints: ```text True 1 ``` The expected result after the fix is: ```text True 0 ``` The allocator should return OOM before indexing the kernel launcher. #### A100 reproduction After warming the successful path, I wrapped the Triton kernel launchers and ran the failing calls with `CUDA_LAUNCH_BLOCKING=1`. Latest upstream `main`: ```text gpu= NVIDIA A100-PCIE-40GB torch_cuda= 12.8 main_sha=a23670ddbf extend_oom, kernel_launches= (True, 1) decode_oom, kernel_launches= (True, 1) ``` The rebased branch in [PR #34400](https://github.com/sgl-project/sglang/pull/34400): ```text gpu= NVIDIA A100-PCIE-40GB torch_cuda= 12.8 patch_sha=1af8b575ed extend_oom, kernel_launches= (True, 0) decode_oom, kernel_launches= (True, 0) ``` The same A100 inputs therefore return the same controlled OOM result without launching either allocation kernel after the fix. The focused regression tests on the patched branch pass: ```text 6 passed, 1 warning in 4.42s ``` #### Logical free-list over-read evidence Using a backing tensor `[7, 99, 101]` and exposing only `backing[:1]` as the logical `free_pages` tensor produced: ```text logical_free_pages= [7] page_ids_used= [7, 99, 101] ``` The backing tensor keeps this reproducer deterministic and in-bounds for the underlying storage. It demonstrates that the kernel reads values beyond the logical free-page list without claiming that every allocator layout produces the same physical memory fault. #### Online relevance This condition is reachable during normal serving when the KV cache is full and no more evictable pages are available. The normal path is: 1. The scheduler calls `alloc_paged_token_slots_extend` or `alloc_paged_token_slots_decode`. 2. The allocator launches the Triton kernel while the free-page list is too short. 3. The allocator returns `None`, and the scheduler raises the expected OOM error. Existing serving reports show the same allocator OOM paths: - [#13765](https://github.com/sgl-project/sglang/issues/13765) reports `alloc_paged_token_slots_extend` failing during EAGLE verify with `available_size=448` and `evictable_size=0`. - [#11581](https://github.com/sgl-project/sglang/issues/11581) reports Decode OOM in a running H100 service with `available_size=3` and `evictable_size=0`. ### Expected behavior The allocator should check page capacity before allocating the output tensor or launching the Triton kernel. When capacity is insufficient, it should return `None` without consuming pages or launching the kernel. With `need_sort=True`, the existing release-page merge may still run before this check. ### Proposed fix Move the existing `num_new_pages > len(self.free_pages)` checks before the Triton kernel launches in both `alloc_extend` and `alloc_decode`. ### Environment - SGLang main commit: `a23670ddbf89678a53230ceb2cc5ed75236c4d85` - Python: 3.10 - PyTorch: 2.9.1+cu128 - CUDA runtime: 12.8 - GPU: NVIDIA A100-PCIE-40GB - Compute capability: 8.0 - Triton: 3.5.1 - sglang-kernel: 0.4.6.post1",
"url": "https://github.com/sgl-project/sglang/issues/34399",
"createdAt": "2026-08-11T09:29:09Z",
"updatedAt": "2026-08-13T10:03:32Z",
"timestamp": "2026-08-13T10:03:32Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "waizuichougou",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34510",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Tracking] PD disaggregation shared-protocol unification",
"text": "# PD Disaggregation Single-Protocol-Layer Unification Plan > Based on the v0.1 draft of [RFC #33861: PD disaggregation: single protocol layer, per-backend transport](https://github.com/sgl-project/sglang/issues/33861) This issue tracks the staged implementation of #33861. ## Step 0 — Behavior and coupling audit - [x] Audit Mooncake/NIXL/Mori protocol behavior - [x] Classify protocol vs Transport responsibilities ## Step 1 — Low-risk behavior alignment - [ ] NIXL Prefill bootstrap timeout - [ ] Mori Decode heartbeat - [ ] Mori ABORT delivery - [ ] Skip stale queued work - [ ] Control-loop fault isolation - [ ] Mori speculative MHA layout - [ ] Mooncake control-thread initialization order I plan to consolidate these low-risk defensive semantic-alignment fixes into a single dedicated PR, and I am already actively working on it. ## Step 2 — Per-backend Transport extraction - [ ] NixlTransport - [ ] MooncakeTransport - [ ] MoriTransport ## Step 3 — Protocol decisions - [ ] Completion/failure delivery ADR - [ ] Abort/quiescence ADR - [ ] Room generation ADR ## Step 4 — Common protocol integration - [ ] Common KVManager/Sender/Receiver - [ ] Replace get_kv_class with make_transport - [ ] Rollout and cleanup ## 1. Document Goals and Implementation Scope Building on [RFC #33861](https://github.com/sgl-project/sglang/issues/33861), this document expands on concrete implementation details in the current code and drafts a stepwise refactoring plan. The original RFC notes that Mooncake, NIXL, and Mori currently each implement a complete set of bootstrap, room lifecycle, transfer orchestration, failure/timeout handling, and third-party engine calls in their own `conn.py`. The goal is not to make the three transport engines use the same API, but to retain only one copy of their common protocol logic while leaving the parts that genuinely depend on third-party engines in each backend Transport. The main scope of this document audits only the Mooncake, NIXL, and Mori direct PD paths covered by the RFC. Other backends such as Ascend are outside the main design and migration scope of this plan; the Fake backend appears only in discussions of migration to `NullTransport` and CPU protocol tests. The expected final structure is: ```text SGLang PD Scheduler / Queues ↓ Single protocol implementation KVManager KVSender KVReceiver KVBootstrapServer ↓ BaseTransport NixlTransport MooncakeTransport MoriTransport NullTransport ↓ Third-party transport engines ``` Where: - The common protocol layer is responsible for understanding requests, rooms, chunks, TP/CP/PP/DCP, aux/state, timeout, heartbeat, failure, abort, completion, and cleanup. - The Transport layer is responsible only for third-party engine lifecycle, memory registration, peer descriptors, submit/progress, engine errors, and backend-private optimizations. The overall refactoring is expected to proceed through the following stages. ### Step 0 — Audit the Current Code and Coupling Review the three Mooncake, NIXL, and Mori `conn.py` implementations class by class and function by function: - What each object and function is currently responsible for; - Which behaviors are the same across the three backends and which differ; - Which differences represent missing common protocol semantics; - Which code couples protocol logic such as room/layout with Transport logic such as engine/descriptor; - Which features are valid backend capabilities rather than behaviors that must be implemented everywhere. Step 0 does not freeze the final interface or perform the actual abstraction. It provides the factual basis and decomposition boundaries for subsequent PRs. The Step 0 section of this document primarily covers this work and provides detailed code context. ### Step 1 — Fill Gaps in Common Semantics with Small PRs Based on the Step 0 results, first align clearly identified common protocol behaviors through small, independent PRs that can each be reviewed and regressed separately. This stage handles only cases where “the same common semantic is implemented more completely in one backend but missing from another.” For example: - A backend lacks a bootstrap timeout; - A backend does not start an existing common heartbeat; - A backend does not correctly receive the common abort message; - A backend's control loop/parser lacks exception isolation already present in other backends; - A backend lacks stale-work rejection, control-loop defenses, or a safe thread startup order already present in other backends. The essence of Step 1 is stability: 1. Add or complete only high-confidence common protocol semantics. 2. Do not remove existing successful paths or introduce the final Transport abstraction. 3. Do not modify the production wire format. 4. Do not address backend differences with disputed semantics. 5. Do not require Mori to add staging/DCP, or require other backends to copy Transport capabilities such as NIXL prepared descriptors. 6. Every fix should have a backend-parameterized regression test. ### Step 2 — First Extract Transport Within Each Backend Within each backend, extract third-party engine-related logic from the current `KVManager/KVSender/KVReceiver` to form independently operable `MooncakeTransport`, `NixlTransport`, and `MoriTransport` implementations. This step initially leaves each backend's current protocol path unchanged. Its goal is to validate: - How the engine should be initialized and closed; - How local memory should be registered and deregistered; - How local/remote descriptors should be represented; - How peers should be established, cached, and closed; - How transfers should be submitted, polled, reported as errors, and canceled; - How capabilities such as staging, mixed memory, prepared handles, and peer probes should be declared. Only after all three real backends can run their existing paths through their own Transports is it appropriate to freeze the minimal `BaseTransport` contract. This document proposes interface sketches and capability ideas, but these still require review by backend owners and must not be treated as a final design ready for implementation. ### Step 3 — Discuss and Unify Remaining Conflicting Common Protocol Semantics Step 1 incrementally fills cases where “other backends already have mature semantics that the current backend lacks”; Step 3 addresses areas where the three backends have genuinely made different choices and it is not possible to directly determine which is correct. Typical questions include: - Whether completion/failure should be delivered through the common control channel or Transport notifications; - Whether `ABORT_ACK` means the message was received or that no in-flight writes remain; - When Decode can safely reclaim destination KV pages; - How Transport execution timeout, cancel, and quiescence should be defined; - How room generation or tombstones prevent late events from contaminating new requests; - Whether descriptors remain valid after memory release/resume; - How the final wire version and rolling upgrade should be implemented. These questions require separate ADRs, backend-owner review, and executable tests; they should not be disguised as simple Step 1 bug fixes. This document records candidate directions but does not freeze a concrete implementation before review is complete. ### Step 4 — Promote the Single Common Protocol and Complete Framework Integration Before entering Step 4, each backend must have run the old protocol path through an independent Transport in Step 2, the backend-local completions in Step 1 must be finished, and the conflicting semantics in Step 3 must have converged. Only after these conditions are met can the common parts be promoted wholesale into the single: ```text KVManager KVSender KVReceiver KVBootstrapServer ``` Backends ultimately retain only Transport implementations, capabilities, and private optimizations, while integration with the Scheduler, queues, factory, and CI is completed. Because the preceding boundaries and semantics have already been validated, Step 4 should focus on mechanical promotion, compatible cutover, and cleanup rather than another round of protocol design.",
"url": "https://github.com/sgl-project/sglang/issues/34510",
"createdAt": "2026-08-12T03:21:24Z",
"updatedAt": "2026-08-13T16:27:56Z",
"timestamp": "2026-08-13T16:27:56Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [],
"author": "jambow0320",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34513",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[RFC] Agent-aware session affinity without routing keys: a router policy",
"text": "## Motivation Multi-turn agent workloads (coding agents, tool-calling loops, multi-agent orchestration) resend the **entire growing conversation prefix on every turn**. SGLang's RadixAttention skips prefill for cached prefixes — but **only when consecutive turns of the same session land on the same worker**. Without session affinity, each turn re-prefills the accumulated context, and the prefix-cache benefit that is central to SGLang's value proposition largely disappears for exactly the workloads that are growing fastest. This matters because agent serving is a primary growth workload for SGLang and prefix-cache efficiency is a headline differentiator. Most real agent clients (frameworks, gateways) do **not** thread a routing key or session header through, and can't be required to — so \"just pass a key\" is not a deployable answer for most users. And the failure we document below is not a corner case: a shared system prompt is the *norm* for agent fleets. ## Existing mechanisms and remaining gap On current main, the routing options for session affinity are: | Mechanism | Stickiness | Caller change | Problem | |---|:---:|:---:|---| | `round_robin` / `random` | none | none | every turn re-prefills the accumulated context | | `cache_aware` (prefix-hash) | partial | none | collapses under widely-shared prefixes (below) | | routing key / session header | yes | **caller must supply one** | impractical for clients that don't expose session affinity; impossible for gateways that forward third-party requests without session context | The remaining gap is therefore narrow but real: **a no-caller-change way to get stickiness**. The two no-caller-change options both fall short in different ways — `round_robin` never reuses the growing prefix, and `cache_aware` collapses precisely on the shared-prefix case it's meant to serve — while the one that works (`routing key`) requires the caller to cooperate. Measured on 4× workers (TP=2, Qwen3-32B), replaying real agent trajectories (DeepSWE) and using SGLang's own `agentic-trace` benchmark (all numbers reproducible from raw run artifacts): - **Without stickiness, the maximum SLO-compliant concurrency drops ~4×.** Closed-loop replay, SLO = mean T2+ TTFT < 3000ms ∧ completion ≥ 99%, knee = max compliant concurrency: sticky sustains **C≈54–64** (~18–19K tok/s) while `round_robin` breaks at **C≈16** (~9.4K tok/s). - **`cache_aware` degrades precisely on the shared-prefix case.** On a 29K-shared-prefix workload it completed only **58/110** sessions (all 52 failures were long-prefix timeouts) while a sticky policy completed **110/110**. An open-loop stress test (SGLang's own `serving.py --dataset-name agentic-trace`, sweeping the injected shared-prefix length) compares all three no-caller-change policies — `cache_aware` breaks the TTFT SLO at **every** length, worse than *no* affinity (`round_robin`), while `agent_aware` stays compliant and roughly flat: | Shared prefix | `round_robin` | `cache_aware` | `agent_aware` | SLO (3000ms) | |---|:---:|:---:|:---:|:---:| | ~1.7K | 1,947ms | 8,385ms ❌ | **1,687ms** | ✅ | | ~3.3K | 1,988ms | 8,575ms ❌ | **1,808ms** | ✅ | | ~4.9K | 2,401ms | 9,173ms ❌ | **2,100ms** | ✅ | | ~6.6K | 2,659ms | 9,785ms ❌ | **2,147ms** | ✅ | ## Recommended contract We prototyped a policy, `agent_aware`, that gets stickiness **without the caller supplying a routing key or session header**: it infers session identity from the request content itself (shared-prefix fingerprint + conversation-prefix match), pins the session to a worker, and uses load-aware placement to avoid clustering. We propose it as an opt-in policy alongside the existing ones. Measurements against `manual` (keyed sticky) and `round_robin` on the same hardware/trajectories. Workloads W1–W6 (110 sessions each, Poisson arrivals, same 4× TP=2 workers): | Workload | Construction | Role | |---|---|---| | W1-Agent | SWE-chat real collaborative coding sessions (median ~38 turns) | core agent load | | W2-Mix | 55 agent + 55 chat sessions | mixed load | | W3-Multi | multiple agent types / system-prompt patterns | multi-fingerprint coexistence | | W4-Chat | ShareGPT, very short prompts (~1060 tokens) | guardrail: no cache benefit to gain, must not regress | | W5-AgentLong | real Claude Code traces, 29K system prompt | maximal cache benefit / long-prefix stress | | W6-MixLong | long agent + chat | long/short interference stability | Latency runs sweep mem-fraction ∈ {0.5, 0.7, 0.9} (KV-cache pressure) × 3 seeds; goodput runs sweep the Poisson arrival rate until the SLO breaks. - **Cache reuse matches keyed sticky, with zero caller change.** Across W1–W6, `agent_aware` achieves the same cache reuse as `manual` (per-session cached-turn coverage identical), and per-session end-to-end time is on par (within noise). - **Higher SLO knee than keyed sticky.** `agent_aware` sustained **C=64** while `manual` broke at **C=54**. At the same C=64, `agent_aware`'s T2+ TTFT was ~31% of `manual`'s and throughput ~44% higher. - **Graceful degradation under overload.** From low concurrency to overload (C=54→C=81), `manual`'s T2+ TTFT blows up ~20×, while `agent_aware` grows only ~7×. - **No collapse under shared prefixes.** `agent_aware` holds the lowest T2+ TTFT at every realistic prefix length (1.7K–6.6K), staying roughly flat (1.7s→2.1s) while `cache_aware` breaks the SLO at every length. Because it strips the shared prefix and hashes the distinguishing part, it is immune to the clustering that sinks `cache_aware`. ## Correctness boundary `agent_aware` provides stickiness **for KV/radix-cache locality only**. It is not suitable where the session identity is a correctness requirement for backend-local state: - Content-inferred identity means **two logically distinct sessions with identical content** (same system prompt + identical conversation so far) are inferred as the same session, pinned to the same worker, and share a KV-cache entry. For cache reuse this is a feature; but clients that depend on per-session isolated backend state must not rely on content-inferred identity. - **Context compaction / summarization breaks prefix matching**: the session re-registers under the new prefix. This is graceful degradation — the system prompt stays cached — but affinity restarts at that point. - **Warmup cost**: the first ~2 requests per new prompt pattern are used to learn the shared prefix, so they are routed by load-balancing fallback (slightly higher T1). T2+ routing is unaffected. ## Suggested validation We validated with two methods (raw artifacts available): - **Closed-loop knee scan**: replay of real agent trajectories with recorded tool-wait gaps; SLO = mean T2+ TTFT < 3000ms ∧ completion ≥ 99%; knee = max compliant concurrency. - **Open-loop shared-prefix stress**: SGLang's own `serving.py --dataset-name agentic-trace`, sweeping the injected shared-prefix length. If this direction is pursued, we'd suggest: unit tests for fingerprint/skip detection and mixed-length variant detection; collision behavior for identical-content sessions; UTF-8 multi-byte boundary safety; a multi-worker benchmark reporting placement, cache/prefill work, throughput, and p95 TTFT across shared-prefix lengths. ## Discussion: we'd like maintainers' read on the direction We recognize that **router-side content inference may not be the direction SGLang prefers** — it is heuristics plus router-side session state, and the community may prefer to solve session affinity elsewhere (e.g. inside the engine, or via a standardized caller-side convention). We open this to get maintainers' read before investing further. Roughly, we see these directions: - **A. Fix `cache_aware` with a global view of the prefix cache** — keep prefix affinity but make it *affinity without clustering*: factor worker load / queue depth into the prefix-match decision, and give the router a global view of which prefixes are cached on which workers rather than relying on the router's self-bookkept approximate tree, which records only its own routing decisions and never observes what workers actually cache or evict. - **B. Feedback-driven routing (control loop)** — workers report real-time signals (queue depth, cache-hit rate, KV-cache occupancy) and the router re-balances by a closed-loop controller rather than static prefix hashing. This direction aligns with the push-based engine-load reporting proposed in #32925, and we'd rather defer to that effort than duplicate it. - **C. (Our prototype) Agent-aware content-inferred affinity** — no routing key / session header; infer session identity from content. Measured above. We're happy to share the full measurement methodology and data, and to help prototype whichever direction the community prefers. If the preferred direction is A or B, we'd welcome that and contribute there instead. CC @merrymercy @hzh0425 @whybeyoung",
"url": "https://github.com/sgl-project/sglang/issues/34513",
"createdAt": "2026-08-12T03:45:32Z",
"updatedAt": "2026-08-13T03:18:00Z",
"timestamp": "2026-08-13T03:18:00Z",
"metrics": {
"reactions": 2,
"comments": 0
},
"labels": [],
"author": "Biaogezi",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34559",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[NPU][Tracking] Ascend A5 MXFP8/MXFP4 Capability and Model Coverage",
"text": "## Background The community-wide Quantization 2026 H2 Roadmap (#31783) already covers the common quantization framework, weight lifecycle, capability validation, and related infrastructure. Therefore, #21584 does not propose another Ascend quantization roadmap. It focuses on: > **Ascend A5 MXFP8/MXFP4 model/operator coverage and the consolidation of A5-specific operator and layout differences.** --- ## 1. Scope This issue mainly tracks the following A5 MXFP schemes: | Scheme | Target | |---|---| | MXFP8 W8A8 | Dense / MoE | | MXFP4 W4A8 | Dense / MoE | | MXFP4 W4A4 | Dense / MoE | Primary model coverage: - Qwen3 Dense / MoE - Qwen3.5 Dense / MoE - Future models should preferably reuse existing Scheme / Kernel implementations instead of introducing model-specific MXFP paths. --- ## 2. A5 MXFP Capability Matrix | Model / Layer | MXFP8 W8A8 | MXFP4 W4A8 | MXFP4 W4A4 | |---|---|---|---| | **Qwen3 Dense** | ✅ Merged [#22352](https://github.com/sgl-project/sglang/pull/22352) | ✅ Merged [#23650](https://github.com/sgl-project/sglang/pull/23650) | ✅ Merged [#23795](https://github.com/sgl-project/sglang/pull/23795) | | **Qwen3 MoE** | ✅ Merged [#30768](https://github.com/sgl-project/sglang/pull/30768) | ✅ Merged [#30318](https://github.com/sgl-project/sglang/pull/30318) | ✅ Merged [#30319](https://github.com/sgl-project/sglang/pull/30319) | | **Qwen3.5 Dense** | 🚧 Validation [#32150](https://github.com/sgl-project/sglang/pull/32150) / Offline fix [#32266](https://github.com/sgl-project/sglang/pull/32266) | 🚧 Validation [#32150](https://github.com/sgl-project/sglang/pull/32150) | 🚧 Validation [#32150](https://github.com/sgl-project/sglang/pull/32150) | | **Qwen3.5 MoE** | 🚧 Online supported / Offline WIP [#30768](https://github.com/sgl-project/sglang/pull/30768) [#32266](https://github.com/sgl-project/sglang/pull/32266) | 🚧 WIP [#32601](https://github.com/sgl-project/sglang/pull/32601) | 🚧 WIP [#32602](https://github.com/sgl-project/sglang/pull/32602) | Suggested status labels: - Production - Experimental - WIP - Planned - Unsupported A merged implementation does not automatically imply production readiness. The final status should consider correctness, accuracy, performance, representative TP/EP scenarios, CI coverage, and known limitations. ### Supporting PRs | Area | PR | |---|---| | MXFP8 Dense online/offline kernel reuse | [#28505](https://github.com/sgl-project/sglang/pull/28505) | | ModelSlim MXFP4 packed-weight loading | [#32013](https://github.com/sgl-project/sglang/pull/32013) | | Qwen3.5 A5 GemmaRMSNorm support | [#32745](https://github.com/sgl-project/sglang/pull/32745) | | Portable GemmaRMSNorm API | [sgl-kernel-npu #638](https://github.com/sgl-project/sgl-kernel-npu/pull/638) | --- ## 3. A5 Device-Specific Quantized Operator Adaptation The common quantization architecture continues to follow #31783: ```text Config → Method → Scheme → Kernel ``` This issue only focuses on hardware-specific differences observed when enabling MXFP on A5, including: - MXFP weight / scale layout differences - A2/A3/A5 operator availability - Dense / MoE operator differences - CANN / torch_npu API differences - Device-specific weight preprocessing The goal is to avoid leaking these details into: ```text Model / QuantMethod / QuantScheme ``` ### Reference: sgl-kernel-npu #638 `sgl-kernel-npu #638` provides a useful pattern for handling hardware-generation differences behind a stable operator API: ```text Stable Operator API │ ┌────────┴────────┐ ↓ ↓ A2/A3 A5 Native Provider A5 Provider ``` The upper SGLang layer does not need to detect the SoC or select the provider. For MXFP, we prefer the same principle where applicable: ```text Quant Scheme ↓ Stable Quantized Operator API ↓ sgl-kernel-npu │ ┌───┴────┐ ↓ ↓ A2/A3 A5 ``` If a hardware difference cannot be fully hidden by the kernel layer, we can then consider a minimal device-adaptation layer, following patterns such as vLLM-Ascend's `DeviceOperator / DeviceAdaptor`, instead of introducing a new Ascend quantization framework. Initial validation cases: 1. **MXFP scale / layout transformation** 2. **MXFP MoE operator differences** --- ## 4. H2 Work Items ### P0 — Capability Closure - [ ] Complete validation of major Qwen3 MXFP Dense / MoE paths - [ ] Complete Qwen3.5 Dense / MoE MXFP coverage - [ ] Validate major Online / ModelSlim paths - [ ] Resolve packed-weight, scale, and layout compatibility issues - [ ] Add necessary accuracy, performance, and CI coverage ### P0 — Device Adaptation - [ ] Identify A2/A3/A5 MXFP operator differences - [ ] Consolidate MXFP scale / layout transformations - [ ] Consolidate hardware-specific Dense / MoE operator dispatch - [ ] Prefer moving hardware differences into `sgl-kernel-npu` - [ ] Evaluate a minimal DeviceAdaptor only when kernel-level abstraction is insufficient ### P1 — Follow-up Based on actual requirements and profiling results: - MXFP Attention - MXFP / INT8 Communication - Additional model coverage --- ## 5. Out of Scope The following topics are not redesigned in #21584: - Common Quant Framework / Weight Lifecycle → #31783 - Global Capability Registry → #31774 - INT8 / FP8 / FP4 KV Cache → corresponding roadmaps/issues - Unified Communication Quantization → #31783 - Future formats such as HiF4 - A new `AscendQuantOps` framework --- ## Success Criteria 1. Close the major A5 MXFP8/MXFP4 Dense and MoE capability gaps. 2. Allow Qwen3.5 and future models to primarily reuse existing Scheme / Kernel implementations. 3. Avoid further propagation of A2/A3/A5 operator/layout differences into Model / QuantMethod code. 4. Prefer stable kernel APIs for hiding hardware-generation differences. 5. Keep #21584 as an accurate A5 MXFP model/format capability matrix. --- ## Related - #31783 — Quantization 2026 H2 Roadmap - #31774 — Capability / Compatibility Validation - #29762 — Ascend A5 MoE MXFP - #24695 — Ascend INT8 KV Cache - #29641 — Ascend FP8 KV Cache - #29913 — FP4 KV Roadmap - #20520 — Ascend TP Communication Compression - [sgl-kernel-npu #638](https://github.com/sgl-project/sgl-kernel-npu/pull/638) — Portable GemmaRMSNorm API",
"url": "https://github.com/sgl-project/sglang/issues/34559",
"createdAt": "2026-08-12T10:02:21Z",
"updatedAt": "2026-08-13T11:17:09Z",
"timestamp": "2026-08-13T11:17:09Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "wangyao-i",
"state": "open",
"assignees": [
"OrangeRedeng"
]
},
{
"id": "github:sgl-project/sglang:issue:34596",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] MI355X Qwen3.5 MTP throughput is significantly behind B200/B300 on realistic agentic workloads",
"text": "### Checklist - [x] I have searched related issues but cannot find a duplicate. - [x] The bug has not been fixed in the latest version. ### Describe the bug Hi @HaiShaw, On realistic Qwen3.5 397B FP4 AgentX workloads, MI355X (gfx950) is significantly behind the B200/B300 SGLang MTP frontiers at comparable P90 E2E normalized interactivity. The official MI355X sweep completed successfully across 16/16 throughput configurations plus eval, using TP2/EP2 and TP4 resident configurations. This makes the gap unlikely to be a failed-run or infrastructure artifact. In this comparison, the MI355X frontier peaks around 22k token throughput per chip, while B200/B300 are substantially higher across the overlapping interactivity range. The MI355X-only view also shows a relatively early throughput ceiling. Could you help investigate gfx950 tuning for Qwen3.5 hybrid/MTP workloads, particularly prefill/decode kernels and TP/EP behavior? We can provide complete engine arguments, per-rank logs, server metrics, and traces. ### Results ### Reproduction Configuration: https://github.com/SemiAnalysisAI/InferenceX/pull/2562 Fully green official sweep: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31581562219 ### Environment - GPU: AMD MI355X (gfx950) - Model: Qwen3.5 397B FP4 - Framework: SGLang v0.5.17, ROCm 7.2 - Speculative decoding: native EAGLE MTP - Tested resident topologies: TP2/EP2 and TP4",
"url": "https://github.com/sgl-project/sglang/issues/34596",
"timestamp": "2026-08-12T16:17:12Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "cquil11",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:issue:34603",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Should the trtllm allreduce fusion accumulate in fp32, like the mnnvl backends do?",
"text": "## Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ## Motivation ### Problem The mnnvl allreduce fusion backends sum the ranks in fp32. The trtllm one sums in bf16, rounding after every rank, unless you pass `fp32_acc`. Nothing ever passes it: ```python # flashinfer_comm_fusion.py:854, fused path -- fp32_acc defaults to False and no caller sets it fp32_acc=fp32_acc, # flashinfer_comm_fusion.py:953, allreduce-only path fp32_acc=False, ``` So at TP=8 trtllm does 7 bf16 roundings where mnnvl does 1, and the same model gets different numerics on SM90 (`auto` -> trtllm) than on Blackwell (`auto` -> mnnvl). ### Proposal Turn it on for TRTLLM to match the mnnvl behaviour. ### Facts | backend | accumulator | `fp32_acc` | | --- | --- | --- | | `trtllm` | bf16, or fp32 iff the flag is set | honors it | | `mnnvl` | always fp32 | ignores it | | `mnnvl-cutedsl` | always fp32 | raises | Also, our benchmark defaults `fp32_acc=True` (`benchmark_fused_collective.py:143`) Some E2E testing is useful. Maybe on some harder multi-turn benchmarks. ## Related resources - #19889 (TRTLLM allreduce fusion for Qwen 3.5) has an unanswered report of infinite generations and JSON schema errors with the fusion on, ~2-5 bad requests per run, while GPQA on the same PR was clean: https://github.com/sgl-project/sglang/pull/19889#issuecomment-4017098753. Might be unrelated, but it is the right shape. - #30700 added the allreduce-only path that also hardcodes `fp32_acc=False`.",
"url": "https://github.com/sgl-project/sglang/issues/34603",
"createdAt": "2026-08-12T17:31:38Z",
"updatedAt": "2026-08-13T11:55:51Z",
"timestamp": "2026-08-13T11:55:51Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"hopper"
],
"author": "b8zhong",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34604",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Kimi-K3 tool call parser fails ~8x/hour in production: TypeError 'string indices must be integers' and json 'unexpected character'",
"text": "## Describe the bug Running Kimi-K3 with `--tool-call-parser kimi_k3` in production (agentic coding workload, Claude-Code-style tool loop), the tool call parser raises on a significant fraction of model outputs. Over a 24h window with moderate traffic we counted **~190 parser failures**: - 106x `Tool call parsing error: string indices must be integers` (TypeError — a parsed call is indexed like a dict but is a str) - 84x `Tool call parsing error: unexpected character: line 1 column 1 (char 0)` (json.loads on a non-JSON argument payload) - 1x `Tool call parsing error: Input is a zero-length, empty document` Logged from `serving_chat.py` (\"Tool call parsing error\") — the request then falls back to returning the raw text instead of a structured `tool_calls` entry, so the client sees a turn without tool calls and the agent loop degrades or retries. ## Environment - SGLang 0.5.17-tree, ROCm (8x MI350X, gfx950), `--tool-call-parser kimi_k3 --reasoning-parser kimi_k3 --enable-strict-thinking`, grammar backend xgrammar - Failures occur both with streaming and non-streaming chat completions ## Analysis `function_call/kimik3_detector.py` extracts calls via `_CALL_RE`/`_ARG_RE` and then: - `json.loads(raw_value)` on each argument value — raises `unexpected character` when the model emits an unquoted/partial value (this is exactly the second error class), and - downstream consumers index `call[\"name\"]`/`call[\"arguments\"]` — the `string indices` TypeError suggests a code path where the parsed call is still a plain string (e.g. malformed/partial `<|open|>call ...` block that the regex only partially matches). ## Expected behavior Malformed tool-call blocks from the model should be handled defensively (skip the malformed call and/or return it as text with a warning), never raise inside the parser. Given Kimi-K3 is explicitly advertised for agentic workloads, hardening this parser has outsized impact — at our traffic every ~7-8 minutes an agent turn silently loses its tool call. ## Reproduction We do not yet have a captured raw failing sample (the error log does not include the payload). We are happy to add debug logging in production and attach concrete failing model outputs if that helps — please tell us the preferred capture point. ## Checklist - [x] Searched existing issues — found tool_call_id-related PRs for kimi-k3 but nothing covering these parser exceptions.",
"url": "https://github.com/sgl-project/sglang/issues/34604",
"createdAt": "2026-08-12T17:36:11Z",
"updatedAt": "2026-08-12T17:36:11Z",
"timestamp": "2026-08-12T17:36:11Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "stefanskiasan",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34611",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] ROCm MI355 HiCache Broken - Poor Performance for Realistic Agentic Workload",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug Hi @HaiShaw @1am9trash , this hicache bug doesnt seem exclusive to just qwen3.5 but affects lots of models ## AI analysis below Both HiCache arms attempted on v0.5.17-rocm720-mi35x-20260811 ([run 31579113764](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/31579113764)) died in scheduler init: 在 `v0.5.17-rocm720-mi35x-20260811` 上尝试的两个 HiCache 分支均在调度器初始化阶段失败： File \"sglang/srt/mem_cache/kv_cache_builder.py\", line 81, in maybe_register_hicache_draft File \"sglang/srt/mem_cache/hybrid_cache/hybrid_pool_assembler.py\", line 936, in build_full_draft_pools if pool.layer_num == 0: AttributeError: 'HybridLinearKVPool' object has no attribute 'layer_num' The hybrid stack itself is healthy on gfx950 — the same logs show 93.37 GB target KV and 50.65 GB Mamba host pools allocating per rank under page_first, then Attached hybrid pool stack to UnifiedRadixCache: pools=KV + MAMBA, transfer_layer_num=60. Only the MTP draft pool crashes. ### Reproduction `v0.5.17-rocm720-mi35x-20260811 ` ### Environment `v0.5.17-rocm720-mi35x-20260811 `",
"url": "https://github.com/sgl-project/sglang/issues/34611",
"createdAt": "2026-08-12T17:56:26Z",
"updatedAt": "2026-08-13T06:38:33Z",
"timestamp": "2026-08-13T06:38:33Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "functionstackx",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34629",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Regression][SM100/SM103] FlashInfer TRTLLM NVFP4 MoE tile-192 path produces NaNs",
"text": "## Summary `flashinfer_trtllm` online NVFP4 MoE becomes numerically invalid after upgrading the aligned FlashInfer trio beyond `0.6.16rc4`. On Blackwell, the newly exposed tile-192 TRTLLM_GEN BMM path produces non-finite MoE output; the registered 200-example GSM8K test consequently scores `0.0`. This is a dependency/integration regression, not a feature-specific regression in the SGLang change where it was first noticed: - the same pre-rebase SGLang checkout fails with FlashInfer `0.6.16.post4` and `0.6.17`; - that exact checkout passes when only the FlashInfer trio is changed to `0.6.15.post1` or `0.6.16rc4`; - forcing a valid tile-128 tactic passes the full correctness workload, while both shape-valid tile-192 tactics fail during startup autotuning before any serving request. ## Public CI symptom (B200) The registered test fails publicly on four NVIDIA B200 GPUs at SGLang `8531fe0a8ca4e3b6d92b4134be399aaac261c636`: - Job: https://github.com/sgl-project/sglang/actions/runs/31607763933/job/94151038518 - Driver: `580.126.09` - PyTorch: `2.13.0+cu130` - FlashInfer: `flashinfer-python==0.6.17`, `flashinfer-cubin==0.6.17`, `flashinfer-jit-cache==0.6.17+cu130` - First `M=896` prefill: `Assertion 'NaN detected! sampler: next_token_logits' failed.` on all four ranks - Result: GSM8K score `0.000` The B200 job independently corroborates the same first-`M=896`, all-rank NaN symptom on SM100. Its later CUDA abort/coredumps are downstream effects and are not separately root-caused here. ## B300 environment ```text Hardware: 8x NVIDIA B300 SXM6 AC (SM103, 267.7 GiB, 148 SMs); server uses TP4/EP4 Driver: 590.48.01 Container: lmsysorg/sglang:nightly-dev-cu13-20260812-c7c03ec5 amd64 digest: sha256:e948f2d4adb7d3ca8336d855f667ab26ab27bc7515702336e3c6f7093121bf23 OS: Ubuntu 24.04.4 LTS, Linux 6.8.0-137-generic, glibc 2.39 Python: 3.12.3 PyTorch / CUDA: 2.13.0+cu130 / 13.0 NCCL: 2.29.7 SGLang package-bisect SHA: 3f3002d4a88d964f2f19ef501755f994cd5d25cc Also reproduced at clean current SHA: 8531fe0a8ca4e3b6d92b4134be399aaac261c636 Model: Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 MoE backend: flashinfer_trtllm Quantization: nvfp4_online Workload: GSM8K completion API, 200 examples, 128 threads, max_tokens=512 ``` The environment held constant across the package bisect: ```text FLASHINFER_NVFP4_4OVER6=1 FLASHINFER_NVFP4_4OVER6_ERR_MODE=MSE FLASHINFER_NVFP4_4OVER6_ERR_USE_FAST_MATH=1 FLASHINFER_NVFP4_4OVER6_E4M3_USE_256=1 SGLANG_FP4_IGNORED_LAYERS=shared_expert,model.layers.40,model.layers.41,model.layers.42,model.layers.43,model.layers.44,model.layers.45,model.layers.46,model.layers.47 SGLANG_ENABLE_ASYNC_ASSERT=true ``` SGLang logs `--disable-shared-experts-fusion is automatically set`; shared-expert fusion is already disabled and is not the failing path. ## Reproduction The model requires Hugging Face authentication. Install SGLang from the tested checkout, then install an aligned FlashInfer trio. Use a fresh server and cache for each version. ```bash git clone https://github.com/zianglih/sglang.git cd sglang git checkout 3f3002d4a88d964f2f19ef501755f994cd5d25cc python3 -m pip install -e python VERSION=0.6.17 python3 -m pip uninstall -y \\ flashinfer-python flashinfer-cubin flashinfer-jit-cache rm -rf /root/.cache/flashinfer python3 -m pip install --no-cache-dir --no-deps \\ \"flashinfer-python==${VERSION}\" python3 -m pip install --no-cache-dir --no-deps \\ --index-url https://flashinfer.ai/whl \\ \"flashinfer-cubin==${VERSION}\" python3 -m pip install --no-cache-dir --no-deps \\ --index-url https://flashinfer.ai/whl/cu130 \\ \"flashinfer-jit-cache==${VERSION}+cu130\" python3 -m pip check python3 -m flashinfer show-config cd test SGLANG_ENABLE_ASYNC_ASSERT=true \\ python3 registered/backends/test_flashinfer_nvfp4_online_moe_backend.py \\ TestFlashinferTrtllmGenMoeBackendNvFp4Online.test_gsm8k -f ``` The test launches: ```text sglang serve \\ --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \\ --moe-runner-backend flashinfer_trtllm \\ --cuda-graph-max-bs-decode 128 \\ --tp-size 4 --ep-size 4 \\ --quantization nvfp4_online \\ --mem-fraction-static 0.7 ``` Repeat with `VERSION=0.6.15.post1`, `0.6.16rc4`, and `0.6.16.post4`. Keep `flashinfer-python`, `flashinfer-cubin`, and `flashinfer-jit-cache` aligned; the JIT cache uses the matching `+cu130` build. ## Package-only bisect on B300 Same SGLang SHA (`3f3002d4...`), image, model, arguments, environment, hardware, and full 200-example workload in every row: | Aligned FlashInfer trio | Result | GSM8K score | Latency | Throughput | First observed failure | |---|---:|---:|---:|---:|---| | `0.6.15.post1` | PASS | `0.960` | `4.2688 s` | `6747.30 tok/s` | none | | `0.6.16rc4` | PASS | `0.945` | `4.7273 s` | `6153.64 tok/s` | none | | `0.6.16.post4` | FAIL | `0.000` | `147.96 s` | n/a | pure prefill, `M=1792`; NaNs on all four ranks | | `0.6.17` | FAIL | `0.000` | n/a | n/a | first concurrent prefill; reduced trace reaches `M=896`; NaNs on all four ranks | This brackets the measured runtime regression after `0.6.16rc4`. FlashInfer source/tag inspection points to the `0.6.16rc5` line as the first carrying the corresponding TRTLLM_GEN BMM artifact/launcher update, but rc5 itself has not yet been run. ## Isolation matrix on 0.6.17 / B300 | Single change | Scope | Result | |---|---|---| | default | 1 example, 1 thread, `max_tokens=512` | PASS | | default | full 200/128/512 | FAIL, score `0.000` | | `--disable-prefill-cuda-graph` | full 200/128/512 | FAIL | | Triton attention + page size 64 | full 200/128/512 | FAIL | | `flashinfer_trtllm_routed` | reduced concurrent run | FAIL | | deferred finalize disabled | reduced concurrent run | FAIL | | `SGLANG_TRTLLM_MOE_PDL_MAX_TOKENS=0` | reduced concurrent run | FAIL | | `--enforce-disable-flashinfer-allreduce-fusion` | full 200/128/512 | FAIL | | `SGLANG_FORCE_COARSE_WAR_BARRIER=1` | full 200/128/512 | FAIL | | `--disable-flashinfer-autotune` | full 200/128/512 | FAIL | | force exact `[128,19]` through `M=16384`, v0.6.17 cubins | full 200/128/512, prefill graph disabled | PASS: score `0.955`, no non-finite marker or async assertion | | same `[128,19]`, substitute exact same-name v0.6.15 FC1+FC2 cubins | 24 requests / 24 threads / `max_tokens=1`, prefill graph disabled | PASS | | force `[192,1]` or `[192,3]`, v0.6.17 cubins | startup autotune, `M=4096` | FAIL: non-finite finalized MoE output on all ranks | | device sync at FP4 entry; stream sync after routing, FC1, explicit quantization, or FC2 | reduced eager run, one probe per run | FAIL at every boundary | | `--disable-overlap-schedule` | full 200/128/512 | PASS, score `0.955` | | `CUDA_LAUNCH_BLOCKING=1` | full 200/128/512 | PASS, score `0.940` | The attention backend and prefill CUDA graph are not required. PDL, routed-vs-FromLogits dispatch, deferred finalize, FlashInfer all-reduce fusion, and SGLang's coarse WAR barrier are insufficient explanations. Since tile 192 becomes non-finite during startup autotuning, SGLang request overlap/concurrency is not required; serialization merely prevents/hides the default serving failure. ## Root-cause status Confirmed: - This is FlashInfer-version dependent: package-only replacement flips pass/fail on identical SGLang code. - It reproduces on B200/SM100 and B300/SM103. - Layer 0 enters the FlashInfer FP4 MoE op with finite input, block scales, per-token scales, and router data, then receives non-finite finalized BF16 output. - It is unrelated to the CuTe DSL feature where the failure was first encountered: the same pre-rebase tree fails, `fp4_utils.py` is not on this runtime activation path, and the test uses `flashinfer_trtllm`. - Synchronizing at FP4 entry and after each high-level internal boundary (routing, FC1, explicit quantization, FC2) does not make the result finite. Strong current FlashInfer lead, not yet a final kernel root cause: - The last passing line pins `b368d003.../batched_gemm-da58956-b4ac80e`; failing versions pin the regenerated `5988e15c.../batched_gemm-3d40263-3e19f0a` artifact. - The artifact update first appears on the release branch in [FlashInfer #4252](https://github.com/flashinfer-ai/flashinfer/pull/4252) and on the v0.6.17 line in [#4180](https://github.com/flashinfer-ai/flashinfer/pull/4180). - The corrupt serving invocation is an aggregated MoE batch (`M=15605` in the finite trace), not the nearby scheduler line that reports 896 new tokens. - Forced tile-128 `[128,19]` passes the full correctness test with the v0.6.17 cubins (score `0.955`); the exact same-name v0.6.15 FC1+FC2 pair also stays finite in the reduced crash reproducer. - Tile 192 has only two shape-valid tactics for `I=768`: `[192,1]` and `[192,3]`. Both fail during the `M=4096` dummy-model forward used by startup autotuning. They change FC1 but share FC2 raw config 787, making that FC2 or its tile-192 integration contract the strongest localization. The finalized-output probe does not yet prove the first corrupt internal tensor. FlashInfer issue with the artifact/tactic analysis and package bisect: https://github.com/flashinfer-ai/flashinfer/issues/4486 ## Mitigation vs. fix Verified mitigations: - pin the aligned FlashInfer trio to `0.6.15.post1` or tested `0.6.16rc4`; or - suppress tile 192 / force a valid tile-128 tactic for this model shape; or - run with `--disable-overlap-schedule` as a serving workaround. `CUDA_LAUNCH_BLOCKING=1` is diagnostic only. The root fix belongs in FlashInfer's tile-192 BMM path; SGLang should also avoid accepting a numerically corrupt tactic and add an integration regression test. No exact SGLang duplicate was found for this version/shape/NaN regression.",
"url": "https://github.com/sgl-project/sglang/issues/34629",
"createdAt": "2026-08-12T20:55:11Z",
"updatedAt": "2026-08-13T01:19:53Z",
"timestamp": "2026-08-13T01:19:53Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "zianglih",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:issue:34631",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Guided decoding resumes one token too early for channel-framed reasoning formats (Muse Glimmer / ATEM)",
"text": "## Summary For a reasoning format where the **model writes its own channel header**, `ReasonerGrammarBackend` re-arms the grammar at the token that follows `think_end_token` — but at that point the model still owes the header of the answer channel. Those header tokens get mask-constrained, so the model is forced to emit `{` in a position where it has never seen `{` during training. Out of distribution, it falls back to the shortest string the schema accepts. Concretely, on Muse Glimmer (`--reasoning-parser muse`), **46.7 %** of guided JSON responses came back as a placeholder object that is schema-valid and useless, while the reasoning channel of those very same calls contained the answer, fully worked out. Repairing the boundary takes that to **0.0 %**. ## Environment - sglang `0.5.18.dev312+g979899449` - model: Muse Glimmer 30B (NVFP4), single GPU, xgrammar backend - flags: `--reasoning-parser muse --tool-call-parser muse` + a `response_format: json_schema` request ## Symptom 26 of the degenerate outputs were the same string, byte for byte: ```json { \"claims\": [ { \"quote\": \"...\", \"claim_type\": \"assertion\", \"expr\": \"...\"} ] } ``` That `\"...\"` appears nowhere in the prompt — it is not template copying. The `reasoning_content` of the same responses contains the finished work: ``` 8: short_interest_shares =13,442,896 . abs diff <=5% 9: days_to_cover =1.6 . abs diff <=5% Return JSON. Need claim_type assertion for each. Provide list. ``` The model does the work and then fails to transcribe it. ## Mechanism The Muse Glimmer chat template ends the generation prompt at `<|start|>assistant` with no recipient and no `<|message|>` (this is by design — `MuseGlimmerDetector` documents it). The model writes the channel framing itself: ``` to=self<|message|> <|eom|> <|start|>assistant to=user<|message|> <|eot|> ``` `<|eom|>` closes the `to=self` channel only. But: - `MuseGlimmerDetector` passes `think_end_token = \"<|eom|>\"` - `ReasonerGrammarObject.transfer_state` sets `tokens_after_end = 0` as soon as `think_end_ids` matches - `ReasonerGrammarObject.fill_vocab_mask` then applies the JSON mask to the **next** token so the 5 tokens of `<|start|>assistant to=user<|message|>` are masked away. The assumption baked into the state machine — *the answer body starts at the token right after the reasoning end token* — holds for ``-style formats and breaks for channel-framed ones. ## Minimal repro No model and no GPU: it builds the grammar object the way the scheduler does and prints which decode positions were mask-constrained. ```python from sglang.srt.constrained.base_grammar_backend import BaseGrammarBackend from sglang.srt.constrained.reasoner_grammar_backend import ReasonerGrammarBackend from sglang.srt.parser.reasoning_parser import ReasoningParser # real ids from the Muse Glimmer tokenizer EOM, START, MESSAGE = 200007, 200022, 200023 ASSISTANT, TO, EQ_USER, EQ_SELF = 140680, 328, 76976, 19669 REASONING_HEADER = [TO, EQ_SELF, MESSAGE] # \" to=self<|message|>\" REASONING = [1001, 1002] ANSWER_HEADER = [START, ASSISTANT, TO, EQ_USER, MESSAGE] # \"<|start|>assistant to=user<|message|>\" BODY = [2001, 2002] # '{\"a\": 1}' STREAM = REASONING_HEADER + REASONING + [EOM] + ANSWER_HEADER + BODY IDS = {\"<|eom|>\": [EOM], \"<|message|>\": [MESSAGE], \" to=self\": [TO, EQ_SELF]} class Tokenizer: def encode(self, text, add_special_tokens=False): return IDS[text] class Recorder: # stands in for the compiled JSON grammar def __init__(self): self.mask_calls = 0 self.accepted = [] def accept_token(self, token): self.accepted.append(token) def fill_vocab_mask(self, vocab_mask, idx): self.mask_calls += 1 class InnerBackend(BaseGrammarBackend): def _init_value_dispatch(self, key, require_reasoning): return Recorder() allocate_vocab_mask = move_vocab_mask = apply_vocab_mask = staticmethod( lambda *a, **k: None ) backend = ReasonerGrammarBackend( InnerBackend(), ReasoningParser(\"muse\", stream_reasoning=False), Tokenizer() ) obj = backend._init_value_dispatch((\"json\", \"{}\"), True) # require_reasoning=True masked = [] for token in STREAM: before = obj.grammar.mask_calls obj.fill_vocab_mask(None, 0) # constrain this position... masked.append(obj.grammar.mask_calls > before) obj.accept_token(token) # ...then take the token drawn at it first = next(i for i, m in enumerate(masked) if m) print(\"constrained :\", masked) print(\"resumes at :\", first, \"| answer body starts at:\", len(STREAM) - len(BODY)) ``` Output on `979899449`: ``` constrained : [False, False, False, False, False, False, True, True, True, True, True, True, True] resumes at : 6 | answer body starts at: 11 ``` Positions 6-10 are the channel header. They must not be constrained. ## Evidence that this is the whole story Two paired cells over 10 production inputs × 3 repetitions, continuing the **same** model-written reasoning through `/generate`, differing only by the 26 characters of channel header inserted before the grammar binds: | cell | grammar binds | degenerate | claims/item | |---|---|---|---| | grammar right after `<|eom|>` (replica of the chat path) | too early | **40.0 %** | 5 | | + `<|start|>assistant to=user<|message|>`, then grammar | correct | **0.0 %** | 13 | Same model, same prompt, same reasoning, same sampler. Sampling parameters and prompt lineage were tested separately and ruled out (T=0 → 43.3 %, T=1/top_p 0.95 → 50.0 %, a neutral minimal prompt → 96.7 % degenerate, i.e. worse). ## Proposed fix Let the detector declare the boundary, and give `ReasonerGrammarObject` one more state between \"thinking\" and \"generating\": - `BaseReasoningFormatDetector.grammar_channel_header_end: Optional[str] = None` — the token that **closes** a channel header. `None` (every existing detector) keeps today's behaviour exactly: the grammar resumes right after `think_end_token`. - `BaseReasoningFormatDetector.grammar_channel_reasoning_header: Optional[str] = None` — a marker meaning \"the header that just closed opens *another* reasoning channel\", so the grammar keeps waiting instead of constraining a second think block. - `MuseGlimmerDetector` declares `\"<|message|>\"` and `\" to=self\"`. - While the header window is open, no mask is filled and the inner grammar is not fed; when the header closes, the grammar resumes exactly at the body. - Fail-safe: if the header does not close within `SGLANG_MAX_CHANNEL_HEADER_TOKENS` tokens (default 16, negative = unlimited), the grammar resumes anyway and logs why — a guided request must never end up silently unconstrained. - `require_reasoning=False` is untouched: there is no reasoning channel to close, and raw `/generate` callers put the header in their own prompt. Rollback across the new state is handled by snapshotting the pre-generation state per accepted token, so speculative decoding and retraction stay correct. Patch and an off-server test suite (13 cases: arming position, a second reasoning channel, the fail-safe cap, rollback in each state, `copy()`, and a check that no other detector opts in) are attached; happy to open a PR. ## Scope Any channel-framed format is exposed, not just `claim_compile` or Muse Glimmer: every guided-decoding request served by such a model hits it. `GptOssDetector` has the same shape on paper — `think_end_token = \"<|end|>\"`, after which harmony writes `<|start|>assistant<|channel|>final<|message|>` — but I have no gpt-oss deployment to measure, so I am reporting that as a lead, not as a second confirmed bug.",
"url": "https://github.com/sgl-project/sglang/issues/34631",
"createdAt": "2026-08-12T21:14:52Z",
"updatedAt": "2026-08-12T21:14:52Z",
"timestamp": "2026-08-12T21:14:52Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "y-edy-os",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34675",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] DP-aware routing rank is dropped for OpenAI Responses API",
"text": "## Bug Description When DP-aware routing is enabled, the model gateway selects a DP rank and injects the rank into the serialized request body. For the OpenAI Responses API, `ResponsesRequest` does not declare the DP rank fields. Pydantic therefore ignores the injected fields during request parsing, and the selected rank is not forwarded to `GenerateReqInput`. As a result, `/v1/responses` requests can lose the DP-aware routing decision before reaching the scheduler. ## Reproduction 1. Enable DP-aware routing with multiple data-parallel ranks. 2. Send a request through the model gateway using `/v1/responses`. 3. Let the gateway inject `data_parallel_rank` into the request body. 4. Inspect the parsed `ResponsesRequest` or the generated `GenerateReqInput`. On the latest `main`, the injected DP rank is absent after `ResponsesRequest` validation. ## Expected Behavior The DP rank selected by the gateway should be preserved and forwarded to `GenerateReqInput`, consistently with the existing Chat Completions and Completions paths. The implementation should preserve the existing precedence between the `X-Data-Parallel-Rank` header and request-body rank fields. ## Additional Context PR #14647 fixed the corresponding DP-aware issue for the Completions and Chat Completions APIs. The Responses API path appears to be missing the equivalent field declaration and propagation logic.",
"url": "https://github.com/sgl-project/sglang/issues/34675",
"createdAt": "2026-08-13T05:23:32Z",
"updatedAt": "2026-08-13T07:29:34Z",
"timestamp": "2026-08-13T07:29:34Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "wanna-01",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34676",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Hybrid Mamba prefill allocation failure kills scheduler instead of returning request to waiting queue",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug occurs on SGLang 0.5.16, build commit `fdebc938f7f4d16fe6b9f55dcd9a767cf0899ea1`. - [x] Environment information and a reproduction configuration are included below. - [x] This is a bug report, written in English. ### Describe the bug Under sustained Kimi-K3 traffic with hybrid Mamba/radix caching, a prefill allocation failure terminates a scheduler rank and subsequently the whole server instead of leaving the request in the waiting queue, rechunking it, retracting work, or failing only that request. Observed fatal error on every scheduler rank: ```text RuntimeError: Prefill out of memory. Try to lower your batch size. Try to allocate 8192 tokens. Available full tokens: 8070656 (full_available_size=7936 + full_evictable_size_=8062720) Available mamba: 760 (available_size=57 + component_evictable_size_=703) ``` The process supervisor then reports that a child failed, sends `SIGQUIT`, and shuts down the complete process tree. There was no host OOM, CUDA Xid, or Docker OOM kill. The surprising part is that the request passes prefill admission but the later real allocation can still fail fatally: 1. `PrefillAdder.rem_total_tokens` for `is_hybrid_ssm_cache` considers the token allocator's `available_size()` plus `tree_cache.full_evictable_size()`. 2. The admitted request is added to `can_run_list` and removed from `waiting_queue` in `_get_new_batch_prefill_raw()`. 3. `new_batch.prepare_for_extend()` subsequently performs the real allocation. 4. `alloc_paged_token_slots_extend()` raises an uncaught `RuntimeError` when `allocator.alloc_extend()` returns `None`. There is no rollback that places the admitted prefill request back in `waiting_queue`, and no retry with a smaller chunk. Decode has a retraction path, but this late prefill allocation failure does not. Expected behavior: capacity pressure should preserve process availability. The request should remain/return to waiting, be retried with a smaller chunk after eviction, be preempted, or receive a request-scoped OOM response. A single allocation miss should not terminate every scheduler rank. ### Reproduction Serve Kimi-K3 with TP8/DCP4 and hybrid Mamba cache settings: ```bash sglang serve \\ --trust-remote-code \\ --model-path moonshotai/Kimi-K3 \\ --served-model-name kimi-k3-dev \\ --tp-size 8 \\ --dcp-size 4 \\ --disable-custom-all-reduce \\ --mem-fraction-static 0.93 \\ --kv-cache-dtype fp8_e4m3 \\ --mamba-ssm-dtype bfloat16 \\ --mamba-full-memory-ratio 7.21 \\ --max-mamba-cache-size 800 \\ --max-running-requests 60 \\ --context-length 262144 \\ --chunked-prefill-size 8192 ``` Run concurrent long-context requests until the Mamba/radix cache is under pressure and submit another request requiring an 8192-token prefill chunk. In our run, the allocator reported the values above and the uncaught exception terminated the server. Even if this particular memory split is too aggressive, the requested fix is graceful backpressure/recovery rather than changing the cache sizing defaults. ### Environment ```text Python: 3.12.3 CUDA available: True GPU 0-7: NVIDIA B300 SXM6 AC (275040 MiB each) Compute Capability: 10.3 CUDA_HOME: /usr/local/cuda NVCC: CUDA 13.0, V13.0.88 CUDA Driver Version: 595.71.05 PyTorch: 2.11.0+cu130 sglang: 0.5.16 sglang build commit: fdebc938f7f4d16fe6b9f55dcd9a767cf0899ea1 sglang-kernel: 0.4.5 flashinfer_python: 0.6.15.post1 flashinfer_cubin: 0.6.15.post1 flashinfer_jit_cache: 0.6.15.post1+cu130 triton: 3.6.0 transformers: 5.12.1 torchao: 0.17.0+cu130 numpy: 2.3.5 aiohttp: 3.14.3 fastapi: 0.140.0 huggingface_hub: 1.24.0 uvicorn: 0.51.0 uvloop: 0.22.1 NVIDIA topology: all eight GPUs connected via NV18 ulimit soft: 1024 ```",
"url": "https://github.com/sgl-project/sglang/issues/34676",
"createdAt": "2026-08-13T05:42:23Z",
"updatedAt": "2026-08-13T05:42:23Z",
"timestamp": "2026-08-13T05:42:23Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "architsinghai2",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34677",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Streaming: one unknown tool name discards or corrupts the other tool calls in the same parallel batch",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug When a model emits parallel tool calls and any one of them names a tool that was not offered, the streaming parser does not just drop that one call — it also loses or corrupts the valid calls around it. The non-streaming path in the same class gets this right, so the two disagree on identical model output. `parse_base_json` skips only the offending entry: ```python # base_format_detector.py for act in action: name = act.get(\"name\") if not (name and name in tool_indices): logger.warning(f\"Model attempted to call undefined function: {name}\") if not envs.SGLANG_FORWARD_UNKNOWN_TOOLS.get(): continue # Skip unknown tools (default legacy behavior) ``` `parse_streaming_increment` throws away the whole buffer instead: ```python # base_format_detector.py, in parse_streaming_increment if \"name\" in obj and obj[\"name\"] not in self._tool_indices: # Invalid tool name - reset state self._buffer = \"\" # also discards the calls batched behind it self.current_tool_id = -1 # the next valid call restarts at index 0 self.current_tool_name_sent = False if self.streamed_args_for_tool: self.streamed_args_for_tool.pop() # deletes the *previous* tool's record return StreamingParseResult() ``` Three separate consequences: 1. **`self._buffer = \"\"`** discards every byte not yet parsed, which for a parallel batch is all the remaining calls. 2. **`current_tool_id = -1`** means formats that resynchronise on their next marker restart numbering at 0, colliding with a call already delivered. 3. **`streamed_args_for_tool.pop()`** removes the entry belonging to the previous, already-streamed tool. `prev_tool_call_arr` and `streamed_args_for_tool` then disagree in length, and `serving_chat._check_for_unstreamed_tool_args` bails out on the empty list, so the end-of-stream argument flush silently goes missing. Nothing is logged on this path, so an operator gets no signal at all. ### Reproduction CPU only, no model or GPU required. Tools offered are `get_weather` and `get_time`; `rm_rf` is not offered. ```python from sglang.srt.entrypoints.openai.protocol import Function, Tool from sglang.srt.function_call.json_array_parser import JsonArrayParser from sglang.srt.function_call.qwen25_detector import Qwen25Detector TOOLS = [Tool(function=Function(name=\"get_weather\", parameters={})), Tool(function=Function(name=\"get_time\", parameters={}))] W = '{\"name\": \"get_weather\", \"arguments\": {\"city\": \"Tokyo\"}}' T = '{\"name\": \"get_time\", \"arguments\": {\"tz\": \"JST\"}}' X = '{\"name\": \"rm_rf\", \"arguments\": {\"path\": \"/\"}}' def stream(detector, text): \"\"\"Rebuild what an OpenAI streaming client would assemble.\"\"\" calls = {} for ch in text: # one character at a time for c in (detector.parse_streaming_increment(ch, TOOLS).calls or []): slot = calls.setdefault(c.tool_index, {\"name\": None, \"arguments\": \"\"}) if c.name: slot[\"name\"] = c.name if c.parameters: slot[\"arguments\"] += c.parameters return calls array = lambda *cs: \"[\" + \",\".join(c.replace('\"arguments\"', '\"parameters\"') for c in cs) + \"]\" tags = lambda *cs: \"\".join(f\"\\n{c}\\n\\n\" for c in cs) print(stream(JsonArrayParser(), array(X, W, T))) print(stream(Qwen25Detector(), tags(W, X, T))) ``` Observed on `bca8ed4af`: ``` {} {0: {'name': 'get_time', 'arguments': '{\"city\": \"Tokyo\"}{\"tz\": \"JST\"}'}} ``` The first line is `tool_choice=\"required\"` returning **no tool calls at all**. The second is worse than a drop: the client receives **one** call whose name has been overwritten and whose arguments are two JSON objects concatenated into a string that will not parse. ### Impact by format Every detector that routes through the shared state machine is affected — that is `JsonArrayParser` (the `tool_choice=\"required\"` / named-tool streaming path), `Qwen25Detector`, `HermesDetector`, `Llama32Detector`, `MistralDetector`, and `TrinityDetector` via `super().parse_streaming_increment`. | format | unknown call first | unknown call in the middle | | --- | --- | --- | | JSON array (`required`) | all calls lost | trailing calls lost | | Llama 3.2 (`;` separated) | all calls lost | trailing calls lost | | Qwen2.5 / Hermes (tag wrapped) | trailing call replayed at index 0 | two calls merged into one | Hallucinated tool names are ordinary model behaviour, especially with large tool lists, so this is reachable in normal agentic traffic rather than only under adversarial input. ### Expected behavior Match `parse_base_json`: drop the one call that names an unknown tool, keep the rest of the batch, leave already-delivered indices alone, and log it. ### Environment Reproduced on `bca8ed4af` (current main). The affected code is pure Python request parsing — no GPU, model, or accelerator involved. ### Related - Non-streaming behaviour is covered by `test/registered/unit/function_call/test_unknown_tool_name.py`; there is no streaming equivalent. - `SGLANG_FORWARD_UNKNOWN_TOOLS` is honoured by `parse_base_json` and by the hunyuan / gpt_oss / pythonic / lfm2 / apertus2509 detectors, but the shared streaming path ignores it entirely. Worth deciding separately whether streaming should honour it too; this report is only about the collateral damage to the *valid* calls.",
"url": "https://github.com/sgl-project/sglang/issues/34677",
"createdAt": "2026-08-13T06:03:58Z",
"updatedAt": "2026-08-13T06:03:58Z",
"timestamp": "2026-08-13T06:03:58Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "tancheng33",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34683",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Radix cache hit changes temperature=0 outputs for Gemma-4 (18/40 greedy flips); --enable-deterministic-inference fails to boot on SM100",
"text": "## Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version (reproduced at 983dfd6a9a1e). - [x] Environment info and a minimal reproducible demo are included below. - [x] Not a general question. - [x] English. ## Describe the bug Two related problems, found while investigating output reproducibility for `google/gemma-4-12B-it`: **1. A radix-cache hit changes temperature=0 outputs.** Submitting the *same request twice* (batch size 1, no concurrency, greedy) produces *different completions*: the second (warm) request takes the cache-hit extend path, whose numerics differ from fresh prefill, and greedy decoding flips at near-tie tokens. Cold-vs-cold is bit-exact (max |Δlogprob| = 0 over 200 output tokens across independently flushed runs), so this is not generic nondeterminism — the divergence is introduced exactly and only by prefix reuse. Measured on 1x B200 at 983dfd6a9a1e (single request, `/flush_cache` between cold runs, details in the repro below): | config | cold vs cold | cold vs warm (same request re-sent) | |---|---|---| | trtllm_mha (default for Gemma-4 on SM100), page 64, single prompt | bit-exact | same greedy tokens, max abs. dlogprob = 0.1007 | | trtllm_mha, page 64, 40-prompt sweep (InstructCoder, seed 1234) | bit-exact | **18/40 prompts flip** | | trtllm_mha, `--page-size 32`, single prompt | bit-exact | greedy text flips at output token 8 | | triton (page 1), single prompt | bit-exact | greedy text flips at output token 89 | | triton, 40-prompt sweep | bit-exact | **19/40 prompts flip** | | `--disable-radix-cache` | bit-exact | bit-exact | | triton + `--enable-deterministic-inference`, 40-prompt sweep | bit-exact | bit-exact, **0/40 flips** | Speculative decoding is unrelated: with FROZEN_KV_MTP on top, the per-position delta profile is bit-for-bit identical to the non-spec run. The flips are classic near-tie argmax flips: e.g. in the page-32 run the cold pass had a top-2 logprob margin of exactly 0.0 at the flip position; the warm pass shifts logits by ~1e-3 at the first decode step and up to ~0.1 later, and the tie resolves differently. Flipped continuations differ materially (e.g. `\"...invalid file types, and inappropriate content\"` vs `\"...invalid file extensions, and prohibited content\"`). Mechanism (source-level): on a radix hit only the uncached suffix rows go through the extend path — `cu_seqlens_q` built from `extend_seq_lens` when `extend_prefix_lens > 0` in `python/sglang/srt/layers/attention/trtllm_mha_backend.py`, and the 2-stage prefix+extend attention in `python/sglang/srt/layers/attention/triton_backend.py` — which is numerically different from computing the same rows inside a fresh full prefill. The deterministic-inference path already solves this for triton via the unified 1-stage extend kernel (`triton_backend.py::_forward_extend_unified`), and my measurements confirm it: 0/40 flips, max |Δlogprob| = 0. **2. The designed escape hatch cannot boot for Gemma-4 on SM100.** `--enable-deterministic-inference` with no `--attention-backend` fails at startup: ``` ValueError: Currently only ['ascend', 'fa3', 'fa4', 'flashinfer', 'triton'] attention backends are supported for deterministic inference, but you explicitly specified 'trtllm_mha'. ``` The user did not specify `trtllm_mha`. `_gemma4_overrides` (`python/sglang/srt/arg_groups/overrides.py`) injects `attention_backend=\"trtllm_mha\"` as the Gemma-4 default on SM100 *before* `_deterministic_attention_backend` runs, and the latter cannot distinguish a model-injected default from a user choice, so it errors instead of falling back. Note `_llama4_overrides` has the same pattern (forces `trtllm_mha` on SM100) and should hit the same failure. Since triton is Gemma-4-compatible, deterministic-capable, radix-cache-compatible under deterministic inference, and empirically lossless on cache hits (0/40 above), the Gemma-4 override should pick triton when deterministic inference is enabled. PR with the fix and a unit test: #34684. Suggested longer-term follow-ups (out of scope of the PR): cache-hit-invariant extend for trtllm_mha, and/or a docs note that radix-cache hits are not numerics-preserving in the default configuration. ## Reproduction Server (defaults; the model override selects trtllm_mha, page 64 on SM100): ```bash python -m sglang.launch_server --model-path google/gemma-4-12B-it \\ --context-length 8192 --mem-fraction-static 0.75 --port 30000 ``` Client: the attached script sends one fixed single-message chat completion (a ~276-token prompt from the public `likaixin/InstructCoder` dataset) three times with `temperature=0`, `max_tokens=200`, `logprobs`: coldA (after `POST /flush_cache`), coldB (after `/flush_cache` again), warm (immediately after coldB → radix hit). ``` coldA vs coldB (control): texts IDENTICAL, max |dlogprob| = 0.000000 coldB vs warm (cache hit): max |dlogprob| = 0.100713 # flips text with --page-size 32 or --attention-backend triton ``` repro.py ```python #!/usr/bin/env python3 \"\"\"Reproduce cold-vs-warm greedy divergence on a radix-cache hit. Usage: python repro.py [--port 30000] [--model google/gemma-4-12B-it] Sends the same single-user-message chat completion three times, greedy (temperature=0), batch size 1, no concurrency: 1. coldA - after POST /flush_cache 2. coldB - after POST /flush_cache again 3. warm - immediately after coldB, so the prompt prefix is a radix hit Compares texts and token logprobs. Expected: all three identical. Observed: coldA == coldB bit-exact, but warm diverges. \"\"\" import argparse import json import urllib.request PROMPT = \"\"\"Simplify and improve the readability of a complex SQL query while maintaining its functionality. \\\"SELECT p.patient_id, p.name, p.age, c.condition_name, t.treatment_name FROM patients AS p JOIN patient_conditions AS pc ON p.patient_id = pc.patient_id JOIN conditions AS c ON pc.condition_id = c.condition_id JOIN patient_treatments AS pt ON p.patient_id = pt.patient_id JOIN treatments AS t ON pt.treatment_id = t.treatment_id WHERE (c.condition_name LIKE '%diabetes%' OR c.condition_name LIKE '%hypertension%') AND t.treatment_name LIKE '%insulin%' ORDER BY p.age DESC;\\\"\"\"\" def post(url, payload=None, timeout=300): data = json.dumps(payload).encode() if payload is not None else b\"\" req = urllib.request.Request( url, data=data, headers={\"Content-Type\": \"application/json\"}, method=\"POST\" ) with urllib.request.urlopen(req, timeout=timeout) as r: return json.loads(r.read() or b\"{}\") def chat(base, model): r = post( f\"{base}/v1/chat/completions\", { \"model\": model, \"messages\": [{\"role\": \"user\", \"content\": PROMPT}], \"max_tokens\": 200, \"temperature\": 0.0, \"logprobs\": True, \"top_logprobs\": 2, }, ) ch = r[\"choices\"][0] lps = [(t[\"token\"], t[\"logprob\"]) for t in ch[\"logprobs\"][\"content\"]] return ch[\"message\"][\"content\"], lps def compare(tag, a, b): (ta, la), (tb, lb) = a, b max_dlp = max( (abs(x[1] - y[1]) for x, y in zip(la, lb)), default=float(\"nan\") ) if ta == tb: print(f\"{tag}: texts IDENTICAL, max |dlogprob| = {max_dlp:.6f}\") else: pos = next( (i for i, (x, y) in enumerate(zip(ta, tb)) if x != y), min(len(ta), len(tb)), ) print(f\"{tag}: texts DIFFER at char {pos} (max |dlogprob| = {max_dlp:.6f})\") print(f\" first: ...{ta[max(0, pos - 40):pos + 60]!r}\") print(f\" second: ...{tb[max(0, pos - 40):pos + 60]!r}\") def main(): ap = argparse.ArgumentParser() ap.add_argument(\"--port\", type=int, default=30000) ap.add_argument(\"--model\", default=\"google/gemma-4-12B-it\") args = ap.parse_args() base = f\"http://localhost:{args.port}\" post(f\"{base}/flush_cache\") cold_a = chat(base, args.model) post(f\"{base}/flush_cache\") cold_b = chat(base, args.model) warm = chat(base, args.model) # radix hit on the prompt prefix compare(\"coldA vs coldB (control)\", cold_a, cold_b) compare(\"coldB vs warm (cache hit)\", cold_b, warm) if __name__ == \"__main__\": main() ``` Boot failure repro: ```bash python -m sglang.launch_server --model-path google/gemma-4-12B-it --enable-deterministic-inference # ValueError: ... but you explicitly specified 'trtllm_mha'. ``` ## Environment ``` Python: 3.12.13 (main, Apr 14 2026, 14:29:00) [Clang 22.1.3 ] CUDA available: True GPU 0,1: NVIDIA B200 GPU 0,1 Compute Capability: 10.0 CUDA_HOME: /usr/local/cuda NVCC: Cuda compilation tools, release 13.1, V13.1.115 CUDA Driver Version: 595.71.05 PyTorch: 2.13.0+cu130 sglang: 0.0.0.dev1+g983dfd6a9 sglang-kernel: 0.4.6.post1 flashinfer_python: 0.6.15.post1 flashinfer_cubin: Module Not Found flashinfer_jit_cache: Module Not Found triton: 3.7.1 transformers: 5.12.1 torchao: 0.17.0 numpy: 2.3.5 aiohttp: 3.14.3 fastapi: 0.141.1 huggingface_hub: 1.27.0 interegular: 0.3.3 modelscope: 1.39.1 orjson: 3.11.9 outlines: 0.1.11 packaging: 26.3 psutil: 7.2.2 pydantic: 2.13.4 python-multipart: 0.0.32 pyzmq: 27.1.0 uvicorn: 0.52.1 uvloop: 0.22.1 vllm: Module Not Found xgrammar: 0.2.1 openai: 2.6.1 tiktoken: 0.13.0 anthropic: 0.121.0 litellm: Module Not Found torchcodec: 0.15.0 NVIDIA Topology: \u001b[4mGPU0 GPU1 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID\u001b[0m GPU0 X NV18 SYS SYS 0-47,96-143 0 N/A GPU1 NV18 X SYS SYS 0-47,96-143 0 N/A NIC0 SYS SYS X PIX NIC1 SYS SYS PIX X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node ``` --- *Disclosure: this investigation and the accompanying PR were prepared with AI assistance (reviewed and validated end-to-end on hardware before filing).*",
"url": "https://github.com/sgl-project/sglang/issues/34683",
"createdAt": "2026-08-13T06:23:54Z",
"updatedAt": "2026-08-13T06:25:00Z",
"timestamp": "2026-08-13T06:25:00Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "yashagar-cmu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34709",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] benchmark/endpoint.py forks the server process after the parent initializes the accelerator (breaks XPU, latent on CUDA)",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug launch_or_reuse_server starts the server with multiprocessing.Process (python/sglang/benchmark/endpoint.py:62), i.e. the platform default start method — fork on Linux. By that point the CLI parent has already initialized the accelerator in ServerArgs.__post_init__, so the child inherits a dead device context: - XPU: server_args.py:3594 → get_device_memory_capacity() → get_xpu_memory_capacity() → torch.xpu.mem_get_info() (utils/common.py:764) - CUDA: server_args.py:3677 → _handle_environment_variables() → get_device_sm() → torch.cuda.get_device_capability() (utils/common.py:611), reached whenever SGLANG_OPT_FP8_WO_A_GEMM is on — it defaults to True (environ.py:1133) launch_server itself forces spawn (entrypoints/engine.py:1703), but that runs inside the already-forked child, so it cannot help. The crash surfaces as soon as the child touches the device — on a multimodal model the warmup request's image preprocessing does image.to(device) — and is reported only as Server process exited during startup (exit code -9). On CUDA the same fork happens but usually goes unnoticed: the forked child is the HTTP/tokenizer-manager process, which does no device work for text-only models, and the scheduler processes it creates are spawned. ### Reproduction Steps as below ``` # NVIDIA python3 -m sglang.benchmark.one_batch_server --model-path Qwen/Qwen3.5-9B \\ --trust-remote-code --batch-size 1 --input-len 128 --output-len 8 --tp-size 4 #Error : RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method #INTEL python3 -m sglang.benchmark.one_batch_server --model-path Qwen/Qwen3.5-9B \\ --trust-remote-code --batch-size 1 --input-len 128 --output-len 8 --tp-size 4 --device xpu #Error : RuntimeError: Cannot re-initialize XPU in forked subprocess. To use XPU with multiprocessing, you must use the 'spawn' start method ``` ### Environment Nvidia Environment - sglang : 0.0.0.dev16528+gef3043a51 torch : 2.13.0+cu130 | cuda: 13.0 | hip: None | xpu: None python : 3.12.3 | Linux-5.15.0-1071-nvidia-x86_64-with-glibc2.39 device : 8 x NVIDIA A100-SXM4-40GB driver : 580.95.05 | mem GB: - 40GB Intel Environment - sglang : 0.5.18.dev364+g667e18d99 torch : 2.12.0+xpu | cuda: None | hip: None | xpu: 20250302 python : 3.12.13 | Linux-6.14.0-37-generic-x86_64-with-glibc2.41 device : 8 x Intel(R) Arc(TM) Pro B60 Graphics driver : 1.14.36300+8 | mem GB: 22.7",
"url": "https://github.com/sgl-project/sglang/issues/34709",
"createdAt": "2026-08-13T08:08:27Z",
"updatedAt": "2026-08-13T09:36:51Z",
"timestamp": "2026-08-13T09:36:51Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "dayanandav",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34716",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] /v1/responses: `created_at` is a float in streaming events but an int in non-streaming responses",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug The `/v1/responses` endpoint serializes `created_at` in two different wire formats depending on the request mode: - **Non-streaming** response body: `\"created_at\": 1786587534` (integer) - **Streaming** snapshot events (`response.created`, `response.in_progress`, `response.completed`): `\"created_at\": 1786588600.0` (float with a trailing `.0`) Root cause (from reading current `main`): - The non-streaming path serializes SGLang's own pydantic model, which types the field as an int: [`protocol.py` — `ResponsesResponse.created_at: int = Field(default_factory=lambda: int(time.time()))`](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/openai/protocol.py) - The streaming path wraps the snapshot in the official `openai` SDK event types (`ResponseCreatedEvent` / `ResponseInProgressEvent` / `ResponseCompletedEvent`, see [`serving_responses.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/entrypoints/openai/serving_responses.py)). The SDK's [`Response.created_at` is typed `float`](https://github.com/openai/openai-python/blob/main/src/openai/types/responses/response.py), so pydantic coerces the int to a float and `event.model_dump_json()` emits `1786588600.0`. So the same field on the same endpoint has two formats, and neither matches the other. For reference, the real OpenAI API emits an integer here in both modes. Impact: downstream consumers that parse `created_at` strictly as an integer work fine against non-streaming responses, then fail to deserialize exactly the three streaming snapshot events. In our case (a Go API gateway in front of SGLang) the dropped `response.completed` event silently broke usage accounting for every streaming request. We have since made our parser tolerant, but the inconsistency inside SGLang remains a compatibility trap. Suggested fix: emit an integer consistently in the streaming path as well (e.g. normalize `created_at` when building/serializing the snapshot events, since the field type lives in the `openai` SDK and cannot be changed there). ### Reproduction **Minimal demonstration — no server or GPU needed.** The coercion happens entirely inside the pydantic models, so it can be shown with just the `openai` SDK (openai 2.48.0, pydantic 2.13.4): ```python from openai.types.responses import Response, ResponseCreatedEvent resp = Response( id=\"resp_123\", created_at=1786588600, # int in error=None, incomplete_details=None, instructions=None, metadata=None, model=\"test\", object=\"response\", output=[], parallel_tool_calls=True, temperature=None, tool_choice=\"auto\", tools=[], top_p=None, ) event = ResponseCreatedEvent(response=resp, sequence_number=0, type=\"response.created\") print(type(event.response.created_at).__name__) # => float (coerced by the SDK's `created_at: float`) print(event.model_dump_json()) # => ... \"created_at\":1786588600.0 ... ``` This is exactly what the streaming path in `serving_responses.py` does (`event.model_dump_json()` on SDK event types), while the non-streaming path serializes SGLang's own `ResponsesResponse` with `created_at: int` — hence the two formats on the wire. **Against a live server** — any model reproduces it; nothing model-specific. ```bash python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-0.5B-Instruct --port 30000 ``` ```bash # Non-streaming: integer curl -s http://127.0.0.1:30000/v1/responses \\ -H \"Content-Type: application/json\" \\ -d '{\"model\": \"Qwen/Qwen2.5-0.5B-Instruct\", \"input\": \"hi\", \"max_output_tokens\": 16}' \\ | grep -o '\"created_at\":[0-9.]*' # => \"created_at\":1786587534 # Streaming: float with trailing .0 in response.created / response.in_progress / response.completed curl -sN http://127.0.0.1:30000/v1/responses \\ -H \"Content-Type: application/json\" \\ -d '{\"model\": \"Qwen/Qwen2.5-0.5B-Instruct\", \"input\": \"hi\", \"max_output_tokens\": 16, \"stream\": true}' \\ | grep -o '\"created_at\":[0-9.]*' | sort -u # => \"created_at\":1786588600.0 ``` ### Environment Apologies — I cannot provide `python3 -m sglang.check_env` output: the affected deployment is a managed GPUStack cluster (NVIDIA RTX Pro 6000D GPUs) serving DeepSeek-V4-Flash, and I have no shell access to the serving nodes. The bug should be environment-independent, though: it is a pure serialization/typing issue, root-caused above with source references to current `main`, and the float coercion is fully reproducible with the client-side snippet in the Reproduction section (run on macOS, Python 3.9, openai 2.48.0, pydantic 2.13.4 — output matches our production capture byte for byte).",
"url": "https://github.com/sgl-project/sglang/issues/34716",
"createdAt": "2026-08-13T08:50:10Z",
"updatedAt": "2026-08-13T08:50:10Z",
"timestamp": "2026-08-13T08:50:10Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "nikuleo",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34718",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] DeepSeek-V4 sparse attention indexer (`fp8_paged_mqa_logits`) illegal memory access with long-context requests",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug ## Summary Serving DeepSeek-V4-Flash-0731 on sglang 0.5.17 (8x H100, TP=8), the server reliably crashes with `CUDA_ERROR_ILLEGAL_ADDRESS` inside the sparse attention indexer's `fp8_paged_mqa_logits` DeepGEMM kernel when a long-context request (roughly 80K–130K input tokens) reaches its first decode step. It looks related to #26427 and #21532, but those are filed under HiSparse specifically; this reproduces without it. All 8 TP ranks crash simultaneously with the identical traceback, and the bug is 100% reproducible by sending a single long-context request to a freshly started server — no concurrency needed. ## Investigation notes (ruling out other causes) We went through several false leads before isolating this, noting them here in case they're useful: - **Not speculative decoding**: reproduces with `--speculative-algorithm` entirely unset (no DSpark/EAGLE). - **Not CUDA graphs, structurally**: with CUDA graphs enabled (default), the crash is instead first detected inside `decode_cuda_graph_runner.py` → `full_cuda_graph_backend.py:129` (`self._graphs[shape_key].replay()`), and in other runs at essentially random subsequent CUDA API calls (`torch.cat`, `TensorImpl::~TensorImpl()` during a stream-capture-status check, `result.copy_done.synchronize()`), consistent with CUDA's async error reporting (\"CUDA kernel errors might be asynchronously reported at some other API call\"). This made it look like a CUDA-graph-replay or radix-cache bug. Only after combining `CUDA_LAUNCH_BLOCKING=1` with `--disable-cuda-graph` did the traceback resolve to the true origin below — and the crash still occurs with `--disable-cuda-graph`, ruling out CUDA graphs as the root cause. ## Traceback (with `--disable-cuda-graph`) ``` File \".../sglang/srt/managers/scheduler.py\", line 3594, in run_batch batch_result = self.model_worker.forward_batch_generation( File \".../sglang/srt/managers/tp_worker.py\", line 593, in forward_batch_generation out = self.model_runner.forward( File \".../sglang/srt/model_executor/model_runner.py\", line 1614, in _forward_raw ret = self.eager_runner.execute( File \".../sglang/srt/model_executor/runner/eager_runner.py\", line 244, in _execute_decode return model_runner.model.forward( File \".../sglang/srt/models/deepseek_v4.py\", line 2423, in forward hidden_states, prev_residual, prev_post, prev_comb = layer( File \".../sglang/srt/models/deepseek_v4.py\", line 1698, in forward hidden_states = self.self_attn( File \".../sglang/srt/models/deepseek_v4.py\", line 1241, in forward q, kv = self._forward_prepare( File \".../sglang/srt/models/deepseek_v4.py\", line 1157, in _forward_prepare self.indexer( File \".../sglang/srt/layers/attention/dsv4/indexer.py\", line 961, in forward return attn_backend.forward_c4_indexer( File \".../sglang/srt/layers/attention/dsv4/indexer.py\", line 774, in forward_c4_indexer logits = fn( File \".../deep_gemm/__init__.py\", line 207, in fp8_paged_mqa_logits return _C.fp8_paged_mqa_logits(q, kv_cache, weights, context_lens, block_table, schedule_meta, max_context_len, clean_logits, indices) File \"python/tvm_ffi/cython/function.pxi\", line 968, in tvm_ffi.core.Function.__call__ tvm.error.InternalError: CUDA driver error (/deepgemm/csrc/apis/../jit_kernels/impls/runtime_utils.hpp:144): 700 (CUDA_ERROR_ILLEGAL_ADDRESS, an illegal memory access was encountered) ``` One of the 8 ranks instead surfaced a secondary/cascading symptom (the CUDA context on that rank was likely already poisoned by the same root fault before its next CUDA call): ``` File \".../sglang/srt/layers/attention/dsv4/indexer.py\", line 679, in forward_c4_indexer q_indexer, weights = self._forward_prepare_normal( File \".../sglang/srt/layers/attention/dsv4/indexer.py\", line 468, in _forward_prepare_normal weights = c4_indexer.compute_weights(x, skip_scale=True) File \".../sglang/srt/layers/attention/dsv4/indexer.py\", line 946, in compute_weights out, _ = self.weights_proj(x) File \".../sglang/srt/layers/linear.py\", line 282, in forward output = self.quant_method.apply(self, x, bias) File \".../sglang/srt/layers/quantization/unquant.py\", line 255, in apply return F.linear(x, layer.weight, bias) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasGemmEx(...)` ``` ## Expected behavior Long-context single requests (well within the configured `--context-length 262144`) should not crash the server via illegal memory access in the sparse attention indexer. ## Related issues - #26427 — DeepSeek-V4 HiSparse crashes/wedges with multiple long-context requests (same `fp8_paged_mqa_logits` / C4 indexer path, but filed with `--enable-hisparse` on; this report reproduces without it) - #21532 — CUDA illegal memory access when enabling HiSparse on disaggregated decode node with long prompts ### Reproduction ## Launch command ```bash sglang serve --host 0.0.0.0 --port 8000 \\ --model-path /path/to/DeepSeek-V4-Flash-0731 \\ --tp-size 8 --max-running-requests 256 \\ --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 \\ --mem-fraction-static 0.82 --moe-runner-backend marlin \\ --model-loader-extra-config '{\"enable_multithread_load\": true, \"num_threads\": 16}' \\ --chunked-prefill-size 8192 --context-length 262144 --allow-auto-truncate \\ --served-model-name deepseek-v4-flash-0731 --trust-remote-code --log-level debug --enable-metrics ``` ### Environment - sglang: 0.5.17 - Model: DeepSeek-V4-Flash-0731 - Hardware: 8x H100, single node - `flashinfer` version: 0.6.15.post1 1. Start the server fresh (cold start). 2. Send a single request (concurrency = 1) with ~80K+ input tokens. 3. The server crashes on all 8 TP ranks as soon as the request transitions from prefill to its first decode step. Observed with real production traffic (a single ~130K-token request, first request after server startup).",
"url": "https://github.com/sgl-project/sglang/issues/34718",
"createdAt": "2026-08-13T09:12:31Z",
"updatedAt": "2026-08-13T09:12:31Z",
"timestamp": "2026-08-13T09:12:31Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "Simcs",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34719",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Scheduler crashes with AttributeError ('list' object has no attribute 'tolist') on mixed batches with token_ids_logprob — prefill and decode paths, v0.5.14–v0.5.17",
"text": "## Summary The scheduler process dies with ``` AttributeError: 'list' object has no attribute 'tolist' ``` whenever a request using `token_ids_logprob` shares a batch with a request that does not. One scoring client can take the whole server down. This affects the released versions **v0.5.14, v0.5.16, and v0.5.17** (verified by reading each tag's source), in **both** the prefill and decode result paths of `scheduler_components/batch_result_processor.py`. ## Mechanism `next_token_token_ids_logprobs_val` holds one entry per request in the batch. Requests that asked for token-ids logprobs contribute GPU tensors; co-batched requests that did not contribute plain `[]` entries (in decode, `get_token_ids_logprobs_raw` appends a bare `[]` for `token_ids is None`; the prefill delayed-D2H path produces plain lists as well). The consumer then calls `.tolist()` unconditionally on every entry: ```python # batch_result_processor.py, v0.5.17 — same in v0.5.14/v0.5.16 if logits_output.next_token_token_ids_logprobs_val: logits_output.next_token_token_ids_logprobs_val = [ v.tolist() for v in logits_output.next_token_token_ids_logprobs_val ] ``` The same unguarded pattern appears twice — in the prefill-path `move_logprobs_to_cpu` and in `_normalize_decode_outputs` — and `next_token_top_logprobs_val` / `next_token_top_logprobs_idx` carry the identical latent bug at both sites. Solo scoring requests survive (batch of one, all tensors), so light testing passes; the first *mixed* batch in production kills the scheduler. The HTTP server keeps answering `/health` with 503, so under Kubernetes the pod sits dead until a liveness probe restarts it, and the next such request kills it again — a crash loop. ## Reproduction Server: `sglang serve --port 8000` (observed on Qwen3.5-9B-based checkpoints, but the bug is model-independent). Send two concurrent requests so they co-batch, one scoring + decoding, one plain: ```bash curl -s localhost:8000/generate -H 'Content-Type: application/json' -d '{ \"text\": \"The capital of France is\", \"sampling_params\": {\"max_new_tokens\": 8, \"temperature\": 0}, \"return_logprob\": true, \"token_ids_logprob\": [3837, 715]}' & curl -s localhost:8000/generate -H 'Content-Type: application/json' -d '{ \"text\": \"Water boils at a temperature of\", \"sampling_params\": {\"max_new_tokens\": 8, \"temperature\": 0}}' & wait ``` Observed traceback (v0.5.14, overlap event loop, decode path): ``` File \".../managers/scheduler.py\", line 1578, in event_loop_overlap pop_and_process() File \".../managers/scheduler.py\", line 1544, in pop_and_process self.process_batch_result(tmp_batch, tmp_result) File \".../managers/scheduler.py\", line 3375, in process_batch_result self.batch_result_processor.process_batch_result_decode(batch, result) File \".../scheduler_components/batch_result_processor.py\", line 657, in process_batch_result_decode next_token_ids, next_token_logprobs = self._normalize_decode_outputs( File \".../scheduler_components/batch_result_processor.py\", line 783, in _normalize_decode_outputs v.tolist() for v in logits_output.next_token_token_ids_logprobs_val AttributeError: 'list' object has no attribute 'tolist' ``` The prefill-path twin of this crash (`move_logprobs_to_cpu` site) was hit the same way by prefill-only scoring traffic (`max_new_tokens=0`, `logprob_start_len=N-1`). ## History and related work - I fixed this once in #25183 (May 2026), but the refactor #25709 deleted `scheduler_output_processor_mixin.py` the next day, so I closed the PR; the refactored `batch_result_processor.py` reintroduced the unguarded pattern at both sites. - #34075 (opened today) fixes the **decode producer** by keeping `token_ids is None` entries tensor-typed under `no_copy_to_cpu` — it would fix the decode crash, but not the prefill-path variant, and the consumer sites stay fragile (`next_token_top_logprobs_val/_idx` included). - #30188 reports an adjacent but distinct mixed-batch crash in the prefill token-id logprob extraction (`torch.tensor(None)`). ## Suggested fix We run this consumer-side guard in production (as an image patch on v0.5.14 and v0.5.16); it is small and mirrors the producer-side precedent in `GenerationBatchResult.copy_to_cpu_async` (`v.to('cpu', non_blocking=True) if torch.is_tensor(v) else v`): ```python v.tolist() if torch.is_tensor(v) else v for v in logits_output.next_token_token_ids_logprobs_val ``` applied to all six comprehensions (`token_ids_logprobs_val`, `top_logprobs_val`, `top_logprobs_idx` × prefill/decode sites). Happy to submit this as a PR (a revival of #25183 against the current layout) — it composes with #34075 rather than conflicting with it. ## Environment - sglang v0.5.14 (`lmsysorg/sglang:v0.5.14-cu130`) and v0.5.16 (`lmsysorg/sglang:v0.5.16-cu130-runtime`), crash observed in production on both; v0.5.17 verified affected by source inspection. - Single B200, TP=1, dense Qwen3.5-9B fine-tune, `--attention-backend trtllm_mha`, overlap scheduling (default).",
"url": "https://github.com/sgl-project/sglang/issues/34719",
"createdAt": "2026-08-13T09:36:02Z",
"updatedAt": "2026-08-13T09:36:02Z",
"timestamp": "2026-08-13T09:36:02Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "treekmk",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34720",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] [XPU] Qwen3.5 GDN + speculative decode: causal_conv1d_update_xpu() got an unexpected keyword argument 'intermediate_conv_window'",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug On --device xpu, any speculative-decoding run on a hybrid GDN/linear-attention model (Qwen3.5) crashes in the target-verify path during the server's own warmup generation, before any user request. gdn_backend.py:43-48 binds the XPU kernel as-is: elif is_xpu(): from sgl_kernel import causal_conv1d_fn_xpu, causal_conv1d_update_xpu causal_conv1d_update = causal_conv1d_update_xpu but the is_target_verify branch at gdn_backend.py:539 calls it with five spec-only kwargs that the XPU op schema does not have — intermediate_conv_window, intermediate_state_indices, retrieve_next_token, retrieve_next_sibling, retrieve_parent_token. The XPU signature stops at conv_state_indices / pad_slot_id. File \"python/sglang/srt/layers/attention/linear/gdn_backend.py\", line 539, in forward_extend mixed_qkv_processed = causal_conv1d_update( TypeError: causal_conv1d_update_xpu() got an unexpected keyword argument 'intermediate_conv_window' The scheduler dies, and the client then reports a downstream ConnectionError (secondary noise). Non-spec decode on the same model/backend is unaffected — only TARGET_VERIFY takes this branch. The NPU and CPU bindings a few lines below (gdn_backend.py:49-60) are wired the same way and look exposed to the identical gap; unverified. Issue is induce #31250 fix. ### Reproduction With simple below steps able to reproduce issue ``` python -m sglang.benchmark.one_batch_server \\ --model-path $MODEL_TAR --trust-remote-code \\ --device xpu --tp-size 4 \\ --mem-fraction-static 0.8 --disable-radix-cache --chunked-prefill-size -1 \\ --max-total-tokens 8192 --max-prefill-tokens 4096 --max-running-requests 1 \\ --batch-size 1 --input-len 512 --output-len 8 \\ --speculative-algorithm NEXTN TypeError: causal_conv1d_update_xpu() got an unexpected keyword argument 'intermediate_conv_window' ``` Note : #34712 is required to reproduce the above error using shell command. ### Environment Intel Environment - sglang : 0.5.18.dev364+g667e18d99 torch : 2.12.0+xpu | cuda: None | hip: None | xpu: 20250302 python : 3.12.13 | Linux-6.14.0-37-generic-x86_64-with-glibc2.41 device : 8 x Intel(R) Arc(TM) Pro B60 Graphics driver : 1.14.36300+8 | mem GB: 22.7",
"url": "https://github.com/sgl-project/sglang/issues/34720",
"createdAt": "2026-08-13T09:40:34Z",
"updatedAt": "2026-08-13T09:41:33Z",
"timestamp": "2026-08-13T09:41:33Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "dayanandav",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34737",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Stale staging watermark subscribers survive prefill failure",
"text": "## Checklist - [x] I searched related issues but found no solution. - [x] The bug persists on the latest `main` (`74c032234201`). - [x] Environment information and a deterministic reproduction are included below. - [x] This report is written in English. ## Describe the bug In heterogeneous-TP disaggregated serving with a staging buffer, the decode-side `DecodeStagingHandler._wm_subscribers` registry retains entries for failed or restarted prefill nodes for the lifetime of the decode process.\\n\\nHere, **staging** specifically means SGLang's official heterogeneous-TP staging/scatter protocol (`DecodeStagingHandler` plus its ZMQ `WATERMARK` messages). It does not mean a UCX/NIXL transport-internal host-staging implementation. This defect is in SGLang's Python control plane, and the common handler is used by both the NIXL and Mooncake backends. `CommonKVManager._handle_node_failure()` removes the failed node from `connection_pool`, disconnects its cached sockets, and fails affected rooms, but it does not remove the corresponding staging watermark subscriber. In addition, `register_wm_subscriber()` currently keeps the first request-scoped receiver for an endpoint instead of refreshing it. Every staging allocation release calls `_free_and_send_watermark()`, which broadcasts to every retained subscriber. After a prefill failure/restart, this continues sending to dead endpoints. On current `main`, #31144 bounds each ZMQ send with a 1-second timeout, but stale entries can accumulate and multiply that delay on every allocation release. On SGLang 0.5.16, before #31144, the same path can block indefinitely and stall benchmark progress. This is separate from #31217: that PR releases outstanding staging allocations and handles failed rooms, but the process-lifetime watermark subscriber registry is not pruned. It is also separate from #29978, which retries/replays staging control messages rather than managing failed subscriber lifecycle. ## Reproduction Topology used in production: - SGLang's official heterogeneous-TP staging/scatter path, using the NIXL backend - two prefill instances, each `--tp-size 2` - one decode instance, `--tp-size 1` - sustained benchmark traffic - restart or lose one prefill instance after it has registered for watermark updates Representative launch arguments (model and orchestration details omitted because the bug is model-independent): ```bash # Prefill (two instances) ... --tp-size 2 \\ --disaggregation-mode prefill \\ --disaggregation-transfer-backend nixl # Decode ... --tp-size 1 \\ --disaggregation-mode decode \\ --disaggregation-transfer-backend nixl ```\\n\\nAfter the failed prefill exceeds the heartbeat failure threshold: 1. Its entries are removed from `CommonKVManager.connection_pool`. 2. Its TCP endpoints are disconnected. 3. Its key remains in `DecodeStagingHandler._wm_subscribers`. 4. Every later `_free_and_send_watermark()` still attempts to send to that stale receiver. A deterministic unit reproduction is: 1. Register two receivers with different bootstrap-info tuples. 2. Populate `connection_pool` with those tuples. 3. Call `_handle_node_failure()` for one prefill bootstrap address. 4. Observe that the failed receiver remains in `_wm_subscribers` on current `main`. In the patched production deployment, the cleanup path removed the failed subscriber and the existing #31217 teardown path then released 11 outstanding unscattered staging allocations. ## Expected behavior When a prefill node is declared failed, all staging watermark subscribers associated with its cached bootstrap-info groups should be removed. Registering the same endpoint again should refresh the request-scoped receiver/session entry. ## Environment Prefill nodes: ```text Python: 3.12.4 GPU: NVIDIA GeForce RTX 4090 CUDA: 12.9 CUDA driver: 590.48.01 PyTorch: 2.11.0+cu129 SGLang: 0.5.16 sglang-kernel: 0.4.5+cu129 pyzmq: 27.1.0 ``` Decode node: ```text Python: 3.12.4 GPU: NVIDIA H20-3e CUDA: 12.9 CUDA driver: 575.57.08 PyTorch: 2.11.0+cu129 SGLang: 0.5.16 sglang-kernel: 0.4.5+cu129 pyzmq: 27.1.0 ``` The affected registry and failure path remain present on latest `main` at `74c032234201`.",
"url": "https://github.com/sgl-project/sglang/issues/34737",
"createdAt": "2026-08-13T12:25:25Z",
"updatedAt": "2026-08-13T12:37:48Z",
"timestamp": "2026-08-13T12:37:48Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "fly-go-run",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34740",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] SGLANG_SIMULATE_ACC_LEN silently degrades detokenization to O(n²) — `predict.fill_(100)` emits a byte-fallback token, and the `endswith(\"\\ufffd\")` commit gate then never advances the incremental-detokenization offsets",
"text": "### Checklist - [x] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [ ] Please use English. Otherwise, it will be closed. ### Describe the bug Two independent defects that compose into one failure mode. **Defect A (benchmark path).** `generate_simulated_accept_index()` overwrites every accepted token id with the hardcoded constant `100`: ```python num_correct_drafts.fill_(simulate_acc_len - 1) predict.fill_(100) # some legit token id ``` The comment only establishes that `100` is *in range*; it does not establish that id `100` **decodes to a complete character**. In any byte-fallback vocabulary the low ids are single-byte tokens. On DeepSeek-V4-Pro (`len(tokenizer) == 129280`) ids **97–111 all decode to U+FFFD** (`�`), and `100` sits inside that range: ``` 90:'x' 91:'y' 92:'z' 93:'{' 94:'|' 95:'}' 96:'~' 97:'�' 98:'�' 99:'�' 100:'�' 101:'�' ... ``` So with `SGLANG_SIMULATE_ACC_LEN` enabled, the entire model output becomes a run of `�`. **Defect B (production path).** `DetokenizerManager` decides whether incremental detokenization may *commit* using a character-level heuristic: ```python new_text = read_texts[i][len(surr_texts[i]) :] if recv_obj.finished_reasons[i] is None: if new_text and not new_text.endswith(\"\\ufffd\"): s.append_decoded_text(new_text) s.surr_offset = s.read_offset # only place the left edge advances s.read_offset = len(s.decode_ids) ... else: printable = find_printable_text(new_text) # no commit; offsets frozen ``` `endswith(\"\\ufffd\")` is used as a proxy for \"the trailing bytes form an incomplete character\". These are **not** the same condition: U+FFFD is also a perfectly legal, complete character that a model can emit, or that can survive from the input. When the two are conflated, the commit gate can be held shut indefinitely. `surr_offset` is the *only* left edge from which a window may be decoded, because it marks the last position known to be a character boundary. Freezing it means the window `decode_ids[surr_offset:]` grows to cover the entire generated sequence, and — since `tokenizer.decode()` is a stateless API with no resumable decoder state — **every output step must re-decode the whole history from the beginning**. Per-request detokenization cost goes from O(n) to O(n²). **Composed effect.** Defect A manufactures exactly the input that trips Defect B: an output stream whose tail is `\\ufffd` forever. The result is a single-process CPU bottleneck that back-pressures the whole engine. #### Impact measured Same server config (graph128 / `mem-fraction 0.92` / `max-running-requests 1024` / `tokenizer-worker-num 32` / EAGLE 3-4 / tp8 dp8 dp-attention), same client (concurrency 1024, 16384 requests, input 60000 / output 1000, `ignore_eos`), only `SGLANG_SIMULATE_ACC_LEN` and `--detokenizer-worker-num` varied: | acceptance | `detokenizer_worker_num` | client throughput (tok/s) | server throughput (tok/s) | running-req / rank | median TTFT | |---|---:|---:|---:|---:|---:| | natural | 1 | 36,591 | 38,893 | 108.12 | 148.6 ms | | natural | 32 | 37,151 | 41,525 | 125.54 | 161.3 ms | | fixed 0.92 (`SIMULATE_ACC_LEN=3.76`) | 1 | 28,699 | 29,538 | **59.60** | **18,013.8 ms** | | fixed 0.92 (`SIMULATE_ACC_LEN=3.76`) | 32 | 39,264 | 43,917 | 126.42 | 154.5 ms | Reading: 1. Turning on the simulation switch costs **−21.6% client throughput** and inflates median TTFT **121×** (148.6 ms → 18.0 s), purely from detokenization. 1. Raising `--detokenizer-worker-num` to 32 recovers it entirely (+36.8% over the degraded run, and above the natural-acceptance baseline). On the natural side the same change is worth only +1.5%, i.e. the amplification is specific to the simulated path. 1. The engine is *starved*, not saturated: during the degraded run GPU was 96% busy, KV usage only 43%, and `num_prealloc_queue_reqs` / `num_transfer_queue_reqs` / `num_queue_reqs` were 0 throughout. Requests finish decoding on the GPU but their text cannot reach the client, so a closed-loop client never frees slots. 1. Client-visible symptom: **all 16,384 responses came back with `generated_text` empty** while `meta_info` reported 1000 generated tokens each, because `find_printable_text()` returns `\"\"` for a run of `�` (no space to break on). The natural-acceptance run returned non-empty text for all 16,384 (median 998 chars). #### Why the cost is O(n²) With output length 1000 and accept length 3.76 there are ~266 output events per request. | | window decoded per event | times each output token is re-decoded | total token-decodes per request | |---|---:|---:|---:| | offsets advance (normal) | ~11 tokens (constant) | ~3 | ~2,900 | | offsets frozen (this bug) | 5 → 1,005 tokens | ~136 average, 266 for the first token | ~136,000 | `5×266 + 3.76×(266×267/2) ≈ 134,900` — the `1+2+…+n` term is the O(n²). The ratio is ≈45× at 1000 output tokens and **grows linearly with output length**. The same factor applies to the Python-side work per event: the `read_texts[i][len(surr_texts[i]):]` slice, the `rfind(\" \")` inside `find_printable_text()`, and the `decode_ids[surr_offset:]` list copy all scale with the window. Secondary risk: `decode_status[rid].decode_ids` never stops growing, so detokenizer memory grows with output length, and entries evicted under `SGLANG_DETOKENIZER_MAX_STATES` pressure raise the `RuntimeError` at `detokenizer_manager.py:339-346`. ### Reproduction Environment: DeepSeek-V4-Pro, tp8 / dp8, `--enable-dp-attention`, EAGLE MTP (`--speculative-num-steps 3 --speculative-num-draft-tokens 4`), `--tokenizer-worker-num 32`, `--detokenizer-worker-num 1` (default). 1. Start the server with `SGLANG_SIMULATE_ACC_LEN=3.76` and `SGLANG_SIMULATE_ACC_METHOD=match-expected`. 1. Drive a closed-loop streaming load (input 60000, output 1000, `ignore_eos`, concurrency 1024) against `/generate`. 1. Observe: median TTFT in the tens of seconds, `#running-req` roughly half of the KV-permitted batch, `full token usage` well below 1.0, all queue depths 0, GPU busy — and every response's `text` empty. 1. Restart with `--detokenizer-worker-num 32`, change nothing else: throughput and TTFT return to the natural-acceptance level. Defect B can also be reproduced without the simulation switch, using the tokenizer alone — this mirrors the exact two-window arithmetic in `_decode_batch_token_id_output`: ```python from transformers import AutoTokenizer tk = AutoTokenizer.from_pretrained(MODEL) FFFD = \"\\ufffd\" def commits(tid, rounds=4, per_round=4, tail=(100000, 20, 300, 4000, 50)): ids, surr, read = list(tail), 0, len(tail) for r in range(rounds): ids.extend([tid] * per_round) new = tk.decode(ids[surr:], skip_special_tokens=True)[ len(tk.decode(ids[surr:read], skip_special_tokens=True)):] if not (new and not new.endswith(FFFD)): return False, r + 1 # offsets frozen from here on surr, read = read, len(ids) return True, None commits(100) # -> (False, 1) byte-fallback token: never commits commits(67) # -> (True, None) 'a': commits every round ``` Full-vocabulary scan on DeepSeek-V4-Pro (129,280 ids, `skip_special_tokens=True`), classified by the commit gate's two requirements (non-empty and not ending in U+FFFD): | class | count | share | trips the bug? | |---|---:|---:|---| | decodes to a string ending in U+FFFD (byte-fallback) | 1,279 | 0.99% | yes — 0/100 sampled ids were safe | | decodes to `\"\"` (special tokens skipped) | 1,230 | 0.95% | yes — `new_text` is falsy, same else-branch | | decodes to a complete visible character | 126,771 | 98.06% | no — 0/300 sampled ids were unsafe | ## Expected behavior 1. `SGLANG_SIMULATE_ACC_LEN` should change *how many* draft tokens are accepted, without changing the cost profile of the return path. Today it silently makes the measurement it exists to enable unreliable. 1. Incremental detokenization should keep its window bounded regardless of output content, so that a stream containing U+FFFD does not stall streaming or degrade to O(n²). ### Proposed fixes **Fix A — do not hardcode the simulated token id** (`python/sglang/srt/speculative/spec_utils.py:337`). A safe id is vocabulary-dependent (here `100` is a byte token while `67` is `'a'`; another tokenizer may invert that), so a constant cannot be correct in general. Derive it from the tokenizer, e.g. `tokenizer.encode(\"a\", add_special_tokens=False)[0]`, or pick the first id whose single-token decode is non-empty and does not end in U+FFFD. **Fix B — test byte completeness, not the U+FFFD character** (`python/sglang/srt/managers/detokenizer_manager.py:354`). Compare bytes consumed versus bytes decoded, or use a tokenizer API that reports trailing incomplete bytes, instead of `new_text.endswith(\"\\ufffd\")`. This is the more important fix: it is reachable on production traffic whenever real output contains U+FFFD (in our natural-acceptance run, 17 of 16,384 responses contained it). **Fix C — bound the window defensively.** Even with Fix B, cap how far `surr_offset` may lag (a complete character cannot span more than a handful of tokens). Force-commit once the uncommitted tail exceeds that bound, so no input can drive this path quadratic. Note that `find_printable_text()` (`python/sglang/utils.py:369-385`, borrowed from HF `TextStreamer`) participates in the symptom: for a run of `�` with no space it returns `\"\"`, so streaming clients receive empty chunks. In `TextStreamer` this heuristic only delays printing; using it behind a *commit* gate turns a display delay into unbounded state growth. ### Environment - sglang v0.5.14 (`SGLANG_BUILD_COMMIT=49e384ce9d304648e9959666ecb8ce8cd98d0deb`), torch 2.11.0+cu130, CUDA 13.0, NVIDIA driver 580.126.09 - Model: DeepSeek-V4-Pro, weights and KV cache `fp8_e4m3`, attention backend `dsv4`, MoE via MegaMoE - 8×GPU single node, tp8 / dp8 / ep8, `--enable-dp-attention`, PD-disaggregated decode node (`--disaggregation-mode decode --disaggregation-transfer-backend fake`) - `--tokenizer-worker-num 32`; `--detokenizer-worker-num` 1 (default) versus 32 ### Note on the mitigation and its precondition `--detokenizer-worker-num 32` fully absorbs the amplification, but it only works because request routing supplies enough distinct keys. `MultiDetokenizerRouter._pick()` shards on `zlib.crc32(recv_obj.http_worker_ipc) % num_workers` (`python/sglang/srt/managers/multi_tokenizer_mixin.py:500-501, 521, 540`) — the key is the owning **tokenizer** worker, not the request id. Effective parallelism is therefore capped by `--tokenizer-worker-num`; with `--tokenizer-worker-num 1`, every request hashes to the same detokenizer worker and `--detokenizer-worker-num 32` has no effect at all. This is worth documenting alongside the flag.",
"url": "https://github.com/sgl-project/sglang/issues/34740",
"createdAt": "2026-08-13T12:49:56Z",
"updatedAt": "2026-08-13T12:51:41Z",
"timestamp": "2026-08-13T12:51:41Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "hope1262946533",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:34758",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Router GEMM should keep fp32 output under deterministic inference (DeepSeek V3/V4)",
"text": "### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Motivation The DeepSeek router (MoE gate) GEMM produces **fp32** logits on every code path *except* deterministic inference, where it silently drops to **bf16**. The precision drop is not required by batch invariance — it is purely an epilogue/store-dtype choice, and the batch-invariant kernel we already run accumulates in fp32 and even has an fp32 store branch that we never take. **Where it happens** — `DeepseekV2MoEGate.forward`, `python/sglang/srt/models/deepseek_v2.py:512`: ```python if get_exec().deterministic.enable_deterministic_inference: return F.linear(hidden_states, self.weight, None) # -> bf16 logits ``` Every other branch in that same function returns fp32: | path | output dtype | | --- | --- | | `dsv3_router_gemm(..., out_dtype=torch.float32)` (CUDA, M ≤ 4/16) | fp32 | | `linear_bf16_fp32(...)` (cuBLAS bf16 x bf16 -> fp32) | fp32 | | prefill-CP path (`linear_bf16_fp32`) | fp32 | | **deterministic (`F.linear`)** | **bf16** | **Why fp32 output does not conflict with batch invariance.** Batch invariance is a property of the *reduction schedule* (tiling, split-K, accumulation order), not of the epilogue dtype. Under deterministic mode `F.linear` dispatches to `aten::mm`/`aten::addmm` -> `matmul_persistent` -> `matmul_kernel_persistent`, which already does: ```python accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32) ... elif c_ptr.dtype.element_ty == tl.float32: c = accumulator.to(tl.float32) # batch_invariant_ops.py:167 — branch exists, never reached ``` The fp32 accumulator is already there and the kernel already has an fp32 store branch. It is *the exact same kernel with the exact same reduction order* — only the final `tl.store` dtype changes. What blocks it is that `_matmul_persistent_triton` hardcodes the output allocation to the input dtype (`dtype = a.dtype`, `batch_invariant_ops.py:185`), so there is no way to ask `matmul_persistent` for an fp32 result. The same applies to the JIT `dsv3_router_gemm` kernel: one CTA per expert, a fixed `warp::reduce_sum` + fixed `kNumWarps` tree over K, and the reduction order is independent of `kNumTokens` — the `kOutFloat` template parameter changes only the final `cast(final_sum)` store. Same kernel, different epilogue. Related: `_mm_dtype_compat` (`batch_invariant_ops.py:967`), registered for `aten::mm.dtype`, honors the requested dtype but not the requested precision: ```python def _mm_dtype_compat(self, mat2, out_dtype): return matmul_persistent(self.contiguous(), mat2.contiguous()).to(out_dtype) ``` This rounds fp32 accumulator -> bf16 -> fp32. A caller asking for `out_dtype=torch.float32` gets bf16-valued fp32 tensors, which is exactly what the caller was trying to avoid. **Why it matters.** Router logits feed `noaux_tc` top-k with an fp32 `e_score_correction_bias`, over 256 (V3/V3.2) or 384 (K2) experts. bf16 has an 8-bit mantissa, so adjacent expert scores routinely fall inside one bf16 ULP; rounding the logits before group-limited top-k can flip expert selection. That is a discontinuous change, not a small numeric one — a token routed to a different expert set produces a materially different output. So deterministic mode today is self-consistent but systematically diverges from the non-deterministic path, and it adds an avoidable source of divergence to prefill-vs-decode logprob (KL) consistency, where a routing flip shows up as a spike rather than as noise. ### Proposed change 1. Thread an `out_dtype` (or a preallocated `output`) through `matmul_persistent` / `_matmul_persistent_triton` so the fp32 store branch in `matmul_kernel_persistent` is actually reachable. No change to tiling, split, or accumulation order, so batch invariance is preserved by construction. 2. Make `_mm_dtype_compat` use that instead of the `.to(out_dtype)` post-cast, so `aten::mm.dtype` under deterministic mode stops double-rounding. 3. In `DeepseekV2MoEGate.forward`, stop early-returning bare `F.linear` and instead request an fp32-out batch-invariant matmul, so deterministic and non-deterministic modes both hand fp32 logits to top-k. 4. Keep a single kernel for all M under deterministic mode. The non-deterministic path switches kernels at `max_router_gemm_tokens` (4 on SM100/103, 16 elsewhere); that token-count-dependent switch is itself batch-variant, so deterministic mode should not adopt it even though `dsv3_router_gemm` is internally batch-invariant. 5. `_matmul_persistent_deepgemm` allocates `out` as `a.dtype` and calls `deep_gemm.bf16_gemm_nn`; either confirm an fp32-out entry point exists or route fp32-out requests to the Triton path. Perf cost is negligible here — the router GEMM is `[M, 7168] x [7168, 256]`, and the change is a store dtype, not an extra kernel. ### Related resources - `python/sglang/srt/models/deepseek_v2.py:512` — the deterministic early return - `python/sglang/srt/batch_invariant_ops/batch_invariant_ops.py:167,185,275,967` — fp32 accumulator, hardcoded out dtype, `matmul_persistent`, `_mm_dtype_compat` - `python/sglang/kernels/jit/csrc/gemm/dsv3_router_gemm.cuh` — `kOutFloat` epilogue switch - #12095 — `[2/2] Deepseek deterministic: support deepseek v3 deterministic inference on 8 x H200`, which introduced the `F.linear` fallback",
"url": "https://github.com/sgl-project/sglang/issues/34758",
"createdAt": "2026-08-13T17:36:25Z",
"updatedAt": "2026-08-13T17:36:25Z",
"timestamp": "2026-08-13T17:36:25Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "b8zhong",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:5979",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Feature] Add KV cache usage prometheus metrics",
"text": "### Motivation It would be great to track prometheus metrics for KV cache utilization. ### Related resources vLLM already offers KV cache utilization prometheus metrics, see [here](https://docs.vllm.ai/en/stable/serving/metrics.html), at `vllm:gpu_cache_usage_perc`.",
"url": "https://github.com/sgl-project/sglang/issues/5979",
"createdAt": "2025-05-02T14:59:34Z",
"updatedAt": "2026-08-13T08:37:37Z",
"timestamp": "2026-08-13T08:37:37Z",
"metrics": {
"reactions": 5,
"comments": 11
},
"labels": [
"good first issue"
],
"author": "jorgeantonio21",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:issue:9867",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Long DeepGEMM v2 warmup time in latest SGLang leading to NCCL timeout.",
"text": "### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed. - [x] 5. Please use English, otherwise it will be closed. ### Describe the bug We've observed that when initializing the latest version of SGLang with DeepGEMM v2, the warmup process for DeepGEMM is excessively long, frequently causing NCCL c10d timeout. Specifically, we are seeing the following startup times: * The prefill nodes take ~23 minutes to startup. * The decode nodes take ~11 minutes to startup. Notably, these long warmup times persist even on subsequent (non-first) launches. As a temporary workaround, we have set `TORCH_DISTRIBUTED_DEFAULT_TIMEOUT=1800` to prevent the NCCL c10d timeout. * DeepGEMM warmup time on prefill nodes: * NCCL c10d timeout: ``` 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Scheduler hit an exception: Traceback (most recent call last): 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 384, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.capture() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 492, in capture 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] ) = self.capture_one_batch_size(bs, forward) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 663, in capture_one_batch_size 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] run_once() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 652, in run_once 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] logits_output_or_pp_proxy_tensors = forward( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py\", line 120, in decorate_context 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return func(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 2257, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] hidden_states = self.model( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 2128, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] hidden_states, residual = layer( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 1939, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] hidden_states = self.mlp( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 465, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self.forward_deepep(hidden_states, forward_batch) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 620, in forward_deepep 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] final_hidden_states = self.experts( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/ep_moe/layer.py\", line 431, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] dispatch_output = self.dispatch( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/ep_moe/layer.py\", line 497, in dispatch 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self.deepep_dispatcher.dispatch( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/two_batch_overlap.py\", line 974, in dispatch 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._execute(\"dispatch\", **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/two_batch_overlap.py\", line 971, in _execute 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return getattr(self._inners[tbo_subbatch_index or 0], name)(**kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 831, in dispatch 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.dispatch_a(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 851, in dispatch_a 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] inner_state = self._get_impl(forward_batch).dispatch_a( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 492, in dispatch_a 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] buffer = self._get_buffer() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 609, in _get_buffer 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return DeepEPBuffer.get_deepep_buffer( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 185, in get_deepep_buffer 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] cls._buffer = Buffer( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/deep_ep/buffer.py\", line 87, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] device_ids = all_gather_object(local_device_id) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/deep_ep/buffer.py\", line 68, in all_gather_object 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] dist.all_gather_object(object_list, obj, group) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return func(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py\", line 3168, in all_gather_object 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] all_gather(object_size_list, local_size, group=group) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return func(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py\", line 3879, in all_gather 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] work = group.allgather([tensor_list], [tensor], opts) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] torch.distributed.DistBackendError: [10] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: wait timeout after 600000ms, keys: /default_pg/0//3//cuda//0 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Exception raised from doWait at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:597 (most recent call first): 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x80 (0x7f57fd97eeb0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #1: + 0x5cd0381 (0x7f57ef0d0381 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #2: + 0x1361e6c (0x7f57ea761e6c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #3: c10d::TCPStore::doGet(std::__cxx11::basic_string, std::allocator > const&) + 0x33 (0x7f57ef167fe3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #4: c10d::TCPStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x114 (0x7f57ef1690f4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #5: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #6: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #7: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #8: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::__cxx11::basic_string, std::allocator > const&, int) + 0x5c4 (0x7f57ae645bd4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::__cxx11::basic_string, std::allocator > const&, c10::Device&, c10d::OpType, int, bool) + 0x1bba (0x7f57ae64867a in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #11: c10d::ProcessGroupNCCL::allgather(std::vector >, std::allocator > > >&, std::vector >&, c10d::AllgatherOptions const&) + 0x1689 (0x7f57ae655ec9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #12: + 0x5d088c0 (0x7f57ef1088c0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #13: + 0x5d15d19 (0x7f57ef115d19 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #14: + 0x53413e5 (0x7f57ee7413e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #15: + 0x5d22cbc (0x7f57ef122cbc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #16: + 0x5d2eb2a (0x7f57ef12eb2a in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #17: + 0xcb1789 (0x7f57fe6b1789 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #18: + 0x381cdf (0x7f57fdd81cdf in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #19: sglang::scheduler_DP10_TP10_EP10() [0x4fdcf7] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #20: _PyObject_MakeTpCall + 0x25b (0x4f747b in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #21: sglang::scheduler_DP10_TP10_EP10() [0x509d6f] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #22: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #23: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #24: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #25: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #26: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #27: _PyEval_EvalFrameDefault + 0x13b3 (0x4ef4e3 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #28: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #29: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #30: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #31: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #32: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #33: _PyEval_EvalFrameDefault + 0x31f (0x4ee44f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #34: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #35: _PyObject_FastCallDictTstate + 0x17d (0x4f687d in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #36: sglang::scheduler_DP10_TP10_EP10() [0x5075b8] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #37: _PyObject_MakeTpCall + 0x2ab (0x4f74cb in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #38: _PyEval_EvalFrameDefault + 0x56d2 (0x4f3802 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #39: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #40: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #41: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #42: _PyEval_EvalFrameDefault + 0x731 (0x4ee861 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #43: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #44: _PyEval_EvalFrameDefault + 0x13b3 (0x4ef4e3 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #45: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #46: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #47: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #48: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #49: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #50: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #51: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #52: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #53: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #54: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #55: _PyEval_EvalFrameDefault + 0x13b3 (0x4ef4e3 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #56: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #57: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #58: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #59: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #60: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #61: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #62: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] . This may indicate a possible application crash on rank 0 or a network set up issue. 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] During handling of the above exception, another exception occurred: 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Traceback (most recent call last): 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/managers/scheduler.py\", line 2612, in run_scheduler_process 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] scheduler = Scheduler( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/managers/scheduler.py\", line 325, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.tp_worker = TpWorkerClass( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py\", line 67, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.worker = TpModelWorker( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/managers/tp_worker.py\", line 84, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.model_runner = ModelRunner( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/model_runner.py\", line 245, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.initialize(min_per_gpu_memory) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/model_runner.py\", line 350, in initialize 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.init_device_graphs() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/model_runner.py\", line 1622, in init_device_graphs 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] CudaGraphRunner(self) if not _is_npu else NPUGraphRunner(self) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 386, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] raise Exception( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Exception: Capture cuda graph failed: [10] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: wait timeout after 600000ms, keys: /default_pg/0//3//cuda//0 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Exception raised from doWait at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:597 (most recent call first): 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x80 (0x7f57fd97eeb0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #1: + 0x5cd0381 (0x7f57ef0d0381 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #2: + 0x1361e6c (0x7f57ea761e6c in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #3: c10d::TCPStore::doGet(std::__cxx11::basic_string, std::allocator > const&) + 0x33 (0x7f57ef167fe3 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #4: c10d::TCPStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x114 (0x7f57ef1690f4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #5: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #6: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #7: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #8: c10d::PrefixStore::get(std::__cxx11::basic_string, std::allocator > const&) + 0x30 (0x7f57ef117700 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #9: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::__cxx11::basic_string, std::allocator > const&, int) + 0x5c4 (0x7f57ae645bd4 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #10: c10d::ProcessGroupNCCL::initNCCLComm(std::__cxx11::basic_string, std::allocator > const&, c10::Device&, c10d::OpType, int, bool) + 0x1bba (0x7f57ae64867a in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #11: c10d::ProcessGroupNCCL::allgather(std::vector >, std::allocator > > >&, std::vector >&, c10d::AllgatherOptions const&) + 0x1689 (0x7f57ae655ec9 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #12: + 0x5d088c0 (0x7f57ef1088c0 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #13: + 0x5d15d19 (0x7f57ef115d19 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #14: + 0x53413e5 (0x7f57ee7413e5 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #15: + 0x5d22cbc (0x7f57ef122cbc in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #16: + 0x5d2eb2a (0x7f57ef12eb2a in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #17: + 0xcb1789 (0x7f57fe6b1789 in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #18: + 0x381cdf (0x7f57fdd81cdf in /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #19: sglang::scheduler_DP10_TP10_EP10() [0x4fdcf7] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #20: _PyObject_MakeTpCall + 0x25b (0x4f747b in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #21: sglang::scheduler_DP10_TP10_EP10() [0x509d6f] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #22: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #23: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #24: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #25: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #26: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #27: _PyEval_EvalFrameDefault + 0x13b3 (0x4ef4e3 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #28: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #29: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #30: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #31: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #32: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #33: _PyEval_EvalFrameDefault + 0x31f (0x4ee44f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #34: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #35: _PyObject_FastCallDictTstate + 0x17d (0x4f687d in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #36: sglang::scheduler_DP10_TP10_EP10() [0x5075b8] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #37: _PyObject_MakeTpCall + 0x2ab (0x4f74cb in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #38: _PyEval_EvalFrameDefault + 0x56d2 (0x4f3802 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #39: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #40: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #41: _PyFunction_Vectorcall + 0x6f (0x4fe13f in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #42: _PyEval_EvalFrameDefault + 0x731 (0x4ee861 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #43: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #44: _PyEval_EvalFrameDefault + 0x13b3 (0x4ef4e3 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #45: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #46: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #47: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #48: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #49: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #50: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #51: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #52: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #53: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #54: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #55: _PyEval_EvalFrameDefault + 0x13b3 (0x4ef4e3 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #56: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #57: _PyEval_EvalFrameDefault + 0x4b26 (0x4f2c56 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #58: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #59: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #60: _PyEval_EvalFrameDefault + 0x2b79 (0x4f0ca9 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #61: sglang::scheduler_DP10_TP10_EP10() [0x509a7e] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #62: PyObject_Call + 0xb8 (0x50a5a8 in sglang::scheduler_DP10_TP10_EP10) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] . This may indicate a possible application crash on rank 0 or a network set up issue. 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Possible solutions: 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 2. set --cuda-graph-max-bs to a smaller value (e.g., 16) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 3. disable torch compile by not using --enable-torch-compile 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] ``` ### Reproduction Please refer to start-up command-line arguments in PR https://github.com/sgl-project/sglang/pull/9660 ### Environment N/A",
"url": "https://github.com/sgl-project/sglang/issues/9867",
"createdAt": "2025-09-01T09:34:19Z",
"updatedAt": "2026-08-13T12:09:33Z",
"timestamp": "2026-08-13T12:09:33Z",
"metrics": {
"reactions": 0,
"comments": 18
},
"labels": [
"high priority",
"inactive"
],
"author": "Sulfur6",
"state": "closed",
"assignees": [
"Alcanderian",
"FlamingoPg",
"BBuf",
"ishandhanani"
]
},
{
"id": "github:sgl-project/sglang:pull_request:17788",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] top k int32 overflow",
"text": "## Motivation Fixes #17646 When a user configures top_k with a value exceeding int32 range (e.g., 2147483648), the server crashes with: RuntimeError: value cannot be converted to type int32 without overflow This happens because the top_ks tensor is created with dtype=torch.int32, and values beyond 2147483647 cause an overflow. While this is an outlier configuration, we cannot enforce constraints on user behavior, and the service should handle it gracefully rather than crashing. ## Modifications Sanitize top-k values before creating the tensor in sampling_batch_info.py Clamp values >= vocab_size to vocab_size (semantically equivalent to \"no top-k filtering\") Handle TOP_K_ALL sentinel value explicitly Optimize need_top_k_sampling flag: skip unnecessary top-k sampling work when all requests have top_k >= vocab_size This follows the same approach used by vLLM: if 0 < top_k < self.vocab_size: self.top_k_reqs.add(req_id)else: top_k = self.vocab_size ## Accuracy Tests N/A - This change does not affect model outputs. ## Benchmarking and Profiling N/A - This change does not impact inference speed. The optimization to need_top_k_sampling may slightly improve performance for edge cases where users pass very large top-k values. ## Checklist [x] Format your code according to the Format code with pre-commit. [ ] Add unit tests according to the Run and add unit tests. [ ] Update documentation according to Write documentations. [ ] Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed. [x] Follow the SGLang code style guidance.",
"url": "https://github.com/sgl-project/sglang/pull/17788",
"createdAt": "2026-01-27T00:58:10Z",
"updatedAt": "2026-08-12T20:15:38Z",
"timestamp": "2026-08-12T20:15:38Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "JingwenGu0829",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:18157",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Multimodal] Add multimodal special tokens for video&audio in Qwen_vl",
"text": "## Motivation Update qwen_vl's `MultimodalSpecialTokens` to support video&audio token's regex ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance).",
"url": "https://github.com/sgl-project/sglang/pull/18157",
"createdAt": "2026-02-03T06:52:21Z",
"updatedAt": "2026-08-12T20:34:43Z",
"timestamp": "2026-08-12T20:34:43Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "coding-famer",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:18432",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Doc] Replace model docs with redirect stubs to cookbook",
"text": "## Motivation Addresses [sglang#18427](https://github.com/sgl-project/sglang/issues/18427) — remove duplicated documentation between `sglang/docs` and `sgl-cookbook`. A companion PR to `sgl-cookbook` (see [https://github.com/sgl-project/sgl-cookbook/pull/141]) merges all model-specific content from `sglang/docs` into the cookbook. **This PR handles the `sglang` side:** replacing model docs with redirect stubs so that existing URLs don't 404. **The change is a SITUATION 2 change as mentioned in the issue since there's a lot of unique contents on sglang document compared to cookbook.** ## Modifications ### Approach: Redirect stubs instead of hard delete Existing bookmarks, external links, and search engine results point to `docs.sglang.io/basic_usage/.html`. Deleting the files would cause 404 errors. Instead, each file is replaced with a short redirect stub that: 1. Clearly states the content has moved. 2. Links directly to the corresponding cookbook page(s). Three sglang docs covered multiple models in a single file. These become \"disambiguation\" stubs listing all relevant cookbook pages: | sglang doc | Cookbook pages | |---|---| | `deepseek_v3.md` | DeepSeek-V3, DeepSeek-V3_1, DeepSeek-R1 | | `glm45.md` | GLM-4.5, GLM-4.6, GLM-4.7 | | `glmv.md` | GLM-4.5V, GLM-4.6V | The other 7 docs are 1-to-1 redirects. ### Files changed (11 total) **Landing page** - `docs/basic_usage/popular_model_usage.rst` — Updated intro to direct users to the cookbook. Toctree preserved so the stub pages remain discoverable. **1-to-1 redirect stubs (7 files)** - `docs/basic_usage/gpt_oss.md` → [GPT-OSS](https://cookbook.sglang.io/autoregressive/OpenAI/GPT-OSS) - `docs/basic_usage/minimax_m2.md` → [MiniMax-M2](https://cookbook.sglang.io/autoregressive/MiniMax/MiniMax-M2) - `docs/basic_usage/qwen3.md` → [Qwen3-Next](https://cookbook.sglang.io/autoregressive/Qwen/Qwen3-Next) - `docs/basic_usage/qwen3_vl.md` → [Qwen3-VL](https://cookbook.sglang.io/autoregressive/Qwen/Qwen3-VL) - `docs/basic_usage/llama4.md` → [Llama4](https://cookbook.sglang.io/autoregressive/Llama/Llama4) - `docs/basic_usage/deepseek_ocr.md` → [DeepSeek-OCR](https://cookbook.sglang.io/autoregressive/DeepSeek/DeepSeek-OCR) - `docs/basic_usage/deepseek_v32.md` → [DeepSeek-V3_2](https://cookbook.sglang.io/autoregressive/DeepSeek/DeepSeek-V3_2) **1-to-many redirect stubs (3 files)** - `docs/basic_usage/deepseek_v3.md` → DeepSeek-V3, DeepSeek-V3_1, DeepSeek-R1 - `docs/basic_usage/glm45.md` → GLM-4.5, GLM-4.6, GLM-4.7 - `docs/basic_usage/glmv.md` → GLM-4.5V, GLM-4.6V ## Accuracy Tests N/A — documentation-only change. No model code, kernel, or inference logic is affected. ## Benchmarking and Profiling N/A — documentation-only change. No performance impact. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance).",
"url": "https://github.com/sgl-project/sglang/pull/18432",
"createdAt": "2026-02-08T01:17:13Z",
"updatedAt": "2026-08-12T20:15:30Z",
"timestamp": "2026-08-12T20:15:30Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"documentation",
"deepseek"
],
"author": "JingwenGu0829",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:18589",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Make per‑token expert‑distribution recorder use asynchronous data syncing",
"text": " ## Motivation The existing MoE expert distribution recorder on \"per-token\" mode introduces too much overhead to be viable in production, especially when collecting traces from real‑world traffic at scale. As a result, we cannot reliably capture expert‑routing patterns, which are crucial for diagnosing expert imbalance and understanding MoE behavior under load. This also prevents any possibility of using live expert distribution signals to drive dynamic scaling or scheduling decisions. In addition, when the recorder runs in \"per‑token\" mode, its buffer grows without bound. Without safeguards or limits, this can easily lead to out‑of‑memory failures, making the feature unreliable and unsafe to enable in both stress testing and production environments. ## Modifications ## Accuracy Tests ## Benchmarking and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review Process 1. Ping Merge Oncalls to start the PR flow. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - `/tag-run-ci-label`, `/rerun-failed-ci`, `/tag-and-rerun-ci` 4. After green CI and required approvals, ask Merge Oncalls to merge. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/18589",
"createdAt": "2026-02-11T04:11:45Z",
"updatedAt": "2026-08-13T09:46:38Z",
"timestamp": "2026-08-13T09:46:38Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "zonghaoliu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:19889",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Use TRTLLM allreduce fusion for Qwen 3.5",
"text": "Before: 21.5 us After 10.4 us This PR is mainly authored by @vincentzed",
"url": "https://github.com/sgl-project/sglang/pull/19889",
"createdAt": "2026-03-04T17:14:25Z",
"updatedAt": "2026-08-12T17:36:27Z",
"timestamp": "2026-08-12T17:36:27Z",
"metrics": {
"reactions": 1,
"comments": 8
},
"labels": [
"run-ci"
],
"author": "b8zhong",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:21831",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[npu] Install vocab for gpt-oss in docker",
"text": " ## Motivation When running the gpt-oss-120b-bf16 model, the service failed to start with the error: openai_harmony.HarmonyError: error downloading or loading vocab file. ## Modifications docker/npu.Dockerfile ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #26940694162](https://github.com/sgl-project/sglang/actions/runs/26940694162) Latest PR Test (Extra): :x: [Run #26940693900](https://github.com/sgl-project/sglang/actions/runs/26940693900) ",
"url": "https://github.com/sgl-project/sglang/pull/21831",
"createdAt": "2026-04-01T08:09:58Z",
"updatedAt": "2026-08-13T00:52:35Z",
"timestamp": "2026-08-13T00:52:35Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"npu"
],
"author": "gjsheu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:21961",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: correct typos in benchmark scripts",
"text": "Two small typo fixes: 1. **bench_one_batch.py**: \"United Kindom\" → \"United Kingdom\" 2. **bench_serving.py**: \"Fail to load tokenizer config\" → \"Failed to load tokenizer config\"",
"url": "https://github.com/sgl-project/sglang/pull/21961",
"timestamp": "2026-08-12T16:30:07Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [],
"author": "Ricardo-M-L",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:22068",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: remove duplicate words in comments and docstrings",
"text": "## Summary - Fix 7 instances of duplicate words across the codebase: - `for for` → `for` (3 instances) - `with with` → `with` (3 instances) - `from from` → `from` (1 instance) ## Test plan - [ ] Verify no functional changes, comments/docstrings only 🤖 Generated with [Claude Code](https://claude.com/claude-code)",
"url": "https://github.com/sgl-project/sglang/pull/22068",
"createdAt": "2026-04-03T18:30:26Z",
"updatedAt": "2026-08-12T16:41:36Z",
"timestamp": "2026-08-12T16:41:36Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"sgl-kernel",
"diffusion"
],
"author": "Ricardo-M-L",
"state": "closed",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:22085",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: correct typos and grammar in code comments",
"text": "## Summary - Fix typo: `datastrucutre` → `datastructure` - Fix grammar: `consider simplify` → `consider simplifying` - Fix missing apostrophe: `dont` → `don't` (2 instances) - Fix grammar: `Avoid to divide zero` → `Avoid division by zero` (3 instances, matching 9+ existing uses in codebase) - Fix possessive: `it's own` → `its own` - Fix grammar: `allows to fallback` → `allows falling back` - Fix spelling + double space: `loose` → `lose` ## Test plan - [ ] Comment-only changes, no functional impact 🤖 Generated with [Claude Code](https://claude.com/claude-code)",
"url": "https://github.com/sgl-project/sglang/pull/22085",
"createdAt": "2026-04-04T00:58:43Z",
"updatedAt": "2026-08-12T16:41:39Z",
"timestamp": "2026-08-12T16:41:39Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"deterministic"
],
"author": "Ricardo-M-L",
"state": "closed",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:22191",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] model: support LongCat-AudioDiT",
"text": "## Motivation This PR adds support for LongCat-AudioDiT, a state-of-the-art (SOTA) diffusion-based text-to-speech (TTS) model that directly operates on the waveform latent space. * Huggingface: https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B * GitHub: https://github.com/meituan-longcat/LongCat-AudioDiT ## Modifications ### Serve - http_server.py: handle audio-only outputs (output_batch.audio as primary output); forward prompt_audio_path / prompt_text / guidance_method instance fields in vertex_generate. - entrypoints/utils.py: add DataType.AUDIO path in post_process_sample — writes WAV via soundfile; rename encode_video_to_base64 → encode_file_to_base64. - gpu_worker.py: extend save_output guard to cover audio-only models (output_batch.output is None but output_batch.audio is not None). ### Pipeline Config - Added LongCatAudioDiTPipelineConfig (task_type=T2A): DiT runs in bfloat16, WAV-VAE in float16. - Added configs/models/dits/longcat_audiodit.py: LongCatAudioDiTConfig (HuggingFace PreTrainedConfig) and LongCatAudioDiTVaeConfig. - Added configs/sample/longcat_audiodit.py: LongCatAudioDiTSamplingParams (num_inference_steps=16, guidance_scale=4.0, guidance_method cfg/apg). - sampling_params.py: added DataType.AUDIO, prompt_audio_path / prompt_text / guidance_method fields; --guidance method CLI argument. - base.py: added ModelTaskType.T2A. ### Model Architecture - DiT: runtime/models/dits/longcat_audiodit.py — LongCatAudioDiTModel (PreTrainedModel), LongCatAudioDiTTransformer (24-layer DiT with global AdaLN, RoPE, cross-attention, ConvNeXt-V2 text conv, long skip), inline Euler ODE integrator replacing torchdiffeq, CFG and APG guidance. - VAE: LongCatAudioDiTVae — WAV-VAE with Snake activation, weight-norm Conv1d, 2048× downsampling ratio (24kHz → ~11.7 frames/s), latent_dim=64. - Text encoder: frozen UMT5-base loaded as a sub-module via HuggingFace sub_configs. ### Pipeline - longcat_audiodit.py — LongCatAudioDiTPipeline: bypasses Diffusers component loading (no model_index.json); loads LongCatAudioDiTModel directly via from_pretrained; registers AutoConfig / AutoModel so no external audiodit package is needed. - LongCatAudioDiTInferenceStage: monolithic stage driving text encoding → ODE solve → VAE decode in a single forward; handles voice cloning(prompt audio VAE encode + duration estimation) and text-only TTS. ### Model Loading LongCat-AudioDiT ships as a single HuggingFace PreTrainedModel checkpoint (no model_index.json). The pipeline loads it with LongCatAudioDiTModel.from_pretrained, then applies mixed precision in-place: transformer → bfloat16, VAE → float16. ## How to Use ### TTS ```shell sglang generate --model-path /root/models/LongCat-AudioDiT-1B \\ --prompt \"今天晴暖转阴雨，空气质量优至良，空气相对湿度较低。\" \\ --output-file-name tts.wav ``` [tts.wav](https://github.com/user-attachments/files/26508639/tts.wav) ### Voice cloning ```shell sglang generate --model-path /root/models/LongCat-AudioDiT-1B \\ --prompt \"今天晴暖转阴雨，空气质量优至良，空气相对湿度较低。\" \\ --prompt-text \"小偷却一点也不气馁，继续在抽屉里翻找。\" \\ --prompt-audio /root/code/LongCat-AudioDiT/assets/prompt.wav \\ --output-file-name clone.wav ``` [prompt.wav](https://github.com/user-attachments/files/26508666/prompt.wav) [clone.wav](https://github.com/user-attachments/files/26508642/clone.wav) ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31701888721](https://github.com/sgl-project/sglang/actions/runs/31701888721) Latest PR Test (Extra): :x: [Run #31701888691](https://github.com/sgl-project/sglang/actions/runs/31701888691) ",
"url": "https://github.com/sgl-project/sglang/pull/22191",
"createdAt": "2026-04-06T12:57:59Z",
"updatedAt": "2026-08-13T12:50:36Z",
"timestamp": "2026-08-13T12:50:36Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"documentation",
"dependencies",
"npu",
"diffusion"
],
"author": "Dragonliu2018",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:22369",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: correct insert_sort logic in add_req_state for ignore_eos scheduling",
"text": "## Summary Fix an off-by-one bug in `PrefillAdder.add_req_state()` where the `insert_sort` path fails to append elements that are larger than all existing elements in `self.req_states`. ## Bug In `schedule_policy.py` lines 662-667, the sorted insertion logic uses a `for` loop to find the correct position: ```python # Before (buggy) i = 0 for i in range(len(self.req_states)): if tokens_left <= self.req_states[i][0]: break self.req_states.insert(i, (tokens_left, tokens_occupied)) ``` When the new `tokens_left` value is **larger than all existing elements**, the `for` loop completes without hitting `break`. In Python, `i` retains the value of the **last iteration** (i.e., `len(self.req_states) - 1`), so `insert(i, ...)` places the new element **before** the last element instead of **after** it. ### Example Given `req_states = [(10, ...), (20, ...), (30, ...)]` and a new element with `tokens_left = 50`: - The loop runs `i = 0, 1, 2` — the condition `50 <= req_states[i][0]` is never true - `i` ends at `2` (last index) - `insert(2, (50, ...))` produces `[(10, ...), (20, ...), (50, ...), (30, ...)]` — **wrong order** - Expected: `[(10, ...), (20, ...), (30, ...), (50, ...)]` ## Fix Use Python's `for/else` construct: the `else` block runs only when the loop completes without `break`, setting `i` to the end of the list: ```python # After (fixed) for i in range(len(self.req_states)): if tokens_left <= self.req_states[i][0]: break else: i = len(self.req_states) self.req_states.insert(i, (tokens_left, tokens_occupied)) ``` ## Impact This bug affects `ignore_eos` requests during the incremental sorted insertion path (line 679). The corrupted sort order causes `_prefill_one_req` to use incorrect `tokens_left` values when estimating remaining token budget, potentially leading to: - Over-scheduling prefills (underestimating memory needed) - Under-scheduling prefills (overestimating memory needed) The initial bulk sort on line 677 is unaffected since it uses Python's built-in `sort()`.",
"url": "https://github.com/sgl-project/sglang/pull/22369",
"timestamp": "2026-08-12T16:31:50Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "Ricardo-M-L",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:22451",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Test] Add unit tests for model_loader/utils.py and weight_utils.py",
"text": "## Summary Adds comprehensive CPU-only unit tests for `srt/model_loader/utils.py` and `srt/model_loader/weight_utils.py`, covering **86 test cases** in **1046 lines**. Closes #20865 (partial) ### `test_utils.py` — 38 tests | Module under test | What's covered | |---|---| | `set_default_torch_dtype` | Context manager restore, nesting, exception safety | | `_is_moe_model` | Architecture name matching (`moe`, `mixtral`), numeric/bool/list/dict/string attribute detection for `num_local_experts`, `num_experts`, `num_experts_per_tok`, `moe_intermediate_size`, `n_routed_experts` | | `_is_sequence_classification_model` | Classification and reward model detection, case insensitivity, empty list | | `_get_transformers_backend_arch` | All 8 combinations of generation/pooling × multimodal × MoE; config mismatch multimodal detection | | `_model_impl_from_architecture` | Transformers/MindSpore/SGLang dispatch | | `should_async_load` | CPU tensor (True), meta device (False), None device (False), missing attribute | | `maybe_executor_submit` | Sync vs async mode, args/kwargs, default kwargs, multiple submissions | ### `test_weight_utils.py` — 48 tests | Module under test | What's covered | |---|---| | `get_lock` | File lock creation in temp/custom dirs, hash naming, suffix, acquire/release | | `_shared_pointers` | Shared storage detection, views, empty dict | | `replace_prefix` / `replace_substrings` | Weight name prefix/substring remapping, chained replacements, no-match pass-through | | `_check_index_files_exist` | Complete index, missing files, empty weight_map, malformed JSON | | `filter_duplicate_safetensors_files` | Index-based dedup, consolidated vs model.safetensors special case | | `filter_files_not_needed_for_inference` | Training artifact blacklist filtering | | `convert_pyslice_to_tensor` | Tensor pass-through, PySafeSlice-like conversion | | `default_weight_loader` | Shape matching, scalar broadcast, mismatch assertion | | `maybe_remap_kv_scale_name` | `.kv_scale` → `.attn.k_scale`, k/v_scale remap, modelopt `k_proj`/`v_proj` prefix, `.mixer.` prefix, quark `output_scale`/`prob_output_scale` remap, missing-in-params → None | | `get_actual_shard_size` | Normal, exceeds weight, end < start, exact boundary | | `reset_param_data_if_needed` | Zeros range, length-zero no-op, negative length assertion | | `narrow_padded_param_and_loaded_weight` | Normal narrow, partial shard at end, with offset | | `KVCacheQuantSchema` | FP8 dtype validation, TP size context validation | | `QuantParamSchema` | Model type mismatch with context, None model_type | ### Conventions - [x] Tests in `test/registered/unit/model_loader/` (mirroring `srt/model_loader/`) - [x] Does NOT launch a server or load real model weights - [x] Includes edge cases, not just happy paths - [x] Uses `CustomTestCase` from `sglang.test.test_utils` - [x] Registered with `register_cpu_ci(est_time=5, suite=\"stage-a-test-cpu\")` ### Test command ```bash # run locally PYTHONPATH=python pytest test/registered/unit/model_loader/test_utils.py test/registered/unit/model_loader/test_weight_utils.py -v ``` Note: Full test execution requires Python 3.10+ with sglang dependencies (torch, pydantic, filelock, safetensors). Tests are designed for CI execution. 🤖 Generated with [Claude Code](https://claude.com/claude-code)",
"url": "https://github.com/sgl-project/sglang/pull/22451",
"timestamp": "2026-08-12T16:31:53Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [],
"author": "Ricardo-M-L",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:22596",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: reset batch_is_full after prefill-only filter_batch",
"text": "## Summary Fixes #22518 - **Bug**: Embedding/reward models (prefill-only) permanently deadlock after processing one full batch. When requests fill `max_running_requests`, `batch_is_full` is set to `True`. For prefill-only models, completed requests are removed via `filter_batch()` on `running_batch` (line 2306-2307), but `batch_is_full` is never reset. This causes `get_new_batch_prefill()` to return `None` forever (line 2401), blocking all subsequent requests. - **Fix**: After `filter_batch()` for prefill-only batches, reset `batch_is_full = False` when the batch size decreases. This mirrors the existing pattern at lines 2286-2287 that already does this for the `last_batch` filter path. - **Test**: Added unit tests covering the deadlock scenario (all requests finish), partial filtering, and non-prefill-only batch behavior. ## Changes - `python/sglang/srt/managers/scheduler.py`: 3-line fix in `get_next_batch_to_run()` — record batch size before `filter_batch()`, reset `batch_is_full` if size decreased - `test/registered/unit/managers/test_scheduler_embedding_batch_full.py`: New unit test file with 4 test cases ## Reproduction (from issue) ```bash sglang serve --model-path Qwen/Qwen3-Embedding-0.6B \\ --is-embedding --max-running-requests 2 --port 8234 # This request sends 3 inputs but max-running-requests=2, triggers deadlock: curl http://127.0.0.1:8234/v1/embeddings \\ -H \"Content-Type: application/json\" \\ -d '{\"model\":\"Qwen/Qwen3-Embedding-0.6B\",\"input\":[\"a\",\"b\",\"c\"]}' ``` ## Test plan - [ ] Unit tests pass (`pytest test/registered/unit/managers/test_scheduler_embedding_batch_full.py`) - [ ] Verify with embedding model that requests above `max_running_requests` no longer hang - [ ] Verify normal (decode) model behavior is unaffected 🤖 Generated with [Claude Code](https://claude.com/claude-code)",
"url": "https://github.com/sgl-project/sglang/pull/22596",
"timestamp": "2026-08-12T16:31:55Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [],
"author": "Ricardo-M-L",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:23112",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add fmha_v2 attention backend for SM90/120",
"text": " ## Motivation Enables TRT-LLM's fmha_v2 prefill attention backend for SM90/120. This backend is more performant than the current default (FA3), and also enables skip-softmax feature, which can further boost performance for long-context / applications where attention output is sparse. ## Modifications Baseline enablement of trtllm's fmha_v2 prefill kernel for SM90/120. https://github.com/sgl-project/sglang/pull/32272 is needed for further perf improvement, and https://github.com/sgl-project/sglang/pull/32268 / https://github.com/sgl-project/sglang/pull/32269 are used for the MTP accuracy reporting. ## Accuracy Tests ``` python3 -m sglang.launch_server \\ --model /SGL/models/Qwen3-Next-80B-A3B-Thinking-FP8 \\ --tp 4 --mem-fraction-static 0.8 \\ --disable-radix-cache --mamba-ssm-dtype float32 \\ --moe-runner-backend deep_gemm --attention-backend trtllm_mha \\ --trust-remote-code --port 30000 # --kv-cache-dtype fp8_e4m3 ``` | Config | KV dtype | Score | Repeat range | |---|---|---|---| | FA3 | bf16 | 0.725 | 0.712-0.742 | | fmha_v2 (trtllm_mha) | bf16 | 0.723 | 0.702-0.753 | | FA3 + MTP | bf16 | 0.734 | 0.712-0.758 | | fmha_v2 prefill + MTP (fa3 verify) | bf16 | 0.721 | 0.707-0.732 | | FA3 | **fp8_e4m3** | 0.727 | 0.707-0.747 | | fmha_v2 (trtllm_mha) | **fp8_e4m3** | 0.724 | 0.682-0.753 | ### MTP Add to server launch: `--speculative-algorithm NEXTN --speculative-num-steps 3 \\ --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` | Config | Accept len | Accept rate | GPQA-20 score | |---|---|---|---| | FA3 + MTP (bf16) | 2.37 (2.35) | 0.46 (0.45) | 0.750 | | fmha_v2 prefill + MTP (bf16) | 2.45 (2.30) | 0.48 (0.43) | 0.700 | | pure trtllm_mha + MTP w/ XQA mask fix (bf16, 10q) | 2.20 | 0.40 | 0.70 | ## Speed Tests and Profiling The e2e performance is not improved with just this PR because it requires a large copy operation before the compute kernel to interleave KV. https://github.com/flashinfer-ai/flashinfer/pull/3575 and https://github.com/sgl-project/sglang/pull/32272 address this issue. Unit Kernel Measurements show ~15% improvement over FA3 over a variety of problem sizes, reaching 20% at small problems. ### Skip-Softmax This feature can provide additional speedup, especially for long-context inference where the attention logits are sparse. We define the threshold factor `\\lambda := a / L` where a is a hyperparameter, and L is the sequence length. To demonstrate the potential speedup, we keep a = 1000 here for simplicity. | ISL | Effective threshold (1000 / ISL) | bf16 gain | fp8 gain | | --- | --- | --- | --- | | 8k | 0.122 | +1.8% | +0.2% | | 32k | 0.031 | +5.2% | +1.7% | | 128k | 0.008 | +12.9% | + 3.5% | ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31564176242](https://github.com/sgl-project/sglang/actions/runs/31564176242) Latest PR Test (Extra): :x: [Run #31564175995](https://github.com/sgl-project/sglang/actions/runs/31564175995) ",
"url": "https://github.com/sgl-project/sglang/pull/23112",
"createdAt": "2026-04-18T00:22:35Z",
"updatedAt": "2026-08-12T20:07:14Z",
"timestamp": "2026-08-12T20:07:14Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"blackwell"
],
"author": "akhilg-nv",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:23274",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] model: support LongCat-Image",
"text": "## Motivation This PR adds support for LongCat-Image, a new text-to-image model based on a Flux-like MMDiT architecture with a Qwen2.5-VL (7B) text encoder. * Huggingface: https://huggingface.co/meituan-longcat/LongCat-Image * GitHub: https://github.com/meituan-longcat/LongCat-Image ## Modifications New components: - `LongCatImageTransformer2DModel`: Flux-like MMDiT, 19 double + 38 single-stream blocks, 24×128 dim, 2×2 latent packing, RoPE (axes=[16,56,56]), AdaLN-Zero. USPAttention (FA3/FA4) + fused qk-norm; weights load without remapping. - `LongCatPromptRewriteStage`(model-specific): optional prompt rewrite via Qwen2.5-VL `.generate()` (max_new_tokens=512, top_k=1 → effectively greedy; sampling params from checkpoint's generation_config.json). Loads the HF encoder in-stage and shares it with TextEncodingStage; sets CPU generator for seed reproducibility. - `LongCatImagePipeline`: PromptRewriteStage → TextEncoding → LatentPrep → TimestepPrep → Denoising → Decoding (only the first is model-specific). - `LongCatImagePipelineConfig` hooks: quotation-aware tokenization with fixed system prefix/assistant suffix (512-token body), 2×2 pack/unpack, 3D modality-aware pos ids (txt=0 / img=1 @ offset 512), seq-len-dependent flow-match shift, CFG renorm (cond_norm/noise_norm clamped to [min, 1.0]). - Bilingual (EN/ZH) few-shot rewrite system prompts. - LongCatImageSamplingParams: guidance_scale=4.5, steps=50, 1024×1024, cfg_renorm + prompt_rewrite on by default. Shared changes: - `SamplingParams`: add enable_cfg_renorm / cfg_renorm_min / enable_prompt_rewrite (+ CLI). - `AutoProcessorLoader`: add \"text_processor\" to component_names. - `registry`: register LongCat-Image (\"longcat\" detector, excludes \"edit\"). ## Examples ```shell sglang generate \\ --model-path /root/models/LongCat-Image \\ --prompt \"一个年轻的亚裔女性，身穿黄色针织衫，搭配白色项链。她的双手放在膝盖上，表 情恬静。背景是一堵粗糙的砖墙，午后的阳光温暖地洒在她身上，营造出一种宁静而温馨的氛围 。镜头采用中距离视角，突出她的神态和服饰的细节。光线柔和地打在她的脸上，强调她的五官 和饰品的质感，增加画面的层次感与亲和力。整个画面构图简洁，砖墙的纹理与阳光的光影效果 相得益彰，突显出人物的优雅与从容。\" \\ --height 768 --width 1344 \\ --enable-prompt-rewrite=True \\ --enable-cfg-renorm=True \\ --num-inference-steps=50 \\ --negative-prompt \"\" \\ --seed=43 \\ --guidance-scale=4.0 \\ --save-output ``` ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31676886716](https://github.com/sgl-project/sglang/actions/runs/31676886716) Latest PR Test (Extra): :white_check_mark: [Run #31676886686](https://github.com/sgl-project/sglang/actions/runs/31676886686) ",
"url": "https://github.com/sgl-project/sglang/pull/23274",
"createdAt": "2026-04-20T16:41:12Z",
"updatedAt": "2026-08-13T11:48:27Z",
"timestamp": "2026-08-13T11:48:27Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"run-ci",
"diffusion",
"run-ci-extra"
],
"author": "Dragonliu2018",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:23317",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bug Fix] Sync FlashInfer autotune tactic selection across TP ranks",
"text": "## What this does FlashInfer's `AutoTuner.choose_one` runs **independently on every TP rank**, so ranks frequently pick **different kernel tactics** for the same op (sub-millisecond timing noise pushes the per-rank `argmin` around). This cross-rank tactic divergence is undesirable for **any** tensor-parallel run, and under `--enable-symm-mem` it becomes a **hard deadlock**. This PR wires in FlashInfer's opt-in `set_autotune_process_group` ([flashinfer#3187](https://github.com/flashinfer-ai/flashinfer/pull/3187)) to all-reduce the per-tactic timings across ranks, so every rank's `argmin` sees identical timings and converges on the same tactic. **Gated on `tp_size > 1` — it applies to all TP, not just symmetric memory** (symm-mem is simply where the divergence turns into a deadlock). ## The deadlock (the acute symptom) Divergent tactics allocate **different-sized scratch tensors**; under `--enable-symm-mem` those go through the NCCL allocator (`ncclMemAlloc` + `ncclCommWindowRegister(NCCL_WIN_COLL_SYMMETRIC)`) — a collective that **deadlocks** unless every rank registers the same size in the same order. Repro: `Qwen/Qwen3-235B-A22B-Instruct-2507`, 8×B200, `--tp 8 --attention-backend flashinfer --enable-symm-mem` hangs silently during CUDA-graph capture; py-spy shows a straggler rank in the MoE kernel while the others busy-spin in NCCL: ``` # 7× \"State: R (running)\" (NCCL busy-wait) + 1× \"State: S\" Process …scheduler_TP1: trtllm_bf16_moe_op (flashinfer/fused_moe/core.py:1706) Process …scheduler_TP0: … init_all_cuda_graphs (scheduler.py:842) # deadlock during capture ``` ## Fix In `flashinfer_autotune_context` (the single context manager all autotune paths funnel through): - Import `set_autotune_process_group` alongside `autotune` (no guard — see the Note on the pin). - Active for `tp_size > 1`; uses the TP gloo `cpu_group` (a sub-millisecond CPU reduce, no CUDA stream). - `try/finally` resets the group, so only the one-time autotune warmup is affected and the serving path is untouched; symmetric memory stays **active during capture and replay**. Full analysis: flashinfer [#3186](https://github.com/flashinfer-ai/flashinfer/issues/3186). ## Evidence (8×B200, current `main`; only variable = this PR) Deadlock fix — BEFORE (no fix) hangs; AFTER (this PR) completes: | | BEFORE | AFTER | |---|---|---| | CUDA-graph capture (`--enable-symm-mem`) | **hangs** | **completes 52/52** → \"fired up and ready to roll\" | | GSM8K (200 q, 5-shot) | N/A (never serves) | **Accuracy 0.975, Invalid 0.000** | | `SGLANG_DEBUG_SYMM_MEM=1` warnings | N/A | **0** (symm-mem active at replay) | Unit tests: this PR's `test_flashinfer_autotune_tp_sync.py`; flashinfer's `tests/autotuner/test_autotuner_distributed.py` → 3 passed. **Cross-rank tactic-divergence (before/after) + a serving perf-non-regression check: measuring now on 8×B200 (TP=8, without symm-mem so both sides serve) — will update.** ## Note `set_autotune_process_group` first shipped in **flashinfer v0.6.16** — the v0.6.15 release branch was cut before [flashinfer#3187](https://github.com/flashinfer-ai/flashinfer/pull/3187) merged, so only the `nightly-v0.6.15-*` builds carry it. `main` now pins `flashinfer_python[cu13]==0.6.17` (`python/pyproject.toml`), which exports it, so the import needs no `ImportError` guard and the sync is **active**, not a no-op. ## Checklist - [x] Format with pre-commit (isort / black / ruff `F401,F821,UP037`); `check-registered-tests` passes - [x] Add unit tests - [x] Accuracy (GSM8K) + symm-mem-active check on 8×B200 - [ ] Tactic-divergence before/after + perf-non-regression numbers (in progress) - [ ] CI (fork PR — needs a maintainer `run-ci` label) --- ### CI States Latest PR Test (Base): :x: [Run #31615935561](https://github.com/sgl-project/sglang/actions/runs/31615935561) Latest PR Test (Extra): :x: [Run #31615935357](https://github.com/sgl-project/sglang/actions/runs/31615935357) ",
"url": "https://github.com/sgl-project/sglang/pull/23317",
"createdAt": "2026-04-21T03:53:41Z",
"updatedAt": "2026-08-12T17:25:37Z",
"timestamp": "2026-08-12T17:25:37Z",
"metrics": {
"reactions": 0,
"comments": 22
},
"labels": [
"run-ci"
],
"author": "thanhhao98",
"state": "open",
"assignees": [
"b8zhong"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:23487",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: replace bare except with specific exception types in model_runner.py",
"text": "## Motivation Bare `except:` clauses catch all exceptions including `SystemExit` and `KeyboardInterrupt`, which can mask critical errors and prevent clean process shutdown. This follows the same pattern as #22947. ## Modifications Replace two bare `except:` clauses in `model_runner.py` with specific exception types: 1. **EAGLE config access** (line 389): `except:` → `except (AttributeError, KeyError, TypeError):` - Handles missing `eagle_config` attribute, missing dict keys, and `None.get()` calls 2. **mooncake import** (line 1008): `except:` → `except (ImportError, RuntimeError):` - Handles missing mooncake package and device filter setup failures ## Checklist - [x] Format: `pre-commit run --all-files` passes - [x] No new tests needed (exception handling refinement only) --- ### CI States Latest PR Test (Base): :x: [Run #27469901539](https://github.com/sgl-project/sglang/actions/runs/27469901539) Latest PR Test (Extra): :x: [Run #27469901474](https://github.com/sgl-project/sglang/actions/runs/27469901474) ",
"url": "https://github.com/sgl-project/sglang/pull/23487",
"timestamp": "2026-08-12T16:31:57Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "Ricardo-M-L",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:23488",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(nixl): avoid UnboundLocalError in create_backend error handler",
"text": "## Motivation In `NixlBackendSelection.create_backend()`, if `agent.get_plugin_list()` (line 101) raises an exception, the `except` block at line 158-161 references `plugin_list` and `initparams` — variables that were never assigned. This turns the original exception into an `UnboundLocalError`, hiding the real failure. ## Modifications Use `locals().get()` to safely reference these variables in the error log message. ## Checklist - [x] `format.sh` and `lint.sh` passed locally",
"url": "https://github.com/sgl-project/sglang/pull/23488",
"createdAt": "2026-04-22T16:37:39Z",
"updatedAt": "2026-08-12T16:31:59Z",
"timestamp": "2026-08-12T16:31:59Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "Ricardo-M-L",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:23806",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: guard against None new_accepted_tokens in vocab boundary check",
"text": "## Summary _check_vocab_boundary_finish crashes with TypeError when new_accepted_tokens is None and vocab_size is also None (e.g. embedding requests). Add explicit None check to prevent the error. 🤖 Generated with Claude Code",
"url": "https://github.com/sgl-project/sglang/pull/23806",
"createdAt": "2026-04-27T05:36:43Z",
"updatedAt": "2026-08-12T16:32:04Z",
"timestamp": "2026-08-12T16:32:04Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "Ricardo-M-L",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:23913",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "perf(radix-cache): replace SHA256 with xxhash64 for cache block hashing",
"text": "## Summary Replace SHA256 with xxhash64 for radix cache block hashing in `hash_page()` and `get_hash_str()`. xxhash64 is ~7.7x faster than SHA256 for non-cryptographic hash operations. The radix cache hashing is called on every KV cache store/remove operation, making this a meaningful hot-path optimization. ## Changes - `python/sglang/srt/mem_cache/radix_cache.py`: Replace `hashlib.sha256()` with `xxhash.xxh64()` in `RadixKey.hash_page()` - `python/sglang/srt/mem_cache/utils.py`: Replace `hashlib.sha256()` with `xxhash.xxh64()` in `get_hash_str()` - `python/pyproject.toml`: Add `xxhash>=3.0.0` dependency ## Test plan - [ ] Existing radix cache unit tests pass - [ ] KV cache store/evict operations work correctly with new hash - [ ] Benchmark shows meaningful improvement in cache operations 🤖 Generated with [Claude Code](https://claude.com/claude-code)",
"url": "https://github.com/sgl-project/sglang/pull/23913",
"createdAt": "2026-04-28T06:29:14Z",
"updatedAt": "2026-08-12T16:32:07Z",
"timestamp": "2026-08-12T16:32:07Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"dependencies"
],
"author": "Ricardo-M-L",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:24132",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Modify low errors",
"text": " ## Motivation Modify low errors ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.",
"url": "https://github.com/sgl-project/sglang/pull/24132",
"createdAt": "2026-04-30T07:48:32Z",
"updatedAt": "2026-08-13T01:11:38Z",
"timestamp": "2026-08-13T01:11:38Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"documentation",
"quant",
"deepseek",
"npu"
],
"author": "caiwenwen11",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:24891",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add nightly AIME25 test for DeepSeek-V4-Pro on B200",
"text": "## Summary - Add nightly AIME25 accuracy test with Think Max reasoning for DSV4-Pro on 8x B200 - Suite: `nightly-8-gpu-b200` - Two variants: - Low-Latency: TP=8, FP4 MoE (flashinfer_mxfp4), EAGLE (3-step) - MegaMoE: TP=8, DP=8, dp-attention, DeepEP + MegaMoE, EAGLE (1-step) --- ### CI States Latest PR Test (Base): :x: [Run #31689194545](https://github.com/sgl-project/sglang/actions/runs/31689194545) Latest PR Test (Extra): :x: [Run #31689194161](https://github.com/sgl-project/sglang/actions/runs/31689194161) ",
"url": "https://github.com/sgl-project/sglang/pull/24891",
"createdAt": "2026-05-10T13:42:19Z",
"updatedAt": "2026-08-13T10:02:11Z",
"timestamp": "2026-08-13T10:02:11Z",
"metrics": {
"reactions": 1,
"comments": 6
},
"labels": [
"deepseek"
],
"author": "yhyang201",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:24911",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Profiling Enhancements [2/3]: detailed execution step annotations",
"text": " ## Motivation The PR introduces changes to enable enhanced profiling of workloads. It introduces detailed trace annotations which encapsulate KV length and its distribution across context and generation phases. ## Modifications ### Roofline Annotations Added a new argument in the profiling end point call:`roofline_annotations` [[2]](). This adds more detailed annotation to the trace regarding number of requests and total KV length during the particular execution step. ## Accuracy Tests The way to test this is the following: In all the tests, the following flags are set: ``` export SGLANG_TORCH_PROFILER_DIR=/home/mohbasit/profile export SGLANG_PROFILE_WITH_STACK=True export SGLANG_PROFILE_RECORD_SHAPE=True ``` ### Execution steps 1. The result for the execution step trace saved will be similar to how it is now. But some operations would be missing shape, for instance when using GLM5 the _tilelang_kernel_tilelang_sparse_fwd_586_ operation. 3. This time in the profiling call we set `roofline_annotations` to True, the resulting trace would have detailed annotations as described above in the execution steps of the GPU stream. This is the same behavior for the framework whether run in eager mode or graph mode. ## Speed Tests and Profiling ## Setup | Item | Value | | --- | --- | | Model | `deepseek-ai/DeepSeek-V3` (native fp8) | | Hardware | MI355X (gfx950), GPUs 0-3 | | Parallelism | TP=4 | | Attention backend | `aiter` | | KV cache | `fp8_e4m3` | | CUDA graph max bs | 32 | ]| Workload | random ISL=1024, OSL=1024, `--random-range-ratio 0.8` | | Load (profiled) | 32 prompts, 64 warmups | - **Baseline (`main`)** = commit `27b15349e` — the exact main commit `feat/roofline_annotations` is merged up to (the PR footprint vs it is just the 7 roofline files). - **My branch** = `feat/roofline_annotations` @ `539611559`. - The `roofline_annotations` toggle is passed via the `/start_profile` request body --- ## Runs | Run | Dir | Branch | `/start_profile` body | | --- | --- | --- | --- | | 1 | `main_profile` | main `27b15349e` | `{\"record_shapes\": true}` | | 2 | `branch_profile` | feat `539611559` | `{\"roofline_annotations\": false, \"record_shapes\": true}` | | 3 | `branch_profile_roofline` | feat `539611559` | `{\"roofline_annotations\": true, \"record_shapes\": true}` | --- ## Results (profiling ON, 32-prompt profiled window) | Run | Branch | roofline | Out tok/s | Req/s | Dur (s) | Mean TTFT (ms) | P99 TTFT (ms) | Mean TPOT (ms) | Mean ITL (ms) | OK | | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | | 1 | main | off (n/a) | 302.08 | 0.32 | 98.51 | 1089.19 | 1117.18 | 23.54 | 23.53 | 32/32 | | 2 | feat/roofline_annotations | off | 303.36 | 0.33 | 98.09 | 1103.15 | 1131.86 | 23.23 | 23.21 | 32/32 | | 3 | feat/roofline_annotations | on | 298.81 | 0.32 | 99.59 | 1090.30 | 1118.77 | 23.91 | 23.90 | 32/32 | ### Analysis - **Run 2 vs Run 1 (my branch, feature off vs main):** 303.36 vs 302.08 tok/s (+0.4%), TPOT 23.23 vs 23.54 ms. Identical within noise → the PR does not regress the existing profiled path when the feature is off (the default). - **Run 3 vs Run 2 (roofline on vs off):** 298.81 vs 303.36 tok/s, TPOT 23.91 vs 23.23 ms, The impact on performance is negligible ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31706941006](https://github.com/sgl-project/sglang/actions/runs/31706941006) Latest PR Test (Extra): :x: [Run #31706940703](https://github.com/sgl-project/sglang/actions/runs/31706940703) ",
"url": "https://github.com/sgl-project/sglang/pull/24911",
"createdAt": "2026-05-10T23:56:47Z",
"updatedAt": "2026-08-13T15:02:04Z",
"timestamp": "2026-08-13T15:02:04Z",
"metrics": {
"reactions": 1,
"comments": 10
},
"labels": [
"documentation",
"quant",
"deepseek",
"npu",
"run-ci"
],
"author": "mohbasit",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:24959",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "XPU: Enable GLM5.1 (GlmMoeDsaForCausalLM) DSA Attention",
"text": " ## Motivation GLM5.1 uses Dynamic Sparse Attention (DSA/NSA) with an FP8 indexer that scores KV pages before sparse attention. This PR enables the path on XPU. ## Modifications **1. server_args.py** When GLM5.1 (or any DSA model) runs on XPU, automatically set: - `decode_attention_backend = \"dsa\"` — puts `DeepseekSparseAttnBackend` as the decode backend inside `HybridAttnBackend`, so it can manage the DSA index K-cache and FP8 logit scoring. - `prefill_attention_backend = \"dsa\"` — We use the dense version of prefill MLA for now since no XPU sparse prefill kernel exists yet in sgl-kernel-xpu. - Disable `SGLANG_OPT_USE_TOPK_V2` — the v2 topk uses TileLang/TVM JIT compilation which requires CUDA. **2. hybrid_attn_backend.py** In `HybridAttnBackend`, when the selected backend is not the decode backend (i.e. during prefill when Triton is selected), also call `decode_backend.init_forward_metadata()` and `get_indexer_metadata()` from the decode backend. This ensures the DSA decode backend (`DeepseekSparseAttnBackend`) remains responsible for the DSA indexer even during prefill passes. **3. dsa_backend.py** Guard `flashmla_metadata` computation with `not _is_xpu`. The `get_mla_metadata` function is from `sgl_kernel.flash_mla` which is CUDA-only. **4. dsa/dsa_indexer.py** Replace `torch.cuda.get_device_properties` and `torch.cuda.mem_get_info` with XPU-aware equivalents in `_get_mqa_logits_budget_bytes`. **5. rotary_embedding/base.py** In `forward_xpu`, handle 2D input tensors `(N, head_size)` for the fallback `sgl_kernel.rotary_embedding` kernel which expects 3D `(N, num_heads, head_size)`. The DSA indexer passes a single-head `k_rope` as 2D. **6. jit_kernel/dsa/__init__.py** Guard the CuteDSL import (NVIDIA CUDA DSL) on XPU, matching the existing ROCm guard. ## Accuracy Tests N/A since the original model is too big for BMG. ## Speed Tests and Profiling N/A since the original model is too big for BMG. ## Checklist - [X] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [X] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31657272811](https://github.com/sgl-project/sglang/actions/runs/31657272811) Latest PR Test (Extra): :x: [Run #31657272745](https://github.com/sgl-project/sglang/actions/runs/31657272745) ",
"url": "https://github.com/sgl-project/sglang/pull/24959",
"createdAt": "2026-05-11T08:03:43Z",
"updatedAt": "2026-08-13T09:50:46Z",
"timestamp": "2026-08-13T09:50:46Z",
"metrics": {
"reactions": 1,
"comments": 8
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"lora",
"Multi-modal",
"deepseek",
"speculative-decoding",
"hicache",
"sgl-kernel",
"blackwell",
"intel",
"npu",
"xpu",
"run-ci",
"diffusion",
"model-gateway",
"mthreads",
"apple-silicon",
"jit-kernel"
],
"author": "Xia-Weiwen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:25199",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Remove silent ×0.85 mem_fraction_static derate for aiter + ctx>8K",
"text": " ## Motivation The hard-coded `mem_fraction_static *= 0.85` when attention_backend=aiter and context_len > 8192 silently shrinks the user-set --mem-fraction-static by 15% (e.g. 0.95 becomes 0.8075 effectively before the KV pool sizer runs). Delete it for clean up purpose. ## Modifications few line deletion. ## Accuracy Tests no affect. ## Speed Tests and Profiling KV-Cache Cap larger 15%, and stable in local test. MiniMax-M2.5 TP=2 EP=1, 64K input, K=16, --mem-fraction-static 0.95: derate=0.85 (current) -> 4,154,465 input TPM, KV cap 2.03 M, 100% success derate=0.95 (patched) -> 4,220,304 input TPM, KV cap 2.49 M, 100% success ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.",
"url": "https://github.com/sgl-project/sglang/pull/25199",
"createdAt": "2026-05-13T23:46:09Z",
"updatedAt": "2026-08-13T00:55:57Z",
"timestamp": "2026-08-13T00:55:57Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [],
"author": "kzjeef",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:25871",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "add fid accuracy benchmark for sglang diffusion t2i model",
"text": "## Motivation Add an **FID accuracy benchmark** for SGLang diffusion text-to-image generation. It provides a reproducible workflow to (1) build a reference batch from COCO val2014 prompts + images, (2) generate samples with `sglang generate`, and (3) compute FID (Fréchet Inception Distance) between the reference and generated batches. Adapted from the xDiT FID benchmark: https://github.com/xdit-project/xDiT/tree/main/benchmark/fid ## Modifications New directory `python/sglang/multimodal_gen/benchmarks/fid/`: - **`preprocess_dataset.py`** — samples `N` COCO val2014 caption/image pairs (one caption per image) into a `prompt.txt` and a fixed-size reference-image folder. Defaults to resizing references to `256x256` (xDiT-style); supports COCO captions JSON as well as TSV/CSV. - **`compute_fid.py`** — computes FID between a reference folder and a generated folder using `pytorch-fid` (InceptionV3, `dims=2048`). - **`README.md`** — end-to-end instructions (download COCO → preprocess → `sglang generate` → compute FID). ## Accuracy Tests Validated end-to-end following the README protocol on **8× AMD Instinct MI350X** (gfx950, ROCm) with **SGLang v0.5.12.post1** (`--attention-backend aiter`, `--performance-mode speed`). - **Dataset:** COCO **val2014** captions (`captions_val2014.json`), one caption per image, `--seed 42 --shuffle`. - **Reference batch:** 10,000 COCO images resized to **256×256** (`--ref-size 256`, the README default). - **Generation:** **256×256**, **30 inference steps** (sufficient for FLUX-family models), fixed `--seed 42`. - **Metric:** `pytorch-fid`, InceptionV3, `dims=2048`. | Model | Resolution | Steps | Guidance | Samples | FID ↓ | |---|---|---|---|---|---| | **FLUX.1-dev** | 256×256 | 30 | 3.5 | 9,983 | **30.43** | | **FLUX.2-dev** | 256×256 | 30 | 4.0 (model default) | 9,989 | **25.74** | (Generated counts are slightly below 10,000 because a few COCO captions are duplicated and collapse to the same output filename — immaterial to the score.) The values are in the expected range for COCO-256 text-to-image FID, and FLUX.2-dev's lower (better) FID vs FLUX.1-dev matches the newer/larger model. Generated images are coherent and prompt-accurate on spot checks. > **Note on sample count:** FID uses 2048-dim Inception features, so a full-rank covariance estimate needs at least ~2048 samples; more samples reduce the well-known positive bias of FID. Use **≥ a few thousand** samples for a meaningful number (a tiny run such as `--num-samples 100` yields a hugely inflated, meaningless FID). These results use ~10k. ### Reproduction ```bash pip3 install pytorch-fid pillow # 1) Prepare reference batch (prompts + 256x256 reference images) python3 -m sglang.multimodal_gen.benchmarks.fid.preprocess_dataset \\ --caption-file /path/to/annotations/captions_val2014.json \\ --coco-val-dir /path/to/val2014 \\ --out-dir /path/to/fid_runs \\ --num-samples 10000 --shuffle --seed 42 --ref-size 256 # -> /path/to/fid_runs/n10000_seed42_ref256/{prompt.txt, *.jpg} # 2) Generate samples (FLUX.1-dev shown; swap model + guidance for FLUX.2-dev) sglang generate \\ --model-path black-forest-labs/FLUX.1-dev \\ --num-gpus 1 --tp-size 1 \\ --attention-backend aiter --performance-mode speed \\ --prompt-file-path /path/to/fid_runs/n10000_seed42_ref256/prompt.txt \\ --height 256 --width 256 --num-inference-steps 30 --guidance-scale 3.5 --seed 42 \\ --save-output --output-path /path/to/fid_runs/n10000_samples_flux1 # 3) Compute FID python3 -m sglang.multimodal_gen.benchmarks.fid.compute_fid \\ --ref-dir /path/to/fid_runs/n10000_seed42_ref256 \\ --sample-dir /path/to/fid_runs/n10000_samples_flux1 \\ --device cuda ``` ## Speed Tests and Profiling Not a perf-sensitive change (offline benchmark tooling), but a throughput note for anyone reproducing large runs: For a **batch of small (256²) images**, generating with **N independent single-GPU workers** (split the prompt file across GPUs, `--num-gpus 1 --tp-size 1` each) is far faster than a single tensor-parallel `tp8` job — a 256² image fits comfortably on one MI350X, so tensor parallelism only adds cross-GPU communication per denoise step. Measured on MI350X (256², 30 steps): | Model | denoise/img (1 GPU) | denoise/img (tp8) | 10k images via 8× single-GPU | |---|---|---|---| | FLUX.1-dev | 0.98 s | 1.73 s | ~24 min | | FLUX.2-dev | 1.79 s | — | ~41 min | So the results above were produced by sharding `prompt.txt` across the 8 GPUs and merging the outputs into one folder before computing FID (equivalent output to the single `sglang generate` command shown above). ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31725952272](https://github.com/sgl-project/sglang/actions/runs/31725952272) Latest PR Test (Extra): :x: [Run #31725952138](https://github.com/sgl-project/sglang/actions/runs/31725952138) ",
"url": "https://github.com/sgl-project/sglang/pull/25871",
"createdAt": "2026-05-20T08:09:58Z",
"updatedAt": "2026-08-13T17:30:12Z",
"timestamp": "2026-08-13T17:30:12Z",
"metrics": {
"reactions": 1,
"comments": 6
},
"labels": [
"documentation",
"run-ci",
"diffusion"
],
"author": "sushildubey171",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:26510",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix _GenerationStreamAccumulator logprob_end off-by-one under retract",
"text": "## Summary Fix an output-logprob cursor off-by-one when overlap scheduling retracts an in-flight request before its prefill result is processed. The fix is deliberately retraction-specific. Non-retracted prefill-only requests with `max_new_tokens=0` retain the existing behavior of returning their first logprob. ## Root cause Under overlap scheduling and retraction: 1. A prefill forward pass is launched. 2. The request is retracted while that pass is in flight, setting `req.is_retracted = True`. 3. `process_batch_result_prefill` skips appending the sampled token for that request, but still passes the full batch to the output streamer. 4. For a non-streaming request with no output IDs, the forced-stream condition is true because `0 % DEFAULT_FORCE_STREAM_INTERVAL == 0`. 5. `max(len(output_ids_), 1)` advances `send_output_token_logprobs_offset` to 1 while `send_token_offset` remains 0. Every later output is therefore shifted by one logprob, and the final response has one fewer output logprob than output token. ## Fix ```python logprob_end = ( len(output_ids_) if req.is_retracted else max(len(output_ids_), 1) ) ``` For a retracted request with no output IDs, both cursors stay at zero. For a legitimate non-retracted prefill-only request, the existing first-logprob behavior remains intact. ## Tests - Added CPU unit coverage that verifies a retracted empty-output request does not advance its logprob cursor. - Added CPU unit coverage that verifies a non-retracted `max_new_tokens=0` request still returns its first logprob. - Kept the end-to-end forced-retraction regression and registered it in CUDA `base-b` and AMD `stage-b-test-1-gpu-small-amd` CI. - Ran all pre-commit hooks on the changed files successfully. - Ran Python syntax compilation successfully. - The focused runtime unit test was not run locally because this bare checkout lacks SGLang runtime dependencies; it is registered with CPU CI. The original end-to-end reproduction produced 2/32 mismatched responses in five consecutive pre-fix trials and 0/32 in eight consecutive post-fix trials. ## Checklist - [x] Format code with pre-commit. - [x] Add regression coverage. - [x] Preserve prefill-only logprob behavior raised in review. - [x] Rebase onto current upstream `main`. cc @Qiaolin-Yu --- ### CI States Latest PR Test (Base): :x: [Run #31644678346](https://github.com/sgl-project/sglang/actions/runs/31644678346) Latest PR Test (Extra): :x: [Run #31644678243](https://github.com/sgl-project/sglang/actions/runs/31644678243) ",
"url": "https://github.com/sgl-project/sglang/pull/26510",
"createdAt": "2026-05-27T22:41:05Z",
"updatedAt": "2026-08-12T21:56:49Z",
"timestamp": "2026-08-12T21:56:49Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [],
"author": "shenxiul",
"state": "open",
"assignees": [
"Qiaolin-Yu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:27010",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] Fix PP inconsistency with HiCache L3 (#22607)",
"text": "## Motivation When deploying HiCache and PP, multiple PP ranks go diverge and eventually cause sglang crash. The detailed reason was discussed in issue #22607. ## Modifications This patch introduces two kinds of synchronization mechanisms to fix the issue. I will illustrate this idea with an example of pp_size=2 and tp_size=1. 1. **pp_sync** across scheduler threads. pp_sync is a new name introduced by this patch. PP0's scheduler thread determine the number of completion events in its local queues (ack_load_queue, ack_prefetch_queue, etc). Then, PP1 must consume the **exact** same number of events from its queues. This prevents divergence. 2. **all_reduce** across prefetch threads. Consider a case that PP0 and PP1 have different lengths on their L3 storage for a given sequence. They have to reach an agreement on the length, by perform all_reduce(MIN). This behavior is consistent with TP. This patch also changes how `check_prefetch_progress` works. Before this patch, it simply does all_reduce(MIN) to determine the final tokens prefetchted. However, this would not work on PP, as we cannot do that in the scheduler thread. This patch adds a new background thread `prefetch_sync_thread`, which does the all_reduce work instead. ## Accuracy Tests Added `test_unified_radix_cache_hicache_pp_kl.py`. I have also tested SWE-bench verified with DSv4 flash model. I ran two groups: - Group A: The baseline, commit 5747ed3b, without this PR, TP=8, L3 enabled. Resolved 356/500. - Group B: This PR, PP=4, TP=2, L3 enabled. Resolved 353/500. I analyzed the difference, and found 351 cases successful in both group, 5 are successful only in group A, and 2 are successful only in group B. The 7 cases that are successfuly in one group but failed in another, are due to container exits after a long time. Here are commands for anyone has interested in reproducing this test. Hints to make L3 are in-use: - Use large number of workers (128), leading to large number of running requests. - Limit number of tokens in L1 and L2. ```sh # Group A sglang serve \\ --model=deepseek-ai/DeepSeek-V4-Flash \\ --tp-size 8 \\ --moe-runner-backend flashinfer_mxfp4 \\ --disable-cuda-graph \\ --mem-fraction-static 0.85 \\ --reasoning-parser deepseek-v4 \\ --tool-call-parser deepseekv4 \\ --host 0.0.0.0 \\ --port 30000 \\ --max-total-tokens=500000 \\ --swa-full-tokens-ratio=0.4 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"512gb\", \"protocol\": \"tcp\"}' # Group B sglang serve \\ --model=deepseek-ai/DeepSeek-V4-Flash \\ --pp-size 4 \\ --tp-size 2 \\ --moe-runner-backend flashinfer_mxfp4 \\ --disable-cuda-graph \\ --mem-fraction-static 0.85 \\ --reasoning-parser deepseek-v4 \\ --tool-call-parser deepseekv4 \\ --host 0.0.0.0 \\ --port 30000 \\ --max-total-tokens=500000 \\ --swa-full-tokens-ratio=0.4 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"512gb\", \"protocol\": \"tcp\"}' # SWE bench agent python -m minisweagent.run.benchmarks.swebench \\ --subset verified \\ --split test \\ -c swebench.yaml -c environment.container_timeout=24h \\ -o /root/swe_test/result \\ -m \"openai/deepseek-ai/DeepSeek-V4-Flash\" \\ -w 128 # SWE evaluation python -m swebench.harness.run_evaluation \\ --predictions_path /root/swe_test/result/preds.json \\ --dataset_name princeton-nlp/SWE-Bench_Verified \\ --split test \\ --max_workers 32 \\ --run_id test-mini-swe-agent \\ --timeout 1200 ``` ## Speed Tests and Profiling I tested with L1 only, L1+L2 and L1+L2+L3. The size of L1 and L2 are configured not to hold all tokens. L3 is large enough to hold all tokens. `request-rate` is set to `1` so that L3 prefetch are performed and not limited by budget. ```sh # L1 sglang serve \\ --model=Qwen/Qwen3-32B \\ --tp-size=2 \\ --pp-size=4 \\ --max-total-tokens=$((256*1024)) \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen25 \\ --page-size=64 \\ --enable-metrics # L2 sglang serve \\ --model=Qwen/Qwen3-32B \\ --tp-size=2 \\ --pp-size=4 \\ --max-total-tokens=$((256*1024)) \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen25 \\ --page-size=64 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-mem-layout=page_first # L3 sglang serve \\ --model=Qwen/Qwen3-32B \\ --tp-size=2 \\ --pp-size=4 \\ --max-total-tokens=$((256*1024)) \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen25 \\ --page-size=64 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-mem-layout=page_first \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"256gb\", \"protocol\": \"tcp\"}' # Benchmark python benchmark/hicache/bench_multiturn.py \\ --num-clients=128 \\ --num-rounds=10 \\ --disable-auto-run \\ --model=Qwen/Qwen3-32B \\ --enable-round-barrier \\ --request-rate 1 ``` ``` L1 Per-round metrics: Round 0: Average TTFT = 0.14s, Cache Hit Rate = 0.000000 (128 requests, 128 clients) Round 1: Average TTFT = 0.17s, Cache Hit Rate = 0.470588 (128 requests, 128 clients) Round 2: Average TTFT = 0.18s, Cache Hit Rate = 0.653846 (128 requests, 128 clients) Round 3: Average TTFT = 0.26s, Cache Hit Rate = 0.630580 (128 requests, 128 clients) Round 4: Average TTFT = 0.77s, Cache Hit Rate = 0.363459 (128 requests, 128 clients) Round 5: Average TTFT = 1.22s, Cache Hit Rate = 0.239976 (128 requests, 128 clients) Round 6: Average TTFT = 1.44s, Cache Hit Rate = 0.146925 (128 requests, 128 clients) Round 7: Average TTFT = 1.94s, Cache Hit Rate = 0.143266 (128 requests, 128 clients) Round 8: Average TTFT = 2.01s, Cache Hit Rate = 0.055469 (128 requests, 128 clients) Round 9: Average TTFT = 8.92s, Cache Hit Rate = 0.035112 (128 requests, 128 clients) L2 Per-round metrics: Round 0: Average TTFT = 0.15s, Cache Hit Rate = 0.000000 (128 requests, 128 clients) Round 1: Average TTFT = 0.17s, Cache Hit Rate = 0.470588 (128 requests, 128 clients) Round 2: Average TTFT = 0.18s, Cache Hit Rate = 0.653846 (128 requests, 128 clients) Round 3: Average TTFT = 0.19s, Cache Hit Rate = 0.742857 (128 requests, 128 clients) Round 4: Average TTFT = 0.18s, Cache Hit Rate = 0.795455 (128 requests, 128 clients) Round 5: Average TTFT = 0.23s, Cache Hit Rate = 0.830189 (128 requests, 128 clients) Round 6: Average TTFT = 0.19s, Cache Hit Rate = 0.854839 (128 requests, 128 clients) Round 7: Average TTFT = 0.70s, Cache Hit Rate = 0.652729 (128 requests, 128 clients) Round 8: Average TTFT = 1.16s, Cache Hit Rate = 0.474805 (128 requests, 128 clients) Round 9: Average TTFT = 1.86s, Cache Hit Rate = 0.379213 (128 requests, 128 clients) L3 Per-round metrics: Round 0: Average TTFT = 0.19s, Cache Hit Rate = 0.000000 (128 requests, 128 clients) Round 1: Average TTFT = 0.19s, Cache Hit Rate = 0.470588 (128 requests, 128 clients) Round 2: Average TTFT = 0.25s, Cache Hit Rate = 0.653846 (128 requests, 128 clients) Round 3: Average TTFT = 0.20s, Cache Hit Rate = 0.742857 (128 requests, 128 clients) Round 4: Average TTFT = 0.23s, Cache Hit Rate = 0.795455 (128 requests, 128 clients) Round 5: Average TTFT = 0.25s, Cache Hit Rate = 0.830189 (128 requests, 128 clients) Round 6: Average TTFT = 0.21s, Cache Hit Rate = 0.854839 (128 requests, 128 clients) Round 7: Average TTFT = 0.25s, Cache Hit Rate = 0.873239 (128 requests, 128 clients) Round 8: Average TTFT = 0.26s, Cache Hit Rate = 0.887500 (128 requests, 128 clients) Round 9: Average TTFT = 0.29s, Cache Hit Rate = 0.898876 (128 requests, 128 clients) ``` It shows that, L1 and L2 hit ratios drop at round 4 and 7 respectively. L3 may hold all rounds. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31661559415](https://github.com/sgl-project/sglang/actions/runs/31661559415) Latest PR Test (Extra): :x: [Run #31661559287](https://github.com/sgl-project/sglang/actions/runs/31661559287) ",
"url": "https://github.com/sgl-project/sglang/pull/27010",
"createdAt": "2026-06-02T04:46:51Z",
"updatedAt": "2026-08-13T11:46:59Z",
"timestamp": "2026-08-13T11:46:59Z",
"metrics": {
"reactions": 3,
"comments": 35
},
"labels": [
"high priority",
"hicache",
"run-ci",
"run-ci-extra",
"release-highlight"
],
"author": "stepinto",
"state": "open",
"assignees": [
"whybeyoung",
"xiezhq-hermann",
"ispobock",
"hzh0425",
"ShangmingCai"
]
},
{
"id": "github:sgl-project/sglang:pull_request:27442",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(diffusion): add OmniDreams autoregressive video world model",
"text": " ## Motivation Adds NVIDIA **OmniDreams** — a 2B distilled autoregressive driving-video world model (Cosmos-Predict2.5-2B based, production runtime = FlashDreams) — to `multimodal_gen`. Given a first frame + a per-frame HD-map (lane lines + actor boxes) control signal, it rolls out a temporally-coherent driving video via 2-step Self-Forcing flow-match denoising over a rolling block KV cache. Served both one-shot (HTTP `/v1/videos`) and realtime (WebSocket `/v1/realtime_video/generate`). Relates to #27214. Cookbook: [/cookbook/diffusion/OmniDreams](https://docs.sglang.io/cookbook/diffusion/OmniDreams/OmniDreams). ## Modifications - **DiT model:** OmniDreams 2B — 3D NeoX RoPE (44:42:42) with `shift_t` chunk positioning, AdaLN-LoRA modulation, flat-checkpoint loading (570-key `.pt`). - **AR pipeline (3 stages):** text/i2v/HD-map pre-encode → block-causal rollout (`BlockKVCache`: sink + rolling window) → seam-free Wan-2.1 VAE decode (clean per-chunk latents concatenated and decoded in one pass so the causal 3D cache flows across chunk boundaries). - **Conditioning:** 2-step flow-match scheduler (sigmas `{1.0, 0.8036, 0.0}`, shift=5.0) with first-frame i2v pinning; Cosmos-Reason1-7B `full_concat` text embedding; per-frame HD-map raster sequence VAE-encoded as one causal clip and sliced into per-chunk latent groups. - **Pure-Python runtime (no custom CUDA):** the vendored native CUDA FP8 DiT extension tree was removed; the runtime is now eager bf16 DiT + weight-only FP8 (pre-quantized E4M3 weights dequantized to bf16, cached for reuse), with an optional FP8-compute path (`torch._scaled_mm`). Self-attention uses SDPA; a SageAttention-3 FP4 backend and a CUDA-graph capture of the steady-state AR step are split out into separate follow-up PRs. The DiT file tree follows the single-file convention every sibling DiT uses (9 helper files → 4). - **Realtime online serving:** WebSocket `/v1/realtime_video/generate` streaming adapter (per-chunk HD-map control events + incremental frame output); `OmniDreamsRealtimeAdapter` reuses the `RealtimeModelAdapter` base class. Open-loop today (repeats last HD-map frame when the client lags); closed-loop per-chunk conditioning is planned. - **Serving:** TP via `Column/RowParallelLinear` head sharding (cross-view attention scaffolded, fail-loud gated off for single-view); HTTP API `hdmap_path` / `num_views`, with a guard rejecting local FS paths over HTTP (only `http(s)://` / `data:`) to close an arbitrary-file-read vector. - **Docs:** OmniDreams cookbook (`docs_new/cookbook/diffusion/OmniDreams/OmniDreams.mdx`) + card in the diffusion cookbook overview. ## Accuracy Tests Two fixes were needed to match the reference distribution and produce sharp output: - **Text encoder runs without an `attention_mask`** — the checkpoint is trained (FlashDreams' `CosmosReason1TextEncoder`) on the full padded sequence unmasked, and the DiT cross-attends over all 512 token embeddings (valid + padding). Adding a mask pushes padding-token conditioning OOD → washed-out rollouts (measured: normalized latent std collapses to ≈0.52 vs ≈0.84 unmasked). - **Clean first-frame re-injected during context-noise KV writes** so the pinned frame-0 doesn't accumulate noise in the rolling cache. **Distribution match:** after the text-conditioning fix the generated rollout's normalized latent std recovers `0.52 → 0.84`, matching real GT `≈0.84` — sharp, temporally coherent (no washed-out collapse). **Generalization, not memorization:** stays sharp across unseen seeds, mismatched first-frame/HD-map pairings, and a held-out scene; gen-vs-GT PSNR declines monotonically with rollout depth (`40.8 dB` @f1 → `23.6 dB` @f29). **CPU unit tests** — 101 cases in one file `test_omnidreams.py` (117 passed, 4 skipped with the lighttae/lightvae checkpoints available): ``` python -m pytest python/sglang/multimodal_gen/test/unit/test_omnidreams.py -q ``` ## Speed Tests and Profiling **rtx6kd (RTX 6000D 85GB, sm_120) — 13 frames, 2 denoise steps, default WanVAE:** | Path | Config | Denoise | Total | Peak VRAM | Notes | | --- | --- | --- | --- | --- | --- | | offline eager | `native_dit_acceleration=disabled` | 2.93s | 7.08s | 44.8GB | bf16 baseline | | offline **fp8_compute** | `native_dit_acceleration=fp8_compute` | 3.03s | 6.99s | 45.8GB | real FP8: 196 linears on `torch._scaled_mm`, weights stay FP8, SDPA attention. Faster than eager. | | offline `weight_only_fp8` (cold-start) | `native_dit_acceleration=weight_only_fp8` | 40.4s | 44.6s | 48.1GB | **dequant fallback**: loads the offline FP8 `.pt` then dequantizes to bf16 (~36s one-time, cached) and runs eager. For GPUs without FP8-compute; not the headline FP8 path. | | online `sglang serve` + `/v1/videos` | eager + WanVAE | 2.93s | 7.26s | 44.8GB | HTTP one-shot | Output is sharp across all paths (per-frame std ~48–52, frame-to-frame meandiff ~1.9, no gray-mush), numerically identical between eager and fp8_compute. **Two FP8 modes (both consume the offline-quantized `omnidreams_fp8_dit.pt` artifact):** - `fp8_compute` — the proper offline-quantization path: weights quantized to FP8 E4M3 (per-output-channel) and kept FP8 in VRAM; GEMMs via `torch._scaled_mm` (rowwise) + SDPA attention. Faster than eager on Blackwell. This is the recommended FP8 path. - `weight_only_fp8` — a fallback that dequantizes the offline FP8 weights back to bf16 and runs the eager DiT. It trades a one-time ~36s dequant (cached for the process lifetime) for a smaller checkpoint on disk, with no VRAM or steady-state speed benefit. Use it only on GPUs without FP8-compute support. **Benchmarks still missing:** - [ ] Warm-serve fp8_compute latency + realtime per-chunk latency (the SageAttention-3 backend + CUDA-graph capture were split out into follow-up PRs; will re-bench the warm/realtime paths once those land). - [ ] Multi-GPU / TP scaling (rtx6kd is single-GPU; needs a multi-GPU host). Sample inputs: `nvidia/omni-dreams-samples` (gated). HD-map is mandatory — without it the rollout goes OOD and collapses to gray after ~2 frames (expected). ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations) (OmniDreams cookbook + cookbook-overview card). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed) — partial (see Speed section: eager / fp8_compute / weight_only_fp8 done; warm-serve + realtime + TP scaling pending). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31623921878](https://github.com/sgl-project/sglang/actions/runs/31623921878) Latest PR Test (Extra): :x: [Run #31623921368](https://github.com/sgl-project/sglang/actions/runs/31623921368) ",
"url": "https://github.com/sgl-project/sglang/pull/27442",
"createdAt": "2026-06-06T11:24:15Z",
"updatedAt": "2026-08-12T17:42:42Z",
"timestamp": "2026-08-12T17:42:42Z",
"metrics": {
"reactions": 1,
"comments": 15
},
"labels": [
"documentation",
"quant",
"lora",
"sgl-kernel",
"diffusion"
],
"author": "Cerdore",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:27689",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] FlashInfer MLA: remove blocking D2H in spec-decode plan",
"text": "## Motivation On H200, DeepSeek-R1 with MTP/EAGLE (`--speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3`) was **no faster than standard decoding**, which shouldn't happen. nsys showed 3 synchronous device→host copies per attention `plan()` in the target-verify and draft-extend steps. Being blocking, they stall the host thread that launches the next CUDA graph, breaking overlap scheduling — and the stalls negate the speculation benefit. Root cause: `BatchMLAPagedAttentionWrapper.plan()` copies `qo_indptr`/`kv_indptr`/`kv_len_arr` to host via blocking `.to(\"cpu\")` when handed GPU tensors. The normal decode path already avoids this via `fast_mla_decode_plan` (it passes pre-computed CPU tensors), but the spec-decode verify and draft-extend paths still used the stock plan. ## Modifications `python/sglang/srt/layers/attention/flashinfer_mla_backend.py` only: - Allocate pinned host buffers `fast_plan_{qo_indptr,kv_indptr,kv_len_arr}_cpu` in `__init__` (prefill-capable backends only). - **Target-verify (cuda graph):** add `fast_mla_prefill_plan` — a sibling of the existing `fast_mla_decode_plan`, analogous to the merged `fast_prefill_plan` in #28854. It hands host-known `qo/kv indptr + lengths` straight to `_cached_module.plan` (no per-replay D2H) and refreshes the cuda-graph buffers, and is installed on the verify wrapper at capture once the real `plan()` has populated `_cached_module`. The verify branch builds the host arrays from `seq_lens_cpu` + the fixed `num_draft_tokens` tree shape and asserts them (fail-fast; the sync-free plan has no device-readback fallback). - **Eager `DRAFT_EXTEND_V2`** (`init_forward_metadata`) builds the same host arrays and passes them to `plan()` so its `.to(\"cpu\")` no-ops (flashinfer-MLA has no draft-extend cuda graph, so this path is eager). - `kv_indices` stays on GPU throughout. Decode, normal prefill, and ragged paths are unchanged. ### Note on the `seq_lens_cpu` dependency (re: review) This feeds `plan()` `seq_lens_cpu`-derived host arrays; it does not *remove* the `seq_lens_cpu` dependency. Fully removing it — as #26824 did for `trtllm_mla` via `needs_cpu_seq_lens=False` — requires **GPU-side** attention planning. FlashInfer-MLA's `plan()` is **host-side** (it reads the indptr/lengths on the CPU to build the tile schedule), so it fundamentally needs the lengths on the host. The merged #28854 takes the same approach (it also `assert`s `seq_lens_cpu is not None` and sets `kv_lens_host = seq_lens_cpu`). The residual per-iteration `seq_lens_cpu` D2H sync is a scheduler-level cost inherent to host-planned attention + data-dependent speculation, and removing it is a separate, backend-level change. ## Accuracy Tests gsm8k via `lm_eval` (local-completions, greedy), DeepSeek-R1-0528 FP8, TP8, H200, **this PR + MTP**: | filter | exact_match | |---|---| | flexible-extract | **0.954** ± 0.006 | | strict-match | **0.951** ± 0.006 | Speculative decoding is lossless (the target model verifies every draft), and this change only affects how the attention metadata reaches `plan()` — it does not alter model outputs (accept length unchanged, 2.253 → 2.257 below). ## Speed Tests and Profiling DeepSeek-R1-0528 FP8, TP8, H200; ISL/OSL=1024, CONC=8. Clean runs (no profiler attached), on the current branch base: | Config | Total Token throughput (tok/s) | Median TPOT (ms) | avg_spec_accept_length | |---|---|---|---| | main + STP | 1320.6 | 11.76 | — | | main + MTP (no fix) | 1319.8 | 11.87 | 2.253 | | **this PR + MTP** | **1671.9** | **9.35** | 2.257 | Before the fix, MTP (1319.8 tok/s) was **no faster than STP** (1320.6) despite accepting ~2.25 tokens/step — the blocking D2H stalls negated the speculation benefit. With the fix, MTP reaches **1671.9 tok/s (+26.7% vs both MTP-no-fix and STP)** and median TPOT drops 11.9 → 9.35 ms. Accept length is unchanged, so the gain is pure overlap recovery. nsys (profiled window, bs=8), before → after the fix, showing the mechanism: | Metric | Before | After | |---|---|---| | `cudaStreamSynchronize` | 1,120 | 160 | | D2H 36B (`qo`+`kv_indptr`) | 633 | 0 | | tiny D2H / graph-launch | 3.48 | 0.48 | The residual ~160 syncs are the inherent per-iteration `seq_lens_cpu` copy (see the note above), which standard decode doesn't incur. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Provide accuracy and speed benchmark results (above). - [ ] Add unit tests. - [ ] Update documentation. --- ### CI States Latest PR Test (Base): :x: [Run #31560198226](https://github.com/sgl-project/sglang/actions/runs/31560198226) Latest PR Test (Extra): :x: [Run #31560198204](https://github.com/sgl-project/sglang/actions/runs/31560198204) ",
"url": "https://github.com/sgl-project/sglang/pull/27689",
"createdAt": "2026-06-09T14:12:49Z",
"updatedAt": "2026-08-13T02:59:51Z",
"timestamp": "2026-08-13T02:59:51Z",
"metrics": {
"reactions": 0,
"comments": 14
},
"labels": [
"performance",
"deepseek",
"run-ci"
],
"author": "elvischenv",
"state": "closed",
"assignees": [
"Fridge003",
"kpham-sgl"
]
},
{
"id": "github:sgl-project/sglang:pull_request:27723",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[sglang-miles] RDT/NIXL weight sync support for Ray scheduler actors",
"text": "## What sglang-side support for **RDT (Ray Direct Transport / NIXL) weight sync** — lets the [miles](https://github.com/radixark/miles) trainer push RL weights to rollout engines via a zero-copy RDMA pull instead of NCCL broadcast. Paired with the miles-side PR (radixark/miles#1313), which has the perf write-up. ## Changes - **`ray/scheduler_actor.py`** — add `pull_weights()`, which uses `ray.experimental.set_target_for_ref` to RDMA pre-sharded weight buckets directly into the model's `param.data` buffers (no intermediate receive buffers / copies). - **`ray/engine.py` + `ray/data_parallel_controller.py`** — register `SchedulerActor`s as **detached** named actors with the http **port** baked into the name, so the trainer (a different Ray job) can discover them via `list_named_actors` even when several engines share a node; raise `max_concurrency` so a concurrent `pull_weights` is not starved while `run_event_loop` blocks; set `RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1` so the absolute GPU id from `get_accelerator_ids()` stays valid. - **`server_args.py`** — add `enable_engine_info_bootstrap` to start the engine-info bootstrap server and register per-rank parallelism config **without** the mooncake/verbs P2P transfer-engine seeding. - **`entrypoints/engine.py` + `model_executor/model_runner.py`** — honor `enable_engine_info_bootstrap`; make the P2P transfer-engine memory registration non-fatal so EFA/verbs clusters where it fails don't crash the scheduler (RDT only needs the parallelism config). - **`model_loader/loader.py`** — make `post_load_weights` public so the RDT path can invoke it. - **`pyproject.toml`** — require `ray>=2.55.1` (`ray.experimental.set_target_for_ref`). ## Notes Targeting the `sglang-miles` integration branch; rebased onto current `sglang-miles` (#28001, #29339). --- ### CI States Latest PR Test (Base): :x: [Run #31725179310](https://github.com/sgl-project/sglang/actions/runs/31725179310) Latest PR Test (Extra): :x: [Run #31725179141](https://github.com/sgl-project/sglang/actions/runs/31725179141) ",
"url": "https://github.com/sgl-project/sglang/pull/27723",
"createdAt": "2026-06-09T23:17:28Z",
"updatedAt": "2026-08-13T17:20:38Z",
"timestamp": "2026-08-13T17:20:38Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"dependencies"
],
"author": "xyuzh",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:27770",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[P/D disagg] Decode-side radix cache for SWA hybrid models (unified radix tree)",
"text": " This PR enables device-resident decode-side radix reuse for hybrid SWA models on the experimental unified radix tree. It keeps full-attention KV reusable while transferring a fresh SWA window, reducing repeated P/D KV transfer without introducing HiCache or Mamba support. ### How This Was Implemented - Caps the reused full-attention prefix at the SWA window boundary and allocates only the fresh SWA tail. - Caches an unfinished request's computed prefix, deduplicates overlapping full-attention KV, then repoints the live request to the cache-owned slots. - Carries SWA lock boundaries through rejection paths, including when the cap reduces a matched prefix to zero. - Requires the unified radix tree and rejects hierarchical cache, Mamba/SSM, DSA, and SWA-compress variants. - Adds registered device-only gpt-oss-20b P/D coverage. ### Before and After | Unfinished-request state | Before | After | |---|---|---| | Full-attention KV is device-resident but the old SWA window is tombstoned | The normal full+SWA match can return zero indices after insertion has deduplicated the request's full KV. Repointing then fails `new_prefix_len <= len(new_indices)`; continuing would leave the request referencing freed duplicate slots. | This one lifecycle path requests the full device-resident match, rewrites the request table to the cache-owned full-KV slots, and transfers its lock to the deepest full-resident node. Normal scheduler matching remains SWA-window-safe. | Walkthrough #### Mental model Full-attention KV remains reusable across turns, while SWA KV is valid only for the current live window. ```mermaid flowchart LR P[\"Prefill delta\"] --> D[\"Decode admission\"] M[\"Unified-tree prefix match\"] --> D D --> F[\"Reuse full-attention KV\"] D --> W[\"Allocate fresh SWA tail\"] F --> C[\"Cache unfinished request\"] W --> C ``` #### Why a full-device match is required `cache_unfinished_req` is named for the request lifecycle, not for a single pointer operation: it commits the current prefix of a request that will continue running. It inserts the prefix into the shared radix tree, deduplicates request-owned full-KV slots that the tree already owns, resolves the canonical cached slots, rewrites the live request's token table, and moves the request's tree lock. For a four-token reused prefix: ```text request-owned full KV before insert: R0 R1 R2 R3 cache-owned full KV: C0 C1 C2 C3 cache-owned SWA KV: tombstoned insert result: prefix_len = 4; duplicate R0-R3 are released normal match: [] because SWA is tombstoned full-device match: [C0 C1 C2 C3] canonical slots for repointing ``` The normal match would make the safety check `4 <= 0` fail. The full-device match makes it `4 <= 4`, so the request is repointed to `C0-C3` before it continues; `return_full_match` defaults to false and is enabled only at this repoint call, so admission and reuse decisions still require an SWA-valid match. #### Request lifecycle Decode matches and locks the device prefix, caps reuse at the SWA window boundary, checks both full and SWA pool budgets, and allocates the remaining full pages plus the live SWA tail. Admission rejection releases the original matched-node lock even when capping reduces the usable prefix to zero; unfinished caching then commits the live prefix, tombstones out-of-window SWA positions, repoints full-attention slots, and transfers the request lock. #### Boundaries and limitations This change is device-only. SWA decode radix combined with hierarchical cache is rejected, and Mamba/SSM, DSA, and SWA-compress models remain unsupported. ### Validation - `PYTHONDONTWRITEBYTECODE=1 .venv/bin/python test/registered/unit/mem_cache/test_decode_radix_lock_ref.py -v` — 6 passed. - `PYTHONDONTWRITEBYTECODE=1 .venv/bin/python test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py` — 1,818 ran; 758 skipped; OK. - Mocked `build_kv_cache` smoke — device-only path succeeds; SWA + hierarchical cache raises. - `.venv/bin/pre-commit run --files ...` — passed. - Targeted `/rerun-group disaggregation` — SWA decode-radix passed on 8×H20; hybrid-attention passed on 8×H200. ## Motivation Extend the decode-side radix cache (P/D disaggregation, #19746) to **sliding-window-attention (SWA) hybrid models** such as gpt-oss, which are currently rejected at startup. With this, a decode worker can **reuse a cached prefix's full-attention KV across requests/turns** and request only the delta from the prefill worker, instead of receiving the entire prefix KV every time. The feature is gated to the **experimental unified radix tree** (`SGLANG_ENABLE_UNIFIED_RADIX_TREE`). The default `SWARadixCache` and the Mamba/SSM pools remain unsupported on this path, as do DSA (DeepSeek-V4) and SWA-compress (Gemma/MiMo) variants. > Supersedes #26218, which took the same feature through the default `SWARadixCache`. We're standardizing on the unified tree for SWA decode-radix, so that path is dropped here. ## Background: why SWA needs special handling A hybrid SWA model keeps **two** KV caches with different lifetimes: - **Full-attention layers** — keep KV for every token, as long as it's cached. Large, sticky. - **Sliding-window layers** — keep KV for only the last *W* tokens; older slots recycle as the window advances. Bounded. So on the decode worker the prefix's **full-attention** KV can be reused directly, but its **sliding-window** KV cannot — by reuse time the original writer's window has slid past it. Such a node is a **tombstone** (full KV present, SWA recycled). This is normal and harmless for correctness: a reuser only needs the window for its *own* current position, which it gets fresh. ## What this PR does - **`kv_cache_builder.py`** — allow SWA + `--disaggregation-decode-enable-radix-cache` **only when the unified tree is active**; otherwise raise (DSA / SWA-compress still blocked even under unified). - **`decode.py`** — decode-side SWA mechanics: cap full-attention prefix reuse at the sliding-window start so the window lands in the fresh delta; `alloc_extend_swa_tail` on radix hits; mark the non-window prefix `swa_evicted_seqlen` (SWA tombstone); SWA-aware budget/eviction accessors; thread the SWA lock uuid through prefix lock/release. - **`unified_radix_cache.py` + `swa_component.py` + `base_prefix_cache.py`** — two `UnifiedRadixCache` fixes this path needs: 1. **SWA pool leak.** `SWAComponent.prepare_for_caching_req` only forwarded `swa_evicted_seqlen` when `is_finished=True`; on the unfinished path the out-of-window prefix tokens were inserted as *live* SWA and counted as `swa_evictable`, so at idle `available + evictable` exceeds the pool. Forward it on both paths so they become tombstones (`0` on the aggregated path → unchanged there). 2. **`cache_unfinished_req` repoint.** The dedup/repoint operates on full-attention cards but was sized by the SWA-window-safe match. On a reused, tombstoned prefix the window-safe match collapses to `0` while the cards number in the hundreds, tripping `assert new_prefix_len <= len(new_indices)`. Add `MatchPrefixParams.return_full_match`: `_match_prefix_helper` returns the deepest device-resident node, `_match_post_processor` returns the full cards + that node. The unified `inc_lock_ref` already skips tombstones, so no lock change is needed. No `swa_radix_cache.py` changes — the old path is intentionally not supported. ## Usage ``` --disaggregation-mode decode \\ --disaggregation-transfer-backend {nixl|mooncake} \\ --disaggregation-decode-enable-radix-cache \\ --page-size 64 # with SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 in the env ``` ## Test `test/manual/disaggregation/test_disaggregation_decode_radix_cache_swa.py` (gpt-oss-20b; manual because the model isn't in the CI pool) launches with the unified tree via `extra_*_env` and runs the shared decode-radix mixin (multi-turn cache-hit + 2-pass gsm8k). Validated on `openai/gpt-oss-20b`, 1P1D on 2×L40S, `--page-size 64 --attention-backend triton`, strict idle + per-iteration memory checks: | Check | Result | |---|---| | Multi-turn prefix reuse (decode cache hit) | cached tokens grow across rounds (0 → 1792 → 3584) | | gsm8k, 500 ex, 2 passes (2nd hits the cache) | 0.532 / 0.572 | | SWA-pool leak / `cache_unfinished_req` assert | **0 / 0** | ## Deep dive Writeup of the two `UnifiedRadixCache` bugs hit while enabling this path — exact errors, root causes, fix walkthrough, before/after validation: https://gist.github.com/ishandhanani/ed3cd2082381f31d848a7f6fcac977c6 --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31727270612](https://github.com/sgl-project/sglang/actions/runs/31727270612) Latest PR Test (Extra): :hourglass_flowing_sand: [Run #31727270469](https://github.com/sgl-project/sglang/actions/runs/31727270469) ",
"url": "https://github.com/sgl-project/sglang/pull/27770",
"createdAt": "2026-06-10T07:28:25Z",
"updatedAt": "2026-08-13T17:45:19Z",
"timestamp": "2026-08-13T17:45:19Z",
"metrics": {
"reactions": 3,
"comments": 38
},
"labels": [
"amd",
"hicache",
"run-ci",
"run-ci-extra"
],
"author": "ishandhanani",
"state": "open",
"assignees": [
"hzh0425",
"ShangmingCai"
]
},
{
"id": "github:sgl-project/sglang:pull_request:28251",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support Confidential Computing (CC) for Qwen3.5-397B-A17B-FP8 on B300",
"text": "Superseded by #31447. --- Targets `release/v0.5.12`. Enables Confidential Computing (CC) inference on B300 (validated with Qwen3.5-397B-A17B-FP8, TP4). **CC perf fixes** — auto-enabled when NVIDIA CC is detected (`is_confidential_compute()`), byte-identical off-CC: - **Async D2H copy worker** — run the per-step D→H token readback on a dedicated daemon thread, so the forced-synchronous copy under bounce-buffer CC no longer blocks the scheduler and serializes overlap. - **Ungate the FlashInfer AllReduce+RMSNorm fusion under CC** — build it on a multicast-free IPC workspace instead of disabling it (kernel parity with cc_off). **Also included:** port of #23351 (piecewise CUDA graph with NSA) for DSA/NSA models. See `CC_FIXES.md` for details. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/28251",
"createdAt": "2026-06-15T06:55:15Z",
"updatedAt": "2026-08-13T16:40:50Z",
"timestamp": "2026-08-13T16:40:50Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"documentation",
"quant",
"amd",
"lora",
"Multi-modal",
"deepseek",
"speculative-decoding",
"hicache",
"blackwell",
"npu",
"piecewise-cuda-graph",
"diffusion",
"jit-kernel"
],
"author": "elvischenv",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28354",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[FlashInfer v0.6.16] Support FlashInfer CuTe DSL NVFP4 MoE quantization",
"text": "## Motivation @humansand - Add FlashInfer CuTe DSL v2 MoE support to `--quantization nvfp4_online`. - Convert eligible BF16, FP16, or FP8 expert weights to NVFP4 at load time. - Compute and forward online per-token FP32 activation scales. - Support no A2A and FlashInfer A2A; both use CuTe DSL v2. - Keep the quantization contract established by merged upstream work: - `nvfp4_online`: online NVFP4 weight conversion with online per-token FP32 activation scales. - `modelopt_fp4`: serialized or online NVFP4 with per-tensor FP32 activation scales, including a default of `1.0` when a scale is absent. - `SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=1`: opt serialized `modelopt_fp4` checkpoints into the per-token path; existing TRT-LLM behavior is unchanged and this PR adds CuTe DSL v2 support. - DeepEP `low_latency` remains a per-tensor CuTe DSL v1 path and is not exposed as `nvfp4_online`. - Nemotron coverage keeps target and embedded draft on the same per-token contract: - Target: `nvfp4_online` + CuTe DSL v2 + no A2A. - Embedded MTP draft: automatically inherits `nvfp4_online` and CuTe DSL v2/no-A2A; no explicit draft quantization is set. - Related work: - https://github.com/sgl-project/sglang/pull/26083 - Initial online NVFP4 implementation. - https://github.com/sgl-project/sglang/pull/31382 - Added useful embedded-draft reachability, but labeling fixed/per-tensor activation-scale paths as `nvfp4_online` was a contract mistake. - https://github.com/sgl-project/sglang/pull/33115 - Routes online per-tensor behavior through `modelopt_fp4` and provides the shared online-weight loader reused here. - https://github.com/sgl-project/sglang/pull/33092 - Tracks the required FlashInfer 0.6.16.post1 dependency bump; this PR does not duplicate that version-only diff. - FlashInfer dependencies: - Per-token NVFP4 API: https://github.com/flashinfer-ai/flashinfer/pull/3645 - Deterministic non-fused finalize: https://github.com/flashinfer-ai/flashinfer/pull/3976 - Selected-tactic replay ordering is fixed by https://github.com/flashinfer-ai/flashinfer/pull/4192 but not released in 0.6.16.post1, so this PR retains a narrow SGLang workaround. - Broader cross-rank tactic synchronization remains tracked by https://github.com/sgl-project/sglang/pull/23317. ## Modifications Upstreamed overlap remains visible for review history: - ~~Preserve a dedicated embedded-draft `nvfp4_online` special case.~~ Merged [#33115](https://github.com/sgl-project/sglang/pull/33115) now provides general explicit/inherited draft quantization routing. - ~~Carry duplicate online-weight handling and a default `1.0` activation scale from [#31382](https://github.com/sgl-project/sglang/pull/31382) under `nvfp4_online`.~~ [#33115](https://github.com/sgl-project/sglang/pull/33115) owns that per-tensor behavior under `modelopt_fp4`. - ~~Treat [#31382](https://github.com/sgl-project/sglang/pull/31382)'s DeepEP `low_latency` path as `nvfp4_online`.~~ It remains available through `modelopt_fp4`; this PR does not claim it as per-token support. - ~~Keep separate draft backend-selection and dummy-dispatch compatibility shims.~~ Upstream supplies the reachability; this PR keeps only the CuTe DSL v2 per-token integration. Remaining PR-specific changes: - Admit `flashinfer_cutedsl` for `nvfp4_online` only with no A2A or FlashInfer A2A. - Quantize activations with `flashinfer.nvfp4_quantize(..., per_token_activation=True, backend=\"cute-dsl\")` and forward the returned per-token FP32 row scale. - Keep FlashInfer-A2A dispatch in BF16 so the runner can generate that row scale; scalar pre-quantized dispatch cannot represent the per-token contract. - Keep checkpoint source exclusions from suppressing online conversion, allowing embedded MTP experts to inherit `nvfp4_online`. - Construct and cache the per-token 4over6 global scale in the CuTe DSL runner; keep ModelOpt backend-neutral. - Preserve CuTe DSL weight layout, derived scales, frozen parameters, and parameter identity across post-load processing and `/update_weights_from_disk`. - Add `SGLANG_FLASHINFER_MOE_FUSED_FINALIZE`; deterministic inference disables fused atomic finalize. - Synchronize only fused-finalize selected-tactic replay; normal serving, profiling, and non-fused finalize remain asynchronous. - Update the FlashInfer fast-prefill private planner call for the `uniform_q_len` ABI. - Keep the earlier DP-attention, Mamba, and mixed-target experiments explicitly reverted; they add no net diff. - Add focused planner, CuTe DSL MoE, Nemotron MTP/EAGLE, and reload coverage. ## Accuracy Tests - Static validation: - `git diff --check`: passed. - `pre-commit run --all-files`: passed. - Environment: - 8x NVIDIA B200 devbox in `c1/infra` (tests use up to 4 GPUs). - Image: `lmsysorg/sglang:nightly-dev-cu13-20260806-ae5f8c94`. - `flashinfer-python==0.6.16.post1` - `flashinfer-cubin==0.6.16.post1` - `flashinfer-jit-cache==0.6.16.post1+cu130` - `apache-tvm-ffi==0.1.11` - Current-head results: - Fast-prefill planner: `2 passed, 16 warnings in 12.27s`. - CuTe DSL MoE suite: `6 passed, 100 warnings, 14 subtests passed in 18.58s`. - Cold-autotune Nemotron MTP/EAGLE E2E: `1 passed, 15 warnings in 128.68s`. - GSM8K score: `0.965`. - Per-request speculative accept length: mean `3.1385`. - Aggregate `avg_spec_accept_length=3.1067` against the `2.5` threshold. - Live args showed target and draft both on `nvfp4_online`; loading logs confirmed online conversion of `mtp.layers.1.mixer.experts`. - `/update_weights_from_disk`: `1 passed, 15 warnings, 3 subtests passed in 359.76s`. - Passed with CUDA graphs and both `flush_cache=True` and `flush_cache=False`. - Text, token IDs, and logprobs remained identical across reloads. ```bash CUDA_VISIBLE_DEVICES=0 python3 -m pytest -v -s \\ test/registered/unit/spec/test_fast_prefill_plan.py CUDA_VISIBLE_DEVICES=0 python3 -m pytest -v -s \\ test/registered/moe/test_cutedsl_moe.py rm -rf /root/.cache/sglang/flashinfer/autotune CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m pytest -v -s \\ test/registered/backends/test_flashinfer_nvfp4_online_moe_backend.py::TestFlashinferCuteDSLMoeBackendNvFp4Online::test_gsm8k CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m pytest -v -s \\ test/registered/rl/test_update_weights_from_disk_blackwell.py::TestServerUpdateWeightsFromDiskNVFP4CuteDSL::test_parameterized_update_weights_from_disk ``` ## Speed Tests and Profiling - No dedicated speed benchmark was run. - Accuracy-test throughput is not reported as benchmark evidence. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31668676229](https://github.com/sgl-project/sglang/actions/runs/31668676229) Latest PR Test (Extra): :x: [Run #31668675985](https://github.com/sgl-project/sglang/actions/runs/31668675985) ",
"url": "https://github.com/sgl-project/sglang/pull/28354",
"createdAt": "2026-06-16T00:14:09Z",
"updatedAt": "2026-08-13T15:49:04Z",
"timestamp": "2026-08-13T15:49:04Z",
"metrics": {
"reactions": 0,
"comments": 29
},
"labels": [
"documentation",
"quant",
"deepseek",
"blackwell",
"run-ci",
"bypass-fastfail"
],
"author": "zianglih",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:28574",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Kimi K2.5] Fuse deferred MoE finalize into next-layer AR + residual + RMSNorm",
"text": "## Summary Fuse the deferred MoE finalize into the next layer's input-RMSNorm via flashinfer `kMoEFinalizeARResidualRMSNorm` (pattern 7): weighted expert combine + shared-expert add + all-reduce + residual + RMSNorm in one kernel, while keeping `residual_out` available for Eagle3 aux capture. Gated by `SGLANG_ENABLE_MOE_DEFERRED_FINALIZE`. Builds on #28343. ## Key modifications - **deepseek_v2.py**: when deferred finalize + AR fusion are active, carry the un-finalized permuted `gemm2_out` across the layer boundary with a finalize-inputs bundle attached. - **communicator.py / layernorm.py**: input-RMSNorm consumes the bundle through new `RMSNorm.forward_with_moe_finalize_allreduce_fusion` (falls back to separate finalize + AR+RMSNorm fusion if unavailable). - **flashinfer_trtllm.py**: `FlashInferTrtllmMoeFinalizeFusionBundle` to package deferred outputs. - **flashinfer_comm_fusion.py**: new fused custom op; reshape `expanded_idx_to_permuted_idx`/`expert_scale_factor` to `[token, top_k]` (kernel derives `top_k` from `size(-1)` — flat index caused OOB); force trtllm backend (pattern 7 is trtllm-only); lamport workspace sizing + `MAX_COMM_SIZE`/capacity fallbacks. - **model_runner.py**: pre-size the MoE-TP workspace for the permuted buffer before CUDA-graph capture (never grow inside captured graphs). Note: pattern 7 is trtllm-only, so the fusion is intra-node (single NVLink domain). ## HumanEval (164, K2.5-NVFP4 + EAGLE3, attn_tp8_moe_tp8) | Metric | Fused | Baseline | |---|---|---| | Accept length | **3.02** | 3.03 | | Successful | 164/164 | 164/164 | Accept length is unchanged within noise — `residual_out` is correctly preserved for Eagle3 (a broken residual would collapse acceptance toward ~1.0). ## Numerics & speed vs the separate-finalize fallback Compared the fused kernel against the fallback path (deferred finalize + `kARResidualRMSNorm`, which **also emits `residual_out`**): - **Correctness**: no structural/indexing bug. 8-rank kernel equivalence vs fp32 truth — residual/norm mean rel-error **3.4% / 3.8%** (fused) vs **1.9% / 1.9%** (fallback): the fused kernel carries ~2× the bf16 rounding error (it accumulates the expert combine inside the bf16 AR pipeline; `fp32_acc` doesn't reach that step). Negligible on real data — HumanEval accept length **3.02 fused == 3.02 fallback**; only OOD random-token GSP shows a small dip (2.18 vs 2.23). - **Speed** (graph-captured decode, 8×B200, per MoE layer): fused is **slower at every batch size** — bs=1 `15.5 vs 14.4 µs` (+1.0), bs=16 `38.4 vs 26.4 µs` (+12.0). Confirmed the kernel's `allreduce_in` is the padded permuted `gemm2_out` (verified runtime rows **256 → 6272**, i.e. 64–128× the token count); this is trtllm's `do_finalize=False` output that the index map gathers from — the intended input, **not a mis-sized/bugged input**. The cross-rank AR is over the **finalized T rows** though (timing is invariant to the permuted-buffer size; AR-only is ~12–22 µs, latency-bound), so the slowdown is **not** AR bandwidth — it's the fused kernel's in-kernel finalize being ~2–4× the cost of the standalone finalize kernel (4–6 µs). **Takeaway**: under CUDA-graph decode the cheaper fallback is faster, more accurate, and already preserves `residual_out`, so it is preferred there; the fused kernel only wins when launch-bound (eager). Made with [Cursor](https://cursor.com) --- ### CI States Latest PR Test (Base): :x: [Run #27725890409](https://github.com/sgl-project/sglang/actions/runs/27725890409) Latest PR Test (Extra): :x: [Run #27725890323](https://github.com/sgl-project/sglang/actions/runs/27725890323) ",
"url": "https://github.com/sgl-project/sglang/pull/28574",
"createdAt": "2026-06-17T23:17:08Z",
"updatedAt": "2026-08-13T05:17:24Z",
"timestamp": "2026-08-13T05:17:24Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"deepseek"
],
"author": "kpham-sgl",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28581",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Simplify tree_speculative_sampling_target_only signature",
"text": "## Summary - Simplify the Python wrapper for `tree_speculative_sampling_target_only` by allocating the scratch `draft_probs` tensor internally. - Remove caller-side `draft_probs` allocation/passing from DFlash and EAGLE speculative sampling. - Update the direct speculative sampling test call to match the simplified API. Addresses the `draft_probs` cleanup item in #12562. ## Tests - `python -m py_compile sgl-kernel/python/sgl_kernel/speculative.py python/sglang/srt/speculative/dflash_utils.py python/sglang/srt/speculative/eagle_utils.py sgl-kernel/tests/speculative/test_speculative_sampling.py` - `python -m black --check sgl-kernel/python/sgl_kernel/speculative.py python/sglang/srt/speculative/dflash_utils.py python/sglang/srt/speculative/eagle_utils.py sgl-kernel/tests/speculative/test_speculative_sampling.py` - `python -m ruff check --select=F401,F821,UP037 sgl-kernel/python/sgl_kernel/speculative.py python/sglang/srt/speculative/dflash_utils.py python/sglang/srt/speculative/eagle_utils.py sgl-kernel/tests/speculative/test_speculative_sampling.py` - `git diff --check` Not run locally: `PYTHONPATH=sgl-kernel/python python -m pytest sgl-kernel/tests/speculative/test_speculative_sampling.py -q` requires a torch/CUDA/kernel-extension environment; this macOS venv does not have torch/CUDA. --- ### CI States Latest PR Test (Base): :x: [Run #30816528474](https://github.com/sgl-project/sglang/actions/runs/30816528474) Latest PR Test (Extra): :x: [Run #30816527549](https://github.com/sgl-project/sglang/actions/runs/30816527549) ",
"url": "https://github.com/sgl-project/sglang/pull/28581",
"createdAt": "2026-06-17T23:52:53Z",
"updatedAt": "2026-08-13T09:17:39Z",
"timestamp": "2026-08-13T09:17:39Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [
"speculative-decoding",
"sgl-kernel"
],
"author": "David-Wu1119",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28599",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[scheduler] Publish per-scheduler load on a dedicated socket for load-aware routers",
"text": "## Motivation Load-aware routers currently infer a worker's load from a router-side in-flight counter. That counter measures what *this router* has dispatched, not what the engine actually has queued — it misses traffic from other routers and direct clients, and for streaming responses it stays held for the whole response rather than the time the request occupies the scheduler. The engine already builds exactly the right number. `SchedulerLoadInquirer.get_loads()` produces a `LoadSnapshot` every publish cycle, which today feeds `/v1/loads` and DP-attention dispatch. This exposes that same snapshot to out-of-process consumers, so a router can price workers on real queue depth. The router-side consumer is #28600. ## Modifications **A third writer in the existing load-snapshot family.** `PubLoadSnapshotWriter` joins `ShmLoadSnapshotWriter` and `ZmqLoadSnapshotWriter` in `managers/load_snapshot.py`, behind the same `write()` / `publish_interval` / `close()` interface. It is not a variant of the existing ZMQ writer: that one is PUSH and *connects* to the single reader process that owns the PULL end, so an additional consumer would load-balance snapshots away from the DP controller rather than receive copies. Routers need fan-out, so this binds PUB. **Collection and publication get one owner.** `SchedulerLoadInquirer` now holds the writers and gains `publish()`; `Scheduler.publish_load_snapshot` is deleted and its two call sites delegate. The snapshot is built once per cycle no matter how many writers are due — relevant because `get_loads()` walks the running batch, the waiting queue and four disaggregation queues. **Port derivation and advertisement share one decision.** `load_pub_port_base()` decides whether a load range exists and where it starts (`kv_events_port + dp_size`; load rank `r` uses `base + r`). Both the writer and `/server_info`'s `load_endpoint_port_base` route through it, so the engine cannot advertise a range it will not bind — a router subscribing to a port nobody bound waits forever while reporting the worker as an expected publisher. It declines, and the advertisement is omitted, when: - there is no kv-events config, or the publisher is `null`; - the endpoint is not `tcp://` — `ipc://` and `inproc://` serve KV events fine but have no port to offset; - the endpoint has a **concrete host**, which the publisher would connect to rather than bind; nothing listens on the load range, so connecting publishes into a void neither side can detect; - the range would run past the u16 ceiling. A router treats a missing advertisement as \"this engine does not report load\" and falls back to its own signal, so declining is a supported outcome rather than an error. **Wire framing.** Subscribers require a three-frame message — `[topic, big-endian i64 seq, msgpack payload]` — and drop anything else. `ZMQ_CONFLATE` would be the natural fit for a gauge but keeps only a single *frame*, which would corrupt that layout; a small send HWM bounds the backlog instead. The topic frame is empty: this socket carries only load, so subscribers subscribe-all. **Failure handling is asymmetric on purpose.** Each writer is constructed under its own guard, so a bad router-facing endpoint cannot cost the internal writer. Losing the internal writer is logged as an error — without it `/v1/loads` omits the rank and the DP controller stops refreshing its budget for it, so the rank drifts out of the dispatch rotation with no other symptom. Publish failures are counted and reported on the first occurrence and every Nth after: the idle path publishes on every scheduler-loop iteration and `--sleep-on-idle` is off by default, so an unthrottled warning would turn a permanently broken writer into thousands of lines per second. ### Notes for reviewers - **Removed override point.** `Scheduler.publish_load_snapshot` was public; forks overriding it will need to move to `SchedulerLoadInquirer.publish`. - **Port footprint.** `--kv-events-config` now reserves `2 * dp_size` consecutive ports from the configured base rather than `dp_size`. Co-located engines spaced `dp_size` apart will collide. The argument documentation is not yet updated for this. - No new server arguments; cadence reuses `--load-snapshot-publish-interval`. ## Accuracy Tests Not applicable — no changes to model execution or output. ## Speed Tests and Profiling No benchmark run. The one relevant change is a reduction: publishing to N sinks now collects a single snapshot per cycle rather than one per sink, which removes a redundant queue walk on every extend batch. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). Tests added in `test/registered/unit/managers/test_load_snapshot_backends.py`: - the three-frame layout, against the module's own decoder; - advertised port equals bound port, in both the accept and decline directions; - the factory returns both writers, in an order that keeps the internal one from hiding behind the optional one; - `publish()` — single collection per cycle, independent per-writer intervals, `force` semantics, per-writer fault isolation, and the failure throttle; - the encoder's wire shape is a map keyed by field name, so declaring `array_like=True` on `LoadSnapshot` or renaming a field fails here rather than silently breaking consumers. --- ### CI States Latest PR Test (Base): :x: [Run #31657253640](https://github.com/sgl-project/sglang/actions/runs/31657253640) Latest PR Test (Extra): :x: [Run #31657253406](https://github.com/sgl-project/sglang/actions/runs/31657253406) ",
"url": "https://github.com/sgl-project/sglang/pull/28599",
"createdAt": "2026-06-18T03:02:19Z",
"updatedAt": "2026-08-13T01:18:34Z",
"timestamp": "2026-08-13T01:18:34Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [],
"author": "Kangyan-Zhou",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28600",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[router] cache_aware_zmq: load-aware selection from engine LoadSnapshot",
"text": "## Motivation The `cache_aware_zmq` policy prices workers by a router-side in-flight counter. That counter reflects what this router dispatched, not what the engine has queued: it cannot see traffic from other router replicas or from direct clients, and for streaming responses it stays held for the full response rather than for the time the request occupies the scheduler. With #28599 the engine publishes its `LoadSnapshot` on a dedicated socket. This subscribes to it and routes on real queue depth, keeping the in-flight counter as the fallback for any worker whose reported picture is missing or incomplete. Depends on #28599 for the publisher and the `/server_info` advertisement. ## Modifications **Discovery.** Workers advertise `load_endpoint_port_base` in `/server_info`'s `kv_events` block. It is `Option`: an engine that does not publish load simply omits it, and such a worker opens no load sockets and keeps using the router-side signal. Both the introspector and the discovery fallback thread the field. **Subscription.** A `SubKind::Load` registry opens one SUB per advertised DP rank on `load_port_base + dp_rank`, subscribe-all, sharing the existing pump channel. It is a separate registry from the KV one, each with its own handle map, so the two never collide despite sharing the `(url, dp_rank)` key. **Decoding.** The payload is the engine's `LoadSnapshot`, a msgpack map keyed by field name. Every field defaults to zero — the Python side is `omit_defaults` — and unknown keys are ignored, because the snapshot carries considerably more than this policy reads (memory, speculative, LoRA, disaggregation and queue sub-structs) and will keep growing. Tolerating both is what stops an engine-side addition from becoming a router-side decode failure. **A worker is trusted only when its picture is complete and fresh.** `EngineLoadTable::snapshot_fresh` sums a worker's ranks and omits the worker entirely if any rank is stale *or* if fewer ranks have reported than the worker advertised. Freshness alone is not sufficient: an entry that does not exist cannot be judged stale, so a rank whose publisher failed to bind at startup would leave that worker reporting a partial sum indefinitely — looking idle and drawing *more* traffic precisely because it is broken. Workers are recorded with the rank count they advertised, which gives the read path its denominator; a worker that never advertised a load topic has no denominator and is taken at face value. **Selection.** `WorkerLoads` is built once per `select()` so every comparison in one decision sees a consistent view, and `load_of` returns engine-reported depth where a fresh snapshot exists and `Worker::active_load()` otherwise. Both the imbalance fast-path and the matched-set tiebreak go through it. ### Degradation Every failure path lands on the prior behavior rather than an error: - no advertised load port → no load sockets opened; - an undecodable frame → logged and dropped, the subscriber survives; - entries older than the freshness window, or an incomplete rank set → the worker is omitted from the table; - worker removal clears its entries so a re-added worker cannot inherit stale load. ### Known gap `load_of` can mix measurement bases within a single selection: engine queue depth for workers with a complete fresh snapshot, and the router-side in-flight counter for the rest. Those quantities are not interchangeable, so a fleet where only some workers report load can produce a fabricated spread in the imbalance check. Making the choice all-or-nothing per selection is a follow-up. ## Accuracy Tests Not applicable — routing policy only, no change to model execution or output. ## Speed Tests and Profiling No benchmark run. `snapshot_fresh` is computed once per selection so per-worker lookups are O(1); it does allocate per worker on the routing path, which is a candidate for `Arc` keys if it shows up in profiles. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). Tests cover the wire contract and each degradation path: decoding a load frame into `WorkerEvent::Load`; ignoring unknown fields and defaulting absent ones; rank sums, staleness, partial freshness, and a worker missing a rank entirely; engine load overriding `active_load`, the matched-set tiebreak using it, and stale load falling back; clearing a removed worker; and the `/server_info` introspection path both with and without the load port advertised.",
"url": "https://github.com/sgl-project/sglang/pull/28600",
"createdAt": "2026-06-18T03:02:28Z",
"updatedAt": "2026-08-12T22:53:24Z",
"timestamp": "2026-08-12T22:53:24Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"npu"
],
"author": "Kangyan-Zhou",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28655",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] GDN linear out-proj fusion",
"text": "## Motivation Each GatedDeltaNet (GDN) layer ends a token step with an **output gated-RMSNorm** followed by the `out_proj` linear. On AttnFP8-style Qwen3.5 checkpoints `out_proj` runs as an **a8w8 GEMM** (per-token activation × per-channel weight FP8), so the bf16 gated-RMSNorm output has to be quantized to FP8 per-token *before* the GEMM. Today that is three passes over the GDN output on the decode-adjacent critical path: gated RMSNorm → write bf16 → separate per-token activation quant → write fp8 → GEMM. The intermediate bf16 round-trip is pure overhead. This PR fuses the **gated-RMSNorm + FP8 per-token activation quant** into a single AITER kernel (`gated_rmsnorm_fp8_per_token_quant`) that emits the `(fp8, scale)` pair directly, which is fed straight into the same `out_proj` a8w8 GEMM the non-fused path uses. The numerics are identical (same weight layout, same scales) — it only removes the bf16 intermediate and the redundant standalone quant. The path is ROCm/aiter-only, auto-enabled where applicable, and a clean no-op fallback everywhere the per-layer capability check doesn't match, so default behavior is unchanged on non-AttnFP8 checkpoints. **Please be noted that this PR works along with Aiter#3844 to take effect.** ## Modifications - **`layers/quantization/fp8_utils.py`**: Add `apply_fp8_linear_aiter_prequant`, a dedicated helper for an activation that a fused producer has **already FP8-quantized**. Given `(qinput_fp8, x_scale)` it routes straight to `gemm_a8w8_bpreshuffle` using `weight.T` and per-channel `weight_scale`, mirroring the AITER per-token branch of `apply_fp8_linear` exactly (same layout/scales) but skipping the activation quant. Keeping this in its own function avoids overloading `apply_fp8_linear`'s input contract with a tuple fast-path. - **`models/qwen3_5.py`**: - Add module-level gate `_fuse_gdn_oproj` (auto-on under AITER; force off with `SGLANG_DISABLE_GDN_OUT_PROJ_FUSION=1`). - `_can_fuse_oproj` (`cached_property`): the real per-layer capability check, evaluated once on first `forward()` (weights/scales are loaded by then) and cached on the instance. It fuses only when `out_proj` is a genuine FP8 a8w8 **per-token** GEMM — detected via `Fp8LinearMethod.use_per_token_if_dynamic` **or** Quark `QuarkW8A8Fp8` `scheme.per_token` (either guarantees the pre-shuffled `bpreshuffle` weight layout) — and the weight dtype is **`torch.float8_e4m3fn`**, `head_v_dim == 128`, v-heads-per-rank ≤ 128, and the output gate is SiLU/none. The dtype is restricted to OCP `e4m3fn` (AttnFP8 weights are always `e4m3fn`; the fused kernel does not handle `fnuz`). Anything else falls back to the original path (fusing without the pre-shuffled weight would corrupt the GEMM output). - `_fused_norm_quant_out_proj`: calls the AITER `gated_rmsnorm_fp8_per_token_quant` kernel to produce `(q_fp8, q_scale)` from `core_attn_out`, `z`, `norm.weight`, and `eps`, then feeds them into `apply_fp8_linear_aiter_prequant` with `out_proj.weight` / `out_proj.weight_scale`. - `forward`: when `_can_fuse_oproj` is true, take the fused path; otherwise the original `norm` → reshape → `out_proj` path is unchanged. > Default behavior is preserved: on any checkpoint where `out_proj` is not FP8 per-token a8w8 (or shapes/gate/dtype don't match), `_can_fuse_oproj` is `False` and the original code runs. ## Usage The fusion is **on by default** on the AITER/HIP path and engages automatically for AttnFP8 checkpoints whose `out_proj` is FP8 per-token a8w8. No new flag is needed to enable it; disable it for debugging/A-B with `SGLANG_DISABLE_GDN_OUT_PROJ_FUSION=1`. ```bash SGLANG_USE_AITER=1 SGLANG_USE_AITER_FP8_PER_TOKEN=1 \\ python3 -m sglang.launch_server \\ --model-path /path/to/Qwen3.5-397B-A17B-MXFP4-AttnFP8 \\ --trust-remote-code \\ --tensor-parallel-size 2 \\ --attention-backend aiter \\ --kv-cache-dtype fp8_e4m3 ``` ## Accuracy Tests `lm_eval` GSM8K 5-shot, full 1319 questions, Qwen3.5-397B-A17B-**MXFP4-AttnFP8**, TP=2, aiter backend, `fp8_e4m3` KV cache, `--enable-dense-fp8 --enable-mixed-chunk`. Fusion is the only variable (`OFF` = `SGLANG_DISABLE_GDN_OUT_PROJ_FUSION=1`). To rule out single-seed sampling noise, the A/B was repeated over **3 seeds** (seed changes the 5-shot exemplar sampling): | config | strict-match (mean ± std) | flexible-extract (mean ± std) | per-seed strict | |----------------|---------------------------|-------------------------------|---------------------| | fusion **OFF** | 0.9388 ± 0.0056 | 0.9335 ± 0.0068 | 0.9378 / 0.9462 / 0.9325 | | fusion **ON** | 0.9378 ± 0.0108 | 0.9295 ± 0.0082 | 0.9409 / 0.9234 / 0.9492 | The strict-match mean gap is **+0.0010 (off − on)** — essentially zero and far below the per-seed spread (± ~0.01). ON is in fact *higher* than OFF in 2 of the 3 seeds. The fused kernel feeds the identical weight/scales into the identical GEMM, so the path is numerically equivalent — accuracy is unaffected. `lm_eval` GPQA-Diamond CoT (`gpqa_diamond_cot_zeroshot`, all 198 questions, generative chain-of-thought via the chat template, 32k generation budget so the reasoning trace completes), same checkpoint and serving config (Qwen3.5-397B-A17B-**MXFP4-AttnFP8**, TP=2, aiter backend, `fp8_e4m3` KV cache, `--enable-dense-fp8 --enable-mixed-chunk`), fusion the only variable. Metric is `exact_match` (flexible-extract). To control for sampling noise the A/B was repeated over **2 seeds** with the recommended Qwen sampling (`temperature=0.6, top_p=0.95, top_k=20`): | config | GPQA-Diamond (CoT), mean ± std | per-seed | |----------------|--------------------------------|-----------------| | fusion **OFF** | 0.8687 ± 0.0000 | 0.8687 / 0.8687 | | fusion **ON** | 0.8712 ± 0.0178 | 0.8838 / 0.8586 | The mean gap is **+0.0025 (on − off)**, far inside the ±0.018 seed-to-seed spread — statistically indistinguishable. Both arms match the official BF16 model card (GPQA **88.4**) to within ~1 point, i.e. the MXFP4-AttnFP8 quantization is effectively lossless here and the out_proj fusion does not perturb it. ## Speed Tests and Profiling TP=2, `--attention-backend aiter`, `--page-size 16`, `--chunked-prefill-size 8192`, `--mem-fraction-static 0.8`, allreduce-fusion on, tuned bf16 GEMM present, measured on our tuned serving config for this checkpoint (fused-AR-quant and `--enable-dense-fp8` both on). **Out-proj fusion is the only knob toggled** (A = OFF via `SGLANG_DISABLE_GDN_OUT_PROJ_FUSION=1`, B = ON). **Throughput is total token throughput (input+output) across the TP=2 group** (`tok/s/gpu × 2`); TPOT is median per-output-token latency. ### MXFP4-AttnFP8 — ISL/OSL = 1024/1024 (decode-bound) | concurrency | total tok/s OFF→ON | Δ throughput | median TPOT OFF→ON (ms) | Δ TPOT | |-------------|--------------------|--------------|-------------------------|--------| | 4 | 840 → 842 | +0.24% | 9.15 → 9.07 | −0.87% | | 8 | 1366 → 1376 | +0.73% | 11.36 → 11.23 | −1.14% | | 16 | 2226 → 2236 | +0.45% | 14.18 → 14.07 | −0.78% | | 32 | 3122 → 3146 | +0.77% | 20.15 → 19.98 | −0.84% | | 64 | 4306 → 4352 | +1.07% | 29.44 → 29.17 | −0.92% | ### MXFP4-AttnFP8 — ISL/OSL = 8192/1024 (prefill-heavy) | concurrency | total tok/s OFF→ON | Δ throughput | median TPOT OFF→ON (ms) | Δ TPOT | |-------------|--------------------|--------------|-------------------------|--------| | 4 | 3448 → 3482 | +0.99% | 9.87 → 9.77 | −1.01% | | 8 | 5330 → 5370 | +0.75% | 12.75 → 12.66 | −0.71% | | 16 | 7992 → 8048 | +0.70% | 17.18 → 17.02 | −0.93% | | 32 | 10426 → 10480 | +0.52% | 26.44 → 26.31 | −0.49% | | 64 | 13062 → 13138 | +0.58% | 43.03 → 42.85 | −0.42% | The fusion is a **small, consistent, no-downside win** at every concurrency on both shapes: TPOT −0.4% to −1.1% (always faster) and total throughput +0.2% to +1.1%. TTFT additionally improves **−3% to −5%** at 1k/1k (short-prompt, decode-adjacent path) and is **neutral at 8k/1k** (TTFT there is prefill-dominated, which this GDN-output fusion doesn't touch — expected). No TPOT regression anywhere. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31689452132](https://github.com/sgl-project/sglang/actions/runs/31689452132) Latest PR Test (Extra): :x: [Run #31689451726](https://github.com/sgl-project/sglang/actions/runs/31689451726) ",
"url": "https://github.com/sgl-project/sglang/pull/28655",
"createdAt": "2026-06-18T12:53:32Z",
"updatedAt": "2026-08-13T10:09:58Z",
"timestamp": "2026-08-13T10:09:58Z",
"metrics": {
"reactions": 1,
"comments": 13
},
"labels": [
"high priority",
"run-ci"
],
"author": "mqhc2020",
"state": "open",
"assignees": [
"yichiche"
]
},
{
"id": "github:sgl-project/sglang:pull_request:28666",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Fuse shared_expert_gate GEMV into the MoE append kernel (HIP/aiter)",
"text": "## Motivation On the AITER shared-expert-fusion path, computing the fused shared-expert routing weight launches a standalone GEMV — `self.shared_expert_gate(hidden_states)`, a `[M, hidden] × [hidden, 1]` matrix-vector op (the `Cijk_…MT1x2x512…` kernel, ~8.9 µs in decode) — whose only output feeds the subsequent `_fused_append_shared_experts_with_weights_kernel`. At decode batch sizes this GEMV is pure kernel-launch overhead (≈0 TFLOPs), so it is a prime candidate to fold into the append kernel. ## Modifications - `python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` - `_fused_append_shared_experts_with_weights_kernel`: add `hidden_ptr, wgate_ptr, scale` args and `FUSE_GATE / HIDDEN / BLOCK_H` constexprs. When `FUSE_GATE`, each program (one token) reduces `hidden[pid, :] · W_gate[:]` in fp32, then broadcasts `sigmoid(logit) * scale` to the shared slots — replacing the `shared_weights_ptr` load. - `fused_append_shared_experts_with_weights(...)`: add `fuse_gate=False, hidden_states=None, gate_weight=None, scale=1.0`. The default (`fuse_gate=False`) path is byte-for-byte unchanged. - `python/sglang/srt/models/qwen2_moe.py` - Add `_shared_expert_scale()` (1/ep_size pre-scale helper). - `_append_shared_to_topk_output`: on `_use_aiter`, call the kernel with `fuse_gate=True` (no separate gate GEMM launch); otherwise keep the original `_get_shared_expert_weights` path. The non-aiter (CUDA) path is unchanged. Net effect: the shared_expert_gate GEMV + sigmoid + scale collapse into the existing append kernel — one kernel instead of two. ## Accuracy Tests GSM8K (200 questions, parallel 2000, greedy), Qwen3.5-397B-A17B-MXFP4, tp=2, AITER backend, MI35x: | Config | Samples | Mean | |---|---|---| | Baseline | 0.620 / 0.640 / 0.625 | 0.628 | | Gate-fused | 0.635 / 0.615 / 0.560 | 0.603 | Within run-to-run variance (greedy + continuous-batching nondeterminism, σ≈0.045). Offline kernel test: the in-kernel fp32 GEMV matches the `F.linear → sigmoid → scale` fp32 reference exactly (max|diff| = 0.0), routed weights bit-identical, shared-expert id correct. ## Benchmarking and Profiling torch profiler trace (decode), same config. ATen kernels keep the `_cuda` suffix on ROCm/HIP. | Kernel | Before | After | |---|---|---| | `Cijk_…MT1x2x512…` (shared-gate GEMV) | ~8.9 µs, separate launch | **0 occurrences (eliminated)** | | `_fused_append_shared_experts_with_weights_kernel` | ~4.0 µs | ~5.2 µs (now does GEMV+sigmoid+append) | Per-MoE-layer: 12.9 µs → ~5.2 µs ≈ **−7.5 µs**, by removing a kernel launch (the GEMV is launch-bound, not compute-bound, so tuning it does not help — fusion does). ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review Process 1. Ping Merge Oncalls to start the PR flow. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests). - `/tag-run-ci-label`, `/rerun-failed-ci`, `/tag-and-rerun-ci` 4. After green CI and required approvals, ask Merge Oncalls to merge. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31400910619](https://github.com/sgl-project/sglang/actions/runs/31400910619) Latest PR Test (Extra): :x: [Run #31400910477](https://github.com/sgl-project/sglang/actions/runs/31400910477) ",
"url": "https://github.com/sgl-project/sglang/pull/28666",
"createdAt": "2026-06-18T17:13:04Z",
"updatedAt": "2026-08-13T00:34:35Z",
"timestamp": "2026-08-13T00:34:35Z",
"metrics": {
"reactions": 1,
"comments": 25
},
"labels": [
"amd",
"run-ci",
"jit-kernel"
],
"author": "yichiche",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28929",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix Qwen3.5 ModelOpt NVFP4 checkpoint loading",
"text": "## Motivation Fixes #22117. `natfii/Qwen3.6-27B-VLM-NVFP4-MTP` fails during TP=2 checkpoint loading with a shape assertion: ```text AssertionError: param_data.shape == loaded_weight.shape ``` This checkpoint uses ModelOpt FP4/NVFP4 quantization and contains packed weights and scale parameters in Qwen GDN and attention submodules. Qwen3.6 VLM reuses the Qwen3.5 model implementation, so the failing path is in the Qwen3.5 loader. In the failing GDN `conv1d` path, the checkpoint tensor shape is `[10240, 1, 4]`, while the TP=2 runtime parameter shape is `[5120, 4]`. The existing TP narrow operation preserves the singleton channel dimension and produces `[5120, 1, 4]`; the default loader then compares it against `[5120, 4]` and fails. Also, Qwen3.5 currently passes `quant_config=None` to linear-attention and full-attention submodules for ModelOpt FP4. That bypasses the existing ModelOpt per-layer config decision and prevents submodules marked as quantized by the recipe from being created and loaded with the packed parameter layout. These submodules should keep the original `quant_config` and let recipe fields such as `exclude_modules` and `packed_modules_mapping` decide the layout. ## Modifications - Keep the original `quant_config` for Qwen linear-attention and full-attention submodules in the ModelOpt FP4 path, so the existing per-layer ModelOpt recipe logic decides quantization or exclusion. - Extend the GDN packed split loader so packed quantized scale parameters such as `weight_scale_2` use the same shard mapping as weights. - Wrap the GDN `conv1d` loader to squeeze/unsqueeze only the singleton channel representation difference between `[N, 1, K]` and `[N, K]`, then delegate to the original loader. - Add unit coverage for ModelOpt FP4 config propagation, `weight_scale_2` loader binding, and GDN `conv1d` singleton channel compatibility. ## Accuracy Tests Added and ran the relevant unit tests in `test/registered/unit/models/test_qwen3_5_packed_weight_loader.py`. The tests cover the packed loader paths, ModelOpt FP4 config propagation, and GDN `conv1d` singleton channel compatibility touched by this PR. ## Speed Tests and Profiling Not applicable. This PR only changes checkpoint loading paths and does not change kernels, scheduling, or model forward computation. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). No user-facing documentation is needed for this PR. - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). This PR does not modify the inference computation path. - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31725229415](https://github.com/sgl-project/sglang/actions/runs/31725229415) Latest PR Test (Extra): :x: [Run #31725229272](https://github.com/sgl-project/sglang/actions/runs/31725229272) ",
"url": "https://github.com/sgl-project/sglang/pull/28929",
"createdAt": "2026-06-22T11:33:26Z",
"updatedAt": "2026-08-13T17:21:22Z",
"timestamp": "2026-08-13T17:21:22Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [],
"author": "ruokee",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:28932",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Add dense-FP8 for MXFP4 checkpoints with fused silu, mul, activation quant",
"text": " ## Motivation Quark MXFP4 checkpoints leave a handful of large dense projections unquantized in bf16 — most notably the Qwen3.5 `shared_expert.down_proj`. On the AMD ROCm/aiter path these leftover bf16 GEMMs are comparatively heavy, and the cost shows up most in prefill-bound regimes where the dense projection is on the critical path. Today there is no way to promote just these excluded layers without re-quantizing the whole checkpoint. This PR adds an opt-in `--enable-dense-fp8` path that promotes only the eligible large bf16 dense layers (which the checkpoint left unquantized) to **online w8a8 FP8**, and fuses the preceding `SiluAndMul` + per-token activation quantization into a single aiter kernel. Instead of running `SiluAndMul`, then a separate activation quant, then the GEMM, the fused kernel emits a pre-quantized `(fp8, scale)` tuple that feeds straight into `down_proj`. The path is numerically a dynamic per-token FP8 quant of an otherwise bf16 layer, is ROCm/aiter-only, and is fully opt-in, so default behavior is unchanged. ## Modifications - **`server_args.py`**: Add the `--enable-dense-fp8` server arg (AMD/aiter, default **off**). Requires `SGLANG_USE_AITER` and `SGLANG_USE_AITER_FP8_PER_TOKEN`. - **`quark/quark.py`**: Give `QuarkConfig` a model-registered dense-FP8 policy. `register_dense_fp8_modules(include, exclude, min_output_size)` lets a model declare which excluded (bf16) layers may be promoted — quark itself owns no module names or thresholds. In `get_quant_method`, a layer that would normally fall back to `UnquantizedLinearMethod` is instead routed to an online dynamic `Fp8LinearMethod` (`Fp8Config(is_checkpoint_fp8_serialized=False, activation_scheme=\"dynamic\")`) when `--enable-dense-fp8` is set and the layer matches the policy (include substring, not excluded, output size ≥ min). - **`models/qwen3_5.py`**: Register the Qwen3.5 policy — include only `.shared_expert.down_proj`, exclude tiny/routing/embedding layers (`conv1d`, `shared_expert_gate`, `mlp.gate`, `in_proj_*`, `lm_head`, `embed`), and require `N ≥ 2048`. Aiter-only, so the tuned policy lives with the model. - **`models/qwen2_moe.py`**: In the MoE block forward, when `down_proj` is running as online w8a8 FP8 per-token on aiter, fuse `SiluAndMul` + per-token quant into a single `aiter.silu_and_mul_quant` call that produces the `(fp8, scale)` tuple consumed directly by `down_proj`, skipping a separate activation + quant pass. Gated on `use_aiter_fp8_per_token`; default off. - **`layers/quantization/fp8.py`**: Teach `Fp8LinearMethod.apply` to accept a pre-quantized `(fp8, scale)` tuple input (from the fused kernel) and route it to the aiter PTPC helper `apply_fp8_ptpc_linear`, since `apply_fp8_linear` has no tuple path. Weight is transposed `(K, N) → (N, K)` to match the helper's expectation. ## Usage Opt-in via `--enable-dense-fp8`. It is AMD/aiter-only and requires aiter FP8 per-token, so set the two aiter env flags. The same command applies to both Qwen3.5 checkpoints (add `--kv-cache-dtype fp8_e4m3` for the MXFP4-AttnFP8 checkpoint): ```bash SGLANG_USE_AITER=1 SGLANG_USE_AITER_FP8_PER_TOKEN=1 \\ python3 -m sglang.launch_server \\ --model-path /path/to/Qwen3.5-397B-A17B-MXFP4 \\ --trust-remote-code \\ --tensor-parallel-size 2 \\ --attention-backend aiter \\ --enable-dense-fp8 ``` The `SiluAndMul`+per-token-quant fusion is on by default whenever `--enable-dense-fp8` is active on the aiter per-token path; it can be disabled for debugging with `SGLANG_DISABLE_DENSE_FP8_SILU_FUSE=1` (not recommended — fusion is neutral-to-faster and saves the bf16 intermediate round-trip). ## Accuracy Tests The same `--enable-dense-fp8` config (online PTPC FP8 + fused `SiluAndMul`+quant) is applied to **both** Qwen3.5-397B-A17B checkpoints, TP=2, aiter backend. GSM8K 5-shot, 1319 questions. **MXFP4-AttnFP8** (fp8_e4m3 KV cache): | config | accuracy (flexible / strict) | |------------------------------------------|------------------------------| | `dense_off` (baseline, down_proj bf16) | 0.9363 / 0.9409 | | `dense_on` (this PR, down_proj FP8) | 0.9393 / 0.9454 | **Plain MXFP4** (3-seed mean ± per-seed stderr ≈ 0.007): | config | flexible (3 seeds) | strict (3 seeds) | |------------------------------------------|-------------------------------|-------------------------------| | `dense_off` (baseline, down_proj bf16) | 0.9340 / 0.9356 / 0.9295 | 0.9401 / 0.9409 / 0.9371 | | `dense_on` (this PR, down_proj FP8) | 0.9340 / 0.9257 / 0.9325 | 0.9401 / 0.9409 / 0.9416 | Accuracy is unchanged within noise on both checkpoints — the per-seed spread (±0.007) exceeds the dense_off↔dense_on difference, so promoting `down_proj` to FP8 is accuracy-neutral. ## Speed Tests and Profiling TP=2, `--attention-backend aiter`, `--page-size 16`, `--chunked-prefill-size 8192`, `--mem-fraction-static 0.8`, allreduce-fusion on, tuned bf16 GEMM present. A = `dense_off` (baseline), B = `dense_on` (this PR); all other knobs identical. **Throughput below is total token throughput (input+output) across the TP=2 group**, i.e. `tok/s = (tok/s/gpu) × 2`. TPOT is the median per-output-token latency. ### MXFP4-AttnFP8 (`--kv-cache-dtype fp8_e4m3`) **ISL/OSL = 1024/1024** (decode-bound): | concurrency | total tok/s OFF→ON | Δ throughput | median TPOT OFF→ON (ms) | Δ TPOT | |-------------|--------------------|--------------|-------------------------|--------| | 4 | 776 → 774 | −0.3% | 9.96 → 9.97 | +0.1% | | 8 | 1362 → 1372 | +0.7% | 11.39 → 11.31 | −0.7% | | 16 | 2238 → 2240 | +0.1% | 14.10 → 14.09 | −0.1% | | 32 | 3124* → 3140 | +0.5% | 20.06* → 20.00 | −0.3% | | 64 | 4378 → 4360 | −0.4% | 28.97 → 29.11 | +0.5% | \\* The captured `dense_off` cc=32 point was a noisy outlier; a repeat baseline run is used here. At 1k/1k the dense projection is too compute-light to move the needle — results are **at parity (no regression)** on both throughput and TPOT. **ISL/OSL = 8192/1024** (prefill-heavy): | concurrency | total tok/s OFF→ON | Δ throughput | median TPOT OFF→ON (ms) | Δ TPOT | |-------------|--------------------|--------------|-------------------------|--------| | 4 | 3170 → 3182 | +0.4% | 10.76 → 10.68 | −0.7% | | 8 | 5216 → 5338 | +2.3% | 13.05 → 12.74 | −2.4% | | 16 | 7874 → 8008 | +1.7% | 17.46 → 17.14 | −1.8% | | 32 | 10260 → 10462 | +2.0% | 27.07 → 26.34 | −2.7% | | 64 | 12840 → 13120 | +2.2% | 44.06 → 42.88 | −2.7% | In the prefill-heavy 8k/1k regime — where the bf16 `down_proj` GEMM actually matters — the fused FP8-dense path gives a **consistent ~2% total-throughput gain** and a **~2–3% lower TPOT** at every concurrency. Net: neutral at 1k/1k, a clear win for prefill-bound workloads, with no TPOT regression anywhere. ### Plain MXFP4 Node-pinned (single NUMA node, fixed GPU pair) to suppress host/neighbor contention. **ISL/OSL = 8192/1024** (prefill-heavy): | concurrency | total tok/s OFF→ON | Δ throughput | median TPOT OFF→ON (ms) | Δ TPOT | |-------------|--------------------|--------------|-------------------------|--------| | 4 | 3258 → 3348 | +2.8% | 10.22 → 10.18 | −0.4% | | 8 | 5140 → 5174 | +0.7% | 13.25 → 13.15 | −0.8% | | 16 | 7798 → 7804 | +0.1% | 17.62 → 17.58 | −0.2% | | 32 | 10184 → 10198 | +0.1% | 27.06 → 27.01 | −0.2% | | 64 | 12878 → 12884 | +0.05% | 43.68 → 43.64 | −0.1% | **ISL/OSL = 1024/1024** (decode-bound) — mean of 3 warm sweeps per arm, with a flat `dense_off` control measured in the same window (control spread ≤0.6%, confirming an uncontended measurement): | concurrency | total tok/s OFF→ON | Δ throughput | median TPOT OFF→ON (ms) | Δ TPOT | |-------------|--------------------|--------------|-------------------------|--------| | 4 | 819 → 819 | 0.0% | 9.43 → 9.42 | −0.1% | | 8 | 1331 → 1341 | +0.8% | 11.68 → 11.58 | −0.9% | | 16 | 2203 → 2199 | −0.2% | 14.30 → 14.32 | +0.1% | | 32 | 3089 → 3091 | +0.1% | 20.28 → 20.34 | +0.3% | | 64 | 4352 → 4323 | −0.7% | 29.08 → 29.32 | +0.8% | On the plain MXFP4 checkpoint the dense-FP8 path is **at parity (no regression)** on this host — both throughput and TPOT are unchanged within run-to-run noise across all concurrencies, verified at **8k/1k** and **1k/1k** (the latter across 3 warm sweeps against a flat control). The PTPC+fusion config is therefore **safe to enable uniformly on both checkpoints**: a clear prefill-bound win on MXFP4-AttnFP8, and neutral on plain MXFP4. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31690508304](https://github.com/sgl-project/sglang/actions/runs/31690508304) Latest PR Test (Extra): :x: [Run #31690508005](https://github.com/sgl-project/sglang/actions/runs/31690508005) ",
"url": "https://github.com/sgl-project/sglang/pull/28932",
"createdAt": "2026-06-22T12:01:06Z",
"updatedAt": "2026-08-13T17:41:03Z",
"timestamp": "2026-08-13T17:41:03Z",
"metrics": {
"reactions": 0,
"comments": 15
},
"labels": [
"amd",
"run-ci",
"bypass-fastfail"
],
"author": "mqhc2020",
"state": "open",
"assignees": [
"yctseng0211"
]
},
{
"id": "github:sgl-project/sglang:pull_request:28993",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] optimize MLA KV buffer gather",
"text": "## Summary - Add a block-split gather path for small Kimi K2.5 MLA batches and a tiled per-loc gather path for larger batches. - Tune the get-buffer dispatch for BF16 Kimi K2.5 rows (`nope=512`, `rope=64`) with PDL/GDC support where available. - Add registered correctness tests covering set/get behavior and the new dispatch boundary. ## Benchmark Local benchmark data collected with Kimi K2.5 MLA rows and longer KV gather chunks (`wrapper` is the new dispatcher): | n_loc | wrapper (us) | old_one_cta (us) | speedup | |---:|---:|---:|---:| | 4096 | 2.028 | 3.265 | 1.61x | | 16384 | 4.611 | 9.646 | 2.09x | | 32768 | 7.688 | 19.041 | 2.48x | | 65536 | 24.789 | 48.261 | 1.95x | | 73728 | 28.316 | 54.061 | 1.91x | | 74240 | 28.495 | 54.314 | 1.91x | | 81920 | 31.377 | 59.900 | 1.91x | ## Test plan - `python3 -m pytest test/registered/jit/test_mla_kv_buffer_triton.py -q` - Pre-commit hooks on commit ## Profile Before After 0.6% faster prefill time for chunked prefix workload `export SGLANG_MAX_KV_CHUNK_CAPACITY=32768` --- ### CI States Latest PR Test (Base): :x: [Run #28060368762](https://github.com/sgl-project/sglang/actions/runs/28060368762) Latest PR Test (Extra): :x: [Run #28060368595](https://github.com/sgl-project/sglang/actions/runs/28060368595) ",
"url": "https://github.com/sgl-project/sglang/pull/28993",
"createdAt": "2026-06-23T04:34:22Z",
"updatedAt": "2026-08-13T05:18:25Z",
"timestamp": "2026-08-13T05:18:25Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "kpham-sgl",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29090",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] optimize mla_kv_pack_quantize_fp8 flat kernel and dispatch",
"text": "## Motivation `mla_kv_pack_quantize_fp8` is a purely memory-bound op (fused `cat(k_nope, broadcast k_pe)` + FP8 quantize for K, FP8 quantize for V). The `_v1_flat_kernel` path spent a meaningful fraction of its time on per-element address arithmetic (`// num_heads`, `% num_heads`, two-term strided offsets) rather than HBM traffic, and the `_pick_kernel` heuristic was tuned on GB300 and keyed only on `s`, so it mis-dispatched at higher head counts. This PR speeds up the common packed-tensor case and re-tunes dispatch for B200 (sm100). ## Modifications - **`_v1_flat_kernel` contiguous fast path:** add a `CONTIGUOUS: tl.constexpr` branch that addresses each `(token, head)` pair as a single flat row (`pair_idx`-based offsets), dropping the per-element `% num_heads` and the two-term strided address math. This is what bounds throughput once the kernel is occupancy-limited. The strided path is kept as a fallback for non-contiguous views, and the fast path is selected only when `k_nope`/`v`/`k_out`/`v_out` are all dense. Output is bit-identical to the strided path. - **`_pick_kernel` retune (B200), keyed on total work `s * num_heads`:** - The flat kernel wins for all but tiny `s`; `v0`'s 2D `(s, head)` grid is kept only for tiny `s`, where it trims launch overhead. - For the flat path, `num_warps` tracks total work: **4 warps** through the mid range (moderate / L2-resident prefix chunks) and **8 warps** only once the launch is large enough to saturate HBM (`>= 200k` pairs). - `num_stages = 3` (strictly `>=` `num_stages=2` across sizes: better mid-range, identical at large). - **Benchmark:** `bench_mla_kv_pack_quantize_fp8.py` now sweeps `num_heads` in addition to `batch_size` (dispatch depends on both axes), refactored to share input/runner helpers. ## Accuracy Tests The contiguous fast path is bit-identical to the existing strided path. The registered correctness test `test/registered/jit/test_mla_kv_pack_quantize_fp8.py` passes (130 cases) across the `(s, num_heads)` grid. ## Speed Tests and Profiling Measured on NVIDIA B200 (DSv3 dims, BF16 -> FP8 e4m3), us/layer. ### vs naive Triton baseline (`num_heads=32`, contiguous) batch_size sweep: | batch_size | hybrid (us) | naive (us) | | ---: | ---: | ---: | | 64 | 1.12 | 2.98 | | 256 | 1.53 | 9.31 | | 1024 | 5.53 | 10.45 | | 4096 | 17.00 | 18.45 | | 16384 | 62.85 | 72.06 | ### Real serving shapes (TP8 DSv3, `num_heads=16`, strided kv-slice inputs) Validated against a TP8 prefill profile where `pack_prefix_chunk_kv` feeds `k_nope = kv[..., :128]` / `v = kv[..., 128:]` (non-contiguous slices, so the kernel takes the strided path). Comparison vs the tokenspeed 2D-grid kernel (`num_warps=4, num_stages=3`): | s | sglang (this PR) | sglang prior config | tokenspeed v0 | PR vs tokenspeed | | ---: | ---: | ---: | ---: | ---: | | 4800 | 6.19 us | 6.91 | 6.18 | 1.00x | | 8192 | 14.95 us | 16.20 | 15.39 | 1.03x | | 16384 (dominant) | 32.93 us | 32.96 | 36.22 | 1.10x | | 32768 | 63.5 us | - | 75.1 | 1.18x | The dispatch retune recovers ~8-11% at the moderate prefix-chunk sizes (s~4800-8192) that the prior `num_warps=8/num_stages=2` config gave up, while leaving the dominant `s=16384` case (already optimal, ~88% of pack time in the profile) untouched. It is a strict improvement over the previous sglang config at every measured size. Note: the pack kernel is a small fraction of end-to-end GPU time (~1.4% in the profile) and at the largest chunks is HBM-bandwidth-bound (in-trace ~5.6 TB/s under contention vs ~7.7 TB/s isolated), so this is a kernel-level cleanup/win rather than a large end-to-end mover. ## Checklist - [x] Format your code according to pre-commit. - [x] Add unit tests / verify existing correctness tests. - [x] Provide accuracy and speed benchmark results. - [x] Follow the SGLang code style guidance.",
"url": "https://github.com/sgl-project/sglang/pull/29090",
"createdAt": "2026-06-23T23:31:00Z",
"updatedAt": "2026-08-13T05:18:28Z",
"timestamp": "2026-08-13T05:18:28Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"quant",
"jit-kernel"
],
"author": "kpham-sgl",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29143",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add intel_xpu to DETERMINISTIC_ATTENTION_BACKEND_CHOICES",
"text": "Add intel_xpu to DETERMINISTIC_ATTENTION_BACKEND_CHOICES --- ### CI States Latest PR Test (Base): :x: [Run #31669990908](https://github.com/sgl-project/sglang/actions/runs/31669990908) Latest PR Test (Extra): :x: [Run #31669990721](https://github.com/sgl-project/sglang/actions/runs/31669990721) ",
"url": "https://github.com/sgl-project/sglang/pull/29143",
"createdAt": "2026-06-24T08:26:09Z",
"updatedAt": "2026-08-13T06:22:02Z",
"timestamp": "2026-08-13T06:22:02Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"intel",
"xpu",
"deterministic"
],
"author": "kalyank007",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29190",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "For SM120, the MoE NVFP4 kernel of B12X is added to SGLang",
"text": " ## Motivation When running Qwen3.5-397B NVFP4 (https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4) on SM120, we found that the MoE component accounts for an extremely large proportion of runtime, nearly 50%, which requires optimization. The test data for this PR is as follows: ## Modifications Directly enable the MoE NVFP4 kernel of b12x (https://github.com/lukealonso/b12x/tree/master) within the MoE backend of SGLang. Currently, this PR uses flashinfer_cutedsl as the backend name, yet it actually links against b12x directly. A new commit will replace the calling interfaces here with the corresponding APIs from FlashInfer. Follow-up: make sure the FlashInfer B12X W4A16 path (NVFP4 weights with bf16 activations) works in SGLang too. See #33710. ## Accuracy Tests | | 1 | 2 -- | -- | -- | -- 397B-MoE-b12x | nvfp4 | 0.884 | 0.8838 397B-MoE-triton | nvfp4 | 0.899 | 0.874 397B-MoE-triton | FP8 | 0.889 | 0.884 ## Speed Tests and Profiling
MoE Backend | EP | ISL | OSL | Concurrency | TTFT (ms) | TPOT (ms) | QPS | Output tok/s/user | Total tok/s | QPS/GPU | Output tok/s/GPU | Total tok/s/GPU -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- triton | 1 | 4096 | 1024 | 4 | 4443.95 | 22.65 | 0.14 | 37.07 | 741.45 | 0.0181 | 18.54 | 92.68 flashinfer_cutlass | 1 | 4096 | 1024 | 4 | 4422.22 | 19.36 | 0.17 | 42.26 | 845.15 | 0.0206 | 21.13 | 105.64 flashinfer_cutedsl_sm120 (b12x) | 1 | 4096 | 1024 | 4 | 4282.32 | 18.18 | 0.17 | 44.74 | 894.79 | 0.0218 | 22.37 | 111.85 ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31514390746](https://github.com/sgl-project/sglang/actions/runs/31514390746) Latest PR Test (Extra): :x: [Run #31514389772](https://github.com/sgl-project/sglang/actions/runs/31514389772) ",
"url": "https://github.com/sgl-project/sglang/pull/29190",
"createdAt": "2026-06-24T16:22:26Z",
"updatedAt": "2026-08-13T07:26:42Z",
"timestamp": "2026-08-13T07:26:42Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"quant"
],
"author": "hsr1234563",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:29202",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Enable draft-extend CUDA graph and reduce bubble for MTP",
"text": " ## Motivation Three independent fixes for DeepSeek-V4 MTP (EAGLE speculative decoding) on the HIP/ROCm path: 1. The `draft_extend` phase was falling back to eager because the HIP draft-extend CUDA-graph gate did not recognize the DSV4 backend, leaving a graph-able phase uncaptured. 2. Running the server with `--profile` after making change to `_attach_unified_kv_prefill_meta `reliably crashed during garbage collection, which blocked profiling/validation of the MTP path entirely. 3. A `repeat_interleave` in the unified-kv prefill metadata build forced an implicit device→host sync that stalled the whole `draft_extend` metadata construction. ## Modifications 1. **Enable draft-extend CUDA graph for the DSV4 HIP backend** (`eagle_worker_v2.py`): the graph gate now also accepts `DeepseekV4HipRadixBackend`, so `draft_extend` runs under CUDA graph instead of eager. 2. **Force GC after profiler stop** (`profiler_manager.py`): the `torch.profiler` pybind11 objects form a reference cycle whose C++ destructor double-frees when reclaimed late by cyclic GC. Drop the reference and `gc.collect()` at the controlled stop point. 3. **Pass `output_size` to `repeat_interleave`** (`deepseek_v4_backend_hip_radix.py`): avoids the implicit device→host sync from reading `extend_seq_lens.sum()` back to host; the caller already knows this value. ## Accuracy Tests gsm8k acc: 0.95 ## Speed Tests and Profiling 5.7%-11.7% improvement in TTT, 9.7%-13.5% improvement in ITL Before: After: ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #29729414329](https://github.com/sgl-project/sglang/actions/runs/29729414329) Latest PR Test (Extra): :x: [Run #29729414083](https://github.com/sgl-project/sglang/actions/runs/29729414083) ",
"url": "https://github.com/sgl-project/sglang/pull/29202",
"createdAt": "2026-06-24T19:36:59Z",
"updatedAt": "2026-08-13T07:50:20Z",
"timestamp": "2026-08-13T07:50:20Z",
"metrics": {
"reactions": 0,
"comments": 17
},
"labels": [
"amd",
"deepseek",
"run-ci",
"bypass-fastfail"
],
"author": "RolaoDenthu",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29328",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs",
"text": "## Summary Adds support in SGLang for loading and inference for ModelOpt (and Quark) NVFP4 checkpoints by dequantizing the supplied NVFP4 weights and requantizing them to MXFP4 at load time to allow efficient inference using supported AMD hardware (e.g. AMD MI355x). Enabled via `--quantization quark_mxfp4`. Weights dequantized then requantized to MXFP4 during loading, so the resulting model occupies MXFP4 memory without ever holding a full-precision copy. Also includes the config-routing and shared-expert-fusion fixes needed to load ModelOpt exports across various architectures using the existing SGLang AMD Quark integration. ## Modifications ### Online Requantization Path - **`layers/quantization/quark/quark.py`**: `_detect_nvfp4_source` detects ModelOpt and AMD Quark NVFP4 layouts; `_parse_nvfp4_excludes` extracts producer exclude lists; `from_config` routing gated on `requantization_method`. `_maybe_disable_shared_experts_fusion` delegates to the existing `can_fuse_shared_expert()` to correctly handle `shared_expert_gate` and `mtp.` exclusions. - **`layers/quantization/quark/schemes/quark_w4a4_mxfp4.py`**: NVFP4 linear scheme: `_create_weights_from_nvfp4` allocates NVFP4 staging buffers directly on device, loads all source shards, then dequant-requant to MXFP4 once all bytes are present. Registers (and discards) the NVFP4 per-tensor `input_scale` slot since MXFP4 uses dynamic activation quantization. - **`layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py`**: NVFP4 fused-MoE scheme: per-expert weight/scale registration, online expert-wise dequant-requant, no-op `input_scale` slots for w13/w2. - **`layers/quantization/dequantization.py`**, **`layers/quantization/online_quantization.py`**, **`layers/quantization/quark/utils.py`**: `Nvfp4SourceConfig` + dequant helpers and `CopyNumelCounter` to track progressive shard loading ### Fixes required to load ModelOpt exports - **Config-loading** (`model_loader/weight_utils.py`): Some ModelOpt NVFP4 checkpoints store quant metadata only in `hf_quant_config.json`; others duplicate it in `config.json`. `get_quant_config` now reads `hf_quant_config.json` preferentially when present and FP4-typed, routing through `from_config`. - **Shared-expert-fusion** (`layers/quantization/quark/quark.py`): Some NVFP4 checkpoints keep shared experts in BF16 while routed experts are FP4. Shared-expert fusion remaps `mlp.shared_experts` into the quantized `FusedMoE`, causing a shape mismatch. `QuarkConfig.__init__` now calls `_maybe_disable_shared_experts_fusion` which delegates to `can_fuse_shared_expert()` (which is the same detection path introduced in #22948 and scoped to Quark in #27057) and sets `disable_shared_experts_fusion` globally before model construction. ### Tests - **`test/registered/quant/test_quark_mxfp4.py`**: Added `TestNVFP4ToMXFP4MOETP1` testing NVFP4→MXFP4 MoE requantization accuracy on `nvidia/Qwen3-30B-A3B-NVFP4`. ### Docs - **`docs_new/docs/advanced_features/quantization.mdx`**: documents the NVFP4-MXFP4 requant path (supported checkpoint layouts, and example command. ## Accuracy Tests All numbers below are obtained using lm_eval using SGLang backend with `--quantization quark_mxfp4` on AMD MI355X GPUs. (`max_gen_toks`=32768) ### GSM8k (lm_eval, flexible-extract) | Model | Original Release (bf16/fp8) | NVFP4-to-MXFP4 | % Recovery | |---|---|---|---| | MiniMax-M2.7-NVFP4 | 0.918 | 0.920 | 100.2% | | GLM-5.1-NVFP4 | 0.952 | 0.928 | 97.5% | | Kimi-K2.6-NVFP4 | 0.939 | 0.930 | 99.0% | | Qwen3.5-397B-A17B-NVFP4 | 0.954 | 0.945 | 99.1% | | Qwen3.5-397B-A17B-NVFP4-V2 | 0.954 | 0.941 | 98.7% | | DeepSeek-R1-NVFP4 | 0.958 | 0.950 | 99.2% | --- ### CI States Latest PR Test (Base): :x: [Run #31619623512](https://github.com/sgl-project/sglang/actions/runs/31619623512) Latest PR Test (Extra): :x: [Run #31619623176](https://github.com/sgl-project/sglang/actions/runs/31619623176) ",
"url": "https://github.com/sgl-project/sglang/pull/29328",
"createdAt": "2026-06-25T20:10:55Z",
"updatedAt": "2026-08-13T17:57:19Z",
"timestamp": "2026-08-13T17:57:19Z",
"metrics": {
"reactions": 1,
"comments": 22
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"lora",
"Multi-modal",
"deepseek",
"speculative-decoding",
"hicache",
"blackwell",
"npu",
"run-ci",
"diffusion",
"mthreads",
"apple-silicon",
"jit-kernel",
"bypass-fastfail"
],
"author": "ColinZ22",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:29593",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[CPU][QUANT] add amx cpu support for auto-round",
"text": " ## Motivation Currently supports offline inference for CUDA devices with auto-round quantized models, but CPU inference is not yet supported. This is a blocker for users who want to run INT4 quantized models on CPU-only machines or in CPU-based serving pipelines using SGLang as the backend. issue description: https://github.com/sgl-project/sglang/issues/27568 ## Modifications add CPU auto-round format quantized model loading support. bugfix of FusedMoE layer loading with marlin backend, pass backend args to apply_quant_layer refactor auto_round into package layout. ## Accuracy Tests Evaluated `tmp/Qwen3-4B-int4-AutoRound` (auto-round INT4, W4G128, sym) served on CPU via SGLang using `lm-eval` (0-shot): | Tasks | Metric | Value | Stderr | |----------------|------------|-------:|-------:| | lambada_openai | acc | 0.5909 | 0.0068 | | lambada_openai | perplexity | 7.7950 | 0.3152 | | piqa | acc | 0.7410 | 0.0102 | | piqa | acc_norm | 0.7443 | 0.0102 | original BF16 reference: | Tasks | Metric | BF16 | INT4 AutoRound | |----------------|------------|-------:|---------------:| | lambada_openai | acc | 0.6024 | 0.5909 | | lambada_openai | perplexity | 7.3054 | 7.7950 | | piqa | acc | 0.7497 | 0.7410 | | piqa | acc_norm | 0.7497 | 0.7443 | moe model generate and eval : ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31575342235](https://github.com/sgl-project/sglang/actions/runs/31575342235) Latest PR Test (Extra): :x: [Run #31575342321](https://github.com/sgl-project/sglang/actions/runs/31575342321) ",
"url": "https://github.com/sgl-project/sglang/pull/29593",
"createdAt": "2026-06-29T01:37:30Z",
"updatedAt": "2026-08-13T07:51:00Z",
"timestamp": "2026-08-13T07:51:00Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"documentation",
"quant",
"intel",
"cpu",
"run-ci",
"run-ci-extra"
],
"author": "WeiweiZhang1",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29668",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] fix: resolve Mooncake local_hostname per node for runtime attach",
"text": " ## Motivation When Mooncake HiCache storage is attached at runtime via `PUT /hicache/storage-backend`, `extra_config` is broadcast to all ranks. `load_from_extra_config()` previously fell back to the fixed default `\"localhost\"` when `local_hostname` was omitted, and a head-node `local_hostname` in the attach payload was applied to every rank. This breaks multi-node Mooncake deployments. Fixes sgl-project/sglang#23457 ## Modifications - Add `MooncakeStoreConfig._resolve_local_hostname()` so `MOONCAKE_LOCAL_HOSTNAME` / `LOCAL_HOSTNAME` from the current process take precedence over shared `extra_config`. - Reuse the helper in `from_file()`, `load_from_env()`, and `load_from_extra_config()`. - Document multi-node runtime attach guidance and JSON config precedence in the Mooncake store README. - Add unit tests covering env precedence, broadcast override behavior, and `from_file` env-over-file resolution. - Register tests for CI with `register_cpu_ci` and `CustomTestCase`. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31722158250](https://github.com/sgl-project/sglang/actions/runs/31722158250) Latest PR Test (Extra): :x: [Run #31722157962](https://github.com/sgl-project/sglang/actions/runs/31722157962) ",
"url": "https://github.com/sgl-project/sglang/pull/29668",
"createdAt": "2026-06-29T17:37:06Z",
"updatedAt": "2026-08-13T16:56:12Z",
"timestamp": "2026-08-13T16:56:12Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"run-ci"
],
"author": "stmatengss",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29723",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Add fused all-reduce RMSNorm per-token FP8/MXFP4 quant",
"text": "## Motivation The existing fused AllReduce → RMSNorm → quant path only supports per-group FP8 scales. Checkpoints that use per-token FP8 activation scaling (and MXFP4) therefore fall back to the unfused AllReduce → RMSNorm → quantize sequence — three kernel launches plus intermediate bf16 round-trips — before every tensor-parallel linear. ## Modifications Adds a per-token FP8 variant (and a 1-stage MXFP4 variant) of the fused kernel, collapsing those three launches into a single aiter kernel. The change is additive and gated: when the per-token fast path is not applicable it falls back to the existing per-group / plain AR+RMSNorm path with no behavior change. ### Key changes **Collective entry points** (`distributed/parallel_state.py`, `distributed/communication_op.py`) - `tensor_model_parallel_fused_allreduce_rmsnorm_quant_per_token` returns `(fp8_output, residual_out, per_token_scale)` with `per_token_scale` shaped `(M, 1)`, or `None` when the backend cannot fuse (fallback signal). - `tensor_model_parallel_fused_allreduce_rmsnorm_mxfp4_quant` plus the `_should_use_1stage_mxfp4_ar` heuristic for the MXFP4 1-stage path. - Device-communicator methods `fused_allreduce_rmsnorm_quant_per_token` / `fused_allreduce_rmsnorm_mxfp4_quant`, calling aiter `custom_fused_ar_rms_quant(post_per_token_quant=True)`. **LayerNorm hook** (`layers/layernorm.py`) - `forward_with_allreduce_fusion_quant_per_token` (plus the shared `_forward_with_allreduce_fusion_quant_per_token` impl), with a cached `_aiter_per_token_quant` functor and a non-aiter / `residual is None` fallback. **Communicator gating** (`layers/communicator.py`) - `_try_fused_allreduce_rmsnorm_quant` dispatches to the per-token path when `quant_format == \"fp8_per_token\"` and the LayerNorm exposes the per-token hook; otherwise it falls through to per-group / unfused. **Quant fast path** (`layers/quantization/fp8_utils.py`) - Consume pre-quantized `(fp8, per_token_scale)` inputs directly, skipping the linear's internal quantization. **Qwen3.5 wiring** (`models/qwen3_5.py`) - `_detect_fused_ar_quant_format` returns `\"fp8_per_token\"` under `SGLANG_USE_AITER_FP8_PER_TOKEN`; consumes the fused 2-tuple/3-tuple handoff in both the GDN linear-attention (`in_proj_qkvz` / `in_proj_ba`) path and the full-attention (`qkv_proj`) path. ### Enablement `SGLANG_USE_AITER=1` + `--enable-aiter-allreduce-fusion` (existing), plus `SGLANG_USE_AITER_FP8_PER_TOKEN=1` to select the per-token format. MXFP4 checkpoints auto-select the MXFP4 variant. ROCm/aiter/gfx95-gated; other backends and non-eligible configs are unchanged. ## Accuracy Tests The fused kernel is numerically equivalent to the reference AR → RMSNorm → per-token quant sequence; when the fast path is unavailable it defers to the existing per-group / plain path. Verified on Qwen3.5-397B MXFP4-AttnFP8 with GSM8K (1319q): fusion ON = 0.932, and toggling the per-token FP8 fusion off (`SGLANG_DISABLE_FUSED_AR_QUANT`) leaves accuracy unchanged at 0.932, confirming the fallback is numerically faithful. Unit tests: `test/registered/ops/test_aiter_allreduce_fusion_amd.py` adds per-token FP8 and MXFP4 cases (fused vs unfused reference) and the `None`-fallback contract. ## Speed Tests and Profiling Kernel benchmark: `benchmark/kernels/all_reduce/benchmark_fused_ar_rms_quant_amd.py` adds per-token FP8 and MXFP4 coverage against the 3-launch baseline. ### End-to-end Setup: Qwen3.5-397B MXFP4-AttnFP8, TP2 on MI355X, fp8 KV cache, `--attention-backend aiter`, random dataset, output len 1024, range ratio 0.8, num-prompts = 10 x cc. (1k,1k) uses `--enable-mixed-chunk`; (8k,1k) does not. The arms differ only by `SGLANG_USE_AITER_FP8_PER_TOKEN` (OFF = per-token fusion disabled → per-group/plain fallback; ON = per-token fused path). Accuracy is identical between arms (GSM8K 1319q = 0.932 both, see Accuracy Tests), so this is a like-for-like numerical comparison. Total token throughput (tok/s), higher is better: | cc | (1k,1k) OFF | (1k,1k) ON | Δ | (8k,1k) OFF | (8k,1k) ON | Δ | |---:|---:|---:|---:|---:|---:|---:| | 4 | 804.5 | 805.4 | +0.11% | 3265.4 | 3299.7 | +1.05% | | 8 | 1401.0 | 1411.1 | +0.72% | 5357.8 | 5413.8 | +1.05% | | 16 | 2129.0 | 2182.1 | +2.49% | 7729.3 | 7749.9 | +0.27% | | 32 | 3097.5 | 3168.6 | +2.29% | 10341.4 | 10327.2 | -0.14% | | 64 | 4578.9 | 4590.5 | +0.26% | 13255.2 | 13269.0 | +0.10% | Mean TPOT (ms/token), lower is better: | cc | (1k,1k) OFF | (1k,1k) ON | Δ | (8k,1k) OFF | (8k,1k) ON | Δ | |---:|---:|---:|---:|---:|---:|---:| | 4 | 9.57 | 9.56 | -0.10% | 10.38 | 10.27 | -1.06% | | 8 | 10.95 | 10.92 | -0.27% | 12.58 | 12.45 | -1.03% | | 16 | 14.63 | 14.28 | -2.39% | 17.58 | 17.54 | -0.23% | | 32 | 19.99 | 19.53 | -2.30% | 26.22 | 26.23 | +0.04% | | 64 | 27.05 | 26.98 | -0.26% | 41.44 | 41.36 | -0.19% | The fused path is faster or equal in 9 of 10 cells, and TPOT moves in lockstep with throughput, which is the signature of removing launches from a decode-bound step rather than of noise. The largest gains are ~2.4% at (1k,1k) cc=16/32; the rest is neutral. That is the expected magnitude: the fused region is a small share of a decode step, so collapsing three launches into one removes real work without dominating end-to-end serving throughput. Caveat on provenance: these are single samples per cell, and the ON and OFF arms were collected in separate sessions rather than interleaved, so deltas below ~1% should be read as neutral. ### Applicability The saving is largest when the consumer of the fused output accepts the quantized tensor directly. If the downstream linear is not quantized in a matching format, the kernel must still produce a bf16 side output, and the benefit reduces to the removed launches alone. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 5. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 6. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 7. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31709151314](https://github.com/sgl-project/sglang/actions/runs/31709151314) Latest PR Test (Extra): :x: [Run #31709151159](https://github.com/sgl-project/sglang/actions/runs/31709151159) ",
"url": "https://github.com/sgl-project/sglang/pull/29723",
"createdAt": "2026-06-30T07:34:19Z",
"updatedAt": "2026-08-13T15:17:28Z",
"timestamp": "2026-08-13T15:17:28Z",
"metrics": {
"reactions": 2,
"comments": 15
},
"labels": [
"high priority",
"quant",
"amd",
"run-ci"
],
"author": "mqhc2020",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29858",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Build SWA window kv buffers for the EAGLE draft-extend cuda-graph path",
"text": "## Motivation Intermittent GPU memory access fault during EAGLE multi-layer speculative decode on hybrid sliding-window models (a mix of full-attention and SWA layers, e.g. MiMo-V2.5-Pro). It only shows up with cuda graph enabled and only when the SWA kv pool is smaller than the full pool. With `--disable-cuda-graph` the model runs clean, which is the tell that this is a cuda-graph-only metadata bug and not a kernel bug. Root cause: on hybrid-SWA models the triton backend keeps the SWA layers on a smaller swa kv pool and builds a separate set of window kv buffers (`window_kv_indptr` / `window_kv_indices` and the window offsets) that index that pool. Three of the four buffer-building paths do this: decode cuda graph (`_update_decode_kv_buffers`), target_verify cuda graph (`_update_target_verify_buffers`), and every eager path (`init_forward_metadata` -> `update_sliding_window_buffer`). The EAGLE draft-extend cuda-graph path is the odd one out: `_update_draft_extend_buffers` fills only the full kv buffers, and the `is_draft_extend_v2` branch of `_build_cuda_graph_forward_metadata` hard-codes `window_kv_indices=None` (the decode/verify branches use `... if swa else None`). So on a hybrid-SWA model the draft step runs the SWA layers with no window indices. `forward_extend` takes the sliding-window branch (the layer has a window size), reads `window_kv_indices` (None), and the captured extend kernel indexes the small swa K/V buffer with full-pool locations, an out-of-bounds read. Under rocgdb the fault lands in the extend kernel (`_fwd_kernel`) on the SWA K-buffer load, and the faulting virtual addresses are foreign to every live `kv_indices` buffer, which is the signature of an OOB read off a too-small buffer rather than a stale/freed pointer. ## Modifications Build the swa window buffers for draft-extend the same way target_verify already does: 1. `_update_draft_extend_buffers`: after filling the full kv buffers, call `update_sliding_window_buffer` to fill the persistent swa window buffers (`cuda_graph_window_kv_indices` / offsets). The window covers the prefix portion (`kv_lens`), matching how draft-extend builds its full `kv_indices`. 2. `is_draft_extend_v2` branch of `_build_cuda_graph_forward_metadata`: point `window_kv_indptr` / `window_kv_indices` / `window_num_kv_splits` / `window_kv_offsets` at those buffers when `swa` is set, mirroring the verify branch. Both capture and replay route through `_apply_cuda_graph_metadata` -> `_update_draft_extend_buffers`, so the persistent swa buffers are refilled on every replay and the ForwardMetadata view (built once at capture) keeps pointing at the same address-stable buffers. The new code is gated on `sliding_window_size > 0` and the metadata fields stay None otherwise, so it is a no-op for non-SWA models. Eager is already correct (its extend path calls `update_sliding_window_buffer`), so this change is intentionally cuda-graph-only. ## Accuracy Tests MiMo-V2.5-Pro on MI300X-class hardware, TP8, default hybrid window sizing, cuda graph on. Before the fix the config faults intermittently (~1 in 3 runs, across all ranks). After the fix, full GSM8K runs clean across 6 consecutive runs, strict-match ~0.963-0.967, in line with the baseline. ## Speed Tests and Profiling Cuda-graph throughput is preserved (~195 s/pass on the validation config), so there is no need to fall back to `--disable-cuda-graph` (~4x slower on decode) or to the `--swa-full-tokens-ratio 1.0` workaround (which costs ~17% KV headroom). The change is a metadata build on the draft-extend path with no kernel changes, so no per-kernel profiling delta is expected. Notes for reviewers: - The same defect exists on the `v0.5.12.post1` release tag but in the pre-refactor shape (separate `init_forward_metadata_capture/replay_cuda_graph` + `update_sliding_window_buffer_cuda_graph`), so a backport to that release needs the older-shape version of this fix, not this diff. - Deployments hitting this have been carrying a small guard in `forward_extend` that falls back to the full kv buffers when `window_kv_indices` is None. That guard turns the NoneType failure into the OOB read described above, so it is a symptom mask, not a fix, and is intentionally not included here since this PR makes `window_kv_indices` non-None on the draft path. - Follow-up worth doing separately: a small assertion that SWA forward modes always have a non-None `window_kv_indices` under cuda graph would catch this class of missing-window-build bug early. Also worth a confirmation run on gfx950 (validated on gfx942-class so far). --- ### CI States Latest PR Test (Base): :x: [Run #31189512886](https://github.com/sgl-project/sglang/actions/runs/31189512886) Latest PR Test (Extra): :x: [Run #31189511878](https://github.com/sgl-project/sglang/actions/runs/31189511878) ",
"url": "https://github.com/sgl-project/sglang/pull/29858",
"createdAt": "2026-07-01T16:53:51Z",
"updatedAt": "2026-08-13T16:01:39Z",
"timestamp": "2026-08-13T16:01:39Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [],
"author": "reger-men",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29935",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature][Intel XPU] Add memory saver support for Intel XPU via upstream torch_memory_saver",
"text": "## Motivation Enable release/resume_memory_occupation on Intel XPU using the upstream torch_memory_saver package (Level Zero VMM backend). Replaces in-repo implementation with pip-installable upstream dependency. Features: - Pause/resume physical GPU memory while preserving virtual addresses - Tag-based selective memory management (kv_cache, weights, cuda_graph) - Optional CPU backup for content preservation - Multi-device support (verified via the package's multi-device test) - Sysman-based memory verification (torch accounting doesn't reflect unmapped pages) ## Modifications Changes: - Add TorchMemorySaverAdapter for unified memory saver API - XPU uses hook_mode='torch' (in-process pluggable allocator) - configure_subprocess() and cuda_graph() are no-ops on XPU - Update Dockerfile to build the upstream package for XPU - Add test suite (10 tests, all passing) Installation (XPU builds from source; TMS_PLATFORM=xpu forces the XPU backend and --no-build-isolation lets the build match libsycl to the installed torch): source /opt/intel/oneapi/setvars.sh TMS_PLATFORM=xpu pip install --no-build-isolation \\ git+https://github.com/fzyzcjy/torch_memory_saver.git Usage: python -m sglang.launch_server --model-path --device xpu --enable-memory-saver Depends on: https://github.com/fzyzcjy/torch_memory_saver/pull/78 ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy] (https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). **N/A** - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. cc @mingfeima @fzyzcjy @habaohaba @yueming-yuan @jianan-gu @hnyls2002 @merrymercy --- ### CI States Latest PR Test (Base): :x: [Run #31668261578](https://github.com/sgl-project/sglang/actions/runs/31668261578) Latest PR Test (Extra): :x: [Run #31668261250](https://github.com/sgl-project/sglang/actions/runs/31668261250) ",
"url": "https://github.com/sgl-project/sglang/pull/29935",
"createdAt": "2026-07-02T11:23:20Z",
"updatedAt": "2026-08-13T06:21:30Z",
"timestamp": "2026-08-13T06:21:30Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"documentation",
"run-ci"
],
"author": "siju-samuel",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30051",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DO NOT MERGE][AMD] Bump Mooncake pin to include cross-node RDMA multi-protocol fix",
"text": "> **Status (2026-08-12): blocked on ROCm runtime fix, not on this bump.** > > Root-cause is complete. Bumping the Mooncake pin here makes the HIP dma-buf path > actually compile in (the old `01d1eb2a` had `USE_HIP_DMABUF` on the wrong CMake > target, so it silently fell back to PeerDirect). Once dma-buf is live, cross-node > KV transfer hits a **ROCm runtime NULL-deref SIGSEGV** — root-caused and tracked in > **ROCm/rocm-systems#8517**. > > The two runtime fixes already exist on ROCm `develop` (CLR `7d979ab5a0aa`, > ROCR-Runtime `5400f1e6418b`) but are **absent from all released branches** > (7.0.0, 7.2.0-7.2.4). We have asked AMD to backport them into a 7.0.x / 7.2.x point release. > > **DO NOT MERGE until** the ROCm backport lands (or a patched runtime is available). > Once ROCm ships the fix, this bump unblocks cross-node Mooncake dma-buf zero-copy end-to-end. --- ## Motivation The Mooncake commit currently pinned in `rocm.Dockerfile` (`01d1eb2a`) crashes with a SIGSEGV during cross-node KV transfer on AMD when a GPU buffer is registered under both the `hip` and `rdma` protocols (the standard SGLang PD-disaggregation setup on AMD MI3xx over RoCE). rdma-only / single-protocol deployments are unaffected, which is why it went unnoticed. The root cause and fix are described in kvcache-ai/Mooncake#2724 and fixed by kvcache-ai/Mooncake#2725 (merged into Mooncake `main`): 1. `selectDevice()` / `selectDeviceByLocalHca()` were protocol-blind and returned the empty-`lkey` `hip` twin of a dual-registered address -> OOB read / SIGSEGV. 2. `selectTransport()` hardcoded `hip` priority above `rdma`, routing cross-node requests to the intra-node-only `hip` transport. 3. `USE_HIP_DMABUF` never reached `rdma_context.cpp`, so GPU MRs silently fell back to plain `ibv_reg_mr` (EINVAL on device memory). ## Modification Bump `MOONCAKE_COMMIT` in `docker/rocm.Dockerfile` from `01d1eb2a` to `45b84d36` (the merged fix on Mooncake `main`). No build-flag changes needed: the image already builds with `USE_HIP=ON`, the CMake fix auto-retargets the dmabuf define, and `MC_DISABLE_HIP` is a runtime env. ## Validation Built from this commit and validated SGLang 1P1D PD disaggregation with Mooncake cross-node KV over RDMA on AMD MI3xx, two independent fabrics (Broadcom bnxt_re + ionic RoCE): - Qwen2.5-0.5B 2x2 matrix (both fabrics x cross-node/single-node): GSM8K accuracies match across topologies -> no KV corruption; 100% request success. - DeepSeek-V4-Flash-FP8 TP8 cross-node: GSM8K-1319 = 0.925, 100% request success. Note: on routed-RoCE (L3) fabrics, deployments must also select a routable GID (e.g. `MC_GID_INDEX=1`); this is deployment config, not part of the image change. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #28666222080](https://github.com/sgl-project/sglang/actions/runs/28666222080) Latest PR Test (Extra): :white_check_mark: [Run #30781760658](https://github.com/sgl-project/sglang/actions/runs/30781760658) ",
"url": "https://github.com/sgl-project/sglang/pull/30051",
"createdAt": "2026-07-03T14:16:40Z",
"updatedAt": "2026-08-12T23:06:04Z",
"timestamp": "2026-08-12T23:06:04Z",
"metrics": {
"reactions": 2,
"comments": 4
},
"labels": [
"amd",
"DO NOT MERGE"
],
"author": "Lzy17",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30318",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] Add mxfp4-w4a8 MOE Quantization Support for NPU",
"text": "## Motivation SGLang already supports W4A8 MXFP quantization for linear layers on the Ascend NPU, but the corresponding MoE path is not yet supported. This PR adds W4A8 MXFP support for ModelSlim-quantized MoE models. It also adapts the implementation to the latest refactored NPU MoE architecture, avoiding duplicated linear-layer support that is already available in the main branch. ## Modifications - Add ModelSlimMXFP4W4A8MoE to load ModelSlim W4A8 MXFP MoE checkpoints. -- Load packed FP4 expert weights. -- Load UE8M0 block scales with a block size of 32. -- Support separate w13 and w2 expert weight layouts. - Add NPUMXFP4W4A8MoEMethod to the refactored NPU MoE implementation. -- Convert packed FP4 weights to the NPU-specific storage format. -- Dynamically quantize MoE activations to MXFP8. -- Execute expert computation through the existing AscendRunner and GroupedMatmul abstractions. -- Reuse the current dispatcher and combine-weight pipeline. - Register the W4A8 MXFP MoE scheme in the ModelSlim quantization configuration. - Make MoE weight offsets optional for quantization schemes that do not require offset tensors. - Keep the change scoped to the MoE path; no duplicate W4A8 MXFP linear implementation is introduced. ## Accuracy Tests Tested with Qwen3-235B-A22B mxfp4 w4a8 weights. ``` 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [11:25<00:00, 1.93it/s] Accuracy: 0.941 Invalid: 0.001 Latency: 685.350 s Output throughput: 273.036 token/s metrics={'accuracy': np.float64(0.9408642911296436), 'invalid': np.float64(0.000758150113722517), 'latency': 685.3499282500707, 'output_throughput': 273.03570378681326} metrics['accuracy']=np.float64(0.9408642911296436) ``` ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31718301247](https://github.com/sgl-project/sglang/actions/runs/31718301247) Latest PR Test (Extra): :x: [Run #31718301063](https://github.com/sgl-project/sglang/actions/runs/31718301063) ",
"url": "https://github.com/sgl-project/sglang/pull/30318",
"createdAt": "2026-07-07T01:40:14Z",
"updatedAt": "2026-08-13T17:04:57Z",
"timestamp": "2026-08-13T17:04:57Z",
"metrics": {
"reactions": 0,
"comments": 11
},
"labels": [
"documentation",
"quant",
"npu",
"run-ci"
],
"author": "LinyuanLi0046",
"state": "open",
"assignees": [
"ping1jing2"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30319",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] Add mxfp4-w4a4 MOE Quantization Support for NPU",
"text": "## Motivation SGLang already supports ModelSlim W4A4_MXFP4 quantization for dense linear layers on Ascend NPU, but the corresponding MoE path is still missing. As a result, MoE models whose expert weights are exported with the W4A4_MXFP4 scheme cannot be loaded and executed through the ModelSlim quantization backend. The Ascend MoE implementation has also been refactored into the modular AscendRunner architecture. Therefore, MXFP4 MoE support needs to integrate with the new per-weight-group quantization methods and runner flow instead of restoring the legacy fused MoE implementation. ## Modifications - Add a ModelSlim W4A4_MXFP4 MoE scheme and register it for MoE quantization detection. - Create w13 and w2 expert weights independently, following the existing ModelSlim MXFP4 checkpoint format:FP4 values stored in float8_e4m3fn containers. - UE8M0 block scales stored as uint8 with a block size of 32. - Add an Ascend MXFP4 MoE quantization method compatible with the refactored AscendRunner and GroupedMatmul interfaces. - Repack offline FP8-container weights into float4_e2m1fn_x2, convert them to the required NPU layout, and reshape block scales for grouped matmul. - Dynamically quantize BF16/FP16 activations to MXFP4 before both expert GMM operations. - Keep the dispatcher output in BF16, since the refactored Ascend dispatcher currently supports BF16 and INT8 outputs, and perform MXFP4 quantization inside the MoE method. - Treat MoE weight offsets as optional so offset-free schemes such as MXFP4 can use the common ModelSlim MoE runner. - Reuse the existing routing, activation, and finalization flow provided by the refactored Ascend MoE backend. - Leave the existing W4A4_MXFP4 linear implementation unchanged. ## Accuracy Tests Tested with Qwen3-235B-A22B mxfp4 w4a4 weights. ``` metrics = run_eval(args) 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [11:09<00:00, 1.97it/s] Accuracy: 0.938 Invalid: 0.000 Latency: 669.312 s Output throughput: 263.397 token/s metrics={'accuracy': np.float64(0.9378316906747536), 'invalid': np.float64(0.0), 'latency': 669.312145670061, 'output_throughput': 263.39728203125276} metrics['accuracy']=np.float64(0.9378316906747536) ``` ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #30600742404](https://github.com/sgl-project/sglang/actions/runs/30600742404) Latest PR Test (Extra): :x: [Run #30600742395](https://github.com/sgl-project/sglang/actions/runs/30600742395) ",
"url": "https://github.com/sgl-project/sglang/pull/30319",
"createdAt": "2026-07-07T01:41:53Z",
"updatedAt": "2026-08-13T11:15:44Z",
"timestamp": "2026-08-13T11:15:44Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"quant",
"npu",
"run-ci"
],
"author": "LinyuanLi0046",
"state": "open",
"assignees": [
"ping1jing2"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30345",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Intel][XPU][LoRA] Enable LoRA on Intel XPU",
"text": "Enable the LoRA functionality on XPU (in addition to CUDA/ROCm), and enable the corresponding unit tests. Source changes: - backends (triton/chunked/torch): use torch.device(self.device) instead of a hard-coded \"cuda\". - lora_moe_runners: route XPU to the pure-torch _naive_moe_lora_align_block_size fallback. - rotary_embedding base.py / mrope.py: guard the XPU-only sgl_kernel imports (fused_qk_rope_with_cos_sin_cache_inplace, multimodal_rotary_embedding). - lora_overlap_loader: use self.device_module.current_stream() instead of torch.cuda.current_stream(). ### Supported and Verified following Features - Core dense LoRA (triton + csgmv backends) - cuda-graph + LoRA - Multi-LoRA - MoE-LoRA - Dynamic Load/ Unload - Pinned Adapters - LoRA with Overlap Loading - LoRA with Radix Cache - LoRA with TP - Eviction (LRU/FIFO) - Embedding with LoRA Test changes: - Device-agnostic device selection via get_device() and register_xpu_ci across the kernel/unit and small-model LoRA tests. ## Motivation - Intel XPU is a first-class inference target: SGLang already supports XPU for base model inference; LoRA fine-tuned models are widely used in production and should be deployable on XPU without requiring a separate code path or falling back to the generic Transformers backbone. - Hard-coded \"cuda\" strings are silent correctness bugs on XPU: Several hot paths (init_cuda_graph_batch_info, lora_overlap_loader) referenced torch.cuda directly, causing device mismatches or runtime errors when the active device is an Intel XPU, even though the surrounding logic was otherwise device-agnostic. - CI coverage prevents regressions across backends: Without device-agnostic test infrastructure (get_device(), register_xpu_ci, ROUGE-L tolerance on XPU), XPU-specific breakage in LoRA paths would go undetected until a user report, making the XPU support effectively untested and unreliable. ## Modifications - Wrote a combined feature-level LoRA test suite validating Dynamic Load/Unload, Pinned Adapters, Radix Cache with LoRA, Embedding with LoRA, Multi-LoRA, and MoE-LoRA both individually and in combination across all three attention backends (csgmv, triton, and torch-native) to catch feature interaction bugs across devices and backend configurations - Replaced torch.cuda.current_stream() with self.device_module.current_stream() in lora_overlap_loader to prevent runtime errors on XPU using the existing device module abstraction - Added get_device() helper and register_xpu_ci for device-agnostic device selection across kernel and unit tests so the same test suite runs on both CUDA and XPU ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests] N/A (https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations] (https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy] N/A (https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31726141024](https://github.com/sgl-project/sglang/actions/runs/31726141024) Latest PR Test (Extra): :x: [Run #31726140469](https://github.com/sgl-project/sglang/actions/runs/31726140469) ",
"url": "https://github.com/sgl-project/sglang/pull/30345",
"createdAt": "2026-07-07T05:29:56Z",
"updatedAt": "2026-08-13T17:32:57Z",
"timestamp": "2026-08-13T17:32:57Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"lora",
"intel",
"xpu",
"run-ci",
"run-ci-extra"
],
"author": "AnuSajikumar6264",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30371",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSV4] Fix SWA state pool over-allocation by using storage page size instead of model window",
"text": " ## Motivation For DeepSeek-V4 models, DSV4PoolConfigurator sizes the c4 compressed-state pool using the model's SWA window (cfg.window_size = 128), while the runtime addresses that same pool using the pool's SWA storage page size (server_args.page_size = 256). These two values differ by 2x, causing capacity waste. ## Modifications Align the state-pool paging to the SWA storage page size (server_args.page_size), consistent with how DeepSeekV4TokenToKVPool and CompressStatePool address the pool at runtime: self.swa_storage_page_size = mr.server_args.page_size # 256 c4_state_ratio = self.c4_ring_size / self.swa_storage_page_size c4_state_pool_size = swa_tokens // self.swa_storage_page_size * self.c4_ring_size A comment is added to disambiguate this storage page size from the model's SWA window (cfg.window_size / SWA_WINDOW = 128). This is not a correctness bug — the runtime already divides by 256, so the smaller pool stays within bounds; it purely recovers wasted capacity. ## Accuracy Tests ## Speed Tests and Profiling Validated on GB200 with DeepSeek-V4-Flash (TP4 / DP4, default config, mem-fraction-static unconstrained). Expected improvements from the memory-pool sizing logs: | Metric | Before | After | Change | |--------|--------|-------|--------| | bytes_per_full_token | 7705.45 | 7033.45 | −8.7% | | full_token | 14,211,072 | 15,568,896 | **+9.6%** | | swa | 1,421,056 | 1,556,736 | +9.6% | | c4 | 3,552,768 | 3,892,224 | +9.6% | | c128 | 111,024 | 121,632 | +9.6% | | c4_state | 88,816 | 48,648 | **−45%** | ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31718370719](https://github.com/sgl-project/sglang/actions/runs/31718370719) Latest PR Test (Extra): :x: [Run #31718370126](https://github.com/sgl-project/sglang/actions/runs/31718370126) ",
"url": "https://github.com/sgl-project/sglang/pull/30371",
"createdAt": "2026-07-07T08:41:14Z",
"updatedAt": "2026-08-13T16:00:25Z",
"timestamp": "2026-08-13T16:00:25Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "mitu626",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30394",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: make automatic NUMA binding configurable",
"text": "## Motivation Env variable `SGLANG_AUTO_NUMA_BIND` remained declared and documented ([here on main branch](https://github.com/sgl-project/sglang/blob/main/docs_new/docs/references/environment_variables.mdx#numa--cpu)), but its gate was accidentally dropped during the automatic NUMA configuration refactor, leaving the variable ineffective. ## Modifications - Restore SGLANG_AUTO_NUMA_BIND=0 as a way to disable automatic NUMA binding. - Preserve automatic binding as the default behavior. - Keep explicit --numa-node configuration higher priority. - Clarify the NUMA environment-variable documentation. - Add unit tests for enabled, disabled, and explicit-node behavior. ## Checklist - [x] Format code with pre-commit. - [x] Add unit tests. - [x] Update documentation. - [x] Follow the SGLang code style guidance. --- ### CI States Latest PR Test (Base): :x: [Run #31596088166](https://github.com/sgl-project/sglang/actions/runs/31596088166) Latest PR Test (Extra): :x: [Run #31596088022](https://github.com/sgl-project/sglang/actions/runs/31596088022) ",
"url": "https://github.com/sgl-project/sglang/pull/30394",
"createdAt": "2026-07-07T12:36:56Z",
"updatedAt": "2026-08-13T00:40:29Z",
"timestamp": "2026-08-13T00:40:29Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [
"documentation"
],
"author": "lluki",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30398",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Refactor] New EPD",
"text": " ## Motivation The EPD encoder had grown into a single 4464-line `python/sglang/srt/disaggregation/encode_server.py` that owned the FastAPI application, the DP launcher, the request scheduler, CPU preprocessing, the GPU forward, and both transfer backends at the same time. That coupling caused three concrete problems: **1. Duplicated orchestration.** HTTP non-DP and DP workers each maintained their own copy of encode scheduling, batch/direct branching, backend detection, metadata publication, embedding staging, send scheduling, error handling, resource release and metrics updates. On top of that: - single-request and batch encode had two separate implementations; - Mooncake, ZMQ-to-scheduler and ZMQ-to-tokenizer each carried near-identical send logic; - `send` and `release` were tightly coupled, and embedding cleanup was scattered across the normal, error and timeout branches. **2. Unsafe resource lifetimes.** Because `send` implicitly released embeddings and cleanup lived in several places, concurrent encode / send / release had no ordering guarantee: a release racing an in-flight forward could tear down state that a pending send still needed, and a failed encode could leave a `send` waiting forever. **3. No layer to build on.** There was no protocol-neutral home for scheduling, so gRPC could not reuse the HTTP backend topology, and neither the CPU preprocessing nor the HTTP layer could be replaced by a Rust implementation. This PR restructures the encoder into a package with explicit layers and unifies everything after `receive` into a single pipeline, while keeping the external HTTP/gRPC surface unchanged. ## Modifications ## 2. One inference lifecycle for every path Everything after `receive` now follows a single chain, identically for HTTP non-DP and for DP workers: ``` receive (protocol-specific entry point) → execute_encode_pipeline → MMEncoder.encode / batch_encode → acquire encode ref → preprocess → publish metadata → model forward → stage embedding → release encode ref → delivery.send → release_request ``` - `encode()` becomes a batch-of-one call into `batch_encode()`, so there is no second encode implementation. `EncoderScheduler` only decides *whether* requests coalesce into a batch. - `execute_encode_pipeline()` owns metrics and tracing, direct-vs-batched dispatch, metadata publication, the `MMEncoder` call, final metadata sync, the ZMQ synchronous send or the Mooncake metadata response, and resource release on the normal, error and timeout paths. - `ReqState` holds the whole per-request lifecycle: `embedding_data`, `embedding_ready`, `active_encodes`, `active_sends`, `release_requested`, `preserve_metadata_on_release`, `lifecycle_condition`, driven by `_acquire_encode_ref()` / `_release_encode_ref()`. The resulting guarantees: - a release issued during encode cannot delete request state early; - a release waits until `active_sends == 0`; - once a release is requested, new sends are rejected; - a shared embedding is freed only after all destinations have finished; - a failed encode still stages the error, so a waiting `send` can never block forever. - `EncoderDelivery` isolates the backend differences. The previous `ZmqSchedulerDelivery` and `ZmqTokenizerDelivery` collapse into one `ZmqDelivery` parameterized by `cleanup_receive_state`; `self.transfer_backend` / `self.use_mooncake` are cached on `MMEncoder` instead of re-reading `server_args.encoder_transfer_backend` all over the code. - `embedding_to_send.pop()`, `discard_embedding()`, handler-`finally` cleanup, the Mooncake callback cleanup and the ZMQ rendezvous cleanup all collapse into `release_request()`. - Mooncake marks an embedding ready **only after the CUDA stream is synchronized**, so an RDMA read can no longer observe a GPU buffer whose write has not landed. Global-cache embeddings keep their existing per-send registration semantics. ### 3. Metadata is published right after preprocess Every backend publishes embedding shape, dtype and size as soon as preprocessing completes — before the forward runs. At that point `ReqState.embedding_data` holds a metadata placeholder; the real embedding is staged and validated afterwards, which is what sets `embedding_ready`. For Mooncake non-DP this lets the receiver call `/scheduler_receive_meta_data`, then allocate and register its landing buffer **in parallel with the encoder forward**; the RDMA transfer itself still waits for `embedding_ready`. ZMQ publishes the same metadata today, but has no early consumer yet. ### 4. Preprocessing metadata consolidated `EncoderPreprocessResult` returns `mm_inputs`, `grid_thw` and `token_counts` together. Grid normalization and patch/token counting move out of `MMEncoder` into `EncoderPreprocessor`, so `process_mm_items` parses the grid exactly once and the direct, cache, Mooncake and batch paths reuse the same result. `slice_embedding` and `_encode_missing` no longer recompute token counts from the grid. ### 5. Receiver: one embedding pool for both backends - `EmbeddingPool` absorbs `MooncakeEmbeddingPool` and gains `try_stage` / `release_on_gc`. When `SGLANG_EMBEDDING_POOL_SIZE_MB` is set explicitly, `zmq_to_scheduler` also keeps received embeddings resident in the GPU pool: zero-copy per-modality views, slot lifetime bound to `mm_inputs` garbage collection, and a `PENDING` retry when the pool is full. **Default behaviour is unchanged** — the 4096 MB default still applies to Mooncake only. - RDMA buffer release becomes an ownership transfer to the receiving thread, which removes an up-to-1800 s `join` block on the scheduler main thread. - `WaitingMMRequestBase` (ABC) unifies `recv_embedding`; the backend differences shrink to three abstract methods — `send_encode_request`, `_extract_embedding_from_buffer`, `_prepare_embedding_buffer`. Seven separate failure paths merge into `_fail_and_release`, and the classes are renamed off \"Image\" into the `Zmq` / `Grpc` / `RDMA` families. - Dead code on the `zmq_to_tokenizer` path is removed. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31669306498](https://github.com/sgl-project/sglang/actions/runs/31669306498) Latest PR Test (Extra): :white_check_mark: [Run #31681475881](https://github.com/sgl-project/sglang/actions/runs/31681475881) ",
"url": "https://github.com/sgl-project/sglang/pull/30398",
"createdAt": "2026-07-07T13:21:04Z",
"updatedAt": "2026-08-13T11:02:41Z",
"timestamp": "2026-08-13T11:02:41Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [
"run-ci",
"run-ci-extra"
],
"author": "liusy58",
"state": "open",
"assignees": [
"ShangmingCai"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30497",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSV4] Enable overlap scheduling for online C128 MTP",
"text": "Follow up of #30365. ## Problem The experimental online C128 MTP target-verify path added in #26471 still uses the host prefill planner. #30365 removed the sequence-length D2H dependency from the other DSV4 speculative paths, but intentionally preserved the CPU path for online C128 MTP. That path requires CPU sequence lengths and a pinned plan buffer, introducing a D2H synchronization on every verify step and preventing it from working with the overlap scheduler. ## Solution - Add a fixed-shape CUDA stage 0 to the existing online C128 C++ planner for MTP target verification, and reuse its existing stage 1 request-to-state-slot finalization. - Accept both int32 and int64 device sequence lengths and request-pool indices without a host round trip. - Use the MTP bound of at most 8 draft tokens to emit at most one close-chunk plan and one trailing-partial plan per request. - Mask inactive padded requests with the runtime active batch size, including zero-active/idle replay, so CUDA graphs cannot reuse stale compression plans. - Keep target-verify sequence lengths and compression metadata on device, remove the pinned CPU plan buffer, and remove the separate Triton planner. ### Planner equivalence The CUDA stage 0 emits the native `CompressPlan` records for the MTP target-verify subset, then the existing online C128 stage 1 performs request-to-state-slot finalization. This preserves the plan semantics of the host `_plan_prefill_partial` path while sharing the C++ planner ABI and finalization logic. ## Validation DeepSeek-V4-Flash TP4 serving measurements use 4x GB300, 512 input tokens, 256 output tokens, and concurrency 32. The serving results below use three paired 384-request runs with seeds 42, 43, and 44 (1,152 successful requests per side). Throughput is aggregated across all three runs; latency values are averages of the three run summaries. GSM8K was rerun after the C++/CUDA stage 0 refactor with 200 examples, 5-shot prompting, temperature 0, and 32 concurrent requests. All 200 requests completed successfully; target-verify CUDA graph capture and replay completed without planner or CUDA errors. | Check / metric | Baseline | This PR | Result | | --- | ---: | ---: | ---: | | GSM8K, 200 examples, temperature 0 | 0.970 | 0.975 | No regression | | Output throughput (tok/s) | 1,157.39 | 1,219.69 | +5.38% | | Total throughput (tok/s) | 3,472.17 | 3,659.08 | +5.38% | | Median TTFT (ms) | 240.96 | 232.81 | -3.38% | | Mean TPOT (ms) | 25.89 | 24.48 | -5.45% | | Mean ITL (ms) | 25.92 | 24.52 | -5.40% | | Median ITL (ms) | 5.77 | 5.02 | -12.98% | | ITL p95 (ms) | 106.20 | 103.37 | -2.67% | | ITL p99 (ms) | 210.49 | 205.21 | -2.51% | | Accept length | 2.80 | 2.80 | Match | The paired output-throughput gains for seeds 42/43/44 were +1.54%, +6.87%, and +8.17%, respectively. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31367679692](https://github.com/sgl-project/sglang/actions/runs/31367679692) Latest PR Test (Extra): :x: [Run #31367679482](https://github.com/sgl-project/sglang/actions/runs/31367679482) ",
"url": "https://github.com/sgl-project/sglang/pull/30497",
"createdAt": "2026-07-08T08:06:36Z",
"updatedAt": "2026-08-13T17:14:09Z",
"timestamp": "2026-08-13T17:14:09Z",
"metrics": {
"reactions": 1,
"comments": 8
},
"labels": [
"performance",
"deepseek",
"speculative-decoding",
"run-ci",
"jit-kernel"
],
"author": "weireweire",
"state": "open",
"assignees": [
"hnyls2002"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30575",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Enable Fast Triton Sparse MLA backend",
"text": "# Fast Triton Sparse MLA Kernels for DSA (Prefill + Decode) ## Summary This adds `triton` as an explicit DSA prefill/decode backend: ```bash --dsa-prefill-backend triton --dsa-decode-backend triton ``` The new backend provides pure Triton sparse MLA kernels for the fp8 DSA path on ROCm, validated on MI355X (gfx950) and MI300X (gfx942). It replaces the previous Triton prefill env-var gate, SGLANG_DSA_TRITON_PREFILL, with a faster kernel exposed through the normal backend option and adds a fast Triton kernel for the decode path. ## What Changed - Adds `triton` to `DSA_CHOICES` for `--dsa-prefill-backend` and `--dsa-decode-backend`. - Adds startup validation in both server-args paths so Triton DSA is only used with `--kv-cache-dtype fp8_e4m3`. - Adds standalone Triton prefill and decode dispatch branches in `dsa_backend.py`. - Ensures HIP Triton DSA uses the raw MLA KV layout (`512 nope + 64 rope`) instead of the scaled fp8 layout used by other non ROCm backends. - Extends the gfx950 fused-rope skip path to include Triton, matching the TileLang/AITER raw-layout flow. ## Kernel Notes - Prefill uses a split-dim 4x128 implementation for the 512-wide nope dimension, plus the 64-wide rope tail. - Prefill can split topk work across CTAs and reduce partial online-softmax state in Triton. - Decode uses an adaptive split-K path with a fused single-pass fast path when `kv_splits == 1`. - Kernels use `exp2` with `sm_scale * LOG2E` prescaling. The kernels consume raw fp8 MLA KV cache layout on HIP. They are not intended for NVIDIA's scaled fp8 KV layout. ## Validation - GSM8K 400-question validated: TileLang and Triton both produced 95.5% accuracy with 0% invalid answers. Accuracy also verified on gfx942 - Isolated prefill microbench on MI355X/GLM-5.2 shape (`H=16`, `D_V=512`, `D_TAIL=64`, `topk=2048`) showed about 10% speedup for the split-dim prefill kernel versus the earlier 512-wide-dot Triton env gated kernel, with cosine similarity 1.00000. - End-to-end GLM-5.1-MXFP4 serving on TP4 MI355X (`ISL=8192`, `OSL=1024`) showed Triton improving throughput by roughly 3-5% versus TileLang in the measured concurrency range. - Performance improvement is seen across all concurrencies, TPs, and for short and long context. ## Performance End-to-end GLM-5.1-MXFP4 serving on TP4 MI355X, concurrency 2: | Backend | Total tok/s | Output tok/s | Mean TTFT (ms) | Mean TPOT (ms) | |---|---:|---:|---:|---:| | TileLang | 1,005 | 112 | 834 | 17.1 | | Triton | 1,044 | 116 | 701 | 16.6 | | Improvement | +3.9% | +3.6% | -16.0% | -2.9% | ## Changed Files - `python/sglang/srt/layers/attention/dsa/triton_sparse_mla.py`: updates the Triton prefill implementation with split-dim and split-K paths. - `python/sglang/srt/layers/attention/dsa/triton_sparse_mla_decode.py`: adds the Triton decode implementation. - `python/sglang/srt/layers/attention/dsa_backend.py`: adds Triton prefill/decode dispatch and removes the old env-var-gated prefill path. - `python/sglang/srt/server_args.py`: exposes `triton` as a DSA backend choice. - `python/sglang/srt/arg_groups/overrides.py`: validates Triton DSA override combinations against fp8 KV cache requirements. - `python/sglang/srt/model_executor/model_runner_kv_cache_mixin.py`: selects raw HIP MLA KV layout for Triton DSA. - `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`: includes Triton in the gfx950 fused-rope skip check. --- ### CI States Latest PR Test (Base): :x: [Run #31420729809](https://github.com/sgl-project/sglang/actions/runs/31420729809) Latest PR Test (Extra): :x: [Run #31420729690](https://github.com/sgl-project/sglang/actions/runs/31420729690) ",
"url": "https://github.com/sgl-project/sglang/pull/30575",
"createdAt": "2026-07-09T01:05:05Z",
"updatedAt": "2026-08-13T09:57:16Z",
"timestamp": "2026-08-13T09:57:16Z",
"metrics": {
"reactions": 1,
"comments": 6
},
"labels": [
"amd",
"run-ci",
"jit-kernel"
],
"author": "clintg6",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30691",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[do not merge] check result of CI",
"text": "Do not merge. Opened solely to trigger and observe CI results for `sgl-kernel/tests/test_moe_topk_sigmoid.py` via `/rerun-test`. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31646179281](https://github.com/sgl-project/sglang/actions/runs/31646179281) Latest PR Test (Extra): :x: [Run #31646179021](https://github.com/sgl-project/sglang/actions/runs/31646179021) ",
"url": "https://github.com/sgl-project/sglang/pull/30691",
"createdAt": "2026-07-09T19:55:24Z",
"updatedAt": "2026-08-12T22:40:22Z",
"timestamp": "2026-08-12T22:40:22Z",
"metrics": {
"reactions": 0,
"comments": 28
},
"labels": [
"sgl-kernel"
],
"author": "b8zhong",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30762",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(hicache/umbp): support DeepSeek-V4 hybrid HostPoolGroup (multi-po…",
"text": "…ol v2) UMBPStore assumed mem_pool_host is a single KV-bearing pool. For the DeepSeek-V4 HiCache stack, mem_pool_host is a HostPoolGroup whose KV anchor is a LogicalHostPool that owns only page indices and holds no physical KV tensor (get_page_buffer_meta() returns None by design). The real KV state lives in page_first side pools (SWA / compressed KV / indexer / state), which the controller registers via register_mem_host_pool_v2() and drives through the batch_*_v2 API. Two problems made UMBP unusable as the L3 backend for DSv4: 1. __init__ crashed while probing dram_page_size: it unpacked the return of the logical anchor's get_page_buffer_meta() (None) into `_, esz`, raising \"cannot unpack non-iterable NoneType object\" and killing the prefill worker at startup. 2. UMBPStore never implemented the v2 multi-pool API, so even past the probe it would inherit HiCacheStorage's batch_*_v2 NotImplementedError stubs and could not store/load the V4 side pools at all. Fixes: - Probe: fetch the buffer-meta tuple first and guard None. Logical-anchor groups leave dram_page_size at 0 (mori master default + PoolClient partial-tail safety net); per-pool sizes are handled at v2 I/O time. - Implement the v2 multi-pool API mirroring MooncakeStore / HiCacheHF3FS, specialized for UMBP's page_first single-object-per-page layout: register_mem_host_pool_v2 (skips the logical KV anchor, registers each real side-pool buffer for RDMA zero-copy), batch_exists_v2 (ALL_PAGES / TRAILING_PAGES hit policies), batch_get_v2 / batch_set_v2 (per-pool zero-copy get/put with put dedup). - Guard batch_get_v1 / batch_set_v1 to no-op on the logical KV anchor (its pages carry no physical data), matching MooncakeStore. - Refactor the RDMA registration into a reusable _register_host_buffer_for_zero_copy() helper. ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ### Setup | | | |---|---| | Model | DeepSeek-V4-Pro (FP4), `--kv-cache-dtype fp8_e4m3` | | Image | `lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710` + this PR's `umbp_store.py` | | Hardware | AMD MI355X, **1 Prefill node + 1 Decode node**, TP=8 each (16 GPUs total), EP=1, no DP-attention | | PD transfer | MoRI (`--disaggregation-transfer-backend mori`), RoCE (`ionic_0..7`) | | HiCache | **L3 = MoRI UMBP** (`--hicache-storage-backend mori`), `page_first` + `direct` + `write_through`, `--hicache-ratio 3`, `--page-size 256`; UMBP master on the prefill node (DRAM tier 10 GiB, hugepage-backed, SSD off) | | Attention | `--attention-backend dsv4`, hybrid SWA `--swa-full-tokens-ratio 0.1` | | Workload | agentic trace replay (`aiperf`, public dataset `semianalysis_cc_traces_weka_062126`), ISL median ~100k / p95 ~450k tokens, OSL median ~365; 3600 s measurement window per concurrency point | This is the exact scenario the PR targets: `mem_pool_host` is a DeepSeek‑V4 `HostPoolGroup` whose KV anchor is a logical pool, so UMBP is driven purely through the new `register_mem_host_pool_v2` + `batch_exists_v2` / `batch_get_v2` / `batch_set_v2` multi‑pool path. ### Launch commands **1) UMBP master** (prefill node, before the prefill server): ```bash umbp_master 0.0.0.0:59151 59152 # grpc :59151, http/metrics :59152 ``` **2) Prefill server** (prefill node, 8× MI355X): ```bash python3 -m sglang.launch_server \\ --model-path /models/DeepSeek-V4-Pro \\ --trust-remote-code \\ --host 0.0.0.0 --port 8000 \\ --tp-size 8 \\ --disaggregation-mode prefill \\ --disaggregation-transfer-backend mori \\ --disaggregation-ib-device ionic_0,ionic_1,ionic_2,ionic_3,ionic_4,ionic_5,ionic_6,ionic_7 \\ --kv-cache-dtype fp8_e4m3 \\ --attention-backend dsv4 \\ --page-size 256 \\ --swa-full-tokens-ratio 0.1 \\ --disable-shared-experts-fusion \\ --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 \\ --mem-fraction-static 0.8 \\ --max-running-requests 128 \\ --chunked-prefill-size 8192 \\ --cuda-graph-bs 1 2 3 4 … 128 \\ --watchdog-timeout 3600 \\ --enable-metrics --enable-cache-report \\ --enable-hierarchical-cache \\ --hicache-ratio 3 \\ --hicache-io-backend direct \\ --hicache-mem-layout page_first \\ --hicache-write-policy write_through \\ --hicache-storage-prefetch-policy best_effort \\ --hicache-storage-backend mori \\ --hicache-storage-backend-extra-config '{\"dram_capacity_bytes\": 10737418240, \"ssd_enabled\": false, \"master_address\": \":59151\", \"node_address\": \"\", \"io_engine_port\": \"19600\", \"peer_service_port\": \"19700\", \"cache_remote_fetches\": false, \"kv_events_subscriber\": true, \"kv_events_endpoint\": \"tcp://localhost:6557\"}' \\ --kv-events-config '{\"publisher\": \"zmq\", \"endpoint\": \"tcp://*:6557\"}' ``` **3) Decode server** (decode node, 8× MI355X): ```bash python3 -m sglang.launch_server \\ --model-path /models/DeepSeek-V4-Pro \\ --trust-remote-code \\ --host 0.0.0.0 --port 8000 \\ --tp-size 8 \\ --disaggregation-mode decode \\ --disaggregation-transfer-backend mori \\ --disaggregation-ib-device ionic_0,ionic_1,ionic_2,ionic_3,ionic_4,ionic_5,ionic_6,ionic_7 \\ --kv-cache-dtype fp8_e4m3 \\ --attention-backend dsv4 \\ --page-size 256 \\ --swa-full-tokens-ratio 0.1 \\ --disable-shared-experts-fusion \\ --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 \\ --mem-fraction-static 0.85 \\ --max-running-requests 128 \\ --cuda-graph-bs 1 2 3 4 … 128 \\ --watchdog-timeout 3600 \\ --prefill-round-robin-balance \\ --enable-metrics ``` > `--cuda-graph-bs 1 2 3 4 … 128` = all batch sizes 1–128. Decode keeps CUDA graphs on; prefill keeps CUDA graphs (compute-bound). The decode server does **not** attach a hierarchical/L3 cache (DSv4 hybrid-SWA decode is incompatible with decode radix); only the prefill side offloads KV to the UMBP L3 store. **4) Router** (prefill node): ```bash python3 -m sglang_router.launch_router \\ --pd-disaggregation --port 30000 \\ --policy cache_aware --prefill-policy cache_aware --decode-policy round_robin \\ --prefill http://:8000 \\ --decode http://:8000 ``` **5) Benchmark client**: ```bash aiperf profile --scenario inferencex-agentx-mvp \\ --url http://:30000 \\ --endpoint /v1/chat/completions --endpoint-type chat --streaming \\ --model /models/DeepSeek-V4-Pro \\ --concurrency <32|48> --benchmark-duration 3600 \\ --failed-request-threshold 0.1 \\ --trajectory-start-min-ratio 0.25 --trajectory-start-max-ratio 0.75 \\ --use-server-token-count --tokenizer-trust-remote-code \\ --public-dataset semianalysis_cc_traces_weka_062126 --num-dataset-entries 393 ``` **Required runtime env (DSv4 kernel routing — no `launch_server` CLI equivalent), applied to both P and D:** ```bash SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton SGLANG_USE_ROCM700A=0 AITER_BF16_FP8_MOE_BOUND=0 SGLANG_OPT_USE_FUSED_COMPRESS=true SGLANG_OPT_USE_FUSED_COMPRESS_TRITON=true SGLANG_OPT_USE_AITER_INDEXER=true SGLANG_OPT_USE_TOPK_V2=false SGLANG_OPT_FP8_WO_A_GEMM=false SGLANG_OPT_DEEPGEMM_HC_PRENORM=false SGLANG_OPT_USE_JIT_INDEXER_METADATA=false SGLANG_OPT_USE_TILELANG_INDEXER=false SGLANG_OPT_USE_TILELANG_MHC_PRE=false SGLANG_OPT_USE_TILELANG_MHC_POST=false SGLANG_OPT_USE_MULTI_STREAM_OVERLAP=false SGLANG_ROCM_USE_MULTI_STREAM=false SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1 SGLANG_EAGER_INPUT_NO_COPY=true SGLANG_DEFAULT_THINKING=1 SGLANG_DSV4_REASONING_EFFORT=max # MoRI dispatch/transfer SGLANG_MORI_DISPATCH_DTYPE=auto SGLANG_MORI_COMBINE_DTYPE=auto SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16384 # prefill (decode: 128) MORI_IO_QP_MAX_SEND_WR=32767 MORI_IO_SQ_BACKOFF_TIMEOUT_US=500000 # UMBP per-process (prefill only) UMBP_DRAM_USE_HUGEPAGES=1 UMBP_DISABLE_ZERO_COPY_REGISTER=false MORI_SHMEM_MODE=ISOLATION MORI_SHMEM_HEAP_SIZE=1G ``` ### Results (1P1D, TP8, UMBP L3, 3600 s/point) | Concurrency | Requests (ok / cancelled / err) | Total throughput | ≈ tok/s/GPU (÷16) | TTFT p50 / p90 (s) | ITL p50 / p90 (ms) | ISL median | OSL median | prefix-cache hit | |---|---|---|---|---|---|---|---|---| | **32** | 3,779 / 15 / **0** | 154,957 tok/s | ~9,685 | 1.96 / 7.18 | 31.5 / 33.6 | 104,938 | 365 | 97.5% | | **48** | 3,669 / 37 / **0** | 154,876 tok/s | ~9,680 | 23.77 / 37.06 | 31.2 / 33.4 | 98,943 | 366 | 97.8% | **Takeaways** - UMBP L3 works end‑to‑end with the DeepSeek‑V4 hybrid `HostPoolGroup`: **0 errors** at both concurrencies over the full 3600 s window; the logical KV anchor no longer crashes the prefill worker at startup, and the `batch_*_v2` multi‑pool store/load path serves the SWA / compressed‑KV / indexer / state side pools correctly. - Decode‑bound at this topology: throughput is flat (~155k tok/s ≈ 9.7k tok/s/GPU) from conc 32→48 while TTFT rises (queuing), consistent with 1P1D decode‑KV saturation; decode ITL stays steady (~31 ms). --- ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31583284423](https://github.com/sgl-project/sglang/actions/runs/31583284423) Latest PR Test (Extra): :x: [Run #31583284270](https://github.com/sgl-project/sglang/actions/runs/31583284270) ",
"url": "https://github.com/sgl-project/sglang/pull/30762",
"createdAt": "2026-07-10T07:58:43Z",
"updatedAt": "2026-08-13T07:40:08Z",
"timestamp": "2026-08-13T07:40:08Z",
"metrics": {
"reactions": 1,
"comments": 15
},
"labels": [
"amd",
"hicache",
"run-ci"
],
"author": "AMD-yanfeiwang",
"state": "closed",
"assignees": [
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30797",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[GDN] perf: Fuse the linear-attention prefill prologue for Flashinfer prefill attn",
"text": "## Motivation Hybrid GDN models such as Qwen3.5 and Qwen3-Next execute linear-attention prefill between CUDA-graph segments. The established FlashInfer path launches separate kernels for packed-QKV splitting, gating, Q/K normalization, and the final output copy, making short prefills launch-bound across many GDN layers. ## What this PR does This PR adds an isolated FlashInfer fast path while leaving the established target-verify and fallback paths unchanged. - **Fused prefill prologue:** `gdn_prefill_fused` combines packed-QKV splitting, GDN gating (including `exp(g)`), and Q/K L2 normalization in one Triton launch. `main` already fuses the QKV *split* alone (#26206); this folds gating, both l2norms and the `exp` into that same launch, taking the FlashInfer GDN prologue from 5 kernel launches per layer to 1. The numerically load-bearing formulas are shared with the standalone kernels through `fla_math`, so the fused path is bit-identical to the unfused one. - **Direct output write:** the caller's output slice is forwarded to FlashInfer's `output=`, avoiding the per-layer device-to-device output copy when the fast path is selected. - **Prefix-free state handling:** `initial_state=None` zero-seeds the FlashInfer kernel when no request has a prefix, avoiding the initial-state pool gather while preserving state writeback. The fused route currently requires row-contiguous packed QKV, equal Q/K head counts, 128-dimensional Q/K heads, and a positive power-of-two value-head count. The packed width must match those dimensions but is not capped independently. ## Accuracy main: ``` == running GSM8K (200 questions, 128-way, 5-shot) == 100%|██████████| 200/200 [00:33<00:00, 6.04it/s] Accuracy: 0.970 Invalid: 0.005 ``` PR: ``` == running GSM8K (200 questions, 128-way, 5-shot) == 100%|██████████| 200/200 [00:31<00:00, 6.42it/s] Accuracy: 0.970 Invalid: 0.005 ``` ## ⚠️ Note on the performance numbers below **The serving numbers in this section were measured on 2026-07-13 and are no longer reproducible against current `main`. Please read this before interpreting them.** At the time of that benchmark, `l2norm_fwd_kernel` in `python/sglang/kernels/ops/attention/fla/l2norm.py` declared `NB: tl.constexpr, T: tl.constexpr`, so **every distinct prefill token count forced a fresh Triton compile**. The FlashInfer GDN prefill path calls `l2norm_fwd` 90 times per forward (q and k, 45 layers), and the 8k1k sweep contains 186 distinct `#new-token` values. Because a synchronous Triton compile blocks the scheduler loop, every in-flight request stalls with it — which is why the effect shows up in *median* TTFT and not just the tail. This PR fuses l2norm into `gdn_prefill_fused_kernel`, whose size arguments (`seq_len`/`t_rows`/`nbn`) are plain runtime scalars, so it never launches `l2norm_fwd` and never pays those compiles. **That, not the launch-count reduction, is the dominant source of the 1.10–1.21x reported below.** #31558 (`42a058c76`, merged **2026-07-18**, i.e. *after* this benchmark and after this PR was opened) fixed the same pathology upstream from the other direction, via `@triton.jit(do_not_specialize=[\"T\"])`. With that in `main`, this PR no longer has that win to deliver. Re-measured on 2026-07-29 against `main@32c30c0f9`, the 8k1k end-to-end difference is **within noise (0.98–1.00x)**. Attribution was confirmed experimentally rather than inferred: reverting #31558 on `32c30c0f9` with a cold Triton cache reproduces the old baseline almost exactly (median TTFT 593 / 722 / 1050 ms at concurrency 8 / 16 / 32, vs 601 / 734 / 929 ms for the 2026-07-13 base arm), while median ITL stays identical across all arms — the entire effect is prefill. **What remains attributable to this PR on current `main`**, measured from prefill traces of an isolated 8192-token extend step: 2267 → 2042 kernel launches, 89 → 44 device-to-device memcpys, GPU-busy −1.26 ms, wall −1.6% on the step. The kernel-hygiene benefit — 5 prologue launches per layer down to 1, with q/k never materialized to HBM. ## Speedup (2026-07-13 — see the note above) main linear attention: PR: Qwen/Qwen3.5-397B-A17B-FP8 B200 pareto 8k1k no MTP: ```bash python3 -m sglang.launch_server \\ --model-path=\"${MODEL}\" \\ --host=\"${HOST}\" \\ --port=\"${PORT}\" \\ --trust-remote-code \\ --tensor-parallel-size=4 \\ --data-parallel-size=1 \\ --expert-parallel-size=1 \\ --enable-symm-mem \\ --disable-radix-cache \\ --quantization fp8 \\ --kv-cache-dtype fp8_e4m3 \\ --mamba-ssm-dtype bfloat16 \\ --attention-backend trtllm_mha \\ --moe-runner-backend flashinfer_trtllm \\ --linear-attn-prefill-backend flashinfer \\ --cuda-graph-max-bs 128 \\ --max-prefill-tokens 16384 \\ --chunked-prefill-size 16384 \\ --mem-fraction-static 0.8 \\ --stream-interval 50 \\ --cuda-graph-backend-prefill=breakable \\ --scheduler-recv-interval 30 \\ --tokenizer-worker-num 6 \\ --context-length 9236 ``` ```bash series conc TPS/user (sp) out TPS/GPU (sp) total TPS/GPU (sp) TTFT ms (sp) ---------------------------------------------------------------------------------------------------------------------- base 4 148.53 ( 1.00x) 135.67 ( 1.00x) 1230.91 ( 1.00x) 366.08 ( 1.00x) base 8 111.83 ( 1.00x) 202.51 ( 1.00x) 1842.56 ( 1.00x) 542.41 ( 1.00x) base 16 75.32 ( 1.00x) 286.28 ( 1.00x) 2553.16 ( 1.00x) 671.15 ( 1.00x) base 32 52.16 ( 1.00x) 395.21 ( 1.00x) 3578.48 ( 1.00x) 813.98 ( 1.00x) base 64 34.60 ( 1.00x) 539.48 ( 1.00x) 4875.69 ( 1.00x) 978.56 ( 1.00x) base 128 22.91 ( 1.00x) 693.82 ( 1.00x) 6263.06 ( 1.00x) 1345.94 ( 1.00x) opt 4 150.98 ( 1.02x) 143.51 ( 1.06x) 1301.99 ( 1.06x) 295.00 ( 1.24x) opt 8 122.60 ( 1.10x) 230.37 ( 1.14x) 2096.01 ( 1.14x) 310.17 ( 1.75x) opt 16 90.26 ( 1.20x) 343.61 ( 1.20x) 3064.46 ( 1.20x) 381.25 ( 1.76x) opt 32 62.95 ( 1.21x) 477.41 ( 1.21x) 4322.79 ( 1.21x) 568.09 ( 1.43x) opt 64 42.50 ( 1.23x) 653.23 ( 1.21x) 5903.75 ( 1.21x) 775.03 ( 1.26x) opt 128 27.96 ( 1.22x) 832.00 ( 1.20x) 7510.41 ( 1.20x) 1112.19 ( 1.21x) ``` --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31580932305](https://github.com/sgl-project/sglang/actions/runs/31580932305) Latest PR Test (Extra): :x: [Run #31580932580](https://github.com/sgl-project/sglang/actions/runs/31580932580) ",
"url": "https://github.com/sgl-project/sglang/pull/30797",
"createdAt": "2026-07-10T16:33:19Z",
"updatedAt": "2026-08-13T08:18:15Z",
"timestamp": "2026-08-13T08:18:15Z",
"metrics": {
"reactions": 0,
"comments": 13
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "mattteochen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30805",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103",
"text": " ## Motivation Integrates TRT-LLM attention kernel for DSv4 style attention (CSA, HCA). ## Modifications ## Accuracy Tests ``` SGLANG_DSV4_ATTN_DECODE_BACKEND=flashmla/trtllm_gen \\ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4-Pro \\ --trust-remote-code --tp 8 --moe-runner-backend flashinfer_mxfp4 \\ --chunked-prefill-size 4096 --disable-flashinfer-autotune \\ --mem-fraction-static 0.88 --max-running-requests 32 ``` ### GSM8K Accuracy GSM8K V4-Pro flashmla: 0.965 GSM8K V4-Pro trtllm_gen: 0.985 ### GPQA (add --chat-template to above server command) ``` python3 -m sglang.test.run_eval --port 21000 --eval-name gpqa \\ --num-examples 198 --max-tokens 120000 --repeat 8 \\ --thinking-mode deepseek-v3 ``` FlashMLA: Repeat: 8, mean: 0.885 Scores: ['0.889', '0.879', '0.889', '0.879', '0.869', '0.899', '0.904', '0.874'] TRTLLM: Repeat: 8, mean: 0.907 Scores: ['0.919', '0.904', '0.894', '0.904', '0.909', '0.919', '0.904', '0.904'] ## Speed Tests and Profiling ### Unit Kernel Benchmarking (ran with private benchmarking repo): ``` # prefill (varlen), long contexts python benchmark.py --backends flashmla_dsv4,trtllm_gen_dsv4 --model deepseek-v4 \\ --hw b200 --dtype fp8 --mode prefill --tp 1 \\ --batch-sizes 1,2 --seq-lens 4096,16384,65536 --graph # decode, low/mid concurrency x long contexts python benchmark.py --backends flashmla_dsv4,trtllm_gen_dsv4 --model deepseek-v4 \\ --hw b200 --dtype fp8 --mode decode --tp 1 \\ --batch-sizes 1,4,16,64 --seq-lens 1024,4096,16384,65536,131072 --graph # decode, high concurrency python benchmark.py --backends flashmla_dsv4,trtllm_gen_dsv4 --model deepseek-v4 \\ --hw b200 --dtype fp8 --mode decode --tp 1 \\ --batch-sizes 128,256,512,1024 --seq-lens 256,1024,4096,16384 --graph ``` Prefill: ~1.2x faster Decode: ~1.45x faster ### E2E bench-serving ```bash python -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 21000 \\ --dataset-name random --random-input-len 131072 --random-output-len 128 \\ --random-range-ratio 1 --num-prompts 8 --max-concurrency 4 ``` flashmla: ``` Benchmark duration (s): 59.92 Total input tokens: 1048576 Input token throughput (tok/s): 17500.55 Mean E2E Latency (ms): 29950.67 Mean TTFT (ms): 19009.87 (median 19636.34) Mean TPOT (ms): 86.15 Median ITL (ms): 12.12 ``` trtllm_gen: ``` Benchmark duration (s): 60.09 Total input tokens: 1048576 Input token throughput (tok/s): 17448.78 Mean E2E Latency (ms): 30039.27 Mean TTFT (ms): 16808.71 (median 15734.66) Mean TPOT (ms): 104.18 Median ITL (ms): 13.11 ``` ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31707526521](https://github.com/sgl-project/sglang/actions/runs/31707526521) Latest PR Test (Extra): :x: [Run #31707526162](https://github.com/sgl-project/sglang/actions/runs/31707526162) ",
"url": "https://github.com/sgl-project/sglang/pull/30805",
"createdAt": "2026-07-10T19:01:43Z",
"updatedAt": "2026-08-13T14:34:42Z",
"timestamp": "2026-08-13T14:34:42Z",
"metrics": {
"reactions": 0,
"comments": 37
},
"labels": [
"high priority",
"deepseek",
"blackwell",
"run-ci",
"release-highlight"
],
"author": "akhilg-nv",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30808",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [GLM5] Enable dense-MHA short-context prefill fallback on gfx950",
"text": "## Motivation On gfx950 (MI355X), GLM-5.2 DSA prefill always ran the triton sparse-MLA path, even at short context where the sparse indexer top-k + gather + mask overhead exceeds the KV it prunes. The dense-MHA prefill fallback — already used on NVIDIA SM90/SM100 and gated by `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` (default = model `index_topk`) — was hard-gated to NVIDIA and never taken on ROCm, despite the dense kernel (aiter `flash_attn_varlen_func`) already being available. ## Modifications - Add gfx950 detection (`gcnArchName`) and include it in the `use_mha` prefill device gate in `dsa_backend.py` (previously `device_sm == 90 or 100 <= device_sm < 110`, NVIDIA only). - Guard `_forward_standard_mha` so ROCm routes through aiter `flash_attn_varlen_func`, never the flashinfer/trtllm (Blackwell) path. - NVIDIA behavior unchanged. On gfx950 the dense path triggers only when `max_kv_len <= SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` (GLM-5.2: index_topk = 2048); longer context stays on sparse-MLA. Off switch: set the threshold to 0. ## Accuracy Tests GLM-5.2-MXFP4, MI355X TP4, GSM8K (200 questions): dense fallback = 0.955. ## Speed Benchmarks GLM-5.2-MXFP4, MI355X TP4, graphs-on, random 1024/1024, `sglang.bench_serving`, median. A. Baseline (dense fallback off)*: | concurrency | TTFT (ms) | ITL (ms) | E2EL (ms) | output tok/s | |---|---|---|---|---| | 4 | 277.3 | 11.33 | 11860 | 342.0 | | 8 | 467.6 | 12.98 | 13790 | 592.7 | | 16 | 872.9 | 16.48 | 17767 | 915.8 | | 32 | 1371.1 | 20.25 | 22655 | 1443.9 | | 64 | 1920.5 | 26.05 | 30112 | 2174.0 | B. Dense fallback on (Δ vs baseline): | concurrency | TTFT (ms) | Δ | ITL (ms) | Δ | E2EL (ms) | Δ | output tok/s | Δ | |---|---|---|---|---|---|---|---|---| | 4 | 196.9 | −29.0% | 11.31 | −0.2% | 11775 | −0.7% | 346.9 | +1.4% | | 8 | 366.8 | −21.6% | 13.00 | +0.2% | 13660 | −0.9% | 582.1 | −1.8% | | 16 | 526.6 | −39.7% | 16.46 | −0.1% | 17432| −1.9% | 935.0 | +2.1% | | 32 | 783.2 | −42.9% | 20.48 | +1.1% | 21895| −3.4% | 1498.13 | +3.8% | | 64 | 1125.6 | −41.4% | 26.10 | +0.2% | 29352 | −2.5% | 2227.3 | +2.5% | Aside from the consistent TTFT decreases (−22% to −43%), the ITL, E2EL, and output throughput deltas are within run-to-run noise (dense fallback is prefill-only, so decode is unaffected). *Baseline and Feature both measured with #30519, #30715 and aiter tuned MoE configs for GLM5.2; we expect those will be merged first and aiter version updated in recent images. ## Checklist - [x] Format your code according to the Format code with pre-commit. - [ ] Add unit tests according to the Run and add unit tests. - [x] Provide accuracy and speed benchmark results. - [x] Follow the SGLang code style guidance. ## Review and Merge Process 1. Ping Merge Oncalls to start the process. 2. Get approvals from CODEOWNERS and other reviewers. 3. Trigger CI tests with comments. --- ### CI States Latest PR Test (Base): :x: [Run #31635016485](https://github.com/sgl-project/sglang/actions/runs/31635016485) Latest PR Test (Extra): :x: [Run #31635016282](https://github.com/sgl-project/sglang/actions/runs/31635016282) ",
"url": "https://github.com/sgl-project/sglang/pull/30808",
"createdAt": "2026-07-10T20:30:40Z",
"updatedAt": "2026-08-12T22:22:42Z",
"timestamp": "2026-08-12T22:22:42Z",
"metrics": {
"reactions": 0,
"comments": 16
},
"labels": [
"documentation",
"amd",
"run-ci"
],
"author": "Raiden-Makoto",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30827",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat: add cache salt support to KV cache events",
"text": "## Motivation External KV-aware routers need request cache namespaces to remain distinct from SGLang's caller-defined `extra_key`. Concatenating the two values can collide, and the existing KV-event hashes and payloads do not expose a typed cache namespace. ## Modifications - Add an explicit scalar/batched `cache_salt` contract to `Engine.generate`, `Engine.async_generate`, request normalization, tokenization, scheduling, sessions, and the supported Python radix-cache variants. - Keep `cache_salt` separate from `extra_key` and index salted radix keys with the structured pair `(extra_key, cache_salt)` while preserving existing unsalted key behavior. - Namespace salted external KV-event hashes by seeding the root chain with `SHA256(b\"sglang-cache-salt-v1\\\\0\" + cache_salt.encode())`; internal cache hashes and unsalted event hashes remain unchanged. - Append typed `{ \"cache_salt\": \"...\" }` metadata to salted `BlockStored` events while preserving the legacy unsalted positional wire layout. - Add unit coverage for normalization, OpenAI forwarding, collision resistance, radix isolation, event hashing/removal, and msgspec wire compatibility. Cache-salt-plus-LoRA routing semantics are outside this PR. ## Accuracy Tests Not applicable; this does not modify model computation or generated outputs. ## Speed Tests and Profiling Not run. Unsalted radix keys and KV-event hashes retain their existing paths. Namespace hashing is limited to salted requests with KV events enabled. ## Validation - `python3 -m compileall -q python/sglang/srt ...` for all changed source and test modules - Black 26.1.0, isort 7.0.0, and Ruff 0.15.1 checks on all changed Python files - Direct msgspec checks for legacy 7-slot unsalted events, typed 8-slot salted events, and existing typed `KVEventBatch` consumers - Full SGLang unit/GPU suites were not available in this local checkout; the focused tests are included for CI ## Checklist - [x] Format the changed code with the repository-pinned tools. - [x] Add focused unit tests. - [x] No documentation update is required for this internal routing/event contract. - [x] Accuracy and speed benchmarks are not applicable to model computation. - [x] Follow the SGLang code style guidance. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31618650333](https://github.com/sgl-project/sglang/actions/runs/31618650333) Latest PR Test (Extra): :x: [Run #31618650207](https://github.com/sgl-project/sglang/actions/runs/31618650207) ",
"url": "https://github.com/sgl-project/sglang/pull/30827",
"createdAt": "2026-07-10T22:36:11Z",
"updatedAt": "2026-08-12T23:14:05Z",
"timestamp": "2026-08-12T23:14:05Z",
"metrics": {
"reactions": 0,
"comments": 30
},
"labels": [
"run-ci"
],
"author": "jthomson04",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30929",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support decode radix cache on DeepSeek-V4 (hybrid-SWA, SWA-tail prealloc)",
"text": "## Motivation `--disaggregation-decode-enable-radix-cache` is unconditionally rejected for **every** hybrid-SWA model in `build_kv_cache`: ``` ValueError: --disaggregation-decode-enable-radix-cache is incompatible with sliding window attention (SWA) models ``` But `disaggregation/decode.py` already implements a full **SWA-tail-prealloc decode-radix path** (`DecodeTransferQueue._uses_swa_tail_prealloc` → `_swa_tail_len` / `alloc_extend_swa_tail`) for pools whose allocator exposes `alloc_extend_swa_tail` with `page_size > 1` (e.g. `SWAKVPool` / `DeepSeekV4TokenToKVPool`): decode receives the full KV for full-attention layers and only the sliding-window tail for SWA layers, which is compatible with the prefix-match-and-lock allocation path. The blanket guard is stale — even `server_args` help documents decode radix as incompatible only with `--enable-hisparse`, speculative decoding, and the fake transfer backend, **not SWA**. As a result, decode-side prefix reuse is unavailable on DeepSeek-V4 and other SWA-tail-capable models, forcing redundant KV transfers. ## Modifications 1. **`mem_cache/kv_cache_builder.build_kv_cache`** — only reject SWA models whose allocator lacks the SWA-tail-prealloc capability (`hasattr(allocator, \"alloc_extend_swa_tail\") and page_size > 1`). Mamba/SSM stays rejected. 2. **`mem_cache/unified_radix_cache.UnifiedRadixCache.cache_unfinished_req`** — fix a fatal assert on hybrid-SWA decode radix. `match_prefix()` gates device indices on **all** components, so the SWA validator collapses `device_indices` to length 0 at the first out-of-window tombstone for a long prefix whose full-attention KV is entirely device-resident (`new_prefix_len=, len(new_indices)=0`). Re-point the still-generating request onto the ungated **FULL-component** indices via a read-only walk of the just-inserted path, and protect the deepest full node from eviction. SWA reuse stays correctly window-gated everywhere else. ## Validation Tested on **DeepSeek-V4** (hybrid-SWA, `page_size=256`, `--attention-backend dsv4`) in a 1P1D PD-disaggregated (mori backend) agentic sweep on MI355X: - The startup guard clears; no `cache_unfinished_req` assert fires. - Decode server serves with **~93–96% decode-side prefix-cache hit** and **0 request errors** over the full profiling window. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [x] Provide a description and validation for this change. - [ ] (Maintainers) Run/extend CI for the decode-radix + SWA path. --- ### CI States Latest PR Test (Base): :x: [Run #31696067768](https://github.com/sgl-project/sglang/actions/runs/31696067768) Latest PR Test (Extra): :x: [Run #31696067419](https://github.com/sgl-project/sglang/actions/runs/31696067419) ",
"url": "https://github.com/sgl-project/sglang/pull/30929",
"createdAt": "2026-07-12T10:48:56Z",
"updatedAt": "2026-08-13T11:43:53Z",
"timestamp": "2026-08-13T11:43:53Z",
"metrics": {
"reactions": 2,
"comments": 3
},
"labels": [
"hicache"
],
"author": "AMD-yanfeiwang",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30972",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: Add support for using MXFP8 datatype in flashinfer A2A communicator",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31379963085](https://github.com/sgl-project/sglang/actions/runs/31379963085) Latest PR Test (Extra): :x: [Run #31379963433](https://github.com/sgl-project/sglang/actions/runs/31379963433) ",
"url": "https://github.com/sgl-project/sglang/pull/30972",
"createdAt": "2026-07-13T04:54:27Z",
"updatedAt": "2026-08-12T22:25:30Z",
"timestamp": "2026-08-12T22:25:30Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"quant",
"sgl-kernel"
],
"author": "djns99",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30984",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [Docker] Upgrade Python 3.12 + torch 2.11 + triton 3.7 in ROCm 7.2.4",
"text": "## Motivation Add ROCm 7.2.4 Docker flavors on Python 3.12 with PyTorch 2.11 and Triton 3.7. PyTorch 2.11 for ROCm 7.2 is available from the PyTorch Foundation index. Its dependency initially installs `triton-rocm==3.6.0`, but this PR replaces it at the end of the build with AITER’s pinned Triton 3.7. Installing Triton last prevents later dependency resolution from reverting the validated ROCm stack. | Component | ROCm 7.2.0 flavors | ROCm 7.2.4 flavors | | --- | --- | --- | | Base image | Ubuntu 22.04 / Python 3.10 | Ubuntu 24.04 / Python 3.12 | | PyTorch | 2.9.1 | 2.11.0+rocm7.2 | | Triton | AITER-managed Triton | AITER-managed Triton 3.7 | | AITER | `d9e5ef7` | `d9e5ef7` | ## Modifications ### `docker/rocm.Dockerfile` - Add `gfx942-rocm724` and `gfx950-rocm724` stages. - Upgrade ROCm 7.2.4 to Python 3.12 and torch 2.11. - Build AITER against the installed torch 2.11 ABI. - Install AITER’s pinned Triton 3.7 as the final Python dependency step, replacing the temporary Triton 3.6 dependency pulled by torch. - Validate that torch remains a ROCm build and report the installed Triton and `triton-kernels` versions. - Use AITER commit `d9e5ef7` consistently across ROCm 7.2.0 and 7.2.4. - Apply the `torch.Stream` compatibility workaround to both ROCm 7.2 variants. - Preserve the ROCm IPC memory-leak workaround through `HSA_ENABLE_IPC_MODE_LEGACY=1`. ### `python/pyproject_other.toml` - Add ROCm 7.2.4 extras compatible with torch 2.11. - Use `compressed-tensors==0.16.0`, because 0.15.0 requires `torch<2.11`. ### CI - Reuse the existing ROCm 7.2 PR workflow instead of adding a duplicated ROCm 7.2.4 workflow. - Add a `rocm_version` input, with ROCm 7.2.4 as the default and ROCm 7.2.0 retained for fallback testing. - Extend the existing nightly and tagged-release workflows to build either ROCm 7.2.0 or 7.2.4. - When CI rebuilds AITER on ROCm 7.2.x, run AITER’s Triton installer first and fail closed if installation fails. - Preserve `amd_ci_image` overrides so candidate images can be tested before publication. - This consolidation reduced the proposed additions from 1,921 to 345 lines—an 82% reduction. ## Accuracy Tests Both new stages were built and evaluated on their target hardware. ### `gfx950-rocm724` on MI355X Built with `--build-arg GPU_ARCH=gfx950-rocm724` and ran the in-repo MI35x accuracy evals. Hardware-limited to 2 GPUs, so tensor-parallel = 2 and MXFP4 checkpoints are used for the large models. GSM8K unless noted, on a 200-question slice for turnaround. | Model | Test | TP | Score | Threshold | Result | | --- | --- | --- | --- | --- | --- | | openai/gpt-oss-20b | `accuracy/mi35x/test_gpt_oss_eval_mi35x.py` | 2 | 0.555 | 0.47 | ✅ PASS | | openai/gpt-oss-120b | `accuracy/mi35x/test_gpt_oss_eval_mi35x.py` | 2 | 0.860 | 0.79 | ✅ PASS | | DeepSeek-R1-MXFP4 | `accuracy/mi35x/test_deepseek_r1_mxfp4_tp2_mi35x.py` | 2 | 0.955 | 0.93 | ✅ PASS | | Qwen3.5-397B-A17B-MXFP4 | `test_qwen3p5_triton_dcp.py` | 2 | 0.930 | 0.90 | ✅ PASS | The same set was re-run against an `ENABLE_MORI=1` build of the same stage, and all four remained above threshold. MORI does build on the Python 3.12 / noble base, so the nightly and release workflows enable it as they do for the other flavors. Notes on the 2-GPU adaptation (registered test files unchanged; edits applied only to throwaway in-container copies): - gpt-oss: ran with `--tp 2` (test default is 8) and `--disable-custom-all-reduce`. - DeepSeek-R1-MXFP4: the `..._tp2_...` test is already tp=2; pointed at a local MXFP4 checkpoint. - Qwen3.5-397B-A17B-MXFP4: the registered test is tp=8 + dcp=2; reduced to tp=2 / dcp=1 and pointed at a local MXFP4 checkpoint. ### `gfx942-rocm724` on MI300X Built with `--build-arg GPU_ARCH=gfx942-rocm724` and ran the gpt-oss evals on 8 GPUs, unmodified. | Model | Test | TP | Score | Threshold | Result | | --- | --- | --- | --- | --- | --- | | lmsys/gpt-oss-20b-bf16 | `accuracy/mi30x/test_gpt_oss_eval_amd.py` | 8 | 0.505 | 0.45 | ✅ PASS | | lmsys/gpt-oss-120b-bf16 | `accuracy/mi30x/test_gpt_oss_eval_amd.py` | 8 | 0.875 | 0.75 | ✅ PASS | All test scripts live under `test/registered/amd/`. ## Speed Tests and Profiling N/A — Docker image/dependency change only; no kernel or model forward code is modified. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31723276916](https://github.com/sgl-project/sglang/actions/runs/31723276916) Latest PR Test (Extra): :x: [Run #31723276693](https://github.com/sgl-project/sglang/actions/runs/31723276693) ",
"url": "https://github.com/sgl-project/sglang/pull/30984",
"createdAt": "2026-07-13T06:25:57Z",
"updatedAt": "2026-08-13T16:58:18Z",
"timestamp": "2026-08-13T16:58:18Z",
"metrics": {
"reactions": 0,
"comments": 15
},
"labels": [
"amd",
"dependencies",
"jit-kernel"
],
"author": "chuyeh",
"state": "open",
"assignees": [
"sogalin",
"bingxche",
"yctseng0211"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31031",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Intel GPU] Upgrade pytorch xpu version to 2.13",
"text": "Align toml file ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31664875494](https://github.com/sgl-project/sglang/actions/runs/31664875494) Latest PR Test (Extra): :x: [Run #31664875325](https://github.com/sgl-project/sglang/actions/runs/31664875325) ",
"url": "https://github.com/sgl-project/sglang/pull/31031",
"createdAt": "2026-07-13T12:57:10Z",
"updatedAt": "2026-08-13T03:46:17Z",
"timestamp": "2026-08-13T03:46:17Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"documentation",
"dependencies"
],
"author": "polisettyvarma",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31172",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[GDN] Hoist GDN extend preparation across layers",
"text": " ## Motivation https://github.com/sgl-project/sglang/pull/30797 follow up. GDN prefill currently rebuilds layer-invariant metadata for every transformer layer. In particular, each layer recomputes whether requests have initial states and, for FlashInfer, normalizes SSM cache indices and sequence-offset dtypes even though these values are fixed for the whole forward. Preparing this metadata once per forward avoids repeating that work across model depth. ## Modifications - Add a backend-neutral extend-preparation hook and cache one atomic GDN extend context per forward. - Have the FlashInfer GDN kernel prepare reusable cache indices and sequence offsets while preserving the existing SM90 and SM100 sentinel and dtype behavior. Direct kernel callers retain the existing fallback behavior. - Leave Triton and CuTeDSL behavior unchanged through the default no-op preparation hook (future work). - Extend the registered H100/B200 FlashInfer GDN tests to cover the attention block numerically, architecture-specific preparation, reuse across two distinct layer caches, and rebuilding on the next forward. ## Accuracy Tests main: ```bash == running GSM8K (200 questions, 128-way, 5-shot) == /root/dev/sglang/python/sglang/test/few_shot_gsm8k.py:166: DeprecationWarning: sglang.test.few_shot_gsm8k is deprecated. Use sglang.test.run_eval with eval_name='gsm8k' instead. run_eval(args) 100%|██████████| 200/200 [00:33<00:00, 5.93it/s] Accuracy: 0.965 ``` PR: ```bash == running GSM8K (200 questions, 128-way, 5-shot) == /root/dev/sglang/python/sglang/test/few_shot_gsm8k.py:166: DeprecationWarning: sglang.test.few_shot_gsm8k is deprecated. Use sglang.test.run_eval with eval_name='gsm8k' instead. run_eval(args) 100%|██████████| 200/200 [00:33<00:00, 5.96it/s] Accuracy: 0.965 ``` ## Speed Tests and Profiling All these 3 kernels are removed from each linear attention layer: ### Isolated prefill step Measured with `sglang.benchmark.one_batch` (Qwen3.5-397B-A17B-FP8, TP4, B200, `--linear-attn-prefill-backend flashinfer --cuda-graph-backend-prefill=breakable`, batch size 1, 4 measured reps per point). Baseline is `main` @ `b129e8a29`. | prefill ISL | main | this PR | delta | 95% CI | absolute | |---|---|---|---|---|---| | 1k | 0.03454 s | 0.03389 s | **-1.89%** | [-2.42%, -1.27%] | 0.652 ms | | 2k | 0.05120 s | 0.05050 s | **-1.36%** | [-1.67%, -1.07%] | 0.695 ms | | 4k | 0.07980 s | 0.07919 s | **-0.76%** | [-0.83%, -0.68%] | 0.605 ms | | 8k | 0.14304 s | 0.14238 s | **-0.45%** | [-0.56%, -0.35%] | 0.650 ms | Every point's 95% CI (bootstrap on the mean) excludes zero. The column to read is the **absolute** one: the saving is flat at **~0.65 ms/forward across an 8x token range**. That is the expected shape for this change — hoisting layer-invariant metadata removes a fixed amount of per-forward host work (the per-layer `clamp`/dtype conversions for `ssm_cache_indices` and `cu_seqlens`, now done once instead of once per GDN layer), independent of how many tokens flow through it. The percentage varies only because the denominator does, so the effect is largest at short ISL and shrinks as prefill becomes more GPU-bound. Methodology: the first rep at each new input shape is discarded as capture-bucket/autotune warmup (e.g. 4k: 0.107 s on the first forward vs 0.0798 s steady-state). Including it inflates the standard deviation from ~0.05 ms to ~12 ms and leaves the 4k/8k points unresolvable. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31007005071](https://github.com/sgl-project/sglang/actions/runs/31007005071) Latest PR Test (Extra): :x: [Run #31007004651](https://github.com/sgl-project/sglang/actions/runs/31007004651) ",
"url": "https://github.com/sgl-project/sglang/pull/31172",
"createdAt": "2026-07-14T08:44:57Z",
"updatedAt": "2026-08-13T07:53:11Z",
"timestamp": "2026-08-13T07:53:11Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [
"run-ci"
],
"author": "mattteochen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31193",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "(fix) Prevent unknown HTTP paths from overloading Prometheus metrics generation",
"text": "## Motivation This change prevents unbounded Prometheus cardinality growth caused by arbitrary unknown HTTP paths. One real-world example is vulnerability scanners, which can probe tens of thousands of unique random endpoints. When the raw request path is used as the endpoint metric label, every unique unknown path creates a new Prometheus time series for the HTTP request/response metrics. In one production incident, a vulnerability scanner generated approximately **17,700 unique unknown endpoints**, producing metric series such as: ```text sglang:http_requests_total{endpoint=\"/.../.../.../.../.../.../.../.../.../.../../../../../../../../../../etc/passwd\",method=\"GET\"} 1.0 sglang:http_requests_total{endpoint=\"/${jndi:ldap://localhost\",method=\"GET\"} 2.0 ``` This significantly inflated the Prometheus multiprocess metric files: ``` counter_1.db 8 MB gauge_livesum_1.db 4 MB histogram_1.db 64 KB ``` The `/metrics` endpoint rebuilds the Prometheus exposition on every scrape. In multiprocess mode, this means scanning and reading the *.db files, merging samples, serializing all metrics, and optionally compressing the response. This work runs in Python on the main ASGI thread and can execute under the GIL. While reproducing the issue, I continuously sent chat completion requests while sampling the SGLang process with [pyspy](https://github.com/benfred/py-spy). Across all samples, the main thread was consistently inside the Prometheus metrics generation path: for example ``` Thread 1 (active): \"MainThread\" compress (gzip.py:616) _bake_output (prometheus_client/exposition.py:116) prometheus_app (prometheus_client/asgi.py:24) handle (starlette/routing.py:448) ``` or ``` Thread 1 (active+gil): \"MainThread\" add_sample (prometheus_client/metrics_core.py:35) _read_metrics (prometheus_client/multiprocess.py:82) merge (prometheus_client/multiprocess.py:43) collect (prometheus_client/multiprocess.py:158) collect (prometheus_client/registry.py:97) generate_latest (prometheus_client/exposition.py:289) ``` Together with the enlarged multiprocess metric files, this explained why the frontend accepted HTTP connections but stopped making progress on completion requests: the main ASGI thread was repeatedly rebuilding and serializing a high-cardinality metrics payload during `/metrics` scrapes. With this change, even under vulnerability scans or other arbitrary requests, these files remain small: ``` counter_1.db 64 KB gauge_livesum_1.db 64 KB histogram_1.db 64 KB ``` Since the metrics endpoint reconstructs the Prometheus exposition by reading and merging these multiprocess files on every scrape, the increased number of time series significantly increased metrics generation cost. In our production incident, the server eventually spent a significant amount of CPU time generating /metrics, reducing its ability to serve completion requests. ## Modifications Unhandled FastAPI paths are now reported as `endpoint=\"__unhandled__\"` instead of using the raw request path. Known routes continue to use their normalized route paths, preserving the existing metrics for handled endpoints while preventing unbounded cardinality for unknown paths. ## Accuracy Tests ## Speed Tests and Profiling Not applicable. This PR only changes HTTP metrics path labeling for unhandled FastAPI routes and does not affect model outputs, kernels, or model forward code. --- ### CI States Latest PR Test (Base): :x: [Run #31628343531](https://github.com/sgl-project/sglang/actions/runs/31628343531) Latest PR Test (Extra): :x: [Run #31628343324](https://github.com/sgl-project/sglang/actions/runs/31628343324) ",
"url": "https://github.com/sgl-project/sglang/pull/31193",
"createdAt": "2026-07-14T13:20:10Z",
"updatedAt": "2026-08-12T18:35:25Z",
"timestamp": "2026-08-12T18:35:25Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [],
"author": "ntny",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31320",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] [Diffusion] support distributed inference pipeline for GLM-Image",
"text": "## Summary This PR adds GLM-Image distributed serving: the public head batches external AR generation and dispatches each AR-complete request to an independent batch-1 denoiser that runs request preparation, DiT, and VAE decoding. ## Modifications - Add GLM-Image distributed mode for a head configured with `--srt-encoder-url` and `--denoiser-urls`, with encoder and decoder URL lists omitted. - Group compatible text-to-image requests by resolution and send one batched `/generate` call to the external AR server. Batch capacity is measured in output slots, consistent with #30683. - Run the blocking external AR request on a dedicated executor so AR generation overlaps with active denoiser work. - Assign generated prior tokens and usage to each request, then dispatch AR-complete requests across connected denoisers with one active request per worker. - Run glyph/text preparation, sequential batch-1 DiT inference, and VAE decoding on each distributed denoiser. `num_outputs_per_prompt > 1` is processed sequentially and regrouped into the original response. - Transfer only request metadata and CPU prior-token IDs over ZMQ; this mode bypasses Mooncake/RDMA and does not transfer latent or embedding tensors. - Return each completed request immediately to the head, which preserves client ordering, seeds, output paths, output saving, and image URL registration. - Track denoiser connectivity, stop dispatching to disconnected workers, and reclaim request state and worker capacity after timeouts. - Preserve the existing encoder → denoiser → decoder disaggregation path when GLM distributed mode is not enabled. - Validate per-role TP/SP/CFG/DP device requirements and disable only auto-enabled CFG parallelism when incompatible with the worker device count. - Document the topology, deployment commands, limitations, and measured NVIDIA and Ascend results. ## Deployment ### Run 14 independent Cache-DiT denoisers ~~~bash #!/bin/bash DISAGG_SERVER=\"tcp://127.0.0.1:19655\" MODEL_PATH=\"zai-org/GLM-Image\" BASE_MASTER_PORT=29005 export SGLANG_CACHE_DIT_FN=2 export SGLANG_CACHE_DIT_BN=1 export SGLANG_CACHE_DIT_WARMUP=4 export SGLANG_CACHE_DIT_RDT=0.4 export SGLANG_CACHE_DIT_MC=4 export SGLANG_CACHE_DIT_TAYLORSEER=true export SGLANG_CACHE_DIT_TS_ORDER=2 export SGLANG_CACHE_DIT_ENABLED=true for i in $(seq 2 15); do scheduler_port=$((19001 + i)) master_port=$((BASE_MASTER_PORT + i)) sglang serve \\ --model-path \"$MODEL_PATH\" \\ --disagg-role denoiser \\ --disagg-server-addr \"$DISAGG_SERVER\" \\ --srt-encoder-url http://127.0.0.1:30020 \\ --scheduler-port \"$scheduler_port\" \\ --master-port \"$master_port\" \\ --num-gpus 1 \\ --base-gpu-id \"$i\" \\ --denoiser-sp 1 \\ --cfg-parallel-size 1 \\ --batching-max-size 1 \\ --attention-backend fa & done wait ~~~ ### Run the AR server ~~~bash sglang serve \\ --model-path zai-org/GLM-Image/vision_language_encoder/ \\ --tokenizer-path zai-org/GLM-Image/processor/ \\ --enable-multimodal \\ --cuda-graph-max-bs 28 \\ --device npu \\ --attention-backend ascend \\ --disable-fast-image-processor \\ --tp-size 2 \\ --host 0.0.0.0 \\ --port 30020 \\ --mem-fraction-static 0.8 ~~~ ### Run the public head ~~~bash sglang serve \\ --model-path zai-org/GLM-Image \\ --disagg-role server \\ --srt-encoder-url http://127.0.0.1:30020 \\ --srt-encoder-timeout 300 \\ --denoiser-urls \"tcp://127.0.0.1:19003;tcp://127.0.0.1:19004;tcp://127.0.0.1:19005;tcp://127.0.0.1:19006;tcp://127.0.0.1:19007;tcp://127.0.0.1:19008;tcp://127.0.0.1:19009;tcp://127.0.0.1:19010;tcp://127.0.0.1:19011;tcp://127.0.0.1:19012;tcp://127.0.0.1:19013;tcp://127.0.0.1:19014;tcp://127.0.0.1:19015;tcp://127.0.0.1:19016\" \\ --batching-mode dynamic \\ --batching-max-size 28 \\ --batching-delay-ms 30 \\ --enable-batching-metrics \\ --host 0.0.0.0 \\ --port 30052 \\ --scheduler-port 19655 \\ --output-path ./outputs ~~~ ### Run the benchmark Tested on [longtext-bench.zip](https://github.com/user-attachments/files/29779516/longtext-bench.zip). ~~~bash python fetch_images.py \\ --base-url http://localhost:30052/v1 \\ --model GLM-Image-distributed-test \\ --output-dir generated_images \\ --max-concurrency 56 ~~~ ## Limitations - GLM distributed mode is text-to-image only; non-string prompts and image-conditioned requests are rejected. - In-flight jobs are not replayed if a denoiser disconnects. - Distributed-denoiser `RequestMetrics` are not propagated back to the head; median request E2E latency and peak memory below are reconstructed from server logs. ## Results All runs use 1280×1280 output, 30 denoising steps, and Cache-DiT. Configurations are ordered from lowest to highest throughput. The tables retain the sustained-throughput and batch-relative latency values from the original performance summary. Median request E2E latency and peak memory are reconstructed from server logs after excluding the initial BS2 warm-up where applicable. Median request E2E is measured from request arrival. Batch-relative latency is measured from batch/AR dispatch and excludes pre-dispatch waiting; for non-batched configurations, it is approximately the request latency. ### 8-GPU NVIDIA comparison | Metric | No Batch + No External AR *(baseline)* | No Batch + External AR | Monolithic Batch (8 GPUs) + External AR | Disaggregated (7 denoisers) + External AR | |---|---:|---:|---:|---:| | **Sustained throughput (img/s)** | **0.018** | **0.09** | **0.24** | **0.37** | | **Throughput improvement (baseline = 1×)** | **1.0×** | **5.0×** | **13.3×** | **20.6×** | | Median request E2E latency (s) | 55.22 | 10.83 | 134.30 | 72.07 | | Batch-relative latency (s) | ~56 | ~11 | ~80 | ~55 | | Latency improvement | 1.0× | **5.1×** | *0.7×* | ~1.0× | | Peak GPU memory (MB) | 46,698 | 27,296 | 27,982 | 28,046 per denoiser | - Using these sustained-throughput values, seven-denoiser disaggregation is 1.54× faster than monolithic batching and 20.6× faster than the local-AR baseline. ### 16-NPU Ascend comparison | Metric | No Batch + Local AR *(baseline)* | No Batch + External AR | Monolithic Batch (28) + External AR | Disaggregated (7 denoisers) + External AR | Disaggregated (14 denoisers) + External AR | |---|---:|---:|---:|---:|---:| | **Sustained throughput (img/s)** | **0.0087** | **0.0435** | **0.20** | **0.41** | **0.74** | | **Throughput improvement (baseline = 1×)** | **1.00×** | **5.0×** | **23.0×** | **47.1×** | **85.1×** | | Median request E2E latency (s) | 122.18 | 22.96 | 141.82 | 69.57 | 73.56 | | Batch-relative latency (s) | ~115 | ~23 | ~90 | ~52 | ~37 | | Latency improvement | 1.00× | **5.0×** | 1.28× | 2.2× | **3.1×** | | Peak NPU memory (MB) | 48,228 | 28,302 | 28,332 | 28,316 per denoiser | 28,566 per denoiser | - Using these sustained-throughput values, 14-denoiser disaggregation is 3.70× faster than monolithic batching, 1.80× faster than seven-denoiser disaggregation, and 85.1× faster than the local-AR baseline. - The 14-denoiser request E2E latency range is 57.25–129.25 s; median request E2E latency is 73.56 s. ### Methodology - The initial BS2 client warm-up is excluded from every affected median request E2E calculation. - Sustained throughput and batch-relative latency values are retained from the original performance summary. - For batched configurations, batch-relative latency is measured from batch/AR dispatch; median request E2E is measured from request arrival and includes pre-dispatch waiting. - Incomplete trailing BS28 groups are excluded from server-log latency calculations. - Peak memory is the runtime-reported diffusion process or denoiser value; it is not aggregate cluster memory. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://github.com/sgl-project/sglang/blob/main/CONTRIBUTING.md#code-style). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from CODEOWNERS and other reviewers. 3. Trigger CI tests with comments or contact authorized users to do so. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31715006037](https://github.com/sgl-project/sglang/actions/runs/31715006037) Latest PR Test (Extra): :x: [Run #31715005390](https://github.com/sgl-project/sglang/actions/runs/31715005390) ",
"url": "https://github.com/sgl-project/sglang/pull/31320",
"createdAt": "2026-07-15T10:41:56Z",
"updatedAt": "2026-08-13T15:22:54Z",
"timestamp": "2026-08-13T15:22:54Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"documentation",
"npu",
"diffusion"
],
"author": "OrangeRedeng",
"state": "open",
"assignees": [
"ping1jing2"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31324",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [GLM5] Skip DSA decode indexer when kv_len <= index_topk (dense k-only fast path)",
"text": "## Summary - On GLM-5.2 DSA decode, when a request's `kv_len <= index_topk` the top-k selects **all** valid positions, so the indexer's logits GEMM + `paged_mqa_logits` + top-k selection is wasted work. Add a **k-only** fast path that skips the indexer, stores the K cache, and generates the identity index directly (`[0, 1, ..., kv_len-1, -1, ...]`), feeding the same sparse-MLA decode attention kernel. - **CUDA-graph \"Design A\" dual-graph:** capture a dense (k-only) and a sparse (full indexer) decode graph per batch-size bucket and dispatch on `max_kv_len` vs `index_topk` at replay. This is correct for mixed lengths — any request with `kv_len > index_topk` in the batch falls back to the sparse graph. - **Auto-enabled for DSA models, no env toggle.** Since DSA models expose `index_topk` in their HF config, the dual-graph is turned on automatically whenever `index_topk` is present (`dsa_dual_graph = dsa_index_topk is not None`); non-DSA models and other archs are unaffected. Eager decode takes the same fast path per-step when safe. (No `SGLANG_DSA_DECODE_*` / `SGLANG_KONLY_DEBUG` environment variables — an earlier revision gated this behind opt-in/debug flags; those were removed per review.) ## Changes | File | Change | | --- | --- | | layers/attention/dsa/dsa_indexer.py | k-only fast path (`_forward_cuda_k_only`): skip logits/top-k, store K cache; for MLA generate the identity index; extended to DECODE (eager + graph) | | model_executor/runner/decode_cuda_graph_runner.py | Design A dual-graph: capture dense (k-only) + sparse decode graphs per bs bucket; host-dispatch on `max_kv_len` vs `index_topk`; auto-enabled for DSA models via `dsa_index_topk` | | model_executor/runner_utils/capture_mode.py | capture-time DSA variant flag (dense/sparse) read by the indexer skip branch | | model_executor/runner/shape_key.py | include the DSA decode variant in the cuda-graph shape key | ## Scope GLM-5.2 DSA decode on gfx950 (MI355X). Auto-enabled for DSA models; correct for mixed lengths (long context stays on the sparse indexer path). Other archs / hardware unaffected. ## Test plan **Accuracy (GSM8K, MI355X TP4, GLM-5.2-MXFP4):** | | Baseline | This PR | | --- | --- | --- | | GSM8K | 0.922 | 0.941 | Within margin of error. **Performance (MI355X TP4, GLM-5.2-MXFP4, docker `rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260714`, tilelang DSA backend):** token throughput per GPU (tok/s/gpu, higher better) and median TPOT (ms, lower better), dense-decode OFF vs ON (this PR). Isolated effect — same build/backend, only this PR's commits added, everything else identical. Only **i1024 / o1024** is shown: this optimization applies when `kv_len <= index_topk` (2048). At i8192 every decode step has `kv_len > 2048`, so it stays on the sparse path and is unchanged. | Concurrency | TPUT off | TPUT PR | Δ TPUT | TPOT off | TPOT PR | Δ TPOT | | --- | --- | --- | --- | --- | --- | --- | | 4 | 71.8 | 75.5 | +5.2% | 13.28 | 12.62 | −5.0% | | 8 | 126.8 | 133.2 | +5.1% | 15.18 | 14.41 | −5.1% | | 16 | 196.9 | 205.1 | +4.2% | 19.46 | 18.67 | −4.1% | | 32 | 312.4 | 322.8 | +3.3% | 24.66 | 23.90 | −3.1% | | 64 | 468.9 | 480.3 | +2.4% | 32.88 | 32.14 | −2.3% | ``` python3 -m sglang.launch_server \\ --model amd/GLM-5.2-MXFP4 \\ --tp 4 \\ --trust-remote-code \\ --tool-call-parser glm47 \\ --reasoning-parser glm45 \\ --watchdog-timeout 1200 \\ --mem-fraction-static 0.85 \\ --kv-cache-dtype fp8_e4m3 \\ --disable-radix-cache \\ --model-loader-extra-config '{\"enable_multithread_load\": true, \"num_threads\": 32}' \\ --dsa-prefill-backend tilelang \\ --dsa-decode-backend tilelang \\ --tokenizer-worker-num 8 \\ --enable-aiter-allreduce-fusion ``` Token throughput per GPU improves ~2.4–5.2% and TPOT drops ~2.3–5.1% at i1k, from skipping the wasted indexer when the sparse top-k would select all positions anyway. The gain is largest at low concurrency and converges to ~2.4% at conc 64. (Δ is the isolated dense-decode effect — same build/backend, feature off vs on — not a full-stack-vs-stock comparison.) --- ### CI States Latest PR Test (Base): :no_entry_sign: [Run #31655055850](https://github.com/sgl-project/sglang/actions/runs/31655055850) Latest PR Test (Extra): :x: [Run #31655055641](https://github.com/sgl-project/sglang/actions/runs/31655055641) ",
"url": "https://github.com/sgl-project/sglang/pull/31324",
"createdAt": "2026-07-15T11:13:56Z",
"updatedAt": "2026-08-13T05:20:31Z",
"timestamp": "2026-08-13T05:20:31Z",
"metrics": {
"reactions": 1,
"comments": 12
},
"labels": [
"amd",
"run-ci"
],
"author": "Jacob0226",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31362",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Speculative Decoding with NGRAM support for XPU",
"text": "This PR adds support for NGRAM on XPU target. JH...! --- ### CI States Latest PR Test (Base): :x: [Run #31673353486](https://github.com/sgl-project/sglang/actions/runs/31673353486) Latest PR Test (Extra): :x: [Run #31673353388](https://github.com/sgl-project/sglang/actions/runs/31673353388) ",
"url": "https://github.com/sgl-project/sglang/pull/31362",
"createdAt": "2026-07-15T18:28:59Z",
"updatedAt": "2026-08-13T06:23:10Z",
"timestamp": "2026-08-13T06:23:10Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"speculative-decoding",
"jit-kernel"
],
"author": "ANSHUMAN87",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31443",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache]: Optimize hybrid/DSA L3 prefetch result sync and usable-prefix clamping",
"text": " ## Motivation Fix: https://github.com/sgl-project/sglang/issues/30321 Thanks @cfbdsirlijun-maker 、[AlejandroParedesLT](https://github.com/AlejandroParedesLT) for reporting the issue. Thanks @stepinto for reproducing the issue: https://github.com/sgl-project/sglang/pull/31348 ## Modifications - Extracted a helper for the HiCache prefetch completion logic: _sync_and_check_hybrid_prefetch_result (Unified) / _sync_and_clamp_prefetch_result (HiRadix), pulling the cross-ATTN-group sync + usable-prefix decision out of check_prefetch_progress. - Added DSA-style clamp in Unified: for Full + KV-derived ALL_PAGES sidecars (DSA/MiniMax indexer), clamp the usable prefix to min(Full KV completion, each sidecar hit) instead of discarding; SWA/Mamba/DeepSeekV4 keep all-or-nothing. - Fixed PoolTransferResult.update_extra_pool_hit_pages: count the leading contiguous run of successes (rs.index(False)) instead of summing, so a gap in the middle of a fetch can't be over-counted as a longer safe prefix. - ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #29657971984](https://github.com/sgl-project/sglang/actions/runs/29657971984) Latest PR Test (Extra): :white_check_mark: [Run #29658092089](https://github.com/sgl-project/sglang/actions/runs/29658092089) ",
"url": "https://github.com/sgl-project/sglang/pull/31443",
"createdAt": "2026-07-16T08:45:58Z",
"updatedAt": "2026-08-13T11:55:58Z",
"timestamp": "2026-08-13T11:55:58Z",
"metrics": {
"reactions": 1,
"comments": 20
},
"labels": [
"high priority",
"hicache",
"run-ci",
"run-ci-extra"
],
"author": "hzh0425",
"state": "closed",
"assignees": [
"xiezhq-hermann",
"ispobock"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31447",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "perf: fix overlap scheduling and all-reduce fusion for NVIDIA Confidential Computing(CC) on Blackwell",
"text": "## Motivation Two fixes so SGLang runs efficiently under **NVIDIA Confidential Computing (CC)**, where the GPU operates behind a bounce buffer: 1. **Per-step D2H result readback stalls the scheduler.** Under CC, `cudaMemcpyAsync` (D2H) is forced synchronous — it blocks at issue — so the per-step readback of `next_token_ids` / logprobs serializes onto the scheduler's critical path and kills decode overlap. 2. **FlashInfer AR+RMSNorm fusion disables itself.** `create_allreduce_fusion_workspace` allocates a symmetric-memory (cuMulticast) workspace whose preflight fails under CC, so the fusion silently falls back off — even though the fusion kernels themselves are multicast-free. Both paths are gated on `is_confidential_compute()` (NVML, overridable via `SGLANG_CONFIDENTIAL_COMPUTE`); **off-CC behavior is unchanged.** ## Modifications **`cc: async device->host result-readback worker`** - New `python/sglang/srt/managers/async_d2h_copy_worker.py` (`AsyncD2HCopyWorker`): runs the per-step D2H copy on a dedicated thread with its own CUDA stream, off the scheduler's critical path, so overlap is preserved. Wired into `scheduler.py` / `managers/utils.py`; only routed through the worker when CC is detected. - Adds `is_confidential_compute()` to `srt/utils/common.py`. **`cc(fix): enable FlashInfer AR+RMSNorm fusion under Confidential Computing`** (`layers/flashinfer_comm_fusion.py`) - FlashInfer now auto-selects a multicast-free IPC workspace under CC, so SGLang reuses the normal off-CC fusion path: `_resolve_backend` forces the `trtllm` backend under CC (SM100 \"auto\" defaults to `mnnvl`, which needs NVLink multicast), the symmetric-memory preflight is skipped, and the workspace stays a normal `AllReduceFusionWorkspace`. Enabled by default whenever CC is detected. ## Accuracy Tests No model-output change: the AR+RMSNorm fusion produces the same result as the off-CC path (same kernels, only the workspace allocator differs), and the D2H worker only relocates the copy, not its contents. Off-CC paths are untouched. ## Speed Tests and Profiling Intent is to *restore* CC performance to the off-CC baseline — preserve decode overlap and keep AR+RMSNorm fusion enabled under CC. CC-hardware e2e benchmarks pending. `test/registered/core/test_async_d2h_copy_worker.py` validates the worker on a single GPU: correctness off-thread (tiny + vocab-sized tensors), many sequential readbacks, error re-raise, idempotent shutdown, and the CC-detection env override. ## Checklist - [x] Format code with pre-commit (all hooks pass). - [x] Added unit tests (`test/registered/core/test_async_d2h_copy_worker.py`). - [ ] Documentation update — n/a. - [ ] Accuracy/speed benchmark — n/a (no output change; CC-hardware benchmark pending). - [x] Follow the SGLang code style guidance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31664799631](https://github.com/sgl-project/sglang/actions/runs/31664799631) Latest PR Test (Extra): :x: [Run #31664799513](https://github.com/sgl-project/sglang/actions/runs/31664799513) ",
"url": "https://github.com/sgl-project/sglang/pull/31447",
"createdAt": "2026-07-16T10:25:54Z",
"updatedAt": "2026-08-13T05:19:44Z",
"timestamp": "2026-08-13T05:19:44Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"run-ci"
],
"author": "elvischenv",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31466",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] DSpark support prefill/decode disaggregation",
"text": " ## Motivation Roadmap: https://github.com/sgl-project/sglang/issues/30344 Reference PR: #30513 This PR adds DSpark support for prefill/decode disaggregation on DeepSeek-V4. On `main`, DSpark can run in the normal non-disaggregated path, but the PD decode path does not receive the target-model hidden states required to bootstrap the draft-side DSpark state. As a result, decode may enter speculative decoding without valid DSpark `spec_info` / draft KV state. This PR fixes that gap by transferring DSpark target hidden states from Prefill to Decode through the disaggregation protocol, then injecting them into the decode-side draft KV cache before the first draft step. The implementation keeps the existing non-disaggregated DSpark behavior unchanged and scopes the new behavior to PD mode. ## What Changed Compared With main ### DSpark Hidden State PD Transfer - Add a new disaggregation state type: `DSPARK_HIDDEN`. - Let Decode describe the required DSpark hidden transfer in metadata: - target layer ids - hidden start offset - hidden length - decode radix-cache policy - PP-aware slice metadata - dynamic destination buffer information - Let Prefill capture target hidden states only on the PP rank that owns the DSpark target layers. - Transfer DSpark hidden states through Mooncake/NIXL using row-addressed dynamic destination buffers. - Trim hidden transfer windows according to the Prefill-side cached prefix while preserving absolute token offsets. - Keep fail-fast checks when required hidden rows are incomplete before transfer. ### Decode-Side Draft Bootstrap - Assemble received DSpark hidden slices on Decode before committing the transferred request. - Attach the assembled hidden rows to `prefill_tail_hidden_states`. - Build DSpark draft input from the Prefill output token as the first decode anchor. - Inject transferred target hidden states into the decode-side draft KV cache before the first draft step. ### PP-Aware Prefill Support - Add PP-aware hidden slice metadata so only the PP rank owning the target layers sends DSpark hidden states. - Propagate DSpark auxiliary hidden tensors through PP proxy outputs. - Match PP outputs by microbatch id / request identity to avoid FIFO mismatch. - Snapshot PP batch sequence lengths so delayed PP result processing does not read mutated request state. - Drain failed PP prefill bootstrap requests consistently across PP ranks. - Preserve PP admission behavior using side-effect-free resource credit probing before consensus. ### Hidden Buffer and Transfer Resource Management - Add `DSparkHiddenTransferPlan` to describe row/chunk transfer layouts. - Add decode-side `DSparkHiddenPagePool` to reuse registered GPU receive buffers. - Add optional decode hidden receive prewarm: - `SGLANG_DSPARK_PD_HIDDEN_RECV_PREWARM_ROWS` - `SGLANG_DSPARK_PD_HIDDEN_RECV_PREWARM_PAGES` - Release Prefill hidden rows as soon as the hidden transfer finishes in the Mooncake worker, instead of waiting for the full KV request success. - Prefer contiguous hidden row allocation and slice copy on Prefill. - Use `torch.empty` instead of `torch.zeros` when assembling Decode hidden tensors to avoid unnecessary clearing. ### Metadata and Speculative Decode Integration - Extend disaggregation metadata buffers to carry DSpark prefill-tail hidden state information. - Extend `DFlashDraftInputV2` / DSpark draft input to carry: - `prefill_tail_hidden_states` - `prefill_tail_valid_mask` - `prefill_tail_start_positions` - `prefill_tail_hidden_projected` - Add non-padded token count metadata needed by DSpark draft forward batches. - Ensure Decode radix-cache and Prefill radix-cache policy are aligned for DSpark hidden correctness. ### Streaming Hidden Chunk Transfer and Release Semantics - Add streaming DSpark hidden transfer for Mooncake so long prompts do not require materializing the full `prompt_len * hidden_width` tensor on Prefill or Decode. - Split DSpark hidden transfer state from KV request completion: - hidden chunk ACK only controls streaming window reuse. - hidden request done does not imply KV request success. - request success still follows the original KV / metadata success path. - Release Prefill-side hidden source rows after the hidden request is done, before full KV request completion. - Release Decode-side hidden receive rows on normal request release / abort paths. - Serialize Mooncake DSpark hidden chunks per room to preserve chunk ordering under ACK-based flow control. - Send DSpark hidden-only chunks even when the aligned KV page count is zero. - Flush pending streaming hidden chunks before writing the next chunk, preventing a later chunk from overwriting an unsent `dspark_hidden_current_*` slot. - Use offset-based streaming source row writes so multi-chunk hidden transfer preserves the absolute hidden token range. - Add fail-fast checks for source-window overwrite hazards and invalid streaming hidden ordering. - Add backend capability boundaries: - Mooncake supports streaming DSpark hidden transfer. - NIXL / MORI keep the non-streaming/default path and do not enter partially implemented streaming release semantics. ## Why These Changes Are Needed DSpark speculative decoding depends on target-model hidden states to initialize the draft-side state correctly. In PD mode, Prefill and Decode run in separate processes, so Decode cannot derive those hidden states locally. The new `DSPARK_HIDDEN` transfer path makes the hidden state an explicit part of the PD protocol, similar to KV/state transfer, but with PP-aware slicing because only specific Prefill PP ranks own the DSpark target layers. The resource probing and transactional allocation logic are needed to avoid PP rank divergence: all PP ranks must agree on which requests enter the pipeline before any rank performs side-effectful resource allocation. The dynamic registered GPU receive buffer pool avoids CPU bounce and keeps hidden transfer on the GDR path. ## Compatibility - Non-disaggregated DSpark behavior is unchanged. - Existing KV / C128 / auxiliary state transfer paths remain intact. - DSpark PD hidden transfer is activated only when Decode requests `DSPARK_HIDDEN` metadata. - Default decode hidden receive prewarm is disabled, so default memory footprint is unchanged. Current Scope and Future Work This PR supports DSpark PD hidden-state transfer for Prefill with PP. The implementation is PP-aware: each Prefill PP rank only captures and transfers the DSpark target hidden slice owned by its local layer range. Mooncake now supports streaming DSpark hidden transfer with hidden/KV/request completion semantics separated. NIXL and MORI do not yet implement streaming hidden release semantics and therefore keep the default non-streaming capability boundary. Prefill radix cache cannot be enabled alone for DSpark PD today. DSpark hidden transfer must match the KV transfer window. If Prefill radix cache is enabled but Decode radix cache is disabled, Prefill may skip cached prompt tokens and the corresponding target hidden states will be missing on Decode. Therefore, DSpark PD currently requires Prefill and Decode radix-cache policies to be consistent. Until DeepSeek-V4 Decode radix cache is supported, Prefill radix cache should stay disabled. #31097 Future work will extend this to Prefill with PP + TP + CP and streaming support for NIXL/MORI. That requires generalizing the hidden metadata from PP-only layer slicing to a full PP/TP/CP layout, including tensor-parallel hidden sharding and context-parallel token-range reconstruction on Decode. ## Accuracy Tests ``` Prefill SGLANG_PP_LAYER_PARTITION=\"6,5,5,6,5,5,6,5\" SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 NCCL_SOCKET_IFNAME=eth0 NCCL_IB_DISABLE=0 SGLANG_DSV4_FP4_EXPERTS=1 GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path /data00/models/DeepSeek-V4-Flash-DSpark --host 0.0.0.0 --port 30000 --trust-remote-code --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.8 --max-running-requests 64 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --pp-size 8 --attention-backend dsv4 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --disable-overlap-schedule --disable-piecewise-cuda-graph --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --enable-metrics --disaggregation-ib-device mlx5_1,mlx5_2,mlx5_3,mlx5_4 --moe-runner-backend flashinfer_mxfp4 --disable-radix-cache Decode SGLANG_DSPARK_PD_HIDDEN_RECV_POOL_TOKENS=65536 SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_JIT_DEEPGEMM_PRECOMPILE=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=1 SGLANG_OPT_USE_TILELANG_MHC_PRE=1 GLOO_SOCKET_IFNAME=eth0 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8 sglang serve --trust-remote-code --model-path /data00/models/DeepSeek-V4-Flash-DSpark --tp 8 --dp-size 8 --enable-dp-attention --cuda-graph-max-bs 32 --max-running-requests 256 --enable-metrics --host 0.0.0.0 --port 30000 --mem-fraction-static 0.85 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --moe-runner-backend flashinfer_mxfp4 --disaggregation-mode decode --disaggregation-ib-device \"mlx5_1,mlx5_2,mlx5_3,mlx5_4\" --speculative-algo DSPARK --tokenizer-worker-num 8 --enable-dp-lm-head --load-balance-method round_robin --swa-full-tokens-ratio 0.8 ``` Hardward | MMLU | GSM8K | QPQA | aime25 repeats 16 | status -- | -- | -- | -- | -- | -- H20 | 0.885 | 0.954 | 0.910 | 93.96 | PD ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #29816426168](https://github.com/sgl-project/sglang/actions/runs/29816426168) Latest PR Test (Extra): :x: [Run #29816426138](https://github.com/sgl-project/sglang/actions/runs/29816426138) ",
"url": "https://github.com/sgl-project/sglang/pull/31466",
"createdAt": "2026-07-16T13:27:50Z",
"updatedAt": "2026-08-13T11:09:17Z",
"timestamp": "2026-08-13T11:09:17Z",
"metrics": {
"reactions": 2,
"comments": 8
},
"labels": [
"deepseek"
],
"author": "zhangxiaolei123456",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31470",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NVIDIA] Support flashinfer Mega Moe",
"text": "Fork from https://github.com/djns99/sglang/tree/djns99/mega_moe_flashinfer @djns99 is the main author of this PR. ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. # MoE backend benchmark: `flashinfer_trtllm_routed` (baseline) vs `flashinfer_megamoe` Benchmark of the DeepSeek-V4-Flash NVFP4 model comparing two MoE backend configurations on the serving path. `flashinfer_trtllm_routed` (trtllm routed MoE runner + flashinfer A2A) is used as the **baseline**; `flashinfer_megamoe` is the compared configuration. All deltas are reported relative to the trtllm_routed baseline. ## Environment - Model: `nvidia/DeepSeek-V4-Flash-NVFP4` - Parallelism: TP=4, DP=4, DP-attention enabled (`attn_dp_size=4`, so per-rank decode batch = `max_running_requests / 4`) - 1x node, same hardware for both runs ## Server commands **Baseline — `flashinfer_trtllm_routed`:** ```bash python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 4096 \\ --disable-flashinfer-autotune \\ --swa-full-tokens-ratio 0.1 \\ --host 0.0.0.0 --port 30000 \\ --moe-runner-backend flashinfer_trtllm_routed \\ --moe-a2a-backend flashinfer \\ --max-running-requests 1024 ``` **Compared — `flashinfer_megamoe`:** ```bash python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 4096 \\ --disable-flashinfer-autotune \\ --swa-full-tokens-ratio 0.1 \\ --host 0.0.0.0 --port 30000 \\ --moe-runner-backend flashinfer_megamoe \\ --moe-a2a-backend flashinfer_megamoe \\ --max-running-requests 1024 ``` ## Client command ```bash python -m sglang.bench_serving \\ --backend sglang \\ --model nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dataset-name random \\ --random-input-len 8 --random-output-len 1024 --random-range-ratio 1.0 \\ --num-prompts $((5 * C)) \\ --max-concurrency C # C in {32, 128, 1024}, num-prompts = 5 x C ``` ## Results ### Output token throughput (tok/s, higher is better) | max_conc | trtllm_routed (baseline) | megamoe | Δ (megamoe vs baseline) | |---------:|-------------------------:|--------:|------------------------:| | 32 | 876.88 | 674.21 | −23.1% | | 128 | 1215.93 | 1092.54 | −10.1% | | 1024 | 3223.92 | 3245.20 | +0.7% | ### Mean TPOT (ms, lower is better) | max_conc | trtllm_routed (baseline) | megamoe | Δ (megamoe vs baseline) | |---------:|-------------------------:|--------:|------------------------:| | 32 | 34.49 | 38.77 | +12.4% | | 128 | 109.93 | 106.83 | −2.8% | | 1024 | 422.61 | 394.92 | −6.6% | ### Full metrics | config | max_conc | out_tput (tok/s) | mean TPOT (ms) | median TPOT (ms) | p99 TPOT (ms) | achieved conc | mean TTFT (ms) | |--------|---------:|-----------------:|---------------:|-----------------:|--------------:|--------------:|---------------:| | trtllm_routed | 32 | 876.88 | 34.49 | 33.32 | 102.42 | 28.76 | 421 | | megamoe | 32 | 674.21 | 38.77 | 40.27 | 78.13 | 26.53 | 321 | | trtllm_routed | 128 | 1215.93 | 109.93 | 104.71 | 395.33 | 116.12 | 797 | | megamoe | 128 | 1092.54 | 106.83 | 101.22 | 345.54 | 103.68 | 776 | | trtllm_routed | 1024 | 3223.92 | 422.61 | 343.54 | 2429.78 | 994.81 | 3142 | | megamoe | 1024 | 3245.20 | 394.92 | 322.13 | 2356.07 | 951.20 | 3261 | ## Takeaways - **Low/mid concurrency (c32, c128): `flashinfer_trtllm_routed` wins on throughput** — +23% at c32 and +11% at c128 output token throughput, with comparable or better median TPOT. It also sustains higher achieved concurrency (28.8/116.1 vs 26.5/103.7), i.e. it drains the request queue faster. - **High concurrency (c1024): the two are effectively tied on throughput** (3224 vs 3245 tok/s, +0.7% for megamoe — within run-to-run noise). At this point both are decode-bound / saturated (achieved conc < target, p99 TPOT ~7x median), so the MoE backend is no longer the bottleneck. - **TPOT crossover:** megamoe has slightly lower mean TPOT at c1024 (−6.6%) but a worse p99 tail at low load (c32: 102 ms vs 78 ms). trtllm_routed gives more consistent per-token latency at low concurrency. - **Net:** `flashinfer_trtllm_routed` is the better default for latency-sensitive / low-to-mid concurrency serving; at saturation the choice is throughput-neutral. ## FlashInfer MegaMOE env knobs This PR exposes `SGLANG_FLASHINFER_MEGAMOE_COMBINE_DTYPE` for the FlashInfer NVFP4 MegaMOE path. It selects the cross-rank combine wire format passed to `Nvfp4CutedslMegaMoeConfig.combine_dtype`. Supported values: - `bf16`: default, exact combine path. - `mxfp8`: quantized combine traffic with a small accuracy tradeoff. - `nvfp4`: lower combine traffic with a small accuracy tradeoff. Example: ```bash SGLANG_FLASHINFER_MEGAMOE_COMBINE_DTYPE=nvfp4 \\ python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --moe-runner-backend flashinfer_megamoe \\ --moe-a2a-backend flashinfer_megamoe ``` `mxfp8` and `nvfp4` are incompatible with `SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE=1`; server args validation now rejects that combination early. > Note: the `flashinfer_megamoe` in-kernel FC2-reduce variant (`SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE=1`) was excluded from this comparison due to a known issue and is not reflected here. Update 7/27/2026: ## MoE backend comparison — DeepSeek-V4-Flash-NVFP4, TP4/DP4/EP4 Baseline: `trtllm`. Positive throughput and negative latency are both wins. Tokens/rank = per-DP-rank prefill chunk (DP attention routes each request wholly to one rank; `--chunked-prefill-size` is divided by `dp_size` internally). ## Prefill ### 2048 tokens/rank (input len 2048, concurrency 128) | Backend | Input tput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | |---|---|---|---|---| | trtllm (baseline) | 38,884 | 3,084 | 3,233 | 4,104 | | **deepgemm megamoe** | **46,117** | **2,572** | **2,719** | **3,372** | | flashinfer_cutedsl_megamoe_plain | 42,324 | 2,952 | 3,018 | 3,849 | | flashinfer_cutedsl_megamoe_ikr | 41,291 | 3,036 | 3,122 | 3,955 | | flashinfer_cutedsl_megamoe_combine_nvfp4 | 42,408 | 2,953 | 3,044 | 3,838 | Change vs `trtllm`: | Backend | Input tput | Mean TTFT | Median TTFT | P99 TTFT | |---|---|---|---|---| | **deepgemm megamoe** | **+18.60%** | **-16.59%** | **-15.88%** | **-17.84%** | | flashinfer_cutedsl_megamoe_plain | +8.85% | -4.28% | -6.65% | -6.21% | | flashinfer_cutedsl_megamoe_ikr | +6.19% | -1.57% | -3.43% | -3.63% | | flashinfer_cutedsl_megamoe_combine_nvfp4 | +9.06% | -4.23% | -5.84% | -6.48% | > Note: trtllm/deepgemm rows are 640 requests (676,871 input tokens); the flashinfer rows are > 1,280 requests (1,327,019 input tokens). Same input len and concurrency, so the rates are > comparable, but the request counts differ. ### 8192 tokens/rank (input len 8192, 640 requests, 2,665,479 input tokens, concurrency 128) | Backend | Input tput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | |---|---|---|---|---| | trtllm (baseline) | 151,213 | 3,137 | 3,339 | 4,361 | | deepgemm megamoe | 154,359 | 3,047 | 3,286 | 4,163 | | flashinfer_cutedsl_megamoe_plain | 162,342 | 2,945 | 3,159 | 4,146 | | flashinfer_cutedsl_megamoe_ikr | 165,658 | 2,831 | 3,018 | 3,835 | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **169,175** | **2,745** | **2,905** | **3,753** | Change vs `trtllm`: | Backend | Input tput | Mean TTFT | Median TTFT | P99 TTFT | |---|---|---|---|---| | deepgemm megamoe | +2.08% | -2.84% | -1.58% | -4.52% | | flashinfer_cutedsl_megamoe_plain | +7.36% | -6.10% | -5.39% | -4.93% | | flashinfer_cutedsl_megamoe_ikr | +9.55% | -9.73% | -9.60% | -12.06% | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **+11.88%** | **-12.50%** | **-13.00%** | **-13.93%** | ## Decode Input len 32, output len 1024, 10,240 requests, concurrency 8192, 5,256,337 generated tokens. | Backend | Output tput (tok/s) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | |---|---|---|---|---| | trtllm (baseline) | 8,783 | 795.2 | 727.4 | 2,244.7 | | deepgemm megamoe | 8,620 | **762.8** | 721.3 | **1,944.3** | | flashinfer_cutedsl_megamoe_plain | 8,832 | 783.1 | 722.0 | 2,167.6 | | flashinfer_cutedsl_megamoe_ikr | 8,849 | 780.0 | 717.7 | 2,182.1 | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **8,886** | 776.6 | **715.8** | 2,151.7 | Change vs `trtllm`: | Backend | Output tput | Mean TPOT | Median TPOT | P99 TPOT | |---|---|---|---|---| | deepgemm megamoe | -1.86% | **-4.08%** | -0.84% | **-13.38%** | | flashinfer_cutedsl_megamoe_plain | +0.55% | -1.53% | -0.75% | -3.44% | | flashinfer_cutedsl_megamoe_ikr | +0.75% | -1.91% | -1.33% | -2.79% | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **+1.17%** | -2.34% | -1.60% | -4.14% | ## Takeaways - **Prefill: the best backend depends on tokens/rank.** deepgemm wins at 2048 tok/rank (+18.6% throughput, -16.6% mean TTFT) but only edges trtllm at 8192 tok/rank (+2.1%). flashinfer_cutedsl_megamoe is the reverse: +9% at 2048, +11.9% at 8192. - **Prefill: the flashinfer variant ranking does not hold across sizes.** At 8192 tok/rank it is `plain < ikr < combine_nvfp4`. At 2048 tok/rank, `plain` and `combine_nvfp4` are tied (+8.85% vs +9.06%, mean TTFT within 2 ms) and `ikr` is weakest (+6.19%). - **Decode is nearly a wash on throughput.** All backends land within ±2% of trtllm (8,620-8,886 tok/s), far tighter than the prefill spread. At concurrency 8192 this workload is memory-bound, so the MoE kernel is not the bottleneck. - **Decode tail latency is where backends separate.** deepgemm cuts P99 TPOT by 13.4% and mean TPOT by 4.1%, at the cost of 1.9% throughput — the only backend that trades throughput for tail latency. flashinfer improves both modestly. - **combine_nvfp4 is the best flashinfer variant in decode**, leading on every metric (+1.17% throughput, -2.34% mean TPOT), reproducing the 8192 tok/rank prefill ordering. - **Overall recommendation:** `flashinfer_cutedsl_megamoe_combine_nvfp4` for large-prefill and decode-heavy serving; `deepgemm megamoe` if small-prefill throughput or decode P99 dominates. ## Reproduce `` below is whatever selects plain / ikr / combine_nvfp4 — it is not a server arg (all three record `moe_runner_backend=flashinfer_megamoe`). ### Decode — flashinfer_cutedsl_megamoe ```bash # server python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 8192 --max-prefill-tokens 8192 \\ --disable-flashinfer-autotune --swa-full-tokens-ratio 0.1 \\ --moe-runner-backend flashinfer_megamoe --moe-a2a-backend flashinfer_megamoe \\ --max-running-requests 8192 \\ --host 0.0.0.0 --port 30000 # # client python -m sglang.bench_serving \\ --backend sglang --model nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dataset-name random \\ --random-input-len 32 --random-output-len 1024 \\ --num-prompts 10240 --request-rate inf --max-concurrency 8192 ``` The trtllm baseline is the same command with `--moe-runner-backend flashinfer_trtllm_routed --moe-a2a-backend flashinfer`. | Backend | Input tput | Mean TTFT | Median TTFT | P99 TTFT | |---|---|---|---|---| | deepgemm megamoe | +2.08% | -2.84% | -1.58% | -4.52% | | flashinfer_cutedsl_megamoe_plain | +7.36% | -6.10% | -5.39% | -4.93% | | flashinfer_cutedsl_megamoe_ikr | +9.55% | -9.73% | -9.60% | -12.06% | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **+11.88%** | **-12.50%** | **-13.00%** | **-13.93%** | ## Decode Input len 32, output len 1024, 10,240 requests, concurrency 8192, 5,256,337 generated tokens. | Backend | Output tput (tok/s) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | |---|---|---|---|---| | trtllm (baseline) | 8,783 | 795.2 | 727.4 | 2,244.7 | | deepgemm megamoe | 8,620 | **762.8** | 721.3 | **1,944.3** | | flashinfer_cutedsl_megamoe_plain | 8,832 | 783.1 | 722.0 | 2,167.6 | | flashinfer_cutedsl_megamoe_ikr | 8,849 | 780.0 | 717.7 | 2,182.1 | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **8,886** | 776.6 | **715.8** | 2,151.7 | Change vs `trtllm`: | Backend | Output tput | Mean TPOT | Median TPOT | P99 TPOT | |---|---|---|---|---| | deepgemm megamoe | -1.86% | **-4.08%** | -0.84% | **-13.38%** | | flashinfer_cutedsl_megamoe_plain | +0.55% | -1.53% | -0.75% | -3.44% | | flashinfer_cutedsl_megamoe_ikr | +0.75% | -1.91% | -1.33% | -2.79% | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **+1.17%** | -2.34% | -1.60% | -4.14% | ## Takeaways - **Prefill: the best backend depends on tokens/rank.** deepgemm wins at 2048 tok/rank (+18.6% throughput, -16.6% mean TTFT) but only edges trtllm at 8192 tok/rank (+2.1%). flashinfer_cutedsl_megamoe is the reverse: +9% at 2048, +11.9% at 8192. - **Prefill: the flashinfer variant ranking does not hold across sizes.** At 8192 tok/rank it is `plain < ikr < combine_nvfp4`. At 2048 tok/rank, `plain` and `combine_nvfp4` are tied (+8.85% vs +9.06%, mean TTFT within 2 ms) and `ikr` is weakest (+6.19%). - **Decode is nearly a wash on throughput.** All backends land within ±2% of trtllm (8,620-8,886 tok/s), far tighter than the prefill spread. At concurrency 8192 this workload is memory-bound, so the MoE kernel is not the bottleneck. - **Decode tail latency is where backends separate.** deepgemm cuts P99 TPOT by 13.4% and mean TPOT by 4.1%, at the cost of 1.9% throughput — the only backend that trades throughput for tail latency. flashinfer improves both modestly. - **combine_nvfp4 is the best flashinfer variant in decode**, leading on every metric (+1.17% throughput, -2.34% mean TPOT), reproducing the 8192 tok/rank prefill ordering. - **Overall recommendation:** `flashinfer_cutedsl_megamoe_combine_nvfp4` for large-prefill and decode-heavy serving; `deepgemm megamoe` if small-prefill throughput or decode P99 dominates. ## Reproduce `` below is whatever selects plain / ikr / combine_nvfp4 — it is not a server arg (all three record `moe_runner_backend=flashinfer_megamoe`). ### Decode — flashinfer_cutedsl_megamoe ```bash # server python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 8192 --max-prefill-tokens 8192 \\ --disable-flashinfer-autotune --swa-full-tokens-ratio 0.1 \\ --moe-runner-backend flashinfer_megamoe --moe-a2a-backend flashinfer_megamoe \\ --max-running-requests 8192 \\ --host 0.0.0.0 --port 30000 # # client python -m sglang.bench_serving \\ --backend sglang --model nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dataset-name random \\ --random-input-len 32 --random-output-len 1024 \\ --num-prompts 10240 --request-rate inf --max-concurrency 8192 ``` The trtllm baseline is the same command with `--moe-runner-backend flashinfer_trtllm_routed --moe-a2a-backend flashinfer`. ### Prefill 8192 tokens/rank — flashinfer_cutedsl_megamoe ```bash # server python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 32768 --max-prefill-tokens 32768 \\ --disable-flashinfer-autotune --swa-full-tokens-ratio 0.1 \\ --moe-runner-backend flashinfer_megamoe --moe-a2a-backend flashinfer_megamoe \\ --max-running-requests 1024 \\ --host 0.0.0.0 --port 30000 # # client python -m sglang.bench_serving \\ --backend sglang --model nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dataset-name random \\ --random-input-len 8192 --random-output-len 1 \\ --num-prompts 640 --request-rate inf --max-concurrency 128 ``` For the 2048 tokens/rank point, use `--chunked-prefill-size 8192 --max-prefill-tokens 8192` on the server and `--random-input-len 2048 --num-prompts 1280` on the client. Setup & caveats - `kv_cache_dtype=fp8_e4m3`, `page_size=256`, EP4, dp-attention enabled - Best run per metric across repeats (max throughput, min latency) - Repeat counts vary: prefill flashinfer variants 2-6 runs each, trtllm/deepgemm 1-2 runs; decode is a single run per backend - Prefill 2048 tok/rank: the plain/combine_nvfp4 throughput gap (~200 tok/s) is inside the run-to-run spread (~1,100 tok/s), so treat those two as tied rather than ranked - Decode: single run per backend. The flashinfer variants span only 0.6% in throughput and 0.8% in mean TPOT, so their relative ordering is suggestive, not established. The deepgemm P99 TPOT win (-13.4%) is large enough to be credible from one run. - Server commands above are reconstructed from the recorded `server_args`; `mem_fraction_static` was auto-derived (0.8 decode, 0.681 / 0.876 prefill) and is not passed explicitly - FLASHINFER_MOE_EP_KNOB_CACHE point to the autotuned configs. Remaining soft spot: every decode number is a single run, and the three FlashInfer variants differ by well under 1% in throughput. If you want the decode ranking to hold up in review, 2–3 repeats per variant would settle it; the deepgemm P99 result is the only decode finding that stands on its own. ## Accuracy — GSM8K (8-shot, 1316 questions) ```bash python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1316 --parallel 1316 ``` | Backend | Accuracy | vs baseline | Invalid | Latency (s) | Output tput (tok/s) | |---|---|---|---|---|---| | trtllm (baseline) | 0.948 | - | 0.000 | 18.155 | 6,912 | | flashinfer_cutedsl_megamoe_plain | 0.956 | +0.8 pt | 0.000 | 17.773 | 7,041 | | flashinfer_cutedsl_megamoe_ikr | 0.951 | +0.3 pt | 0.000 | 18.563 | 6,805 | | flashinfer_cutedsl_megamoe_combine_nvfp4 | 0.956 | +0.8 pt | 0.000 | 18.482 | 6,749 | **No accuracy regression.** All three flashinfer_cutedsl_megamoe variants match or slightly exceed the trtllm baseline, with zero invalid outputs. At n=1316 the binomial standard error is ~0.6 pt, so the +0.3 to +0.8 pt differences are within noise — the variants should be read as accuracy- equivalent to the baseline, not better than it. > The latency/throughput columns here are not a perf signal: an ~18 s run at `--parallel 1316` is > dominated by warm-up and scheduling, and the ordering contradicts the 10-minute decode benchmark > above. Use the decode section for throughput and TPOT. ## Update 7/29/2026 — latest 2048 tokens/rank prefill rerun Workload: DeepSeek-V4-Flash-NVFP4, 2048 tokens/rank prefill, output len 1, 640 requests, max concurrency 128. | Backend / config | Input tput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P95 TTFT (ms) | P99 TTFT (ms) | |---|---:|---:|---:|---:|---:| | flashinfer_cutedsl_megamoe_combine_nvfp4, without autotune | 46,686.44 | 10,017.09 | 10,971.72 | 13,067.52 | 13,739.66 | | deepgemm megamoe | 51,865.66 | 8,959.21 | 9,848.21 | **11,661.31** | **12,296.46** | | flashinfer_cutedsl_megamoe_combine_nvfp4, with autotuned config | **53,150.67** | **8,592.50** | **9,150.89** | 12,980.44 | 13,935.91 | Takeaway: in this latest setup, the autotuned `combine_nvfp4` config improves throughput and average TTFT over the non-autotuned `combine_nvfp4` path (+13.8% input throughput, -14.2% mean TTFT). Against `deepgemm megamoe`, autotuned `combine_nvfp4` has slightly higher input throughput (+2.5%) and lower mean TTFT (-4.1%), while `deepgemm megamoe` has better tail latency (-10.2% P95 TTFT, -11.8% P99 TTFT). ### CI States --- ### CI States Latest PR Test (Base): :x: [Run #31656310212](https://github.com/sgl-project/sglang/actions/runs/31656310212) Latest PR Test (Extra): :x: [Run #31656310007](https://github.com/sgl-project/sglang/actions/runs/31656310007) ",
"url": "https://github.com/sgl-project/sglang/pull/31470",
"createdAt": "2026-07-16T14:45:50Z",
"updatedAt": "2026-08-13T02:39:56Z",
"timestamp": "2026-08-13T02:39:56Z",
"metrics": {
"reactions": 0,
"comments": 10
},
"labels": [
"documentation",
"high priority",
"quant",
"deepseek",
"run-ci"
],
"author": "wenscarl",
"state": "open",
"assignees": [
"Fridge003",
"YAMY1234"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31479",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "perf(kv-events): coalesce cache events",
"text": "## Summary - Coalesce compatible KV store and removal events at the cache-event queue tail before scheduler publication. - Preserve store-chain semantics and metadata barriers for medium, LoRA ID, block size, event kind, and clear events. - Update cache and manual event tests to consume multi-block payloads. ## Motivation SGLang emitted a separate KV event for each page despite list-valued wire fields, adding avoidable event and serialization overhead. ## Validation - `env PATH=\"$PWD/.venv/bin:$PATH\" TORCH_EXTENSIONS_DIR=\"$PWD/.torch_extensions\" .venv/bin/python -m pytest … -q` — 15 passed, 3 subtests passed - `uv pip check --python .venv/bin/python` - `git diff --check` --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31651848845](https://github.com/sgl-project/sglang/actions/runs/31651848845) Latest PR Test (Extra): :x: [Run #31651848122](https://github.com/sgl-project/sglang/actions/runs/31651848122) ",
"url": "https://github.com/sgl-project/sglang/pull/31479",
"createdAt": "2026-07-16T18:21:03Z",
"updatedAt": "2026-08-13T04:05:42Z",
"timestamp": "2026-08-13T04:05:42Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"run-ci"
],
"author": "jthomson04",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31574",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[EPD] Batch embedding cache host-device range copies",
"text": " ## Motivation The paged multimodal embedding cache may store one embedding across multiple non-contiguous pinned-host page runs. Loading from or storing to this pool currently submits one `Tensor.copy_` operation per run from Python. As the cache becomes fragmented, repeated Python dispatch and CUDA copy submission reduce effective H2D and D2H throughput. Existing KV transfer kernels are tied to KV layouts and index-based gather/scatter semantics, so they are not a good fit for copying contiguous embedding ranges. ## Modifications - Add `transfer_embedding_ranges_direct`, which accepts source starts, destination starts, and lengths for multiple embedding ranges. - Submit the ranges with `cudaMemcpyBatchAsync` on supported CUDA runtimes. - Fall back to individual `cudaMemcpyAsync` calls when the batch API or stream configuration is unsupported. - Use the new operation for embedding-cache host-device transfers while preserving the Python `Tensor.copy_` loop as a compatibility fallback. - Separate storage-transfer buffer construction from host-device transfer-plan construction and add unit coverage for fragmented H2D/D2H plans. ## Accuracy Tests ### CUDA correctness test Added `test_transfer_embedding_ranges_direct` to `test_kvcacheio.py`. It compares both host-to-device and device-to-host range copies against equivalent PyTorch `copy_` operations. ### End-to-end EPD test Benchmark setup and launch command ```bash export MODEL_PATH=/data/cccya-models-pr31574/Qwen3-VL-8B-Instruct export CUDA_VISIBLE_DEVICES=0,1 export MOONCAKE_MASTER=127.0.0.1:50077 export MOONCAKE_PROTOCOL=tcp export MOONCAKE_TE_META_DATA_SERVER=http://127.0.0.1:18086/metadata export MOONCAKE_GLOBAL_SEGMENT_SIZE=4294967296 export MC_MS_AUTO_DISC=0 python3 -m mooncake.http_metadata_server --port 18086 & mooncake_master --port 50077 & ``` Encoder on GPU 0: ```bash sglang serve \\ --model-path ${MODEL_PATH} \\ --trust-remote-code \\ --encoder-only \\ --encoder-transfer-backend mooncake \\ --enable-mm-global-cache \\ --tp 1 \\ --base-gpu-id 0 \\ --host 127.0.0.1 \\ --port 21300 ``` Combined PD on GPU 1 with radix cache disabled: ```bash sglang serve \\ --model-path ${MODEL_PATH} \\ --trust-remote-code \\ --language-only \\ --encoder-urls http://127.0.0.1:21300 \\ --encoder-transfer-backend mooncake \\ --disable-radix-cache \\ --tp 1 \\ --base-gpu-id 1 \\ --host 127.0.0.1 \\ --port 21100 ``` Send the same local `assets/logo_square.png` twice: ```bash IMAGE_BASE64=$(base64 -w 0 assets/logo_square.png) for round in 1 2; do curl -sS http://127.0.0.1:21100/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ --data-binary @- < Send the same request twice. Cache logs: ```text First request: Local Hits: 0 | Global Hits: 0 | Misses (GPU Work): 1 Second request: Local Hits: 1 | Global Hits: 0 | Misses (GPU Work): 0 ``` Both requests returned: ```text A stylized orange icon depicting a branching structure with two circular nodes and a central tag symbol containing HTML-like angle brackets. ``` For this manual run, I temporarily bypassed the scheduler's `batch_encode` path so that image requests used the existing per-request global-cache path. No scheduler change is included in this PR. ## Speed Tests and Profiling Environment: - GPU: NVIDIA H20 - PyTorch: `2.11.0+cu130` - CUDA runtime: `13.0` - Data type: BF16 - Embedding dimension: 3584 - Fragmented range size: approximately 252 KiB - Reported throughput uses median CUDA-event elapsed time | Size | Layout | Ranges | H2D Python loop (GB/s) | H2D direct (GB/s) | D2H Python loop (GB/s) | D2H direct (GB/s) | |---:|---|---:|---:|---:|---:|---:| | 8 MiB | contiguous | 1 | 51.74 | 52.15 | 51.53 | 51.87 | | 8 MiB | fragmented | 33 | 33.12 | 46.19 | 33.55 | 46.41 | | 32 MiB | contiguous | 1 | 54.55 | 54.65 | 54.15 | 54.28 | | 32 MiB | fragmented | 131 | 34.11 | 48.12 | 32.98 | 48.15 | | 128 MiB | contiguous | 1 | 55.28 | 55.30 | 54.66 | 54.71 | | 128 MiB | fragmented | 521 | 34.13 | 48.52 | 32.66 | 48.59 | | 512 MiB | contiguous | 1 | 55.39 | 55.40 | 54.77 | 54.79 | | 512 MiB | fragmented | 2081 | 33.99 | 48.81 | 31.90 | 48.04 | For fragmented transfers, the direct path improves H2D throughput by approximately 42-43%. D2H improves by 28-48%, depending on transfer size. Single-range transfers remain approximately unchanged. Full benchmark script ```shell python bench_embedding_page_run_copy.py ``` ```python import argparse import csv import json import math import statistics import time from pathlib import Path import torch from sgl_kernel.kvcacheio import transfer_embedding_ranges_direct DIM = 3584 DTYPE = torch.bfloat16 TARGET_PAGE_BYTES = 256 * 1024 PAGE_TOKENS = TARGET_PAGE_BYTES // (DIM * torch.tensor([], dtype=DTYPE).element_size()) PAGE_BYTES = PAGE_TOKENS * DIM * torch.tensor([], dtype=DTYPE).element_size() def percentile(values, percentile_value): ordered = sorted(values) index = math.ceil(percentile_value / 100 * len(ordered)) - 1 return ordered[max(index, 0)] def python_loop(src, dst, src_starts, dst_starts, lengths): for src_start, dst_start, length in zip(src_starts, dst_starts, lengths): dst[dst_start : dst_start + length].copy_( src[src_start : src_start + length], non_blocking=True ) def direct_op(src, dst, src_starts, dst_starts, lengths): transfer_embedding_ranges_direct( src, dst, src_starts, dst_starts, lengths ) def build_plan(num_pages, layout, direction, seed): if layout == \"contiguous\": total_tokens = num_pages * PAGE_TOKENS return [0], [0], [total_tokens], total_tokens generator = torch.Generator().manual_seed(seed) permutation = torch.randperm(num_pages, generator=generator).tolist() fragmented_starts = [2 * page * PAGE_TOKENS for page in permutation] contiguous_starts = [page * PAGE_TOKENS for page in range(num_pages)] lengths = [PAGE_TOKENS] * num_pages storage_tokens = 2 * num_pages * PAGE_TOKENS if direction == \"h2d\": return fragmented_starts, contiguous_starts, lengths, storage_tokens return contiguous_starts, fragmented_starts, lengths, storage_tokens def allocate_tensors(num_pages, layout, direction, seed): src_starts, dst_starts, lengths, storage_tokens = build_plan( num_pages, layout, direction, seed ) contiguous_tokens = num_pages * PAGE_TOKENS if layout == \"contiguous\": host_tokens = contiguous_tokens else: host_tokens = storage_tokens host = torch.empty((host_tokens, DIM), dtype=DTYPE, pin_memory=True) device = torch.empty((contiguous_tokens, DIM), dtype=DTYPE, device=\"cuda\") if direction == \"h2d\": return host, device, src_starts, dst_starts, lengths return device, host, src_starts, dst_starts, lengths def check_correctness(stream): num_pages = 7 for direction in (\"h2d\", \"d2h\"): for layout in (\"contiguous\", \"fragmented\"): src, dst, src_starts, dst_starts, lengths = allocate_tensors( num_pages, layout, direction, seed=17 ) if direction == \"h2d\": src.zero_() for page, src_start in enumerate(src_starts): src[src_start : src_start + lengths[page]].fill_(page + 1) else: src.zero_() for page, src_start in enumerate(src_starts): src[src_start : src_start + lengths[page]].fill_(page + 1) expected = torch.zeros_like(dst, device=\"cpu\") source_cpu = src.cpu() if src.is_cuda else src for src_start, dst_start, length in zip( src_starts, dst_starts, lengths ): expected[dst_start : dst_start + length].copy_( source_cpu[src_start : src_start + length] ) dst.zero_() torch.cuda.synchronize() with torch.cuda.stream(stream): direct_op(src, dst, src_starts, dst_starts, lengths) stream.synchronize() actual = dst.cpu() if dst.is_cuda else dst torch.testing.assert_close(actual, expected) del src, dst, actual, expected def benchmark_case( implementation, direction, layout, requested_mib, warmups, target_gib, seed, stream, ): requested_bytes = requested_mib * 1024**2 num_pages = math.ceil(requested_bytes / PAGE_BYTES) actual_bytes = num_pages * PAGE_BYTES iterations = max(30, math.ceil(target_gib * 1024**3 / actual_bytes)) src, dst, src_starts, dst_starts, lengths = allocate_tensors( num_pages, layout, direction, seed ) fn = python_loop if implementation == \"python_loop\" else direct_op for _ in range(warmups): with torch.cuda.stream(stream): fn(src, dst, src_starts, dst_starts, lengths) stream.synchronize() cpu_us = [] gpu_us = [] start_event = torch.cuda.Event(enable_timing=True) end_event = torch.cuda.Event(enable_timing=True) for _ in range(iterations): with torch.cuda.stream(stream): start_event.record(stream) cpu_start = time.perf_counter_ns() fn(src, dst, src_starts, dst_starts, lengths) cpu_end = time.perf_counter_ns() end_event.record(stream) end_event.synchronize() cpu_us.append((cpu_end - cpu_start) / 1000) gpu_us.append(start_event.elapsed_time(end_event) * 1000) gpu_median_us = statistics.median(gpu_us) result = { \"implementation\": implementation, \"direction\": direction, \"layout\": layout, \"requested_mib\": requested_mib, \"actual_mib\": actual_bytes / 1024**2, \"num_pages\": num_pages, \"num_runs\": len(lengths), \"iterations\": iterations, \"cpu_submit_median_us\": statistics.median(cpu_us), \"cpu_submit_p95_us\": percentile(cpu_us, 95), \"gpu_median_us\": gpu_median_us, \"gpu_p95_us\": percentile(gpu_us, 95), \"effective_gbps\": actual_bytes / (gpu_median_us / 1e6) / 1e9, } print(json.dumps(result, sort_keys=True), flush=True) del src, dst torch.cuda.empty_cache() return result def main(): parser = argparse.ArgumentParser() parser.add_argument(\"--sizes-mib\", nargs=\"+\", type=int, default=[8, 32, 128, 512]) parser.add_argument(\"--warmups\", type=int, default=20) parser.add_argument(\"--target-gib\", type=float, default=8) parser.add_argument(\"--seed\", type=int, default=20260717) parser.add_argument(\"--output-dir\", default=\"/data/embedding_cache_bench\") args = parser.parse_args() torch.cuda.set_device(0) stream = torch.cuda.Stream(device=0) check_correctness(stream) print( json.dumps( { \"correctness\": \"PASS\", \"torch\": torch.__version__, \"gpu\": torch.cuda.get_device_name(0), \"dim\": DIM, \"dtype\": str(DTYPE), \"page_tokens\": PAGE_TOKENS, \"page_bytes\": PAGE_BYTES, }, sort_keys=True, ), flush=True, ) results = [] for requested_mib in args.sizes_mib: for direction in (\"h2d\", \"d2h\"): for layout in (\"contiguous\", \"fragmented\"): for implementation in (\"python_loop\", \"direct_op\"): results.append( benchmark_case( implementation, direction, layout, requested_mib, args.warmups, args.target_gib, args.seed, stream, ) ) output_dir = Path(args.output_dir) output_dir.mkdir(parents=True, exist_ok=True) timestamp = time.strftime(\"%Y%m%d_%H%M%S\") json_path = output_dir / f\"page_run_copy_{timestamp}.json\" csv_path = output_dir / f\"page_run_copy_{timestamp}.csv\" json_path.write_text(json.dumps(results, indent=2)) with csv_path.open(\"w\", newline=\"\") as file: writer = csv.DictWriter(file, fieldnames=results[0].keys()) writer.writeheader() writer.writerows(results) print(json.dumps({\"json\": str(json_path), \"csv\": str(csv_path)}), flush=True) if __name__ == \"__main__\": main() ``` ### Realistic fragmented embedding-cache benchmark To measure the improvement over the pre-PR per-range `Tensor.copy_` loop in a real embedding-cache path, I constructed an alternating fragmentation pattern using a Qwen3-VL-8B encoder. Environment: - GPU: NVIDIA H20 - PyTorch: `2.11.0+cu130` - CUDA runtime: `13.0` - Model: Qwen3-VL-8B-Instruct - Cache embedding dtype: BF16 - Cache embedding dimension: 16384 - Page size: 256 KiB - Vision pool: 0.25 GiB / 1024 pages Fragmentation setup: 1. Fill the vision pool with 128 minimum-sized local images. Each image produces 64 embedding tokens and occupies 8 contiguous pages. 2. Refresh the even-numbered entries so that the odd-numbered entries become alternating LRU eviction candidates. 3. Insert one 2016x1960 image containing 3843 embedding tokens. Its 481 pages are allocated across 61 non-contiguous page runs. 4. Repeat this cached image 16 times per request, producing 976 H2D ranges and approximately 1.876 GiB of H2D data per request. Each mode processed 20 requests. The benchmark was repeated in forward and reverse mode order in two fresh processes on the same H20 node, giving 40 measured requests per mode. The same fragmented cache layout was reused across modes within each process. All 640 image copies per mode were local-cache hits, with zero global hits, misses, or D2H copies. Timing covers encoder-side GPU embedding assembly through H2D completion. | Implementation | Submit mean (us) | H2D assembly mean (us) | Effective throughput (GB/s) | |---|---:|---:|---:| | Pre-PR per-range `Tensor.copy_` loop | 9188.12 | 39892.38 | 50.51 | | Batched op, Stream source order | 2309.89 | 37069.82 | 54.35 | | Batched op, Any source order | 2307.46 | 37062.35 | 54.36 | Compared with the pre-PR `Tensor.copy_` loop, the current PR path: - reduces submission time by 74.89%; - reduces complete H2D assembly time by 7.09%, or 2.83 ms per request; - improves effective H2D throughput by 7.64%. `cudaMemcpySrcAccessOrderAny` and `cudaMemcpySrcAccessOrderStream` perform equivalently on this H20. Therefore, the measured improvement comes from batching the fragmented ranges rather than from the source access-order flag. This workload uses alternating holes at the minimum allocation granularity reachable by normal Qwen3-VL image requests. Each freed minimum-sized cache entry contains 8 pages, so most page runs in this test are approximately 2 MiB. The earlier single-page synthetic benchmark represents a more fragmented worst case and therefore shows a larger throughput improvement. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31676899965](https://github.com/sgl-project/sglang/actions/runs/31676899965) Latest PR Test (Extra): :white_check_mark: [Run #31676904137](https://github.com/sgl-project/sglang/actions/runs/31676904137) ",
"url": "https://github.com/sgl-project/sglang/pull/31574",
"createdAt": "2026-07-17T08:45:19Z",
"updatedAt": "2026-08-13T10:43:33Z",
"timestamp": "2026-08-13T10:43:33Z",
"metrics": {
"reactions": 2,
"comments": 5
},
"labels": [
"sgl-kernel",
"run-ci",
"run-ci-extra"
],
"author": "cccccya",
"state": "open",
"assignees": [
"liusy58",
"huangtingwei9988"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31575",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix rope config compatibility and VL/transformers-fallback weight loading",
"text": "## Summary Five independent, small fixes uncovered while enabling additional models. - **`utils/hf_transformers/common.py`**: `get_rope_config()` raised `KeyError: 'rope_theta'` whenever a model's `rope_parameters` dict existed but didn't contain a `rope_theta` key (e.g. `baidu/ERNIE-4.5-VL-28B-A3B-PT`, `allenai/Olmo-3-7B-Instruct`). Now falls back to `config.rope_theta` (default `10000`) in that case. - **`models/ernie45_moe_vl.py`**, **`models/olmo2.py`**: switched from indexing `config.rope_parameters[\"rope_theta\"]` directly to `get_rope_config(config)`, so they benefit from the fallback above. - **`models/qwen.py`**: `load_weights` now skips checkpoint keys absent from `params_dict` (e.g. `transformer.visual.*` on `Qwen/Qwen-VL-Chat`), since this implementation only covers the text backbone and previously crashed with `KeyError` on VL checkpoints sharing the `QWenLMHeadModel` architecture name. - **`models/transformers.py`**: two fixes for the generic `transformers` fallback path — (1) remap legacy `vision_tower.vision_model.*` checkpoint keys when the live model lacks that sub-module (transformers v5 flattened SigLIP/CLIP), and (2) pad 5D `pixel_values` features to the batch-wide max patch count instead of flattening, so anyres-style models with varying per-image tile counts concatenate correctly instead of crashing/misaligning. - **`managers/mm_schedule.py`**: `_get_chunked_prefill_embedding` batches image cache-misses from *all* concurrent per-image requests into one cross-request ViT call (`_batch_encode_per_image_misses`) for throughput, which assumes every item's feature tensor has the same non-batch shape. Phi-4-multimodal-instruct's dynamic-HD preprocessing gives each image its own crop count based on that image's resolution/aspect ratio, so two images from different concurrent requests can carry different crop counts — when such items land in the same batched ViT call, `phi4mm.get_image_feature`'s `torch.cat` crashes (`Sizes of tensors must match except in dimension 0. Expected size 7 but got size 11...`). This same class of bug was already worked around for ROCm/NPU by routing `is_hip`/`is_npu` through the per-request-only `_get_chunked_embedding_by_item` path; XPU had never been added to that list and inherited CUDA's batched path. Adds `is_xpu` to the same condition, so XPU now takes the same safe, already-production-proven per-request path as ROCm/NPU. ## Validation Each fix was validated end-to-end on real hardware (4x Intel Arc Pro B60, XPU backend) via differential baseline-vs-patched testing against the exact checkpoints that originally hit each bug: - `get_rope_config` + `olmo2.py` — `allenai/Olmo-3-7B-Instruct`: baseline reproduces `KeyError: rope_theta`; patched loads, serves, and passes gsm8k (0.75 exact_match/8 examples). - `get_rope_config` + `ernie45_moe_vl.py` — `baidu/ERNIE-4.5-VL-28B-A3B-PT` (TP=4): baseline (common.py excluded from overlay) reproduces the identical `KeyError`; patched loads weights cleanly across all 4 TP ranks. - `qwen.py` — `Qwen/Qwen-VL-Chat`: baseline reproduces `KeyError: 'transformer.visual.positional_embedding'`; patched loads and serves successfully. - `transformers.py` — `llava-hf/llava-onevision-qwen2-0.5b-ov-hf`: loads, serves, mmmu_val (10 examples) gives 30% accuracy, consistent with the documented 32.2% full-run. - `mm_schedle.py` — `microsoft/Phi-4-multimodal-instruct` (XPU): confirmed on hardware that the unpatched routing takes the batched cross-request path (`_is_xpu` absent → falls through to CUDA-style batching) while the patched routing takes the safe per-request path (`_is_xpu=True`). A direct unit-level repro reproduced the exact error text (`Expected size 7 but got size 11`) from concatenating two mismatched-shape items, confirming the mechanism; `_get_chunked_embedding_by_item` is scoped to one request's own items only, so the fix structurally rules out cross-request shape collisions rather than just reducing their probability. ## Test plan - [x] Differential (baseline vs. patched) smoke test per fix on real checkpoints, as above - [ ] CI --- ### CI States Latest PR Test (Base): :x: [Run #31587547509](https://github.com/sgl-project/sglang/actions/runs/31587547509) Latest PR Test (Extra): :x: [Run #31587547169](https://github.com/sgl-project/sglang/actions/runs/31587547169) ",
"url": "https://github.com/sgl-project/sglang/pull/31575",
"createdAt": "2026-07-17T08:49:38Z",
"updatedAt": "2026-08-13T08:00:54Z",
"timestamp": "2026-08-13T08:00:54Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"run-ci"
],
"author": "vshekhawat-hlab",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31586",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[dLLM] Decouple prefill chunk size from decode block size",
"text": "## Motivation This PR addresses the long-prefill optimization proposed in #24644 and #24645. Previously, dLLM prefill and decode shared the same `block_size` (typically 32). As a result, a long prompt could advance by only one decode block per scheduler round, causing excessive scheduling overhead and underutilizing the GPU. This PR decouples the maximum prefill chunk size from the fixed decode block size: - decode still processes exactly one `block_size` block per request; - pure prefill can process multiple aligned blocks in one scheduler round; - the non-aligned prompt tail is handled together with masks by the following decode block. ### Execution-mode separation Pure-prefill requests intentionally use the regular `ForwardMode.EXTEND`, while decode requests continue to use `ForwardMode.DLLM_EXTEND`. Pure prefill has a dynamic extend length and commits only real prompt KV; decode must preserve the fixed `block_size` dLLM semantics and its dedicated CUDA Graph path. **This separation is a key correctness point for review.** The scheduler must determine the final request phase before it constructs the phase-aware `PrefillAdder` and selects the forward mode. A stale or prematurely selected prefill phase could incorrectly send a decode request through regular `EXTEND` instead of `DLLM_EXTEND`. ### Phase-selection correctness problems addressed This PR also fixes two ordering issues exposed by multi-block prefill: 1. `determine_dllm_phase()` previously ran with stale `prefix_indices`. Prefix-cache matching could update the matched prefix without reclassifying the request, so a prompt-tail-plus-mask block could remain incorrectly classified as pure prefill. Phase determination now runs after cache matching has finalized `prefix_indices`. 2. `get_new_batch_dllm()` previously selected `is_prefill`, created the adder, and selected the forward mode before incoming requests were fully prepared. Even if `process_dllm_incoming_reqs()` later changed a request to decode, the current round had already been configured as prefill. All managed dLLM requests are now collected and prepared before phase selection, so the adder and forward mode are constructed from the final cache-matched phases. Related to #24644, #24645. ## Modifications ### Configurable multi-block prefill - Add an optional `prefill_block_size` field to `--dllm-algorithm-config`. - Add `--dllm-prefill-block-size` as a command-line override for the YAML `prefill_block_size` value. - Default `prefill_block_size` to `block_size` to preserve the existing behavior. - Require `prefill_block_size >= block_size` and divisibility by `block_size`. - Require users to explicitly pass `--attention-backend flashinfer` when `prefill_block_size > block_size`, because multi-block dLLM prefill currently depends on FlashInfer's custom attention-mask support. - Reject unsupported attention backend combinations at startup instead of silently using incorrect attention semantics. The existing single-block behavior (`prefill_block_size == block_size`) is unaffected by this requirement. Example: ```yaml block_size: 32 prefill_block_size: 1024 threshold: 0.95 ``` The YAML value can be overridden without editing the configuration file: ```bash python -m sglang.launch_server ... \\ --dllm-prefill-block-size 1024 \\ --attention-backend flashinfer ``` ### Phase-aware dLLM scheduling - Separate pure-prefill scheduling from fixed-size decode scheduling. - Fetch incoming requests and prepare all managed dLLM requests before selecting the round phase. - Re-run phase determination after prefix-cache matching updates `prefix_indices`, preventing stale-prefix phase classification. - Use phase-aware scheduler budgets: - prefill: `max_running_requests * prefill_block_size`; - decode: `max_running_requests * block_size`. - Perform prefix-cache matching before determining the dLLM phase. - Construct the phase-aware adder and select `EXTEND` versus `DLLM_EXTEND` only after the final request phases are known. - Preserve the incoming request lifecycle until admission by `PrefillAdder`. - Prefer staging requests over incoming requests within the same phase. - Prevent unadmitted requests from advancing `dllm_block_offset`. - Align pure-prefill frontiers to `block_size`. For example, with `block_size=32` and a 300-token prompt, pure prefill advances to token 288. The remaining 12 prompt tokens and 20 masks are processed by the next 32-token decode block. ### Block-wise prefill attention Multi-block dLLM prefill uses the following visibility rule, where `B` is the dLLM block size: ```text visible(q, k) = floor(k / B) <= floor(q / B) ``` The custom-mask implementation in this PR is an interim solution until FlashInfer's native [Block Extend Attention support](https://github.com/flashinfer-ai/flashinfer/pull/2722) is merged, released, and available in SGLang's supported FlashInfer version. Once that native operator is available, this path can use its block-expanding mask semantics directly and avoid constructing and passing an additional attention mask. Migrating to the native operator is therefore the intended follow-up rather than keeping the explicit custom-mask path permanently. This provides: - bidirectional attention within the same dLLM block; - visibility into all preceding blocks; - isolation from future blocks. Masked and unmasked ragged prefill use separate persistent FlashInfer wrappers. The custom-mask wrapper is pinned to FA2, preventing an unmasked FA3 plan from being incorrectly reused for a later masked request on SM90. Unsupported combinations such as multi-block custom masks with sliding-window dispatch or multi-item scoring fail explicitly. ### Breakable prefill CUDA Graph - Run pure dLLM prefill through `ForwardMode.EXTEND`. - Allow reuse of Breakable prefill CUDA Graphs when: - CUDA and FlashInfer are used; - the prefill graph backend is Breakable; - the token count exactly matches a captured bucket; - no unsupported input mode is active. - Fall back to eager execution when these conditions are not satisfied. - Propagate dLLM prefill metadata through graph replay so the same block-wise mask is reconstructed. Exact bucket matching is required because upward padding would change dLLM bidirectional attention and KV-write semantics. ### Tests and documentation - Add scheduler/configuration tests for: - phase-aware budgets; - aligned and non-aligned prompt boundaries; - incoming request lifecycle and prefix-cache phase detection; - fixed decode blocks; - request admission and offset updates; - FlashInfer backend validation; - Breakable CUDA Graph capability checks. - Add FlashInfer attention tests for: - block-mask construction; - heterogeneous batches; - single-block fast path; - non-aligned prefixes; - masked/unmasked wrapper isolation; - ragged output against a dense reference; - prefix cascade output and LSE parity; - future-block isolation; - paged fallback parity. - Document the new server argument and scheduling/attention design. ## Accuracy Tests Validation environment: | Item | Value | | ------------------------ | --------------------------- | | GPU | 2x NVIDIA H200 143 GB | | Model | `inclusionAI/LLaDA2.0-mini` | | PyTorch | `2.11.0+cu128` | | FlashInfer | `0.6.12` | | dLLM block size | 32 | | Multi-block prefill size | 1024 | ### GSM8K quality GSM8K was evaluated with 200 examples, `temperature=0`, TP=1, and radix cache disabled. | Configuration | Score | | -------------------------- | ----- | | Main baseline | 0.895 | | Current single-block eager | 0.895 | | Multi-block eager | 0.915 | | Multi-block Breakable | 0.910 | Acceptance results: ```text multi_eager - main = +0.020 multi_eager - single_block = +0.020 multi_breakable - multi_eager = -0.005 ``` Both the multi-block eager and Breakable configurations pass the configured absolute and relative quality thresholds. Single-block and multi-block execution are not expected to be token-identical for longer prompts because changing the prefill chunk boundaries also changes kernel shapes, KV submission boundaries, and floating-point accumulation. This difference was also observed after forcing both paths to FA2. The end-to-end GSM8K quality evaluation shows no accuracy regression. ## Speed Tests and Profiling ### GSM8K latency (H200, TP=1, 200 examples) | Configuration | Latency (s) | Output throughput (tok/s) | | --- | ---: | ---: | | Main baseline | 65.8 | 381.0 | | Current single-block eager | 121.6 | 205.2 | | Multi-block eager | 58.6 | 435.0 | | Multi-block Breakable | 47.1 | 540.7 | Compared with the existing single-block eager configuration: - multi-block eager improves output throughput from `205.2` to `435.0 tok/s` (`2.12x`); - multi-block Breakable improves output throughput to `540.7 tok/s` (`2.63x`); - multi-block Breakable reduces evaluation latency from `121.6s` to `47.1s`. Compared with multi-block eager, Breakable prefill graph replay improves output throughput by approximately `1.24x`. ### Prefill serving matrix (H200, TP=1, exact Breakable buckets) Full matrix under `benchmark_results/pr_prefill_perf_matrix_20260720/`: - configs: `main` (`block_size=32`) + current-branch `prefill_block_size ∈ {32,128,512,1024,2048,4096}` - input lengths: `128, 256, 512, 1024, 4096, 8192, 16384` - `num_prompts=16`, `output_len=64`, `max_concurrency=1`, fixed seeds (`seed = 2000 + input_len`) Input token throughput (tok/s): | input | main | pbs32 | pbs128 | pbs512 | pbs1024 | pbs2048 | pbs4096 | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 128 | 528 | 416 | 451 | 438 | 441 | 441 | 442 | | 256 | 917 | 774 | 897 | 846 | 850 | 847 | 849 | | 512 | 1832 | 1545 | 1967 | 1968 | 1982 | 1989 | 1977 | | 1024 | 2563 | 2251 | 2975 | 3245 | 3250 | 3268 | 3299 | | 4096 | 4784 | 4251 | 8561 | 10759 | 12420 | 12810 | 12320 | | 8192 | 5517 | 4543 | 11845 | 18841 | 23248 | 25174 | 25671 | | 16384 | 5193 | 3757 | 13103 | 22751 | 27713 | 30709 | 29577 | vs `main` input-throughput delta at long context: | input | pbs1024 | pbs2048 | pbs4096 | | ---: | ---: | ---: | ---: | | 4k | +160% | +168% | +158% | | 8k | +321% | +356% | +365% | | 16k | +434% | +491% | +470% | Mean TTFT (ms) at 16k: `main 2996` → `pbs1024 410` / `pbs2048 351` / `pbs4096 365`. Notes: - Current-branch `pbs32` is slightly slower than `main` (expected overhead of the new phase-aware path with the same chunk size). - Gains grow with prompt length; sweet spot on this setup is about `prefill_block_size=2048` for 8k–16k. - Prefill CUDA-graph miss count was `0` across completed sweeps; no OOM on H200 with `mem-fraction-static` 0.75–0.85. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci`. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/31586",
"createdAt": "2026-07-17T12:44:51Z",
"updatedAt": "2026-08-13T09:20:15Z",
"timestamp": "2026-08-13T09:20:15Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"documentation"
],
"author": "javey-q",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31633",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[feature] sglext output ids field",
"text": " ## Motivation Exact token ids behind a completion cannot reliably be recovered by retokenizing the response. Returning exact prompt and sampled token IDs is currently unsupported for `v1/chat/completions` streaming responses (#30917 added `return_token_ids`, but chat streaming rejects it, and its streaming shape interleaves per-chunk id deltas into every choice). This PR exposes both input and output token IDs on the response-level `sglext` extension, delivered once per response, and is opt-in per request or server-wide. ## Modifications - `protocol.py`: add `return_input_ids` / `return_output_ids` request flags and `sglext.input_ids` (shared prompt) / `sglext.output_ids` (per choice) response fields. - `server_args.py`: add `--return-input-ids` / `--return-output-ids` to force the behavior for every request; also settable per request via `x-sglext-return-input-ids` / `x-sglext-return-output-ids` headers. - `serving_chat.py`: - Non-streaming: ids are attached to the response's sglext field. - Streaming: ids are delivered exactly once, in a final response-level sglext chunk before [DONE] - Unit tests in test_serving_chat.py covering streaming/non-streaming, n>1, both streaming modes, and abort paths. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31552048751](https://github.com/sgl-project/sglang/actions/runs/31552048751) Latest PR Test (Extra): :x: [Run #31552048508](https://github.com/sgl-project/sglang/actions/runs/31552048508) ",
"url": "https://github.com/sgl-project/sglang/pull/31633",
"createdAt": "2026-07-18T00:43:16Z",
"updatedAt": "2026-08-12T23:19:25Z",
"timestamp": "2026-08-12T23:19:25Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "amykchang",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31698",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Scheduler] Reuse per-step cuda events uniformly (WAR read_done + copy_done)",
"text": "## Motivation Follow-up to #31468, several paths construct a fresh cuda `Event()` every decode step: the WAR `read_done` in `DecodeCudaGraphRunner.execute()` (twice per DFlash step: draft + verify runners), the same in `EagleDraftExtendCudaGraphRunner`, and the scheduler's `copy_done` (both the overlap and non-overlap spec paths). Each costs a `cudaEventCreate` + Python object churn (~µs/step) — not a host sync, purely allocation/GC hygiene, so no throughput change is expected; a no-regression A/B is attached below. ## Modifications - New `ReusableEventRing` (`sglang/srt/utils/cuda_event_ring.py`): lazy fixed ring of re-recordable events; a slot is re-recorded only after every consumer of its previous record has waited/synchronized, so depth = max records in flight (+ margin). - `read_done` sites (decode graph runner, eagle draft-extend runner): depth **2** per runner — the WAR barrier `wait_event`s and clears `war_fastpath_read_done_event` right after `run_batch`, so at most one record is outstanding. - Scheduler `copy_done` (overlap + non-overlap spec): depth **3** — `result_queue` momentarily holds the previous result plus the just-appended one (≤2 in flight); every consumer (`batch_result_processor`, disaggregation prefill) synchronizes exactly once before a slot recurs. - **Intentionally out of scope:** the unified-memory `forward_done` event — `MultiEndedAllocator` retains and `query()`s old references, so re-recording would silently change what a retained handle reports; it needs its own lifecycle. Likewise the PP microbatch `d2h_event`s, whose lifetime is the PP async depth, not the decode pipeline. ## Test `test/registered/unit/spec/test_cuda_event_ring.py` (CI-registered): lazy fixed allocation + round-robin ordering, distinct-objects-within-depth, depth validation, and a CUDA record/wait/synchronize cycle across ring wrap-around mimicking both lifecycles. ## Benchmarks No-regression e2e posted in the comments: per-cc deltas within the harness noise band (mean +0.6%), GSM8K 0.955/0.965, AL parity. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#code-formatting-with-pre-commit). - [x] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#running-unit-tests-writing-documentation). - [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/developer_guide/contribution_guide.html#running-unit-tests-writing-documentation). - [x] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/developer_guide/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31372129048](https://github.com/sgl-project/sglang/actions/runs/31372129048) Latest PR Test (Extra): :x: [Run #31372128813](https://github.com/sgl-project/sglang/actions/runs/31372128813) ",
"url": "https://github.com/sgl-project/sglang/pull/31698",
"createdAt": "2026-07-19T04:09:00Z",
"updatedAt": "2026-08-13T07:40:31Z",
"timestamp": "2026-08-13T07:40:31Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci"
],
"author": "thanhhao98",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31716",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] Add a ctx axis to the adaptive spec _route (BS × ctx → slot)",
"text": "## Motivation Long-context decode is memory-bandwidth-bound on the per-step KV read; verifying K drafted tokens amortizes that read across K+1 tokens, so the optimal K depends on ctx as well as BS. A batch-only palette has to pick one K per BS and either leaves ctx-amortization on the table for long sequences or over-drafts short ones. This is the SGLang side of the mechanism described in vLLM RFC #48627 (sibling implementation in vLLM PR #48944). ## Modifications ### Config schema (`adaptive_spec_params.py`) - Each per-BS entry may hold `ctx_buckets: {\"lo-hi\": {\"candidate_steps\": ..., ...}}`. Ranges are 1-indexed, inclusive on both ends, and must cover `[1, +inf)` with no gaps (validated at load time). - Legacy 1D entries (no `ctx_buckets` key) still parse; internally the single virtual bucket is `[1, INT_MAX]`, and `_slot_for_ctx` takes a `len(buckets) == 1` fast path. ### Routing - `AdaptiveSpeculativeParams._route(batch_size, ctx_repr=0)` extends the BS-only closest-slot lookup with `_slot_for_ctx(bs, ctx_repr)` (bisect on the per-BS `ctx_hi` list). - `get_steps_for_batch`, `on_verify_complete` grow a `ctx_repr` kwarg, default 0 (inert for legacy 1D configs). - `cuda_graph_bs_for_step` reachability check now unions each BS's ctx buckets' `candidate_steps`; the CUDA-graph capture set stays BS-only. ### Wiring — ctx_repr computation stays fully CPU-side - `adaptive_runtime_state.py`: `AdaptiveController.activate_step_by_batch` and `on_verify_complete` thread `ctx_repr` through to the params. - `base_spec_worker.py` / `eagle_worker_v2.py` / `ngram_worker.py`: `on_verify_complete_cpu` and `activate_step_by_batch` grow `ctx_repr` (default 0), matching `BaseSpecWorker` signature. - `eagle_worker_v2.py:forward_batch_generation`, `frozen_kv_mtp_worker_v2.py:forward_batch_generation`, and `scheduler_components/batch_result_processor.py` compute `ctx_repr` from `batch.reqs` (CPU-side list already in memory): `sorted(req.seqlen for req in batch.reqs)[n//2]`. `Req.seqlen` is a pure-Python property (`len(origin_input_ids) + len(output_ids)`) — same physical quantity as `seq_lens_cpu[i]`, no tensor allocation, no host-device sync in the decode hot path. This matches the no-sync design principle stated in vLLM RFC #48627. ## Accuracy Tests `test/registered/unit/spec/test_adaptive_spec_params.py` adds 15 new cases (46 total, all passing): **Schema parsing (6):** gap/start/key/empty/steps validation, 1-index rule. **Routing (8):** ctx-switch within BS, boundary inclusivity, ctx above last hi, legacy 1D inert to ctx, default ctx, differ-by-ctx `get_steps`, `on_verify_complete` slot isolation, `cuda_graph_bs` union. **Capture-set identity (1):** `test_capture_set_identical_to_1d_union` proves capture-set BS-only invariance: for any 2D config C2 and its 1D-union equivalent C1 (per-BS `candidate_steps` = union of C2's ctx buckets), C1 and C2 return identical `cuda_graph_bs_for_step(step)` for every step and identical `candidate_steps`. Plus `resolve_candidate_steps_unions_across_ctx_buckets` for the buffer sizing path. Existing 31 cases in the same file unchanged and pass. ## Live-server smoke (2D routing works end-to-end) - Hardware: H100 NVL 94GB, driver 580.159 / CUDA 13.0 - Target: `meta-llama/Llama-3.1-8B-Instruct` (bf16) - Draft: `lmsys/sglang-EAGLE3-LLaMA3.1-Instruct-8B` (bf16) - Server: `python -m sglang.launch_server --speculative-algorithm EAGLE3 --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-adaptive --speculative-adaptive-config /tmp/adaptive_2d.json --context-length 8192 --enable-metrics` - 2D config: ```json { \"1\": {\"candidate_steps\": [3]}, \"8\": {\"ctx_buckets\": { \"1-100\": {\"candidate_steps\": [0]}, \"101-100000\": {\"candidate_steps\": [3]} }} } ``` - Observed via `sglang:spec_num_steps` gauge: | Case | BS | Prompt len | Expected slot | Observed `spec_num_steps` | |---|---|---|---|---| | 1 | 1 | ~10 tokens | BS=1 slot → K=3 | 3.0 ✓ | | 2 | 8 concurrent | ~3 tokens | BS=8 ctx<100 → K=0 | 0.0 ✓ | | 3 | 8 concurrent | 262 tokens | BS=8 ctx≥101 → K=3 | 3.0 ✓ | The \"Switch adaptive runtime state: steps 0 → 3\" log line fires on the BS=8 long-prompt case, confirming ctx-based routing takes effect end-to-end. Case 3 also demonstrates the RFC #48627 dose-response prediction (higher K helps at long ctx even under high BS) holds in SGLang. ### Note on the interim commits Commits `c6fa6ce1` (elif GPU fallback) and `7d2825af` (CPU-native rewrite) resolve a bug the smoke exposed: `batch.seq_lens_cpu` is `None` during CUDA-graph decode, so the initial `seq_lens_cpu.median().item()` path silently returned `ctx_repr=0` and 2D routing never fired for the K>0 side. The fallback commit added a `seq_lens.median().item()` branch that fixed correctness but introduced a per-iter host-device sync — flagged by gemini-code-assist as an anti-pattern. `7d2825af` supersedes both by computing the median from `batch.reqs` (CPU-side), which is correctness-safe under CUDA graphs and free of any tensor sync. History kept for review traceability; happy to squash before merge if preferred. ## Speed Tests and Profiling Routing overhead is a single `bisect.bisect_left` per verify complete + activate (already CPU-side). Legacy 1D configs skip the bisect via the `len(buckets) == 1` fast path. CUDA-graph capture set does not change (proved by the capture-set identity test), so no additional graph memory or compile cost. `ctx_repr` computation is `sorted()` over `len(batch.reqs)` Python ints — O(B log B) with B ≤ max_running_requests, dwarfed by the verify step's compute. End-to-end tok/s comparison against a batch-only palette on real workload traces is out of scope here and will be reported in a follow-up once a workload harness is agreed on. ## References - vLLM RFC #48627 — Context-length-aware K in DSD (no-sync design principle) - vLLM PR #48944 — sibling implementation - gemini-code-assist review — flagged the sync-in-hot-path anti-pattern; applied in `7d2825af` --- **AI-assistance disclosure** (per `AGENTS.md`): this patch was prepared and end-to-end verified with Claude Code. Every diff was reviewed by the author; unit tests and the live-server smoke above were executed against the patched module before proposing. --- ### CI States Latest PR Test (Base): :x: [Run #29693721003](https://github.com/sgl-project/sglang/actions/runs/29693721003) Latest PR Test (Extra): :x: [Run #29693720892](https://github.com/sgl-project/sglang/actions/runs/29693720892) ",
"url": "https://github.com/sgl-project/sglang/pull/31716",
"createdAt": "2026-07-19T15:18:05Z",
"updatedAt": "2026-08-13T07:57:38Z",
"timestamp": "2026-08-13T07:57:38Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [],
"author": "seongyun1104",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31730",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Fix Encoder Decoder KV Cache Alignment",
"text": " ## Motivation Encoder Decoder models produces garbage output on the intel_xpu backend (page_size > 1). Two root causes in the paged encoder-decoder KV path: 1. Cross-attention page table not converted to page numbers. encoder_page_table was left token-granular while the paged kernel interprets its entries as page indices — the decoder attended to wrong encoder KV. 2. Decoder KV region not page-aligned. e.g. whisper reserves max_source_positions (1500) encoder slots, but 1500 % page_size ≠ 0, so the decoder KV started mid-page. The paged decode kernel/allocator assume each sequence begins on a page boundary (physical slot = page * page_size + i % page_size), so decoder tokens were read/written at wrong slots. ## Modifications Reserve a page-aligned number of encoder slots (ceil_align(encoder_len, page_size), e.g. 1500 → 1536) so the decoder KV starts on a page boundary, while keeping the true encoder length (1500) for what the encoder writes and what cross-attention reads. The extra padding slots are allocated but never written/read. - whisper.py — pad_input_ids prepends ceil_align(encoder_len, page_size) dummy tokens; num_image_tokens stays the true length. - schedule_batch.py — strip/offset the decoder region by the page-aligned reserve; write encoder KV for only the true length. - mem_cache/allocation.py (alloc_for_decode) — read last_loc and write decoder KV at the page-aligned encoder_offset column. - xpu_backend.py — stride+divide encoder_page_table into page numbers (cross-attn fix); slice the decoder self-attention page_table from the aligned offset. All changes use ceil_align(x, page_size), which is the identity when page_size == 1. CUDA Whisper (flashinfer, page_size=1) and every other model are byte-identical no-ops. The paged encoder-decoder decode path is only exercised on XPU today. --- ### CI States Latest PR Test (Base): :x: [Run #31694867328](https://github.com/sgl-project/sglang/actions/runs/31694867328) Latest PR Test (Extra): :x: [Run #31694867047](https://github.com/sgl-project/sglang/actions/runs/31694867047) ",
"url": "https://github.com/sgl-project/sglang/pull/31730",
"createdAt": "2026-07-19T19:38:44Z",
"updatedAt": "2026-08-13T13:14:42Z",
"timestamp": "2026-08-13T13:14:42Z",
"metrics": {
"reactions": 1,
"comments": 5
},
"labels": [
"intel",
"xpu",
"run-ci",
"run-ci-extra"
],
"author": "SKRohit",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31751",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] upgrade sglang xpu backend to PyTorch 2.13",
"text": "update pytorch to 2.13 for xpu and update other relation. The oneAPI version should be 2026.0 after updated. ## Tests python3 run_suite.py --hw xpu --suite stage-a-test-1-gpu-xpu and python3 run_suite.py --hw xpu --suite stage-b-test-1-gpu-xpu --- ### CI States Latest PR Test (Base): :x: [Run #31715088591](https://github.com/sgl-project/sglang/actions/runs/31715088591) Latest PR Test (Extra): :x: [Run #31715088312](https://github.com/sgl-project/sglang/actions/runs/31715088312) ",
"url": "https://github.com/sgl-project/sglang/pull/31751",
"createdAt": "2026-07-20T05:47:30Z",
"updatedAt": "2026-08-13T16:40:36Z",
"timestamp": "2026-08-13T16:40:36Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"dependencies",
"deepseek",
"intel",
"xpu",
"run-ci"
],
"author": "yuchengliu1",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31785",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DCP][Spec] EAGLE/EAGLE3 support for decode context parallelism — stacked on #21637",
"text": "Makes **EAGLE / EAGLE3** work with decode context parallelism (`--dcp-size > 1`). Stacked on #21637. DFlash × DCP already worked (see #33912, `_dflash_draft_cell_size`); it gets one shared fix here, not new support. **Root cause:** the target KV pool is sharded across DCP ranks, but the draft pool is **replicated**. Draft-side code that reads the DCP topology therefore builds rank-local metadata against a full pool. Every change below follows from that. ## Results 8×B200, K2.5-NVFP4 + EAGLE3 (ns2/topk1/ndt3), `--dcp-comm-backend a2a --dcp-replicate-q-proj`, jobs 1496709–1496996. Accept length is the correctness gate: the DCP arm must match the off arm, since a draft chain fed rank-local metadata degenerates after the first token. | config | off (ms) | DCP=8 (ms) | ratio | AL off → DCP | |---|---|---|---|---| | 50K / cc4 | 4.96 | 5.93 | 0.84× | 2.239 → 2.234 | | 128K / cc16 | 19.58 | 20.44 | 0.96× | 2.160 → 2.150 | | 128K / cc24 | 37.66 | **26.78** | **1.41×** | 2.172 → 2.163 | | 256K / cc8 | 20.63 | 22.19 | 0.93× | 1.756 → 1.781 | | 256K / cc16 | 59.51 | **38.14** | **1.56×** | 1.748 → 1.745 | GSM8K 0.940–0.960 on every arm. Below the KV-pool edge DCP pays the verify addon for no benefit; past it the off arm loses admission first and DCP wins. Attention FLOPs are conserved under context sharding, so there is no headroom to beat the off arm at shallow overcommit — the ~16% cost at 50K/cc4 is the design point, not a regression. ## Tests `test_dcp_draft_guard.py`, `test_dcp_draft_pool_bounds.py`, `test_dcp_spec_topk_guard.py` — all three cover failures that are otherwise **silent**: a guard that stops unsharding draft forwards corrupts the chain with no error, an under-budgeted draft pool OOMs at the first large prefill far from its cause, and a mis-scoped topk validator returns wrong tokens rather than raising. The topk tests pin that DSPARK is *not* gated (`is_dflash_family()` would pull it in and reject the shipping Kimi-Linear + DSPARK + DCP path). Registered DCP/spec suites pass on 8×B200, including main's `test_kimi_linear_dcp4.py`. --- ### CI States Latest PR Test (Base): :x: [Run #31612001671](https://github.com/sgl-project/sglang/actions/runs/31612001671) Latest PR Test (Extra): :x: [Run #31612001435](https://github.com/sgl-project/sglang/actions/runs/31612001435) ",
"url": "https://github.com/sgl-project/sglang/pull/31785",
"createdAt": "2026-07-20T13:39:38Z",
"updatedAt": "2026-08-12T23:42:51Z",
"timestamp": "2026-08-12T23:42:51Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"documentation",
"deepseek",
"speculative-decoding",
"blackwell",
"jit-kernel"
],
"author": "thanhhao98",
"state": "open",
"assignees": [
"kpham-sgl"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31820",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Use piecewise cuda graphs",
"text": "## Motivation Inspired by SGLang's work on breaking graphs into piecewise CUDA graphs, we've built a small, standalone pure-Python package -- piecewise-cuda-graphs (https://github.com/meta-pytorch/piecewise-cuda-graphs). Our goal is to make it easily reusable across the PyTorch community, so that projects don't each have to maintain their own version. This PR swaps SGLang's in-tree piecewise CUDA graph code for the new PyTorch annex package. The behavior is identical in the context of SGLang. We'd love it if you could run your CI on this to see how it holds up in your setup. Whether or not it's a fit to merge is entirely your call -- we completely understand if you'd rather keep it in-tree, and no worries either way. Happy to iterate on anything, and thanks for taking a look! ## Modifications **Code (commit 1).** The `breakable` prefill backend's hand-maintained segmented-capture engine is replaced with the library, called directly at every site: | in-tree (removed) | `piecewise_cuda_graphs` | | --------------------------- | ----------------------- | | `BreakableCUDAGraph` | `CUDAGraphSequence` | | `BreakableCUDAGraphCapture` | `piecewise_graph` | | `eager_on_graph` | `no_graph` | | `break_graph` | `force_no_graph` | `piecewise-cuda-graphs==0.1.0` is added as a dependency. Backend name, config, and CLI flags are unchanged (still `--cuda-graph-backend-prefill breakable`). **Docs (commit 2).** Updates the Breakable CUDA Graph page for the new package. ## Accuracy Tests Qwen3-8B scores **0.840** on mgsm_en with `--cuda-graph-backend-prefill=breakable` (the integration test threshold is 0.80) -- unchanged from the in-tree engine. ## Speed Tests and Profiling No regression from the swap. Benchmarked with `bench_one_batch` on H100, this branch vs. its parent (the in-tree engine), on identical shapes (input 512, output 8, batch 1/4/16) across five models: gpt-oss-120b, Llama-3.1-405B-Instruct-FP8 (tp8), Llama-3-70B-Instruct (tp4), Llama-3-8B-Instruct, and MiniMax-M2.7 (tp4). Every prefill delta -- the only path this swap touches -- is within **+/-1.6%** and mixed in sign. Decode runs on the unchanged `full` backend. Repeating a fixed config on one build puts the harness's run-to-run variance at **~3% std**, which exceeds every prefill delta, so nothing here is distinguishable from noise. ## Limitation after rebase Rebasing onto main picked up four breakable features that `piecewise-cuda-graphs==0.1.0` cannot yet express: `barrier_fn` and `capture_stub` (both #31987), CUDA graph segment dedup (`SGLANG_ENABLE_CUDA_GRAPH_DEDUP`, default off), and XPU (#30273) all dropped here and intended as follow-ups on the package side. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31651768601](https://github.com/sgl-project/sglang/actions/runs/31651768601) Latest PR Test (Extra): :x: [Run #31651768327](https://github.com/sgl-project/sglang/actions/runs/31651768327) ",
"url": "https://github.com/sgl-project/sglang/pull/31820",
"createdAt": "2026-07-20T19:31:28Z",
"updatedAt": "2026-08-12T23:45:16Z",
"timestamp": "2026-08-12T23:45:16Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"documentation",
"dependencies",
"deepseek",
"diffusion"
],
"author": "frgossen",
"state": "open",
"assignees": [
"Oasis-Git"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31821",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] Decode context parallelism (DCP) for DSA models (DeepSeek V3.2, GLM-5.x)",
"text": "Decode context parallelism (DCP) for DSA models (GLM-5.x, DeepSeek V3.2). Extends the MLA DCP path (#14194) to sparse attention. 1. KV cache is interleaved across DCP ranks (`slot % dcp_size`), same as MLA DCP. KV capacity multiplies by dcp_size: 2,292,288 tokens/rank physical -> 4,252,416 logical at dcp2 on the default fp8 KV stack. 2. Indexer K cache is replicated (sized `size * dcp`, indexed by global slot). The indexer is all ReplicatedLinear, so every rank computes identical top-k. Each rank keeps only its owned slots (filter + divide in the index transform kernels). 3. Decode and extend both all-gather q across the DCP group (ranks hold different TP head groups), run sparse attention on the local shard, and LSE-combine (`cp_lse_ag_out_rs_mla`, base-2). 4. trtllm backends only (the sm100 defaults, asserted at launch): the sparse kernel scans a counted valid-first prefix, not -1 skipping. The owner filter is followed by descending-sort compaction, and per-row counts are passed as seq_lens. Needs `return_lse` (flashinfer >= 0.6.14). flashmla_sparse was only used during bring-up and its DCP support is removed (ba054051). 6. Composes with dp-attention (requires `attn_tp_size % dcp_size == 0`) and with EP (TEP4 + DCP4 measured 0.915 GSM8K at 200q). EAGLE works, experimental, needs flashinfer >= 0.6.15 (flashinfer-ai/flashinfer#3238). Prefill CP + DCP is rejected at launch, see F7. 7. `ag_rs` stays the comm-backend default. On a single NVSwitch node at dcp2 it is ~8% faster than `a2a` (multi-node is the opposite, see #21637). Tested on 4x/8x B300 (sm103), GLM-5.2 NVFP4 / FP8 / BF16, flashinfer 0.6.14/0.6.15, sgl-kernel 0.4.5.  The concurrency sweep for each line stops at that config's measured KV capacity, so every point is queue-free. TP4 has the best latency wherever it fits and ends at 68 users (32K) / 17 users (128K); DCP2 and DCP4 continue to 127/180 and 32/45. **Median TPOT (ms) at concurrency 1:** | context | TP4 | DCP2 | DCP4 | |---|---|---|---| | 32K | 6.67 | 11.27 | 11.74 | | 128K | 6.86 | 12.04 | 12.57 | ### KV cache capacity (measured `max_total_num_tokens`, fp8 KV defaults) | parallelism | mem-fraction | physical tokens/rank | logical KV tokens | vs tp4 | |---|---|---|---|---| | tp4 | 0.85 | 2,292,288 | 2,292,288 | 1.0x | | tp4 dcp2 | 0.82 | 2,126,208 | 4,252,416 | 1.85x | | tp4 dcp4 (dcp = tp size) | 0.70 | 1,504,192 | 6,016,768 | 2.62x | Logical capacity scales sub-linearly because mem-fraction has to drop to fund the per-dcp allocations (see F-list). The launch log prints the physical per-rank count; metrics report the logical one. ## Commands Common: `--host 127.0.0.1 --port 31510`. Accuracy evals (fresh server per config, first run discarded for JIT warm-up): - GSM8K: `python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 200 --port 31510` - AIME25: `sgl-eval run aime25 --base-url http://127.0.0.1:31510/v1 --num-examples 30 --n-repeats 4 --num-threads 30 --max-tokens 64000 --temperature 1.0 --top-p 0.95 --thinking` (install: `pip install git+https://github.com/sgl-project/sgl-eval`). Scores are read from metrics.json, error_rate is 0.0 on every row. `--thinking` is required, otherwise the answer stays in the reasoning channel and scoring breaks. More than ~30 threads causes request errors that count as wrong answers. ### Accuracy (GSM8K 8-shot 1319q / AIME25 pass@1 avg-of-4) | checkpoint | stack | parallelism | GSM8K | AIME25 | |---|---|---|---|---| | NVFP4 | trtllm + fp8 KV (defaults) | tp4 | 0.945 | 0.875 | | NVFP4 | trtllm + fp8 KV (defaults) | tp4 dcp2 | 0.939 | 0.925 | | NVFP4 | trtllm + fp8 KV (defaults) | tp4 dcp4 | 0.940 | 0.917 | | FP8 | trtllm + fp8 KV (defaults) | tp4 | 0.944 | 0.908 | | FP8 | trtllm + fp8 KV (defaults) | tp4 dcp2 | 0.940 | 0.883 | | NVFP4 | defaults + dp-attention | tp4 dp2 dcp2 | 0.939 | 0.917 | | NVFP4 | defaults + EAGLE 5-1-6 | tp4 | 0.942 | 0.933 | | NVFP4 | defaults + EAGLE 5-1-6 | tp4 dcp2 | 0.939 | 0.883 | GSM8K 0.939-0.945, AIME25 0.875-0.933 (pass@1 sem ~0.02-0.04 per run). All DCP rows are within noise of baseline. dcp4 needs `--mem-fraction-static 0.70 --cuda-graph-max-bs 256`. Pool sizing does not yet account for the allocations that grow with dcp: the replicated index-K buffers, the trtllm workspace, the indexer capture workspace, and the widened-q extend transients. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 --dcp-size 4 \\ --quantization modelopt_fp4 --chunked-prefill-size 8192 \\ --mem-fraction-static 0.70 --cuda-graph-max-bs 256 --host 127.0.0.1 --port 31510 ``` Bench: `python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 31510 --dataset-name random --random-input-len --random-output-len --random-range-ratio 1.0 --num-prompts --max-concurrency ` (flush cache between points). ### Validated configurations (full commands) **8. Prefill CP only, tp4.** TTFT -48% to -55% at 128k. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --enable-prefill-cp --cp-strategy interleave --quantization modelopt_fp4 \\ --chunked-prefill-size 32768 --mem-fraction-static 0.85 --host 127.0.0.1 --port 31510 ``` **10. trtllm + fp8 KV (sm100 defaults), tp4 baseline** ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.85 --host 127.0.0.1 --port 31510 ``` **11. trtllm + fp8 KV, tp4 + dcp2.** 4,252,416 logical KV tokens. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 --dcp-size 2 \\ --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.82 --host 127.0.0.1 --port 31510 ``` **13. dp-attention, tp4 dp2 baseline, trtllm defaults** ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --dp-size 2 --enable-dp-attention \\ --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.80 --host 127.0.0.1 --port 31510 ``` **14. dp-attention + DCP, tp4 dp2 dcp2, trtllm defaults.** Requires `attn_tp_size % dcp_size == 0`. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --dp-size 2 --enable-dp-attention --dcp-size 2 \\ --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.75 --host 127.0.0.1 --port 31510 ``` **16. EAGLE 5-1-6, no DCP, default kernel stack.** Needs flashinfer >= 0.6.15 (flashinfer-ai/flashinfer#3238). ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --quantization modelopt_fp4 \\ --speculative-algorithm EAGLE --speculative-num-steps 5 \\ --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.85 --host 127.0.0.1 --port 31510 ``` **17. EAGLE 5-1-6 + dcp2, default kernel stack, experimental.** Accept length 4.35-4.61. Verify runs through the trtllm LSE path. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 --dcp-size 2 \\ --quantization modelopt_fp4 \\ --speculative-algorithm EAGLE --speculative-num-steps 5 \\ --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.80 --host 127.0.0.1 --port 31510 ``` ### Not working (full commands) **F1. EAGLE + DCP is experimental (configs 16-17).** Main now pins flashinfer 0.6.15.post1, which has the flashinfer-ai/flashinfer#3238 draft MoE fix, so the stock install works. **F2. bf16 KV cache under DCP.** Works with explicit backends: `--kv-cache-dtype bfloat16 --dsa-prefill-backend trtllm --dsa-decode-backend trtllm` scores 0.935 GSM8K (200q) at dcp2. Without the explicit flags, bf16 KV auto-resolves the prefill backend to flashmla_sparse and DCP rejects it at launch. **F3. dp-attention with dcp > attn_tp.** Rejected at launch: the DCP group would span attention-DP shards (attn_tp = tp/dp = 1 here), which breaks the replicated-indexer invariant. Needs decoupled attn/FFN parallelism (#21637 direction). ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --dp-size 4 --enable-dp-attention --dcp-size 2 \\ --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.75 --host 127.0.0.1 --port 31510 ``` **F8. Fused top-k v2 under DCP.** Disabled: measured incorrect on the current tree (GSM8K 0.000 with fusion on vs 0.920 off, single-commit delta at dcp2). The v2 fused transform stopped composing with the DCP owner filter somewhere in the recent upstream churn; needs root-causing before the decode-perf win can be claimed. Extends additionally need an extend-shaped plan (the v2 plan is decode-shaped). **F4. hisparse + DCP.** Asserted off, the page-translation layer is not DCP-aware. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 --dcp-size 2 \\ --enable-hisparse --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.82 --host 127.0.0.1 --port 31510 ``` **F5. DeepSeek V3.2.** Same code path (`DeepseekV32ForCausalLM` uses the same backend/indexer/model file) but not runtime-tested, no checkpoint on the test box. Expected to work as-is. ```bash python3 -m sglang.launch_server --model-path nvidia/DeepSeek-V3.2-NVFP4 --tp 8 --dcp-size 2 \\ --quantization modelopt_fp4 \\ --chunked-prefill-size 8192 --mem-fraction-static 0.82 --host 127.0.0.1 --port 31510 ``` **F6. PD disaggregation (prefill-CP workers feeding DCP decode workers).** Blocked on sequence-dim KV resharding in PD transfer (#25846). The prefill worker's CP-chunked cache must be re-interleaved to `slot % dcp` ownership during transfer. ```bash # prefill worker (CP): python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 \\ --enable-prefill-cp --cp-strategy interleave \\ --quantization modelopt_fp4 --chunked-prefill-size 32768 --mem-fraction-static 0.85 \\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\ --host 127.0.0.1 --port 31510 # decode worker (DCP): python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 --dcp-size 2 \\ --quantization modelopt_fp4 --mem-fraction-static 0.82 \\ --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\ --host 127.0.0.1 --port 31511 ``` **F7. Prefill CP + DCP.** Rejected at launch. The DCP extend all-gathers q assuming every rank in the group holds the same rows, and prefill CP splits rows across ranks, so the gathered q mixes unrelated tokens (0.000 GSM8K). Fix is to gather position-ordered KV for CP-split extends, like `all_gather_kv_cache_for_mla_extend`. ```bash python3 -m sglang.launch_server --model-path nvidia/GLM-5.2-NVFP4 --tp 4 --dcp-size 2 \\ --quantization modelopt_fp4 --enable-prefill-cp --cp-strategy interleave \\ --chunked-prefill-size 32768 --mem-fraction-static 0.80 --host 127.0.0.1 --port 31510 ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31621859666](https://github.com/sgl-project/sglang/actions/runs/31621859666) Latest PR Test (Extra): :x: [Run #31621859428](https://github.com/sgl-project/sglang/actions/runs/31621859428) ",
"url": "https://github.com/sgl-project/sglang/pull/31821",
"createdAt": "2026-07-20T20:18:45Z",
"updatedAt": "2026-08-12T17:19:22Z",
"timestamp": "2026-08-12T17:19:22Z",
"metrics": {
"reactions": 0,
"comments": 13
},
"labels": [
"deepseek",
"jit-kernel"
],
"author": "vincentzed",
"state": "open",
"assignees": [
"b8zhong",
"kpham-sgl"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:31839",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] register diffusion qknorm-rope benchmark for jit-kernel-benchmark-test-amd",
"text": "## Summary Part of [ROCm/sglang-ci#349](https://github.com/ROCm/sglang-ci/issues/349). Registers the **diffusion QKNorm+RoPE** kernel benchmark for AMD. `diffusion/bench_qknorm_rope.py` compared two providers: `split` (JIT QKNorm + **FlashInfer** RoPE, CUDA-only) and `fused` (SGL JIT). Guard the `split` provider behind `is_hip()` so ROCm benchmarks the fully-fused SGL JIT QKNorm+RoPE kernel alone, and register it for `jit-kernel-benchmark-test-amd`. ## Scope note — this PR is 1 bench, not 4 I initially probed 4 flashinfer-using benches. The probe ([ROCm 7.0 run](https://github.com/sgl-project/sglang/actions/runs/29785219561)) showed only `diffusion/bench_qknorm_rope` actually runs on ROCm. The other 3 fail because their **underlying SGL JIT kernel is itself CUDA-only** (not just the flashinfer provider): - `bench_qknorm` → `qknorm.cuh: 'cuda_bf16.h' file not found` - `bench_norm` → `fused_add_rmsnorm.cuh: 'cooperative_groups/reduce.h' file not found` - `bench_rope` → `pos_enc.cuh: 'cuda_fp16.h' file not found` Those need kernel-source HIP porting (not an `is_hip()` provider guard), so they're reclassified as CUDA-only and tracked on #349 rather than shipped here. ## Verification (`continue_on_error=false` — true gate) - ROCm 7.2 → https://github.com/sgl-project/sglang/actions/runs/29880264279 - ROCm 7.0 → https://github.com/sgl-project/sglang/actions/runs/29880265539 ## Test plan - [x] `diffusion/bench_qknorm_rope` passes on ROCm 7.0 (probe run 29785219561). - [ ] Gate runs green on ROCm 7.0 + 7.2. - [ ] Mark ready for review. --- ### CI States Latest PR Test (Base): :x: [Run #30491658236](https://github.com/sgl-project/sglang/actions/runs/30491658236) Latest PR Test (Extra): :no_entry_sign: [Run #30491810142](https://github.com/sgl-project/sglang/actions/runs/30491810142) ",
"url": "https://github.com/sgl-project/sglang/pull/31839",
"createdAt": "2026-07-20T22:50:01Z",
"updatedAt": "2026-08-13T02:30:51Z",
"timestamp": "2026-08-13T02:30:51Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci"
],
"author": "michaelzhang-ai",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31856",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Accelerate AITER unified-attention decode with scaled FP8 Q",
"text": "### Motivation AITER unified attention already supports FP8 Q + FP8 KV. With BF16 Q, the FP8 KV path cannot select the native FP8-Q matrix-multiply path. Quantizing Q adds one graph node per full-attention layer, but can reduce the much larger `kernel_unified_attention_3d` cost at medium and high decode concurrency. Qwen3.5-397B-A17B-MXFP4 has 15 full-attention layers in one complete decode replay. The added Q-quant cost is therefore nearly constant at 15 launches per replay, while attention-kernel savings grow with batch/concurrency. ### Code changes - Import `scaled_fp8_quant` in `aiter_backend.py`. - Automatically quantize Q when the selected unified-attention decode branch uses an FP8 KV cache; no dedicated environment or architecture gate is required. - Quantize Q only after the request has selected the Triton unified-attention decode branch. - Require an FP8 KV cache before enabling Q quantization. - Prefer `layer.k_scale`, falling back to `self.k_scale`, for Q quantization and `q_descale`. - Pass `q_descale` into `unified_attention` instead of always passing `None`. - Add three registered AMD tests: unified-branch isolation across MLA, vectorized-5D, unified, and legacy decode; a BF16-KV guard that keeps Q in BF16; and an FP8 Q + FP8 KV numerical comparison against a BF16 PyTorch reference at the production decode shape. ## Validation environment - GPU: AMD Instinct MI355X (`gfx950`), TP2 - Docker: `rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260716` - Model: `amd/Qwen3.5-397B-A17B-MXFP4` ## Performance ### 1. Complete decode-trace kernel comparison The trace run uses complete 8K/1K serving requests at concurrency 4, 8, 16, 32, 64, and 128. The analyzer selects the same 30 complete full-batch decode replays across baseline/candidate and TP0/TP1, discards the first selected replay, and reports the slower-TP-rank median of the remaining 29 replays. Each replay contains 15 full-attention layers. `Before` is the default BF16 Q + BF16 KV path. `After` is the FP8 Q + FP8 KV path introduced by this PR and includes one additional Q-quantization launch per layer. Values below are per-call times: each replay-level kernel total is divided by its 15 calls. `After total` is one FP8 attention call plus its corresponding Q-quantization call. | Concurrency | Before: BF16 attention | After: FP8 attention | Added Q quant | After total: FP8 attention + Q quant | Kernel time saved | |---:|---:|---:|---:|---:|---:| | 4 | 10.682 us/call | 11.890 us/call | 4.202 us/call | 16.091 us/call | -50.64% | | 8 | 17.525 us/call | 15.578 us/call | 4.276 us/call | 19.854 us/call | -13.29% | | 16 | 33.639 us/call | 29.317 us/call | 4.279 us/call | 33.596 us/call | +0.13% | | 32 | 62.047 us/call | 52.474 us/call | 4.343 us/call | 56.817 us/call | +8.43% | | 64 | 103.581 us/call | 82.475 us/call | 4.351 us/call | 86.826 us/call | +16.18% | | 128 | 186.339 us/call | 141.419 us/call | 4.436 us/call | 145.855 us/call | +21.73% | The dispatch is trace-confirmed: ```text Before: kernel_unified_attention_3d_..._TILE_SIZE_16_..._IS_Q_FP8_0_IS_KV_FP8_0 After: scaled_quant_kernel kernel_unified_attention_3d_..._TILE_SIZE_32_..._IS_Q_FP8_1_IS_KV_FP8_1 ``` ### 2. Serving E2E Both variants use random 8K input / 1K output, range ratio 0.8, `num_prompts=10*concurrency`, and request rate `inf`. Each concurrency is repeated three times, and all 36 runs completed their expected `10*concurrency` requests. | Concurrency | Before total tok/s | After total tok/s | Throughput change | Before median TPOT | After median TPOT | TPOT change | |---:|---:|---:|---:|---:|---:|---:| | 4 | 3,330.703 | 3,357.509 | +0.80% | 10.305 ms | 10.369 ms | -0.62% | | 8 | 5,418.426 | 5,421.908 | +0.06% | 12.798 ms | 12.777 ms | +0.17% | | 16 | 7,635.264 | 7,628.320 | -0.09% | 18.138 ms | 18.158 ms | -0.11% | | 32 | 10,321.976 | 10,351.448 | +0.29% | 27.323 ms | 27.254 ms | +0.25% | | 64 | 13,149.938 | 13,233.151 | +0.63% | 43.313 ms | 43.009 ms | +0.70% | | 128 | 16,230.473 | 16,435.077 | +1.26% | 70.566 ms | 69.568 ms | +1.41% | #### Long-context 70K/300 serving The long-context run uses random 70K input / 300 output, range ratio 0.8, ignore EOS, `num_prompts=10*concurrency`, request rate `inf`. `Before` is BF16 Q + BF16 KV ; `After` is FP8 Q + FP8 KV. Both use the same pinned image, model, and serving configuration. Each concurrency from 4 through 128 is repeated three times; all 36 runs completed. Values are medians across the three repeats. The TTFT/TPOT columns use the median of each run's mean latency; positive latency change means lower is better. | Concurrency | Before total tok/s | After total tok/s | Throughput change | |---:|---:|---:|---:| | 4 | 18,424.347 | 18,789.606 | +1.98% | | 8 | 19,676.435 | 20,836.995 | +5.90% | | 16 | 20,242.285 | 21,567.647 | +6.55% | | 32 | 20,458.731 | 21,894.467 | +7.02% | | 64 | 20,536.560 | 22,041.995 | +7.33% | | 128 | 20,709.810 | 22,105.603 | +6.74% | | Concurrency | Before mean TTFT | After mean TTFT | TTFT change | Before mean TPOT | After mean TPOT | TPOT change | |---:|---:|---:|---:|---:|---:|---:| | 4 | 3,607.415 ms | 3,470.553 ms | +3.79% | 36.920 ms | 36.333 ms | +1.59% | | 8 | 4,986.867 ms | 4,560.806 ms | +8.54% | 76.553 ms | 72.896 ms | +4.78% | | 16 | 6,872.587 ms | 6,389.923 ms | +7.02% | 159.098 ms | 149.580 ms | +5.98% | | 32 | 10,440.422 ms | 9,966.331 ms | +4.54% | 324.030 ms | 302.379 ms | +6.68% | | 64 | 16,624.100 ms | 15,817.837 ms | +4.85% | 667.008 ms | 619.621 ms | +7.10% | | 128 | 121,580.119 ms | 26,851.283 ms | +77.91% | 989.489 ms | 1,253.606 ms | -26.69% | At C128 this is a TTFT/TPOT trade-off: the FP8 path substantially reduces admission/prefill queueing for the 70K-token burst, while the higher active decode pressure increases per-request inter-token latency. The TTFT gain dominates the TPOT regression: median-of-three mean E2E falls from 389,120.324 ms to 366,323.637 ms (+5.86%), while total throughput rises by 6.74%. An additional independent C128 A/B run reproduced the same direction: +6.77% total throughput, +77.93% TTFT improvement, -26.62% TPOT change, and +5.91% E2E improvement. ## Accuracy GSM8K comparison was run on the exact pinned image/model configuration at generation caps 2048 (`parallel=128`, `temperature=0`): | Variant | Accuracy | Invalid | |---|---:|---:| | BF16 Q + BF16 KV | 0.950 | 0.005 | | FP8 Q + FP8 KV | 0.945 | 0.000 | The candidate delta is -0.005 absolute (-0.5 percentage point), and both variants exceed the 0.92 project gate. ## Activation No separate FP8-Q flag is required. Selecting AITER unified attention with an FP8 KV cache enables the path automatically: ```bash SGLANG_USE_AITER=1 \\ SGLANG_USE_AITER_UNIFIED_ATTN=1 \\ python3 -m sglang.launch_server \\ --model-path amd/Qwen3.5-397B-A17B-MXFP4 \\ --revision 0c14667cee2c50a1412c5d10651675d2d9619b94 \\ --tensor-parallel-size 2 \\ --attention-backend aiter \\ --kv-cache-dtype fp8_e4m3 \\ --page-size 16 \\ ... ``` ## Checklist - [x] Format your code according to [Format code with pre-commit](https://docs.sglang.io/docs/developer_guide/contribution_guide#format-code-with-pre-commit). - [x] Add unit tests according to [Run and add unit tests](https://docs.sglang.io/docs/developer_guide/contribution_guide#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/docs/developer_guide/contribution_guide#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/docs/developer_guide/contribution_guide#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/docs/developer_guide/contribution_guide#benchmark-the-speed). - [x] Follow the SGLang [code style guidance](https://docs.sglang.io/docs/developer_guide/contribution_guide#code-style-guidance). --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31369141567](https://github.com/sgl-project/sglang/actions/runs/31369141567) Latest PR Test (Extra): :x: [Run #31369141291](https://github.com/sgl-project/sglang/actions/runs/31369141291) ",
"url": "https://github.com/sgl-project/sglang/pull/31856",
"createdAt": "2026-07-21T01:10:53Z",
"updatedAt": "2026-08-13T00:39:26Z",
"timestamp": "2026-08-13T00:39:26Z",
"metrics": {
"reactions": 0,
"comments": 11
},
"labels": [
"run-ci"
],
"author": "zijiecode",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31922",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[PD] Prevent outbound ZMQ endpoint cache FD exhaustion",
"text": "Fixes #31766. ## Motivation A prefill worker caches one outbound ZeroMQ PUSH socket and one monitor PAIR socket for every decode endpoint it contacts. Endpoint churn can therefore grow the process FD count until libzmq raises `ZMQError: Too many open files`. In the reported failure, that exception escapes the Mooncake transfer loop as a fatal worker error and leaves in-flight transfers stuck. ## Root cause The outbound cache had no hard capacity. It was also shared by endpoint under a Python lock, although a lock does not make a non-thread-safe ZeroMQ socket safe to use from multiple sender threads. The relevant capacity unit is a `(sender owner thread, decode rank endpoint)` pair, not only an endpoint. With the default four Mooncake transfer shards plus the bootstrap thread that sends `ABORT_ACK`, the conservative stable bound for the reported TP=16, DP=4 topology is: ```text (4 transfer owners + 1 bootstrap owner) * 64 live rank endpoints = 320 pairs ``` During rolling scale or replacement, 64 old and 64 new endpoints can coexist, so the corresponding bound is 640 pairs. Chunk-ready, auxiliary-data, and status sends execute on the same transfer owner and do not add another owner multiplier. Distinct decode instances or ports used by one owner do add distinct live endpoints. ## Changes - Store outbound entries in `threading.local()`, so socket creation, `send_multipart`, monitor consumption, and terminal close all stay on the same owner thread. - `threading.get_ident()` is used only as an ownership assertion; it is not a long-lived cache key. - `_connect()` still returns the raw `zmq.Socket`, preserving existing `send_multipart` arguments, return values, and exceptions. - Enforce a manager-wide hard limit on owner-thread/endpoint pairs before creating either a PUSH or PAIR socket. - Auto-size the default hard limit from the process and libzmq resource ceilings: - one entry uses exactly two ZMQ sockets and normally about three process FDs including the TCP transport; - the outbound cache receives 25% of each resource ceiling, retaining 75% as server headroom; - `auto_limit = min(floor(RLIMIT_NOFILE / 12), floor((ZMQ_SOCKET_LIMIT - 1) / 8))`; - the context `ZMQ_MAX_SOCKETS` setting is raised only as needed for the selected limit and never beyond `ZMQ_SOCKET_LIMIT`. - Keep `SGLANG_DISAGGREGATION_MAX_CACHED_ZMQ_ENDPOINTS` as an explicit positive-integer operator override. - Unset or invalid strings use the automatic calculation (invalid strings also warn). - Zero and negative values fail initialization. - Make entry creation transactional. If PUSH or monitor setup fails, the owner thread attempts to close every partially created socket. - The reserved capacity slot is released immediately only when both the PUSH and monitor PAIR sockets are confirmed closed. - If cleanup is incomplete, the partial entry is retained in an owner-thread pending-cleanup queue and continues to consume its capacity slot. - A later `_connect()` call on the same owner thread retries cleanup before admitting additional endpoints. - Introduce an explicit retiring state for terminal cache entries. - After `EVENT_DISCONNECTED` or `EVENT_CLOSED` is consumed, the entry is marked as retiring and can no longer be returned for sending. - A retiring entry remains referenced and cannot be overwritten by another entry for the same endpoint. - PUSH and monitor cleanup is retried by the owner thread until both sockets are confirmed closed. - The manager-wide capacity slot is released exactly once after complete cleanup. - Propagate cleanup failures through `SocketCacheCleanupError`, alongside `SocketCacheCapacityError`, under the common `SocketCacheError` base class. - Mooncake fails and cleans only the current room while keeping the transfer worker available for later work. - Propagate the dedicated capacity exception through Mooncake send paths. The transfer loop fails and cleans only the current room, then continues processing later work; bootstrap ACK capacity failures are logged without terminating its receive loop. For the local test environment, `RLIMIT_NOFILE=1,048,576` and `ZMQ_SOCKET_LIMIT=65,535`, so the automatic limit is 8,191 pairs. If SGLang raises a lower process limit to its normal 65,535 target, the same formula gives 5,461 pairs. Both cover the 320 stable and 640 two-generation bounds above. ## Why not LRU eviction? A healthy PUSH socket may still own queued control messages. Closing it with `linger=0` merely to admit a new endpoint can silently drop those messages, while waiting for linger can block a sender. This change therefore does not evict healthy or non-terminal entries. An endpoint that never connects and only produces `CONNECT_DELAYED` or `CONNECT_RETRIED` remains admitted. A TTL would have the same message-loss problem. Operators should restore that endpoint, set a larger positive capacity after checking FD headroom, or drain and restart the prefill process. ## Testing CPU-only regression coverage: - New regression file: **9 passed**. - Same pytest process, five complete rounds: **45 passed**. - Sender ownership and multipart-integrity stress in one Python process: **20/20 passed**. - Capacity-full test verifies rejection occurs before any new PUSH/PAIR creation or FD growth, while the existing endpoint still sends and receives. - Terminal recovery verifies both `EVENT_DISCONNECTED` and a deterministic protocol-valid `EVENT_CLOSED` monitor frame release the complete entry, after which a replacement endpoint sends successfully. - Monitor creation rollback returns capacity and closes the partially created PUSH socket. - Mooncake call-chain test verifies capacity failure cleans only the current room and the worker processes the next item. - Latest upstream behavior with 80 unique endpoints: approximately **250 open FDs**; the final minimal-process recheck measured **5 -> 249 (+244)**, with **80 PUSH sockets and 80 monitors** retained. - Patched 80-endpoint churn: **45 -> 52 FDs (+7)**, final cache size **1**, and **all 80 real messages received**. - `py_compile`: passed. - `ruff check`: passed. - `ruff format --check`: passed. - `git diff HEAD^ HEAD --check`: passed. No GPU, model, SGLang server, or Mooncake cluster was started. ## Local environment limitation The pre-existing disaggregation directory cannot complete collection in this shared environment because the installed `transformers.configuration_utils` does not export `PreTrainedConfig`. - Final branch collection: 9 new tests were collected, then 8 pre-existing modules failed collection; the old related suite executed 0 tests. - Exact latest `origin/main` snapshot in the same environment: 0 tests collected and the same 8 collection errors; 0 tests executed. - Both fail at `python/sglang/srt/configs/cohere2_moe.py` with the same `ImportError`. No shared dependency was installed or upgraded. GitHub CI is needed to run the broader suite in its supported dependency environment. ## Operational behavior - Capacity counts owner-thread/endpoint pairs across the manager. - A capacity error includes the endpoint, configured/resolved limit, current cache count, and operator actions. - Healthy entries are never evicted and queued messages are not deliberately dropped. - Terminal entries transition to an owner-thread retiring state after `EVENT_DISCONNECTED` or `EVENT_CLOSED`. - Retiring and partially created entries remain charged against the manager-wide hard limit until both their PUSH and monitor PAIR sockets are confirmed closed. - Incomplete cleanup is retried only by the socket owner thread on a later `_connect()`; no destructor, finalizer, or replacement thread closes ZeroMQ sockets across thread boundaries. - A retiring entry is never returned for sending and cannot be overwritten by a replacement entry for the same endpoint. - Capacity is released exactly once after complete cleanup. It is deliberately not released while a socket may still own an FD. - The cache uses `threading.local()`; Python thread-ID reuse cannot retrieve another thread's socket. - Mooncake transfer workers and the bootstrap receiver thread are long-lived. If an owner exits unexpectedly, its thread-local mapping is no longer reachable by a replacement thread and global capacity accounting remains conservative until a controlled process restart. - This PR does not claim production shutdown teardown: it adds no manager public `close()`, destructor, or finalizer, and it never performs `linger=0` shutdown while a worker may still send. - `CommonKVReceiver` is unchanged. ## Checklist - [x] Scope is limited to issue #31766. - [x] Commit changes exactly three production files and one CPU-only regression file. - [x] Raw socket API and multipart send semantics are preserved. - [x] No cross-thread ZeroMQ socket migration is introduced. - [x] No healthy-cache LRU eviction or queued-message dropping is introduced. - [x] Capacity, rollback, terminal recovery, worker continuation, FD growth, and message integrity are covered. - [x] Rebased onto the latest upstream `main`. - [x] No GPU/model/server/cluster test was run. --- ### CI States Latest PR Test (Base): :x: [Run #29835643649](https://github.com/sgl-project/sglang/actions/runs/29835643649) Latest PR Test (Extra): :x: [Run #29835643221](https://github.com/sgl-project/sglang/actions/runs/29835643221) ",
"url": "https://github.com/sgl-project/sglang/pull/31922",
"createdAt": "2026-07-21T09:24:58Z",
"updatedAt": "2026-08-13T11:08:48Z",
"timestamp": "2026-08-13T11:08:48Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [],
"author": "chengcuiping",
"state": "open",
"assignees": [
"whybeyoung"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31926",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Mooncake] Fix silent SSD offload corruption when TP/PP ranks share ssd_offload_path",
"text": "## Motivation Each scheduler rank embeds its own `MooncakeDistributedStore` client, and `MooncakeStore.__init__ `passes all of them the same ssd_offload_path. Mooncake's SSD tier assumes one client per storage location, and a shared directory corrupts silently: - bucket ids are generated per process (`timestamp<<12` seed with a plain atomic increment, `mooncake-store/src/storage_backend.cpp`) - after a restart every client resumes from the same scanned max_bucket_id, making filename collisions deterministic - bucket files are opened with `O_CREAT|O_TRUNC`, so a collision truncates another rank's bucket and subsequent reads return wrong KV bytes with no error ## Modifications Give every rank a private subdirectory `ssd_offload_path/rank___` (created before setup()). The suffix mirrors the per-rank key convention already used in this file (mha_suffix). When storage_config is None the path is used as-is, unchanged from today. The dp_rank component is required, not cosmetic: with dp-attention, `HiCacheController` reports the attention-group-local rank as tp_rank, which is 0 on every DP rank when attn_tp_size == 1 (e.g. tp8/dp8 DeepSeek), so tp_rank alone collapses all ranks onto one directory. We hit exactly this in a production deployment: 8 clients started in the same second shared one seed and truncated each other's buckets within minutes. `HiCacheStorageConfig` gains a `dp_rank` field (default 0) populated from the dp_rank the controller already computes. Note for existing SSD-offload deployments: previously offloaded buckets at the directory root are orphaned by the layout change (a one-time cache loss, recomputed on miss - not corruption). They can be deleted. ## Accuracy Tests N/A - no change to model outputs (client-side storage path handling only). ## Speed Tests and Profiling N/A - one os.makedirs at engine startup. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31714157632](https://github.com/sgl-project/sglang/actions/runs/31714157632) Latest PR Test (Extra): :x: [Run #31714157430](https://github.com/sgl-project/sglang/actions/runs/31714157430) ",
"url": "https://github.com/sgl-project/sglang/pull/31926",
"createdAt": "2026-07-21T10:08:46Z",
"updatedAt": "2026-08-13T15:13:12Z",
"timestamp": "2026-08-13T15:13:12Z",
"metrics": {
"reactions": 2,
"comments": 1
},
"labels": [
"hicache"
],
"author": "ironashram",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31956",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Optimize MiniMax-M2.7 on CPU",
"text": " ## Motivation Optimize MiniMax-M2.7 on CPU [#issue 26439](https://github.com/sgl-project/sglang/issues/26439) ## Modifications - Add `topk_softmax_cpu` and `topk_sigmoid_cpu` kernels with `correction_bias`, optional renormalization, and FP32 gating output support, and dispatch to them from `fused_topk_cpu`. - Make CPU routing consistently return `int32` `topk_ids` and remove the redundant `int64`-to-`int32` conversion from `fused_experts_cpu`. - Add fused CPU QK RMSNorm kernels for both single-rank and tensor-parallel execution, wire them into `MiniMaxM2QKRMSNorm._forward_cpu`, and register the corresponding CPU compile fake operators. - Add CPU tests covering softmax/sigmoid Top-K routing, correction bias, renormalization, mixed input dtypes, and fused QK RMSNorm under single-rank and TP configurations. ## Accuracy Tests gsm8k: 0.920 hellaswag: 0.770 ```bash # Server python3 -m sglang.launch_server --model MiniMaxAI/MiniMax-M2.7 --disable-radix-cache --trust-remote-code --device cpu --disable-overlap-schedule --tp 6 --tool-call-parser minimax-m2 --reasoning-parser minimax-append-think # Client # gsm8k python -m sglang.test.few_shot_gsm8k --num-questions 200 --num-shots 5 # hellaswag cd benchmark/hellaswag python3 bench_sglang.py --num-questions 200 --num-shots 20 ``` ## Speed Tests and Profiling Benchmark results compared against the baseline (main branch, without the fused optimizations): | Method | E2E Latency (ms) | TTFT (ms) | TPOT (ms) | | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | :----------------------------------------------------------: | | w/ topk_sigmoid, w/ fused_qknorm | 1.30×↑ | 1.13×↑ | 1.30×↑ | ```bash # Env Variable export SGLANG_USE_CPU_ENGINE=1 export SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=0 export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu export LD_PRELOAD=${SGLANG_REPO_DIR}/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so. # server python3 -m sglang.launch_server \\ --model MiniMaxAI/MiniMax-M2.7 \\ --disable-radix-cache \\ --trust-remote-code \\ --device cpu \\ --disable-overlap-schedule \\ --tp 6 \\ --tool-call-parser minimax-m2 \\ --reasoning-parser minimax-append-think # client python3 -m sglang.bench_serving \\ --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json \\ --dataset-name random \\ --random-input 1024 \\ --random-output 1024 \\ --request-rate inf \\ --random-range-ratio 1.0 \\ --num-prompts 1 \\ --max-concurrency 1 ``` --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31655236729](https://github.com/sgl-project/sglang/actions/runs/31655236729) Latest PR Test (Extra): :white_check_mark: [Run #31666597832](https://github.com/sgl-project/sglang/actions/runs/31666597832) ",
"url": "https://github.com/sgl-project/sglang/pull/31956",
"createdAt": "2026-07-21T15:52:00Z",
"updatedAt": "2026-08-13T07:04:17Z",
"timestamp": "2026-08-13T07:04:17Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"sgl-kernel",
"intel",
"cpu",
"run-ci"
],
"author": "xinguozhu-2026",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32059",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feat][DeepSeek V4] Shared KV Cache for Prefill CP",
"text": "## Motivation DeepSeek V4 Prefill context parallelism requires every CP rank to access the complete attention history. The ordinary layout keeps persistent KV replicated across CP ranks and limits long-context capacity. Following the Shared-KV direction introduced in #31435, this PR stores each attention page on one CP owner while exposing one logical address view to all ranks through CUDA VMM. ## Modifications - Add reusable Shared-KV primitives for owner-sharded CUDA VMM storage, logical-to-physical layout, publication, transfer, and row Demand-cache. - Add a thin DeepSeek V4 adapter for owner-sharded SWA/C4/C128 KV and mutable attention compression state. Indexer KV/state remain rank-local replicas. - Use FlashMLA-KV for Prefill and Decode. SM90 and SM100 Prefill use a transient, capacity-guarded one-way Demand-cache; Decode keeps the minimal direct VMM view. Null-cache calls preserve the original FlashMLA direct-read kernels. - Support owner-sharded Prefill-PD transfer and add focused common, DSV4, PD, layout, capacity, and numerical tests. ## Validation Validated on one 8 x NVIDIA H20 node with DeepSeek-V4-Flash-FP8, CP8TP8, FP8 E4M3 KV, FlashMLA-KV Prefill/Decode, page size 256, 8,192-token chunks, and 16,384 maximum Prefill tokens. The 128K/B1 result is the mean of three warmed, cache-flushed runs with identical arguments other than the Shared flag. | Workload | Base CP8 TTFT | Shared KV TTFT | Shared vs Base | | --- | ---: | ---: | ---: | | 128K / B1 | 4,336.64 ms | 4,398.78 ms | +1.43% | The owner-sharded persistent attention cache is single-copy across CP8. The transient Prefill Demand-cache uses 629,145,600 bytes per rank; its storage is reused across layers, but every layer starts a fresh epoch and retains no cross-layer or cross-request validity. The fixed-base release commit passes 290 focused tests plus 90 subtests. Seven SM90 FlashMLA GPU tests produce Base-vs-Demand bit-exact output/LSE, including a 37,376-byte Shared source stride, and the fixed 32K Base/Shared precision case has identical top-1 output with 19/20 top-logprob token overlap. A final 128K/B1/output=1 Shared smoke completed successfully. The FlashMLA extension builds dedicated Demand-cache and direct-read cubins for SM90 and SM100 with CUDA 13; B200 correctness and performance validation remain pending because the available cluster contains only H20/H20-3e nodes. ## Status - [x] Reusable Shared-KV/VMM and Demand-cache foundation - [x] DeepSeek-V4-Flash-FP8 on SM90 with CP8TP8 - [x] FlashMLA-KV Prefill and basic Decode - [x] SM90 transient Prefill Demand-cache - [x] SM100 transient Prefill Demand-cache source and CUDA13 build support - [x] Owner-sharded Prefill-PD path and regression coverage - [x] Correctness, numerical, capacity, and warmed Prefill validation - [ ] Decode performance optimization - [ ] B200/SM100 end-to-end and performance validation - [ ] DeepSeek-V4-Pro-FP8 validation - [ ] Owner-sharded L2/L3 - [ ] Speculative decoding and PP support Enable with `--enable-dsa-shared-kv-cache`. Co-authored with [@foraxe](https://github.com/foraxe) --- ### CI States Latest PR Test (Base): :x: [Run #31615486910](https://github.com/sgl-project/sglang/actions/runs/31615486910) Latest PR Test (Extra): :x: [Run #31615486614](https://github.com/sgl-project/sglang/actions/runs/31615486614) ",
"url": "https://github.com/sgl-project/sglang/pull/32059",
"timestamp": "2026-08-12T16:03:32Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"quant",
"deepseek",
"sgl-kernel",
"jit-kernel"
],
"author": "taoyuanyuan",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:32094",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSA] Add LiteTopk fused indexer top-k prefill path for SM100",
"text": "## Summary This PR is adapted from vLLM PR https://github.com/vllm-project/vllm/pull/48726. Credit to [Heisenberg-Yin](https://github.com/Heisenberg-Yin) Adds **[LiteTopk](https://github.com/Heisenberg-Yin/LiteTopK)** as an opt-in DSA indexer top-k path for prefill (ragged extend): fp8 MQA scoring (tcgen05 UMMA) + an online bucketed gate + a compact exact top-k, fused into **one pass over KV**, so the `[num_q, seq_len]` logits matrix is **never materialized** and the memory-budget chunk loop is skipped entirely. Recall is **exact by construction**. With the per-row affine $b(x) = \\lfloor (x - o) \\cdot \\delta^{-1} \\rfloor$ (code: `origin` $= o$, `inv_delta` $= \\delta^{-1}$) over negated scores $x = -\\text{score}$ (bucket 0 = best), the scan emits exactly $\\{\\, j : b(x_j) \\le \\text{th} \\,\\}$, and every refresh recomputes $\\text{th}$ from counts of genuine row elements — so it only ever *tightens* while staying above $b(x_{(k)})$, the bucket of the true k-th best score: $$\\text{th}_0 \\ge \\text{th}_1 \\ge \\cdots \\ge b\\big(x_{(k)}\\big) \\quad\\Longrightarrow\\quad \\{ j : b(x_j) \\le \\text{th}_t \\} \\supseteq \\text{top-}k \\text{at every } t$$ A stale threshold therefore admits extra candidates but never drops one, and `select` trims the superset exactly. In B200, GLM-5.2 tensors, Q=8192, topk=2048, **1.09–1.25×** over dense logits + top-k at 256K–1M contexts, with **8–32 GB of transient logits memory removed**. It is **opt-in** (`SGLANG_ENABLE_DSA_LITETOPK=1`), **off by default**, SM100 (Blackwell) only, GLM DSA shape only (H=32, D=128), and gated to configurations it can serve exactly (`num_init_tokens == num_local_tokens == 0`); everything else stays on the dense path unchanged, so it adds **zero risk** to existing deployments. ## Background: how LiteTopk replaces the dense top-k The dense prefill path scores **all** KV into a `[num_q, seq_len]` fp32 logits buffer (8–32 GB transient at 256K–1M, forcing a memory-budget chunk loop), then runs top-k over it. LiteTopk streams KV in tiles and keeps only *plausible* candidates: 1. **`seed_prep` (calibration)** — per request, score a bounded KV prefix (`min(8192, kv_len)`) with the existing dense `fp8_mqa_logits`, and derive per-row bucket-space parameters (`origin`, `inv_delta`) plus an initial gate threshold = the bucket of the k-th best sample score. A sample k-th value can only be *worse* than the true global k-th, so the initial threshold is a valid loose bound. 2. **`scan`** — the fused kernel: UMMA-scores each KV tile, folds the per-row bucket affine into the register weights (scores accumulate directly in bucket units), and emits positions whose bucket passes the gate into a compact candidate buffer via warp-local queues. Two spare warps run a threshold-refresh daemon that walks the live bucket histogram and **only tightens** the gate as real counts accumulate. 3. **`select`** — exact top-k over the compact candidates (radix on float bits in bucket space), `-1`-padded to `topk`. ## Modifications **New** - `python/sglang/jit_kernel/csrc/dsa_litetopk/dsa_indexer_kernels.cuh` — the warp-specialized scan kernel (vendored 1:1 from vLLM PR #48726). - `python/sglang/jit_kernel/csrc/dsa_litetopk/dsa_indexer.cuh` — `seed_prep` / `select` kernels, TMA helpers, shape config (vendored). - `python/sglang/jit_kernel/csrc/dsa_litetopk/entry.cuh` — the only sglang-facing layer (TVM-FFI launchers); mirrors the upstream launcher argument order so future syncs stay mechanical. - `python/sglang/jit_kernel/csrc/dsa_litetopk/vendor_deep_gemm/` — minimal vendored DeepGEMM headers (tcgen05 / TMA / UMMA PTX), with the DeepSeek license preserved. - `python/sglang/jit_kernel/dsa/litetopk.py` — Python orchestration: per-request calibration loop, scratch allocation, the three primitive calls. - `test/registered/jit/test_dsa_litetopk.py` — recall tests vs an fp32 torch reference (details below). **Modified** - `python/sglang/srt/layers/attention/dsa/dsa_indexer.py` — `use_dsa_litetopk` gate in `Indexer.__init__` + `_get_topk_ragged_litetopk` branch in `_get_topk_ragged`. - `python/sglang/srt/environ.py` — `SGLANG_ENABLE_DSA_LITETOPK` (`EnvBool`, default `False`). ## sglang deviations - **Per-request gate calibration.** Upstream calibrates every row from a single shared `kv[:sample_len]` prefix of the gathered buffer — in a multi-request ragged batch that samples positions a row can never attend, which can over-tighten the gate. Here each request samples **its own** KV prefix with per-row causal `ke`, so thresholds are causally valid by construction. - **`-1` index padding** (upstream pads `0`), matching the dense ragged transform contract (`fast_topk_transform_ragged_fused`) so downstream sparse attention sees the same invalid-slot marker. Output coordinates are identical to the dense path: gathered-KV absolute positions. - **Real SM count** for the KV-split heuristic instead of the hardcoded 148 (the kernel template keeps `NUM_SMS` for 1:1 vendoring). - **Flat-score bucket-span floor** (second commit). With all-identical K rows the sample span degenerates to ~0, the original $10^{-20}$ floor makes `inv_delta` explode, and cross-kernel fp32 accumulation-order noise then spans whole buckets — the gate dropped *every* candidate. Floor the span at $\\sim \\text{mag}/256$ (one bucket $\\geq \\text{mag} \\cdot 2^{-16}$, ~16× above the noise) plus an absolute $10^{-6}$ floor. Coarser buckets only *loosen* the gate, so the fix is recall-safe by construction. Upstream has the same latent issue (only validated on real tensors); found by the adversarial test below. - **Float4 row-alignment padding** (third commit). `seed_prep` reads sample-logits rows with 16 B `float4` loads; the per-request sample width `min(sample_len, kv_len)` is arbitrary, and any width not a multiple of 4 misaligns every odd row base — a device-side fault upstream never hits (its shared prefix has a fixed width). The wrapper pads the width to a multiple of 4 with `-inf` (already skipped by every pass's `isfinite` guard), and the entry layer fails loudly on misaligned widths instead of faulting. ## Implementation notes - **Recall safety is monotonic end to end**: the initial threshold is a loose upper bound from a causally-valid sample; every refresh recomputes from counts of genuine row elements and only tightens; the gate consumes a one-window-stale prefetched threshold (staleness admits extras, never drops); `select` is exact over the surviving candidates. - The scan keeps the **DeepGEMM 2.5 scoring loop** (register-held weights, row-pair TMEM loads, early UMMA release) with **non-persistent KV-split scheduling** (`blockIdx.x` = q-block, `blockIdx.y` = KV window), which keeps all SMs busy on the small-Q chunks long-context prefill actually produces. - Ragged final q-blocks are handled by forcing padded rows to an empty KV range; TMA out-of-bounds reads zero-fill and are dropped by the per-row range checks. - The JIT build pins `sm_100a` and vendors only the DeepGEMM headers the kernel needs; nothing is added to the AOT `sgl-kernel` wheel. ## Validation Hardware: NVIDIA B200 (SM100), CUDA 13. ### Correctness — recall vs an independent fp32 torch reference `test/registered/jit/test_dsa_litetopk.py`, registered on the `4-gpu-b200` CI runner (`base-b-kernel-unit`). The reference recomputes $$\\text{score}[r, j] \\;=\\; \\sum_{h} w[r, h] \\cdot \\mathrm{relu}\\!\\big(q[r, h, :] \\cdot k[j, :]\\big) \\cdot s[j], \\qquad j \\in [\\, ks[r],\\ ke[r] \\,)$$ in fp32 (with $s[j]$ the per-position dequant scale `kv_scale`); checks are tie-tolerant exactness: every selected index must score $\\geq v_k - \\varepsilon$, and every index scoring $> v_k + \\varepsilon$ must be selected (with $v_k$ the k-th best reference score, $\\varepsilon = 10^{-3} \\cdot \\max(|v_k|, 1)$), plus count / `-1` padding / no-duplicate / causal-validity assertions. - `[(32, 32768)] @ topk=2048` — single long request. - `[(16, 8192), (16, 24576), (8, 4096)] @ topk=2048` — multi-request ragged. - `[(32, 32768)] @ topk=512`. - `[(8, 1024)] @ topk=2048` — short rows: valid < topk → `-1` padding. - `[(9, 4099), (5, 1023)] @ topk=512` — odd sample widths (float4 alignment regression) + ragged final q-block padding. - **Flat-score adversarial** (`[(8, 16384)]`, all-identical K rows): every position ties at the k-th value — worst case for the threshold-bucket boundary and candidate-buffer pressure. Caught the bucket-span bug fixed in the second commit (0 candidates selected pre-fix; count + validity + uniqueness green post-fix). ### Performance — upstream kernel-level measurement From the vLLM PR #48726 evaluation (B200, GLM-5.2 tensors, Q=8192, topk=2048), LiteTopk vs dense `fp8_mqa_logits` + top-k: **1.09× / 1.17× / 1.25×** at 256K / 512K / 1M context, with the 8–32 GB transient logits allocation removed. sglang-side e2e numbers will accompany the PR that turns the path on by default. ## Notes - **Exact top-k SET, unspecified order**: tie-breaking at the k-th value follows atomic arrival order, and the within-row output order is unsorted (the dense radix path is also unordered). Not valid for deterministic mode. - The exact-set guarantee holds while the candidate buffer bound (`cand_cap` $= \\max(4 \\cdot \\text{topk},\\ 16384)$ per row) is not exceeded; the calibration sample must also be $\\geq$ `topk` (default $8192 \\geq 2048$). - Prefill (ragged extend) only; decode keeps the existing paged path. GLM DSA shape only (H=32, D=128); fp8 index-K cache only. - The dense path is untouched and remains the default; the env flag flips a per-`Indexer` boolean, nothing else. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31557797915](https://github.com/sgl-project/sglang/actions/runs/31557797915) Latest PR Test (Extra): :x: [Run #31557797919](https://github.com/sgl-project/sglang/actions/runs/31557797919) ",
"url": "https://github.com/sgl-project/sglang/pull/32094",
"createdAt": "2026-07-22T15:39:13Z",
"updatedAt": "2026-08-13T14:14:31Z",
"timestamp": "2026-08-13T14:14:31Z",
"metrics": {
"reactions": 2,
"comments": 6
},
"labels": [
"documentation",
"deepseek",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "yuan-luo",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:32114",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Delete cutlass_mla, non-Marlin GPTQ, AWQ AOT kernel, and Dual Chunk Flash Attention",
"text": "## Summary - Delete the `cutlass_mla` attention backend (kernel + Python integration): SM10.0-only decode kernel, already disabled on GB300 (SM10.3), falls through to `FlashInferMLABackend` for everything except plain decode, no CI coverage. - Delete the non-Marlin GPTQ CUDA kernel (`gptq_gemm`/`gptq_shuffle`) and the `\"gptq\"` GPU quantization choice: superseded by `gptq_marlin` (fully JIT), which `GPTQMarlinConfig` already auto-upgrades compatible checkpoints to. `auto-round` and `moe_wna16` lose their non-Marlin GPU fallback as a result (their Marlin paths are unaffected). - Delete the AWQ AOT dequantize kernel (`awq_dequantize`): every call site already tries a JIT kernel then a Triton kernel first; the AOT path was an unreachable-in-practice last resort. XPU is unaffected (resolves `sgl_kernel.awq_dequantize` from the separate `sgl-kernel-xpu` package). - Delete Dual Chunk Flash Attention (`dual_chunk_flash_attn`, Qwen-1M long-context backend): single-model-family opt-in, no functional kernel change in ~1 year. - Update `docs_new` (quantization/attention-backend/server-arguments pages, DeepSeek-V3 cookbook) and delete the now-orphaned examples/tests for each of the above. ## Test plan - [x] CI (not triggered on this draft — hold for review) - `python3 -m py_compile` on all modified Python files (done locally) - pre-commit hooks (done locally) Fixes #32111 Fixes #32112 --- ### CI States Latest PR Test (Base): :x: [Run #31394593765](https://github.com/sgl-project/sglang/actions/runs/31394593765) Latest PR Test (Extra): :x: [Run #31394590747](https://github.com/sgl-project/sglang/actions/runs/31394590747) ",
"url": "https://github.com/sgl-project/sglang/pull/32114",
"timestamp": "2026-08-12T16:11:27Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"documentation",
"quant",
"speculative-decoding",
"sgl-kernel",
"blackwell",
"run-ci"
],
"author": "b8zhong",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:32166",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Use SYCL kernels for DeepSeek V4 MHC on XPU",
"text": "## Motivation Enable XPU-specific optimized kernels for the DeepSeek V4 MHC (Multi-Head Compression) path on Intel XPU hardware. The existing CUDA-path kernels (deep_gemm, CUDA-first device selection) are either unavailable or suboptimal on XPU, causing incorrect device dispatch or missing performance opportunities. ## Modifications deepseek_v4.py: On XPU, dispatch hc_head to sgl_kernel.mhc.fused_hc_head (SYCL kernel) instead of the Triton implementation; dispatch mhc_fused_post_pre to the XPU-specific kernel; skip deep_gemm when running on XPU; fix device selection to avoid falling back to CUDA when XPU is available. kernels/ops/layernorm/mhc.py: Guard CUDA-only code paths so they are not triggered on XPU. test/registered/xpu/test_hc_head.py: Add XPU CI test comparing SYCL fused_hc_head against the Triton reference for both accuracy (assert_close, atol/rtol=2e-2) and performance (assertLess median latency). ## Accuracy Tests Tested fused_hc_head (SYCL) vs Triton reference on Intel XPU with hidden_size=7168, hc_mult=4, dtype=bfloat16, num_tokens=[1, 16, 128, 1024]: All configurations pass torch.testing.assert_close(atol=2e-2, rtol=2e-2). Max absolute difference is within bfloat16 precision bounds (≤ 0.03125), mean difference ≈ 0. ## Speed Tests and Profiling Benchmarked fused_hc_head SYCL kernel vs Triton on Intel XPU (BMG), hc_mult=4, dtype=bfloat16 (median latency, 50 iterations): | batch_size | seq_len | hidden_size | dtype | sycl_ms | triton_ms | speedup | |-------------:|----------:|--------------:|:---------|----------:|------------:|----------:| | 1 | 16 | 4096 | bfloat16 | 0.033488 | 0.06552 | 1.957 | | 1 | 16 | 7168 | bfloat16 | 0.054496 | 0.094744 | 1.739 | | 1 | 128 | 4096 | bfloat16 | 0.043862 | 0.075218 | 1.715 | | 1 | 128 | 7168 | bfloat16 | 0.073606 | 0.11596 | 1.575 | | 1 | 1024 | 4096 | bfloat16 | 0.222664 | 0.366938 | 1.648 | | 1 | 1024 | 7168 | bfloat16 | 0.38298 | 0.732524 | 1.913 | | 4 | 16 | 4096 | bfloat16 | 0.038324 | 0.068536 | 1.788 | | 4 | 16 | 7168 | bfloat16 | 0.06162 | 0.110292 | 1.79 | | 4 | 128 | 4096 | bfloat16 | 0.137748 | 0.194922 | 1.415 | | 4 | 128 | 7168 | bfloat16 | 0.23192 | 0.428142 | 1.846 | | 4 | 1024 | 4096 | bfloat16 | 0.746642 | 1.21654 | 1.629 | | 4 | 1024 | 7168 | bfloat16 | 1.29662 | 2.64555 | 2.04 | | 16 | 16 | 4096 | bfloat16 | 0.076388 | 0.09984 | 1.307 | | 16 | 16 | 7168 | bfloat16 | 0.12974 | 0.223678 | 1.724 | | 16 | 128 | 4096 | bfloat16 | 0.390572 | 0.627458 | 1.607 | | 16 | 128 | 7168 | bfloat16 | 0.679328 | 1.32392 | 1.949 | | 16 | 1024 | 4096 | bfloat16 | 2.88233 | 5.00391 | 1.736 | | 16 | 1024 | 7168 | bfloat16 | 5.07629 | 10.7714 | 2.122 | | 64 | 16 | 4096 | bfloat16 | 0.22256 | 0.366912 | 1.649 | | 64 | 16 | 7168 | bfloat16 | 0.383344 | 0.731328 | 1.908 | | 64 | 128 | 4096 | bfloat16 | 1.45382 | 2.46189 | 1.693 | | 64 | 128 | 7168 | bfloat16 | 2.55819 | 5.32249 | 2.081 | | 64 | 1024 | 4096 | bfloat16 | 11.4992 | 20.5583 | 1.788 | | 64 | 1024 | 7168 | bfloat16 | 20.1847 | 43.6769 | 2.164 | --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31376396565](https://github.com/sgl-project/sglang/actions/runs/31376396565) Latest PR Test (Extra): :x: [Run #31376396388](https://github.com/sgl-project/sglang/actions/runs/31376396388) ",
"url": "https://github.com/sgl-project/sglang/pull/32166",
"createdAt": "2026-07-23T07:27:38Z",
"updatedAt": "2026-08-13T06:21:16Z",
"timestamp": "2026-08-13T06:21:16Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"deepseek",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "cyxlily",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32196",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[PD] Keep EAGLE DP graph and token metadata consistent",
"text": "Fixes #32182. In PD decode, target and draft can use different MoE A2A backends. The scheduler kept target-local token counts for the draft worker, so DP ranks disagreed on draft padding and collective shapes. Missing GLM DSA seeds could also make only active ranks fall back to eager. This change synchronizes graph eligibility across DP ranks and keeps draft-specific token counts without changing the target DeepEP path. Validation: - GLM-5.2 EAGLE PD, DP32/TP32, target DeepEP + draft A2A none - 32/32 routed warmup requests returned HTTP 200; no NaN/Inf, traceback, or CUDA error --- ### CI States Latest PR Test (Base): :x: [Run #31573591543](https://github.com/sgl-project/sglang/actions/runs/31573591543) Latest PR Test (Extra): :x: [Run #31573591347](https://github.com/sgl-project/sglang/actions/runs/31573591347) ",
"url": "https://github.com/sgl-project/sglang/pull/32196",
"createdAt": "2026-07-23T11:20:26Z",
"updatedAt": "2026-08-13T10:47:44Z",
"timestamp": "2026-08-13T10:47:44Z",
"metrics": {
"reactions": 0,
"comments": 10
},
"labels": [
"bug",
"run-ci"
],
"author": "weireweire",
"state": "open",
"assignees": [
"Fridge003",
"kpham-sgl"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32214",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Strict bit-exact SWA HiCache for DeepSeek-V4 with unified_kv: SWA-window + c4/indexer state riding across L1/L2/L3",
"text": " ## Motivation DeepSeek-V4's `unified_kv` backend on ROCm packs three KV families in one pool: the content-addressed compressed KV (C4/C128), the per-request **SWA ring** (addressed by `req_pool_idx * window + pos % window`), and the compressor's **c4 / indexer overlap state**. Only the compressed KV is content-stable; the SWA ring and the c4/indexer overlap state are not, and are never written into the radix tree. Two prior PRs made reuse of such prefixes *safe* but only **approximately correct**: - **#29417** enabled unified_kv HiCache, but it offloads **only** the compressed KV (C4/C128). The SWA ring is not preserved, so on a hierarchical hit the trailing sliding window is **re-prefilled** (`swa_reprefill_tail_tokens()`) rather than restored. - **#30339** applied the same **re-prefill** idea to the radix-only path: cap the match by the trailing window and recompute it into the request's own ring, so decode never reads a stale ring. Re-prefill recomputes the trailing window (and, implicitly, the c4/indexer overlap state for that region) instead of reproducing the exact values the original prefill wrote. That is enough to prevent gross stale-ring errors and to keep GSM8K-level accuracy within noise — but it is **not bit-exact**: a partial re-prefill cannot reconstruct the original SWA window and c4/indexer overlap state byte-for-byte, so the reused output can differ from a full fresh compute at the bit level. This PR closes that gap. Instead of re-prefilling an approximation, it **captures and offloads the true values** of the SWA window and the c4/indexer overlap state across L1→L2→L3, and **restores them exactly** on a hierarchical hit — making unified_kv SWA prefix reuse **strict and bit-exact**. A hard reuse gate re-prefills only when the truth is unavailable (never a stale/approximate read). This is an **opt-in** gated by `SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE` (default **off**), and it engages only when **all** of the following hold at once: - running **DeepSeek-V4** on **ROCm/HIP**; - the **`unified_kv_triton`** attention backend is chosen (`SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton`); - **HiCache is enabled** (`--enable-hierarchical-cache`). In every other case, the feature is a strict no-op, no pool is allocated, no capture, no restore; behavior is byte-identical to `main`. ## Modifications Organized as three layers: **1. Substrate (Commit 1–5):** - `SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE` flag + `--hicache-swa-offload-page-stride` arg (`environ.py`, `server_args.py`). - Dedicated pinned-host SWA-window offload pool with capture-done handshake + upstream deferred release (`memory_pool_host.py`). - Two independent, tile-start-addressed L3 pools for c4 state and indexer state, packing (`deepseek_v4_memory_pool.py`, `hybrid_cache/hybrid_pool_assembler.py`). - Version-namespaced L3 keys (`hicache_storage.py`, `storage/nixl/hicache_nixl.py`). - Startup fail-fast guard (`hybrid_pool_assembler.py`): with the strict flag on, raise a clear `ValueError` unless `--hicache-write-policy write_through`, before any host pool is pinned. **2. Capture (Commit 6–8):** - Capture the SWA ring window at each completed prefill/decode page with EAGLE-safe geometry (`deepseek_v4_backend_hip_radix.py`, `model_runner.py`, `deepseek_v4.py`). - Snapshot c4/indexer overlap state for the same window into the independent state pools (`compress_hip.py`, `compressor_v2.py`). **3. Reuse/restore + strict gate (Commit 9–10):** - Ride the captured window + state back to device at the correct `buf_lo` offset on a hierarchical hit; **strict reuse gate** caps the match and re-prefills any tail that isn't provably bit-exact, never a stale read; commit-coupling guard keeps window+state+full-KV committed atomically (`swa_component.py`, `unified_radix_cache.py`, `base_prefix_cache.py`, `schedule_batch.py`, `schedule_policy.py`). ## Note to reviewers This PR can be split into three cumulative sub-PRs on request — substrate / capture / reuse (boundaries C1–5 / C6–8 / C9–10) — each gated by the same flag and a strict no-op by default. ## Accuracy Tests Two detectors (like #30339): a byte-exact harness is the real detector; GSM8K is only a no-regression guard. ### 1. Unit Tests (5 test files, 163 UTs total) **How:** `cd test && python3 -m pytest -q srt/mem_cache/test_swa_*.py` **Expected:** `163 passed` — byte-exact swa+c4_state/c4_indexer_state capture/restore across L1→L2→L3, strict-gate reject, commit-coupling guard, dirty-read protection, L3-only restore. ### 2. E2E (DeepSeek-V4 unified_kv on ROCm, HiCache + strict flag) **How:** launch with the gates on and `SGLANG_SWA_DBG_CHECKSUM=1` (emits a per-window checksum vs a fresh device re-read), trigger prefix reuse, grep the log. ```bash export SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton export SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=1 # dev; set 0 for the \"off\" baseline export SGLANG_SWA_DBG_CHECKSUM=1 # + standard dsv4 env as in #30339 python3 -m sglang.launch_server --model-path /data/models/DeepSeek-V4-Pro --attention-backend dsv4 \\ --page-size 256 --swa-full-tokens-ratio 0.1 --enable-cache-report --port 30001 \\ --enable-hierarchical-cache --hicache-write-policy write_through \\ --hicache-io-backend direct --hicache-mem-layout layer_first \\ --tp 4 --dp 4 --enable-dp-attention 2>&1 | tee server.log # decode + EAGLE spec path (tp8/dp8): add --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-num-draft-tokens 4 --speculative-eagle-topk 1 # L3-only variant: add --hicache-storage-backend file --hicache-storage-prefetch-policy wait_complete # + SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/ssd/dir, populate, kill, restart same dir, resend prefix ``` **Trigger reuse** — send the same long prompt N rounds; `cached_tokens` climbs 0 → large: ```bash python3 - <<'PY' import json, urllib.request P = \"This is a long shared prefix sentence. \" * 4000 # >> page_size (256) tokens for r in range(6): # 6 for prefill; 20 for decode dp8 (every rank must hit) b = json.dumps({\"model\": \"x\", \"prompt\": P, \"max_tokens\": 8, \"temperature\": 0}).encode() u = json.loads(urllib.request.urlopen(urllib.request.Request( \"http://127.0.0.1:30001/v1/completions\", data=b, headers={\"Content-Type\": \"application/json\"})).read())[\"usage\"] print(\"round\", r, \"cached_tokens\", (u.get(\"prompt_tokens_details\") or {}).get(\"cached_tokens\", 0)) PY ``` **Expected** (grep `server.log`): | check | grep | expect | | --- | --- | --- | | byte-exact SWA + c4/indexer | `[C4-STATE-DBG]` state ride bit-exact / `[LB-DEV]` device landing byte-exact / `[SWA-DBG]` restore verified | **L1/L2 same-process reuse**: all three **> 0**. **L3-only cross-restart**: `[LB-DEV]` + `[C4-STATE-DBG]` **> 0** (`[SWA-DBG]` is **0** on this path — the prefill checksum is not persisted across a process restart, so the window's byte-exact verification is carried by `[LB-DEV]` device-landing). | | no failure / no dirty read | `MISMATCH\\|reuse_reject\\|BIND-MISS\\|AssertionError\\|Traceback` | **0** | | L3-only source | `prefetch .*matched=[0-9]+ loaded=[0-9]+` | `matched=0 loaded>0` | **L3-only restore validated** (cold restart, empty L1/L2): `prefetch matched=0 loaded=2304`, the first request immediately reports `cached_tokens=2304`, `[LB-DEV]` + `[C4-STATE-DBG]` byte-exact all pass, zero `MISMATCH`. **gsm8k eval**: Two arms, each run once per server profile — off (SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=0) and dev (=1) — with the cache flushed (/flush_cache) before each pass and temperature=0, so the only variable is the strict flag. GSM8K is a no-regression guard (the byte-exact harness above is the real correctness detector); the acceptance criterion is that dev is not below off and invalid ≈ 0. - non-thinking ``` python3 benchmark/gsm8k/bench_sglang.py \\ --host 127.0.0.1 --port 30001 --num-shots 8 --num-questions 1319 --parallel 256 ``` For off (SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=0) and dev (=1) , all above tests we got acc ~0.94(invalid ≈ 0). - thinking ``` python3 -m sglang.test.run_eval \\ --eval-name gsm8k --host 127.0.0.1 --port 30001 \\ --num-threads 256 --temperature 0.0 --repeat 1 --num-shots 8 --max-tokens 8192 ``` For off, we got 0.933, and for dev, we got 0.942(invalid 0). ## Speed Tests and Profiling Both arms HiCache on, `DBG=0`, `--hicache-ratio 1.0` (single variable = strict flag); customer clients. Just take some of the configs as example. | scenario | metric | off | dev | note | | --- | --- | --- | --- | --- | | Prefill (dp4, 50k ISL, prefix-cache0.6) | tokens/s | ≈ 32.7k | ≈ 30.8k | dev ≈ −5.8%| | Prefill (dp4, 50k ISL, prefix-cache0.8) | tokens/s | ≈ 31.4k | ≈ 29.1k | dev ≈ −7.3%| | Decode TPOT (dp8, EAGLE, batch 32/48/64) | ms/token | ≈ 70.6 / 82.4 / 96.8 | ≈ 70.8 / 83.8 / 98.4 | dev slightly slower, no regression | The prefill cost is the deterministic price of strict correctness (positional H2D of the SWA window + c4/indexer state on reuse, not a content-addressed page copy), applied only with the flag on. **Optional in-repo reproduction.** The table above comes from the customer perf clients. For a self-contained check that needs no external scripts, the stock bench_serving client reproduces the strict overhead directly in-repo (it is a different, smaller workload, so absolute numbers differ). Launch the same server twice with DBG=0 — once SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=1 (dev), once =0 (off) — then run against each: ```bash python3 -m sglang.bench_serving --backend sglang --dataset-name random \\ --random-input-len 50000 --random-output-len 200 --random-range-ratio 1 \\ --num-prompts 64 --max-concurrency 64 --seed 42 --port 30001 ``` Compare peak output token throughput: dev trails off by a small, consistent margin (measured off ≈ 1536 tok/s vs dev ≈ 1472 tok/s, ≈ −4%), the same direction as the customer-client table. Note that at this concurrency/input length the Median/P99 TTFT is dominated by request queueing (~50–100 s) and is not a reliable reuse detector — prefix-reuse correctness is verified separately by the byte-exact harness and the climbing cached_tokens above, not by this client's TTFT. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31686468859](https://github.com/sgl-project/sglang/actions/runs/31686468859) Latest PR Test (Extra): :x: [Run #31686468531](https://github.com/sgl-project/sglang/actions/runs/31686468531) ",
"url": "https://github.com/sgl-project/sglang/pull/32214",
"createdAt": "2026-07-23T13:24:12Z",
"updatedAt": "2026-08-13T09:25:37Z",
"timestamp": "2026-08-13T09:25:37Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"deepseek",
"hicache"
],
"author": "amd-danli103",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32269",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support XQA backend for SpecDec verify",
"text": " ## Motivation Adds support for TRT-LLM XQA decode backend for SM90/120 speculative decoding, by building a causal mask it requires for the verify forward pass (more than 1 q token). Agent generated summary: Before: pure `--attention-backend trtllm_mha` + NEXTN on SM90/120 died at server startup during decode CUDA-graph capture — verify routes to the XQA decode kernel, which demands a mask for `q_len > 1` that the backend never passed: ``` File \"/SGL/flashinfer/flashinfer/xqa.py\", line 332, in xqa assert mask is not None, \"Mask is required for speculative decoding\" AssertionError: Mask is required for speculative decoding ``` Never caught before because the verify→decode-kernel routing was written for SM100 (trtllm-gen needs no mask); XQA enablement for SM90/120 came later and nobody ran spec decoding on that combo. ## Modifications Fix/After: precompute XQA's bit-packed causal mask (constant for chain/topk=1 speculation) at init and pass it on verify — flashinfer already plumbed the argument. ## Accuracy Tests Validated: boots through capture, GPQA-10 0.70, acceptance 2.20/0.40 matching the FA3-verify reference (see MTP data reported in https://github.com/sgl-project/sglang/pull/23112). ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31565406415](https://github.com/sgl-project/sglang/actions/runs/31565406415) Latest PR Test (Extra): :x: [Run #31565405956](https://github.com/sgl-project/sglang/actions/runs/31565405956) ",
"url": "https://github.com/sgl-project/sglang/pull/32269",
"createdAt": "2026-07-24T02:07:30Z",
"updatedAt": "2026-08-13T06:49:43Z",
"timestamp": "2026-08-13T06:49:43Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"performance",
"blackwell",
"run-ci",
"hopper"
],
"author": "akhilg-nv",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32280",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] Enable LLaDA2.2 block routing and graph-captured JTI on Ascend",
"text": "## Summary This PR adds self-contained LLaDA2.2-flash support for block-routing MoE and JointThresholdInDel (JTI) on Ascend NPU. Model-forward graph capture continues to use SGLang's existing NPU dLLM graph path; this PR adds graph capture only for the fixed-width JTI token-update step. It incorporates only the block-routing contract needed from #31768 and the JTI semantics needed from #31773. Those PRs are implementation references, not merge dependencies; this branch remains buildable on `main`. ## Scope and boundaries Model- and algorithm-specific code stays under LLaDA2/JTI: - `python/sglang/kernels/ops/llada2/npu.py`: Ascend block routing and edit-token reductions. - `python/sglang/srt/dllm/algorithm/joint_threshold_indel.py`: fixed-shape eager and NPU-graph JTI. - `python/sglang/srt/models/llada2.py`: selects block routing only for checkpoints that declare `expert_capacity`; the LLaDA2.1 path is unchanged. There are no changes to shared model-executor, prefill-runner, NPU attention, or NPU graph-backend code. Both dLLM prefill and iterative decode already use `ForwardMode.DLLM_EXTEND`, which is captured by the existing `NPUGraphRunner`. ## Block-routing behavior The NPU block-routing path follows this contract: 1. select `expert_capacity` experts from the maximum corrected score in each token block; 2. select each token's ordered top-k within that compact capacity; 3. use unbiased sigmoid scores for the returned weights; 4. resolve exact ties by lower expert ID without perturbing close but unequal scores; 5. normalize in FP32 when `top_k > 1`; preserve the raw sigmoid score for `top_k == 1`. The implementation keeps the CANN top-k fast path and folds `routed_scaling_factor` into `topk_weights`, matching the existing NPU MoE contract. ## JointThresholdInDel The JTI implementation preserves the required termination and accuracy semantics: - the prompt is the immutable contiguous prefix before the first MASK; - an original MASK predicted as MASK uses the best non-MASK fallback; - DELETE/SPLIT move original-mask tracking through compaction; - regular and post-edit budgets force a final scrub that excludes MASK, DELETE, and SPLIT; - a cleanup that changes token IDs gets one final forward to persist KV state; - graph and non-graph paths use fixed-shape state and are covered for parity. `enable_graph` applies only to the fixed-width JTI token-update step and is NPU-only. Model-forward graph capture remains on the existing `NPUGraphRunner`. FDFO keeps the non-graph fixed-shape path because its batch composition changes between rounds. ## Historical hardware results BF16, BS1, TP4, native top-8, block size 32, GSM8K first 200: | Model/path | Aggregate tok/s | Correct | Output tokens | |---|---:|---:|---:| | LLaDA2.1-flash reference | 190.419 | 183/200 (91.5%) | 62,931 | | LLaDA2.2 before routed-scale fix | 204.882 | 156/200 (78.0%) | 56,987 | | LLaDA2.2 with routed-scale fix | 258.354 | 189/200 (94.5%) | 52,329 | | LLaDA2.2 validated standalone tree | **252.332** | **191/200 (95.5%)** | 52,567 | The validated run used one warmup request followed by 200 serial requests on Ascend 910B3 NPU 0-3. Aggregate throughput is total output tokens divided by the sum of request durations; wall-clock throughput was 252.282 tok/s. BFCL-V4 `simple_python` was run once at temperature 1.0 and scored with BFCL's official AST checker: **371/399 (92.98%)**. This is a directional compatibility check, not a reproduction of the model card's five-run aggregate score. These hardware results predate the final scope-only cleanup. The full hardware benchmark was not rerun after that cleanup. Artifacts: - [LLaDA2.2 GSM8K-200 raw JSON](https://huggingface.co/babba82200/test1/resolve/main/llada22_pr32280_gsm8k200_raw.json?download=true) - [Measurement summary](https://huggingface.co/babba82200/test1/resolve/main/llada22_pr32280_gsm8k200_results.md?download=true) - [BFCL raw JSON](https://huggingface.co/babba82200/test1/resolve/main/llada22_bfcl_simple_python_399_raw.json?download=true) - [BFCL scored JSON](https://huggingface.co/babba82200/test1/resolve/main/llada22_bfcl_simple_python_399_scored.json?download=true) - [LLaDA2.1 GSM8K-200 JSON](https://huggingface.co/babba82200/test1/resolve/main/llada21_flash_npu_bf16_bs1_tp4_jt_gsm8k200.json?download=true) - [Warmed one-block Chrome trace](https://huggingface.co/babba82200/test1/resolve/main/llada22_flash_npu_bf16_bs1_tp4_jt_graph_206tps_oneblock.trace.json?download=true) ## Configuration ```yaml threshold: 0.5 edit_threshold: 0.0 enable_graph: true ``` ```text --dllm-algorithm JointThresholdInDel --dllm-algorithm-config /path/to/jti.yaml ``` The model config is the source of truth for `delete_token_id` and `split_token_id`; SGLang does not contain a checkpoint-specific fallback. The currently published LLaDA2.2-flash config does not declare these fields, so it requires: ```text --json-model-override-args '{\"delete_token_id\": 156930, \"split_token_id\": 156931}' ``` ## Current-head validation - JTI graph/non-graph parity, termination, configuration, and LLaDA2 routing tests: **11 passed** on CPU. - Python compilation and `git diff --check`: passed; GitHub Lint is pending on this head. - Rebased cleanly on current `upstream/main`. - NPU kernel regression tests cover tail blocks, exact ties, close unequal scores, ordered top-k, `top_k == 1`, and final scrub behavior. Device CI awaits the protected `run-ci` / `run-ci-extra` labels. ## Known limitations - DeepEP block routing is rejected until its capacity-routing semantics are implemented. - JTI can require more forwards per output token than JointThreshold because DELETE/SPLIT may introduce masks and post-edit rounds. - Current NPU graph mode does not support FDFO state packing. --- ### CI States Latest PR Test (Base): :x: [Run #31691927991](https://github.com/sgl-project/sglang/actions/runs/31691927991) Latest PR Test (Extra): :x: [Run #31691927830](https://github.com/sgl-project/sglang/actions/runs/31691927830) ",
"url": "https://github.com/sgl-project/sglang/pull/32280",
"createdAt": "2026-07-24T03:59:43Z",
"updatedAt": "2026-08-13T10:38:28Z",
"timestamp": "2026-08-13T10:38:28Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"npu",
"jit-kernel"
],
"author": "leedongkun30-arch",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32313",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] Optimize TP LMHead with All-to-All",
"text": " ## Motivation When the model is deployed with pure DP and dp-attention, **--enable-dp-lm-head will suffer from poor gemm efficiency when batchsize of each rank is small**. For example, in DeepseekV4 Pro, we use dp=8 and --enable-dp-lm-head. In such case, the lmhead gemm itself costs **320us** when bs=36 on each dp rank. In small batch, TP LMHead is a better choice. However, current implementation of TP LMHead will introduce allgather communication on vocab_size. In deepseek v4, each rank will send and receive [bs*8, vocab_size // 8 * 7] = **31MB** data, which costs around **150us**. Therefore, we propose to use **All2All to replace the allgather + scatter communication in TP LMHead**. With All2All, the data volume that each rank sends and receives decreases to: - send: [bs * 7, vocab_size // 8]. When bs=36, it's **3.88MB**. - receive: [bs, vocab_size // 8 * 7]. When bs=36, it's **3.88MB**. The all2all costs around 46us in profile. Note that the **all2all optimization is tailored for dp-attention with pure DP parallel**. As for dp-tp hybrid parallel, it's hard to support with significant improvement. ## Modifications 1. Use all2all operation in _get_logits after lmhead computation is completed. 2. Only support pure dp. Dp-lm-head, cp, dp-tp hybrid parallel are not supported to be turned on together. ## Accuracy Tests GSM8k test on deepseek v4 is OK. ## Speed Tests and Profiling Profile | Setup | gemm (us) | communication (us) | Total duration of LMHead (us) | | --- | --- | --- | --- | | dp-lm-head | 320 | \\ | 320 | | tp-lm-head-allgather | 71 | 151 | 278 | | tp-lm-head-all2all | **71** | **46** | **169** | Launch and benchmark command ``` # prefill GLOO_SOCKET_IFNAME=eth1 NCCL_SOCKET_IFNAME=eth1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT=1 SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN=1 SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW=1 SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_OPT_USE_ONLINE_COMPRESS=1 SGLANG_EXPERIMENTAL_ONLINE_C128_MTP=1 SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0 SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1 sglang serve --trust-remote-code --model-path ./models/deepseek-ai/DeepSeek-V4-Pro/ --tp 8 --ep 8 --moe-a2a-backend megamoe --enable-dsa-prefill-context-parallel --dsa-prefill-cp-mode round-robin-split --chunked-prefill-size 63488 --max-prefill-tokens 63488 --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.78 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --tokenizer-worker-num 8 --host 0.0.0.0 --port 9091 > log 2>&1 & # decode GLOO_SOCKET_IFNAME=eth1 NCCL_SOCKET_IFNAME=eth1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_OPT_USE_ONLINE_COMPRESS=1 SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0 SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1 sglang serve --trust-remote-code --model-path ./models/deepseek-ai/DeepSeek-V4-Pro/ --tp 8 --dp 8 --enable-dp-attention --moe-a2a-backend megamoe --mem-fraction-static 0.92 --swa-full-tokens-ratio 0.09 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --enable-hisparse --hisparse-config '{\\\"top_k\\\": 1024, \\\"device_buffer_size\\\": 4096, \\\"host_to_device_ratio\\\": 5}' --tokenizer-worker-num 8 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --host 0.0.0.0 --port 9092 > log 2>&1 & # benchmark python3 -m sglang.bench_serving --model ./models/deepseek-ai/DeepSeek-V4-Pro/ --flush-cache --host 0.0.0.0 --port 9090 --backend sglang --dataset-name generated-shared-prefix --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1024 --gsp-num-groups 1 --gsp-prompts-per-group 1024 --gsp-system-prompt-len 61440 --gsp-question-len 4096 --gsp-output-len 1024 --request-rate 10 --warmup-requests 10 ``` Benchmark result The TPOT decreases **36.97ms** (dp-lmhead) => **36.21ms** (tp-lmhead-allgather) => **35.67ms** (all2all). The E2E latency decreases **3082.24ms** compared with dp-lmhead, **1151.63ms** compared with allgather. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 3. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 4. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 5. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31684326632](https://github.com/sgl-project/sglang/actions/runs/31684326632) Latest PR Test (Extra): :x: [Run #31684326467](https://github.com/sgl-project/sglang/actions/runs/31684326467) ",
"url": "https://github.com/sgl-project/sglang/pull/32313",
"createdAt": "2026-07-24T10:27:12Z",
"updatedAt": "2026-08-13T11:21:00Z",
"timestamp": "2026-08-13T11:21:00Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci"
],
"author": "SYChen123",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32327",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DeepSeek-V4] Add Q8KV8 sparse MLA prefill runtime backend",
"text": "Co-authored-by:@JackChuang ## Summary This PR is part of the roadmap tracked in https://github.com/sgl-project/sglang/issues/25746. This PR ports the Q8KV8 sparse MLA prefill path to the DeepSeek-V4 runtime backend and adds a runtime dispatch path via `--dsv4-prefill-backend flashmla_sparse_q8`. When the KV cache uses `fp8_e4m3`, DeepSeek-V4 can run the FP8 query × FP8 KV sparse MLA kernel during prefill instead of falling back to the existing BF16 sparse prefill path. The runtime side also adds Q-head padding, attention-sink padding, kernel input-contract validation, and regression coverage for a real serving shape that previously caused a DeepSeek-V4 hang. Validation has been completed on DeepSeek-V4-Flash with TP8, FP8 KV cache, `flashinfer_mxfp4` MoE, and CUDA graph default serving. Performance experiments show that Q8KV8 delivers consistent throughput and TTFT gains over the existing `flashmla_sparse` baseline, while GSM8K and LongBench-v2 do not show a material accuracy regression. ## Motivation DeepSeek-V4 already supports FP8 KV cache, but the existing sparse prefill backend still primarily uses the BF16 sparse MLA prefill path. As a result, even when KV is stored in FP8, the runtime cannot fully benefit from the native FP8 Q8KV8 sparse prefill kernel. This PR adds a DeepSeek-V4 Q8KV8 runtime path so prefill can directly use FP8 tensor-core computation. ## Modifications | file | role | |---|---| | `deepseek_v4_backend.py` | DeepSeek-V4 integration + `flashmla_sparse_q8` routing (`_forward_prefill_sparse_q8kv8`), Q/head-sink padding, FP8 workspace, C0/C4/C128 sparse-prefill dispatch | | `dsv4/sparse_prefill_utils.py` | Q8KV8 backend selection helper, debug env override, dtype-aware sparse-prefill workspace reuse | | `dsv4/dequant_k_cache.py` | fused paged KV gather + dequant + requant to FP8; padded Q cast for Q8KV8 prefill | | `sparse_mla_q8kv8_prefill_sm90.py` | fail-fast Python-side input contract validation before launching the SM90 Q8KV8 sparse prefill kernel | | `server_args.py` | `flashmla_sparse_q8` in DeepSeek-V4 prefill backend choices via `--dsv4-prefill-backend` | | `test_q8kv8_sparse_prefill_backend.py` | runtime wrapper tests for Q8KV8 DeepSeek-V4 sparse prefill, including C0/C4/C128 behavior | | `test_server_args.py` | CLI tests for `--dsv4-prefill-backend` choices | ## Usage | Variant | `--dsv4-prefill-backend` | Notes | |---|---|---| | Baseline / q16 | `flashmla_sparse` | Existing sparse prefill backend | | Q8KV8 / q8 | `flashmla_sparse_q8` | New Q8KV8 sparse prefill backend | ```bash python -m sglang.launch_server \\ --model-path \\ --trust-remote-code \\ --tp 8 \\ --kv-cache-dtype fp8_e4m3 \\ --mem-fraction-static 0.85 \\ --max-running-requests 64 \\ --dsv4-prefill-backend flashmla_sparse_q8 \\ --moe-runner-backend flashinfer_mxfp4 \\ --disable-radix-cache \\ --chunked-prefill-size 8192/16384 \\ --port 30000 ``` ## Experimental Results ### Performance(H20) We measured prefill performance with `chunked_prefill_size=8192` and `chunked_prefill_size=16384` under the same DeepSeek-V4-Flash serving setup. For each chunk size, we evaluate both `c=1` and `c=16`, where `c` means request concurrency. `c=1` measures single-request prefill latency and input throughput, while `c=16` measures the saturated serving case with 16 concurrent prefill requests. Q8KV8 consistently improves input throughput and TTFT across all tested input lengths, chunk sizes, and concurrency settings. #### chunk16384 — c=1 single-request latency: | il | q8 tps | q16 tps | Δtps | q8 TTFT | q16 TTFT | ΔTTFT | |---:|---:|---:|---:|---:|---:|---:| | 3.5k | 11,565 | 11,080 | +4.4% | 303 ms | 316 ms | +4.4% | | 8k | 12,257 | 11,631 | +5.4% | 653 ms | 688 ms | +5.4% | | 16k | 12,498 | 11,815 | +5.8% | 1,280 ms | 1,354 ms | +5.8% | | 32k | 11,967 | 11,292 | +6.0% | 2,674 ms | 2,834 ms | +6.0% | | 64k | 11,074 | 10,394 | +6.5% | 5,779 ms | 6,158 ms | +6.5% | | 120k | 9,746 | 9,097 | +7.1% | 12,312 ms | 13,191 ms | +7.1% | | 160k | 9,018 | 8,390 | +7.5% | 17,743 ms | 19,070 ms | +7.5% | #### chunk16384 — saturated throughput (c=16): | il (c) | q8 tps | q16 tps | Δtps | q8 TTFT | q16 TTFT | ΔTTFT | |---:|---:|---:|---:|---:|---:|---:| | 32k (c16) | 12,336 | 11,580 | +6.5% | 23,217 ms | 24,702 ms | +6.4% | | 64k (c16) | 11,389 | 10,628 | +7.2% | 49,643 ms | 53,189 ms | +7.1% | #### chunk8192 — c=1 single-request latency: | il | q8 tps | q16 tps | Δtps | q8 TTFT | q16 TTFT | ΔTTFT | |---:|---:|---:|---:|---:|---:|---:| | 3.5k | 11,587 | 11,048 | +4.9% | 302 ms | 317 ms | +4.9% | | 8k | 12,278 | 11,620 | +5.7% | 652 ms | 688 ms | +5.7% | | 16k | 12,117 | 11,447 | +5.9% | 1,320 ms | 1,398 ms | +5.9% | | 32k | 11,658 | 10,982 | +6.2% | 2,745 ms | 2,914 ms | +6.1% | | 64k | 10,784 | 10,088 | +6.9% | 5,934 ms | 6,344 ms | +6.9% | | 120k | 9,578 | 8,848 | +8.2% | 12,529 ms | 13,562 ms | +8.2% | | 160k | 8,860 | 8,165 | +8.5% | 18,058 ms | 19,595 ms | +8.5% | #### chunk8192 — saturated throughput (c=16): | il (c) | q8 tps | q16 tps | Δtps | q8 TTFT | q16 TTFT | ΔTTFT | |---:|---:|---:|---:|---:|---:|---:| | 32k (c16) | 11,974 | 11,269 | +6.3% | 23,672 ms | 25,114 ms | +6.1% | | 64k (c16) | 11,072 | 10,369 | +6.8% | 50,012 ms | 53,361 ms | +6.7% | ### Accuracy For GSM8K, we use the full evaluation set with `--num-examples 1319` and keep the default deterministic evaluation configuration from `sglang.test.run_eval`. GSM8K is a short-context mathematical reasoning benchmark, so it mainly checks whether the Q8KV8 sparse prefill path preserves the model's normal reasoning behavior under the same serving configuration. Both q8 and q16 runs use the same model, tokenizer, TP size, FP8 KV cache, chunked prefill size, and sampling/evaluation settings; the only changed server-side parameter is `--dsv4-prefill-backend`. For LongBench-v2, we use a filtered long-context subset with `--num-examples 120`, `--min-context-length 10000`, and `--max-context-length 200000`. The context-length filter keeps the evaluation focused on long-prefill workloads while avoiding outlier samples that exceed the model/server context limit after tokenization. We follow the stochastic long-context setting with `temperature=0.6`, `top_p=0.95`, and `max_tokens=8192`, matching the style of prior DeepSeek-V4 sparse prefill accuracy checks. Under this setup, the comparison is intended to test whether the Q8KV8 prefill implementation changes long-context answer quality, rather than to report a full LongBench-v2 leaderboard number. | Dataset(chunk8192) | q8 | q16 | Δ (q8 - q16) | |---|---:|---:|---:| | GSM8K | 0.970 | 0.969 | +0.001 | | LongBench-v2 | 0.509 | 0.515 | -0.006 | The GSM8K result is unchanged, with Q8KV8 slightly higher by 0.001 absolute. On the LongBench-v2 filtered subset, Q8KV8 is lower by 0.006 absolute, which is small for this stochastic long-context evaluation setting. Overall, these results do not show a material accuracy regression from switching the sparse prefill path from `flashmla_sparse` to `flashmla_sparse_q8`. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31698620874](https://github.com/sgl-project/sglang/actions/runs/31698620874) Latest PR Test (Extra): :x: [Run #31698620580](https://github.com/sgl-project/sglang/actions/runs/31698620580) ",
"url": "https://github.com/sgl-project/sglang/pull/32327",
"createdAt": "2026-07-24T14:16:14Z",
"updatedAt": "2026-08-13T14:28:16Z",
"timestamp": "2026-08-13T14:28:16Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [
"quant",
"deepseek",
"run-ci",
"jit-kernel",
"bypass-fastfail",
"run-ci-extra"
],
"author": "shiyang814-cpu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32340",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Amd/dsv4 shared experts fusion top6",
"text": "# [AMD] DeepSeek-V4: fix shared-experts fusion for top-6 ## Summary Enabling shared-experts fusion (`--enforce-shared-experts-fusion`) for DeepSeek-V4 on MI355X (gfx950) crashed at startup. Two independent issues in the fused topk / append path assume DeepSeek-V3 conventions (fp32 correction bias, power-of-two topk) that DeepSeek-V4 (bf16 correction bias, **top-6** routing) violates. This PR fixes both so the fused path runs, and shows it is a **~+8–11% output-throughput win at low concurrency with accuracy preserved**. ## What changed - **`python/sglang/srt/layers/moe/topk.py`** — in `biased_topk_jit_kernel_impl`, cast `correction_bias` to `float32` before calling `moe_fused_gate`. DeepSeek-V4 stores `e_score_correction_bias` in bf16 (for the aiter sqrtsoftplus topk path), but `moe_fused_gate` asserts a float32 bias (`AssertionError: bias must be float32`). Only the fused-gate branch (taken when `num_fused_shared_experts > 0`) is touched. - **`python/sglang/kernels/ops/moe/fused_moe_triton_kernels.py`** — `_fused_append_shared_experts_kernel` indexed the routed/shared lanes with `tl.arange(0, K)` / `tl.arange(0, S)`, which Triton only accepts for power-of-two ranges. DeepSeek-V4 routes top-6 (`K=6`), so it failed with `ValueError: arange's range must be a power of 2`. The kernel now iterates over `next_power_of_2(K)` / `next_power_of_2(S)` blocks with masking, mirroring the existing `_fused_append_shared_experts_with_weights_kernel` sibling. Behavior is unchanged for power-of-two `K`. - **`test/registered/moe/test_fused_append_shared_experts_top6.py`** — new unit test covering non-power-of-two `K`/`S` (incl. the DSV4 `K=6` case) against a pure-torch golden reference. ## Validation Model: `deepseek-ai/DeepSeek-V4-Pro`. Hardware: MI355X (gfx950), TP4. Image: `rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260724`. Baseline = `--disable-shared-experts-fusion`, Fusion = `--enforce-shared-experts-fusion`. ### Unit test ``` $ python3 -m pytest test/registered/moe/test_fused_append_shared_experts_top6.py -q 3 passed, 7 subtests passed ``` ### Accuracy (GSM8K, 2000 questions × 3 runs) | run | baseline | fusion | |----:|:--------:|:------:| | 1 | 0.942 | 0.943 | | 2 | 0.939 | 0.937 | | 3 | 0.936 | 0.940 | | **avg** | **0.939** | **0.940** | Invalid ≈ 0.000 for all runs. Accuracy is preserved (within run-to-run noise). ### Performance (ISL=8192, OSL=1024, output token throughput) | concurrency | baseline (tok/s) | fusion (tok/s) | Δ | |------------:|:----------------:|:--------------:|:-----:| | 1 | 69.59 | 77.06 | **+10.7%** | | 2 | 122.31 | 135.33 | **+10.6%** | | 4 | 216.26 | 238.91 | **+10.5%** | | 8 | 366.29 | 403.37 | **+10.1%** | | 16 | 569.47 | 613.60 | **+7.7%** | ### Commands used Server (per config; `$FLAG` = `--enforce-shared-experts-fusion` or `--disable-shared-experts-fusion`): ```bash export SGLANG_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0 export SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton sglang serve --model-path deepseek-ai/DeepSeek-V4-Pro \\ --tensor-parallel-size 4 --attention-backend dsv4 --trust-remote-code \\ --kv-cache-dtype fp8_e4m3 --page-size 256 --mem-fraction-static 0.9 \\ --cuda-graph-max-bs 16 --max-running-requests 16 \\ --context-length 10240 --chunked-prefill-size 8192 $FLAG ``` Accuracy: ```bash python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 1200 ``` Performance (per concurrency `C` in 1 2 4 8 16): ```bash python3 benchmark_serving.py --model deepseek-ai/DeepSeek-V4-Pro --backend sglang \\ --dataset-name random --random-input-len 8192 --random-output-len 1024 \\ --random-range-ratio 0.8 --num-prompts $((C*10)) --max-concurrency $C --ignore-eos ``` ## Test plan / Checklist - [x] `--enforce-shared-experts-fusion` no longer crashes for DeepSeek-V4 (top-6) on gfx950. - [x] New unit test `test_fused_append_shared_experts_top6.py` passes (non-pow2 K/S incl. K=6). - [x] GSM8K accuracy preserved vs baseline (0.940 vs 0.939). - [x] Output throughput improves ~+8–11% at concurrency 1–16 (TP4, 8k/1k). - [x] Power-of-two `K` behavior unchanged (regression guard in the unit test). --- ### CI States Latest PR Test (Base): :x: [Run #31675757901](https://github.com/sgl-project/sglang/actions/runs/31675757901) Latest PR Test (Extra): :x: [Run #31675757742](https://github.com/sgl-project/sglang/actions/runs/31675757742) ",
"url": "https://github.com/sgl-project/sglang/pull/32340",
"createdAt": "2026-07-24T18:32:44Z",
"updatedAt": "2026-08-13T09:11:49Z",
"timestamp": "2026-08-13T09:11:49Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"amd",
"deepseek",
"run-ci",
"jit-kernel"
],
"author": "karverma-amd",
"state": "open",
"assignees": [
"kkHuang-amd",
"1am9trash",
"RolaoDenthu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32382",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Make checkpoint_engine worker device-agnostic",
"text": "> [!IMPORTANT] > **Depends on https://github.com/MoonshotAI/checkpoint-engine/pull/96** > (adds Intel XPU support to checkpoint-engine). That PR must merge first > `pyproject_xpu.toml` tracks checkpoint-engine's upstream `main`, which only > resolves to XPU-capable code once #96 lands. ## Motivation SGLang's checkpoint-engine integration enables fast in-place weight updates for a running server (the RL / online-serving use case), but the worker hardcoded `torch.cuda` for device UUID/id resolution. As a result the ZMQ handshake with checkpoint-engine's `ParameterServer` only worked on CUDA. This makes the worker device-agnostic so the same path works on Intel XPU, and along the way fixes a latent NPU bug: the ParameterServer keys NPU devices as `NPU-`, but the old CUDA fallback would have keyed them as `GPU-` — a key that never resolves. ## Modifications - `srt/checkpoint_engine/checkpoint_engine_worker.py` - `_accelerator_type()` now delegates to SGLang's device-agnostic `get_device()` instead of assuming CUDA; device-module lookups go through `get_device_module()` (removing the `getattr(torch, ...)` indirection). - `get_device_uuid()` mirrors `checkpoint_engine.ps::_get_physical_gpu_id` exactly: NPU → `NPU-`, every other accelerator (CUDA, XPU) → `GPU-`. This is the fix for the NPU key mismatch above. - `python/pyproject_xpu.toml` - Add the `checkpoint-engine` optional dependency for the XPU build. XPU support is not yet in a released checkpoint-engine, so it tracks upstream source (`git+https://github.com/MoonshotAI/checkpoint-engine.git`) until a release ships it, mirroring how `sgl-kernel-xpu` is referenced in the same file. - `test/registered/unit/checkpoint_engine/test_checkpoint_engine_worker.py` - CPU-mockable tests for cuda/xpu/npu key-format routing and the `AssertionError`→`ValueError` wrapping. - A hardware-gated XPU test asserting the worker's real device UUID equals the key the `ParameterServer` independently derives (the actual handshake contract). - Registered for both CPU (`base-a-test-cpu`) and XPU (`stage-b-test-1-gpu-xpu`) CI. ## Accuracy Tests Not applicable — this changes device-identity resolution for the weight-update handshake, not model forward/kernels. Model outputs are unaffected. ## Speed Tests and Profiling Not applicable — no change to the inference hot path. `get_device()` / `get_device_module()` are called only during the weight-update handshake, not per token. ## Checklist - [x] Format your code according to the Format code with pre-commit. - [x] Add unit tests according to the Run and add unit tests. - [ ] Update documentation according to Write documentations. - [ ] Provide accuracy and speed benchmark results (N/A — see above). - [x] Follow the SGLang code style guidance. --- ### CI States Latest PR Test (Base): :x: [Run #31661962279](https://github.com/sgl-project/sglang/actions/runs/31661962279) Latest PR Test (Extra): :x: [Run #31661962159](https://github.com/sgl-project/sglang/actions/runs/31661962159) ",
"url": "https://github.com/sgl-project/sglang/pull/32382",
"createdAt": "2026-07-25T05:24:48Z",
"updatedAt": "2026-08-13T04:41:05Z",
"timestamp": "2026-08-13T04:41:05Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"dependencies",
"run-ci"
],
"author": "siju-samuel",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32405",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MoE Refactor] Migrate SM100 trtllm-gen mxfp4 MoE onto MoeRunner",
"text": "## Motivation Part of the MoE refactor tracked in #8715. cc @ch-wan `Mxfp4MoEMethod` was the last quant method with a partially migrated path: the SM90 CUTLASS branch went through `MoeRunner` (#26489), but the SM100 trtllm-gen branch still called the kernel inline from `apply`, with `create_moe_runner` falling through to `pass` and a `TODO`. This migrates it, so both FlashInfer MXFP4 GPU paths in `Mxfp4MoEMethod` now use the shared runner pipeline. ## Modifications - Add `FlashInferTrtllmGenMxfp4MoeQuantInfo` and `_fused_experts_flashinfer_mxfp4_sm100_trtllm_gen`, holding the relocated kernel call. - `fused_experts_none_to_flashinfer_mxfp4` becomes a dispatcher: both mxfp4 paths register under the single `(\"none\", \"flashinfer_mxfp4\")` key quant_info type selects the kernel. - `create_moe_runner` accepts `trtllm_sm100` alongside `cutlass_sm90` and now raises `NotImplementedError` for unhandled combinations instead of silently leaving `self.runner` unset. New tests - `test_mxfp4_trtllm_gen.py` (`base-b`, `4-gpu-b200`): asserts that `apply` and a direct `trtllm_fp4_block_scale_moe` call produce equal output. Both reach the same kernel, so what this actually checks is that `apply` assembles the kernel arguments correctly. ## Accuracy Tests Validated on a single B200 (driver 580.159.04, CUDA 13.0, flashinfer 0.6.15.post1, torch 2.11.0+cu130) with `openai/gpt-oss-20b` (mxfp4), `--moe-runner-backend flashinfer_mxfp4`, at ep_size=1 tp_size=1. Bitwise logprob comparison, base `8727d105d` against this branch, 6 fixed prompts, greedy, 192 scored output tokens per cell: | precision | text identical | token ids equal | logprobs bitwise equal | |---|---|---|---| | default | yes | yes | yes, max delta 0.000e+00 | | bf16 | yes | yes | yes, max delta 0.000e+00 | GSM8K was also run with `few_shot_gsm8k --num-questions 200`. The accuracy column is the fraction of the 200 questions whose parsed answer was correct, so 0.520 means 104 of 200: | commit | precision | accuracy | invalid | |---|---|---|---| | `8727d105d` base | default | 0.525 | 0.155 | | this branch | default | 0.520 | 0.165 | | `8727d105d` base | bf16 | 0.540 | 0.155 | | this branch | bf16 | 0.520 | 0.140 | | `8727d105d` base, rerun of the same commit | bf16 | 0.570 | 0.145 | The last row is the important one. Identical code scored 0.540 and then 0.570, a 0.030 swing larger than any gap between base and branch, so GSM8K at 200 questions cannot resolve this comparison in either direction. The bitwise comparison above is the meaningful result. ## Speed Tests and Profiling N/A ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31725851714](https://github.com/sgl-project/sglang/actions/runs/31725851714) Latest PR Test (Extra): :x: [Run #31725851338](https://github.com/sgl-project/sglang/actions/runs/31725851338) ",
"url": "https://github.com/sgl-project/sglang/pull/32405",
"createdAt": "2026-07-25T18:55:22Z",
"updatedAt": "2026-08-13T17:29:07Z",
"timestamp": "2026-08-13T17:29:07Z",
"metrics": {
"reactions": 1,
"comments": 17
},
"labels": [
"run-ci",
"bypass-fastfail"
],
"author": "Beihao-Zhou",
"state": "open",
"assignees": [
"ch-wan"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32414",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add Reasoning-Aware Compression (RAC) pruning recipe for reasoning models",
"text": "Co-authored with Ryan Lucas (MIT) and Kayhan Behdin (LinkedIn). ## Motivation Compressing reasoning models with standard pruning does more damage than compressing a conventional LLM, and it can make the model **slower**. One-shot pruning picks weights by minimizing a layer-wise reconstruction error against a calibration activation matrix `X`: ``` min_{W'} || W X - W' X ||_F^2 s.t. ||W'||_0 <= S ``` `X` is conventionally built from **prompt** tokens (C4, or task prompts). That is a fair proxy when the prompt dominates the token count. Reasoning models invert the ratio: they emit thousands of chain-of-thought tokens per query, so nearly every forward pass the pruned model will ever run is over a token it generated itself. Calibrating on prompts alone optimizes the pruned weights for a distribution the model barely visits. The resulting failure mode is worse than a graceful accuracy drop — the pruned model rambles, emitting more thinking tokens for a less accurate answer, so pruning *increases* end-to-end latency. From the paper (DeepSeek-R1-Distill-Qwen-7B, MATH-500, SparseGPT @ 50% sparsity, 1M calibration tokens): | Calibration set | acc@1 | Eval wall clock | | --- | --- | --- | | Dense (no pruning) | 0.936 | 23.3 min | | C4 | 0.744 | 135.0 min | | Task prompts only | 0.812 | 115.6 min | | **RAC (prompts + on-policy CoT)** | **0.900** | **35.3 min** | Reasoning-Aware Compression (RAC) fixes this by calibrating on the dense model's own on-policy rollout, reconstructing prompt and decode activations jointly: ``` X_RAC = [ X_prompt , X_decode ] ``` The solver is untouched, so RAC is a drop-in calibration-set swap for any existing SparseGPT/Wanda workflow. From [*Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction*](https://arxiv.org/abs/2509.12464) (Lucas, Behdin, Wang, Tang, Song, Mazumder; ICLR 2026). Reference implementation: [RyanLucas3/Reasoning-Aware-Compression](https://github.com/RyanLucas3/Reasoning-Aware-Compression). ## Why this belongs in SGLang, and what deliberately does not Collecting the rollout is Phase I of the paper's Algorithm 1 and is the expensive half — the paper's budget is 1M on-policy CoT tokens per calibration set. That is batched autoregressive generation, which is what SGLang is for; the reference implementation does it with a Hugging Face `generate` loop. The pruning solver itself is training-time code and is **not** proposed for `python/sglang/srt/`. This PR lands as an offline recipe under `examples/usage/`, matching the existing `examples/usage/modelopt_quantize_and_export.py` precedent, and delegates the solver to [`llm-compressor`](https://github.com/vllm-project/llm-compressor): | Phase | Script | Role | | --- | --- | --- | | I | `rac_collect_traces.py` | `sgl.Engine` samples on-policy CoT → `traces.jsonl` | | II | `rac_prune.py` | `llm-compressor` runs SparseGPT/Wanda against those activations | | III | `rac_serve_and_eval.py` | SGLang serves the sparse checkpoint and scores MATH-500 | `llmcompressor` is imported lazily and is **not** added to SGLang's dependencies — Phases I and III need only SGLang. Also out of scope, and better as separate PRs: sparse-serving kernels (2:4 / cuSPARSELt runtime paths), and vendoring a SparseGPT solver into the engine. ## Modifications - **New** `examples/usage/reasoning_aware_compression/` — three scripts plus a README with full reproduction commands for the paper's DeepSeek-R1-Distill-Qwen-1.5B @ 50% row. - **New** `docs_new/docs/advanced_features/reasoning_aware_compression.mdx`, registered in `docs_new/docs.json` after the quantization page. Design points worth reviewer attention: - **Token-in-token-out.** Phase I runs the engine with `skip_tokenizer_init=True` and emits token ids; Phase II consumes them directly. The sequence the pruner reconstructs is exactly the one the model produced, with no detokenize/retokenize drift. - **Lazy, chunked corpus reads.** The paper's math corpus is 220k rows while a 1M-token budget touches only a few hundred prompts, so prompts are chat-templated and rolled out per chunk rather than up front. - **Batch size 1 during calibration.** Batching variable-length sequences would require padding, and pad-token activations would enter the layer-wise Hessian as if they were real — precisely the calibration contamination RAC exists to avoid. - **`--calibration-mode prompt_only`** reproduces the paper's ablation baseline from the same prompts, so the comparison motivating the method is runnable from the shipped code. - **Phase III reports mean CoT length and wall clock next to accuracy**, because accuracy alone hides the rambling failure mode above. - **Magnitude pruning is intentionally not exposed.** `llm-compressor`'s magnitude modifier is a gradual training-time modifier, not a one-shot solver; the README says so rather than shipping an option that cannot work. ## Accuracy Test The smoke path a reviewer with a GPU can run in a few minutes: ```bash cd examples/usage/reasoning_aware_compression pip install \"llmcompressor>=0.12.0\" python rac_collect_traces.py --model-path Qwen/Qwen3-0.6B \\ --dataset open-r1/OpenR1-Math-220k --prompt-column problem \\ --target-tokens 20000 --max-new-tokens 1024 --output-dir /tmp/rac_traces python rac_prune.py --model-path Qwen/Qwen3-0.6B \\ --calibration /tmp/rac_traces/traces.jsonl --sparsity 0.5 --output-dir /tmp/rac_pruned python rac_serve_and_eval.py --model-path /tmp/rac_pruned --num-problems 50 --max-new-tokens 2048 ``` Phase I should report a decode share well above 50%; Phase II a realized sparsity within a hair of the target. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#code-formatting-with-pre-commit). - [x] Add documentation as needed (`docs_new/docs/advanced_features/reasoning_aware_compression.mdx`). --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #30185462791](https://github.com/sgl-project/sglang/actions/runs/30185462791) Latest PR Test (Extra): :x: [Run #30185462708](https://github.com/sgl-project/sglang/actions/runs/30185462708) ",
"url": "https://github.com/sgl-project/sglang/pull/32414",
"createdAt": "2026-07-26T01:04:54Z",
"updatedAt": "2026-08-12T19:54:40Z",
"timestamp": "2026-08-12T19:54:40Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"documentation"
],
"author": "PKUWZP",
"state": "open",
"assignees": [
"Qiaolin-Yu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32444",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] Suppress multiprocessing resource_tracker warnings in compile_deep_gemm (#32378)",
"text": "## Issue Description Fixes #32378. ## Cause When running `python3 -m sglang.compile_deep_gemm`, the script starts a temporary SGLang server process using `multiprocessing.Process` to precompile DeepGEMM kernels and then terminates the server via `kill_process_tree(proc.pid)`. In Python 3.12, `multiprocessing.resource_tracker` tracks shared memory segments and semaphores created during process execution. Because the server processes are terminated via SIGKILL, they do not unregister their resources, causing `resource_tracker.py` to issue warnings on exit: `UserWarning: resource_tracker: There appear to be 8 leaked semaphore objects...` `UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects...` ## Solution Patch `multiprocessing.resource_tracker.register` to a no-op in `sglang.compile_deep_gemm`, matching the established pattern in `sglang/srt/distributed/parallel_state.py`, to suppress false-positive warnings when the precompilation launcher exits. --- ### CI States Latest PR Test (Base): :x: [Run #31695614015](https://github.com/sgl-project/sglang/actions/runs/31695614015) Latest PR Test (Extra): :x: [Run #31695613811](https://github.com/sgl-project/sglang/actions/runs/31695613811) ",
"url": "https://github.com/sgl-project/sglang/pull/32444",
"createdAt": "2026-07-26T16:39:29Z",
"updatedAt": "2026-08-13T11:28:38Z",
"timestamp": "2026-08-13T11:28:38Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [],
"author": "GiGiKoneti",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32467",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[BugFix] Fix race in c128 prefill plan kernel on ragged extend",
"text": " ## Motivation In DSpark compact ragged-verify mode, CUDA graph capture could fail with an illegal memory access in `c128_v2.cuh:506` when `max-running-requests` is large (e.g. 96). The failure was nondeterministic: one TP rank could build a correct plan while another emitted out-of-bounds `ragged_id` values. Root cause: in `plan_compress_prefill_kernel0`, warp 0 initializes all `warp_min`/`warp_max` scratch slots while each warp later writes its own slot in Stage B. Without a barrier between init and reduction, warp 0 can clobber another warp's min result. That makes ragged `extend_lens` (e.g. `[4]×72 + [3]×24`) look uniform (`s_min == s_max`), wrongly selecting the MTP fast path and generating `ragged_id` up to `batch_size × E - 1`, past the actual ragged input rows. The write kernel has no bounds check on CUDA, so this surfaces as an IMA. This is a general correctness bug for any non-uniform prefill extend; compact ragged capture just makes it easy to hit. ## Modifications - Add `__syncthreads()` in `c_plan.cuh` between warp-scratch initialization and per-warp min/max reduction in `plan_compress_prefill_kernel0`. - Ensures ragged extend correctly takes Path 2 (general prefill) instead of being misclassified as MTP-uniform. ## Accuracy Tests ## Speed Tests and Profiling **Setup:** DeepSeek-V4-Flash-DSpark, DSPARK block-size 5, TP=2, Marlin MoE, `max-running-requests=96`, radix cache disabled. Metric: `mean_output_throughput`. | batch | compact (`SGLANG_RAGGED_VERIFY_MODE=compact`) | default | |-------|-----------------------------------------------|---------| | 1 | 336.46 | 333.14 | | 32 | 48.12 | 48.49 | | 64 | 32.78 | 33.21 | | 96 | 24.75 | 25.73 | **compact command:** ```bash CUDA_VISIBLE_DEVICES=6,7 \\ SGLANG_RAGGED_VERIFY_MODE=compact \\ sglang serve \\ --trust-remote-code \\ --model-path /models/deepseek-ai/DeepSeek-V4-Flash-DSpark/ \\ --speculative-algorithm DSPARK \\ --speculative-dspark-block-size 5 \\ --tp 2 \\ --moe-a2a-backend none \\ --mem-fraction-static 0.75 \\ --moe-runner-backend marlin \\ --reasoning-parser deepseek-v4 \\ --max-running-requests 96 \\ --disable-radix-cache \\ --host 0.0.0.0 \\ --port 8000 ``` **default command:** ```bash CUDA_VISIBLE_DEVICES=6,7 \\ sglang serve \\ --trust-remote-code \\ --model-path /models/deepseek-ai/DeepSeek-V4-Flash-DSpark/ \\ --speculative-algorithm DSPARK \\ --speculative-dspark-block-size 5 \\ --tp 2 \\ --moe-a2a-backend none \\ --mem-fraction-static 0.75 \\ --moe-runner-backend marlin \\ --reasoning-parser deepseek-v4 \\ --max-running-requests 96 \\ --disable-radix-cache \\ --host 0.0.0.0 \\ --port 8000 ``` Before this fix, compact mode failed during CUDA graph capture at large batch tiers; after the fix, capture completes and serving is stable at bs=96. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31582596399](https://github.com/sgl-project/sglang/actions/runs/31582596399) Latest PR Test (Extra): :x: [Run #31582595666](https://github.com/sgl-project/sglang/actions/runs/31582595666) ",
"url": "https://github.com/sgl-project/sglang/pull/32467",
"createdAt": "2026-07-27T03:15:26Z",
"updatedAt": "2026-08-13T05:40:59Z",
"timestamp": "2026-08-13T05:40:59Z",
"metrics": {
"reactions": 0,
"comments": 23
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "EanWang211123",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32491",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(router): enforce lifecycle-based admission control",
"text": "## Summary - replace token-bucket-based concurrency limiting with lifecycle-owned admission leases - enforce a hard per-Pod in-flight limit, bounded waiter count, and per-request absolute queue deadline - hold leases through the complete response body lifecycle, including streaming responses and client disconnects - keep local QPS rate limiting independent from request concurrency - apply admission only to inference routes and release queued waiters during graceful shutdown - expose admission gauges, decisions, and queue-wait latency, and document the new semantics ## Motivation The existing gateway uses the same token bucket for both `--max-concurrent-requests` and local rate limiting. This mixes two different concepts: request lifecycle concurrency and time-based QPS. The existing queue also bounds only the internal channel. The queue processor removes entries from that channel before asynchronously waiting for a token, so the number of real outstanding waiters can exceed `--queue-size`. A configured queue timeout can therefore behave like a token-refill wait instead of an absolute admission deadline. For streaming responses, overload protection needs to keep a concurrency slot until the response body completes or is dropped. Returning the first token must not admit another request. ## Behavior - `--max-concurrent-requests N` limits complete in-flight request lifecycles per Router Pod. Values `<= 0` disable admission control. - `--queue-size M` is a hard cap on outstanding waiters. A full queue returns `429 queue_full`. - `--queue-timeout-secs T` starts when the request enters the queue. Expiration returns `408 queue_timeout`. - `--rate-limit-tokens-per-second R` remains an independent local-QPS token bucket. Unset or `0` disables it. - inference gates run in this order: mesh/global QPS, local QPS, then admission - control and observability routes bypass admission; response cancellation remains available during overload - queued requests are released with `503 shutting_down` during graceful shutdown - background Responses fail closed with `503 background_admission_unsupported` while lifecycle admission is enabled ## Observability This adds: - `smg_admission_inflight` - `smg_admission_queued` - `smg_admission_queue_wait_seconds` - `smg_admission_total{decision=\"...\"}` The gauges report the current number of lifecycle leases and real outstanding waiters in each Router Pod. ## Validation - `cargo fmt --all -- --check` - `cargo test --tests --no-run` - `cargo test core::admission::tests::` — 5 passed - `cargo test admission_decision_labels_are_stable` — 1 passed - `cargo test --test load_guard_raii_test` — 6 passed Manual streaming validation with `N=8`, `M=4`, and `T=10s` observed eight active requests and two real waiters under ten concurrent coding-agent sessions. Releasing or cancelling sessions drained the queue and returned both gauges to zero. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31677890993](https://github.com/sgl-project/sglang/actions/runs/31677890993) Latest PR Test (Extra): :x: [Run #31677890388](https://github.com/sgl-project/sglang/actions/runs/31677890388) ",
"url": "https://github.com/sgl-project/sglang/pull/32491",
"createdAt": "2026-07-27T07:07:26Z",
"updatedAt": "2026-08-13T07:29:20Z",
"timestamp": "2026-08-13T07:29:20Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"documentation",
"model-gateway"
],
"author": "Chen-BUPT",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32500",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(hicache): support Ascend Mamba states with FIA and async IO",
"text": "## Motivation Qwen3.6 hybrid models require both Full KV and the matching Mamba state when a HiCache prefix is reused on Ascend. PR #32275 introduced the correct Torch-based Ascend host-transfer fallback for these Mamba state sidecars. This PR preserves that fallback and extends the integration in two areas found during Qwen3.6 NPU validation: 1. FIA exposes KV cache buffers as per-layer views, while Ascend HiCache's all-layer exchange path requires the original contiguous allocation. HiCache write-through can otherwise fail or hang when FIA is enabled. 2. The Torch Mamba transfer path materializes the layout before D2H/H2D. A native asynchronous transfer operator can avoid that overhead when the matching `sgl-kernel-npu` capability is installed. This PR therefore supersedes #32275 while preserving its commits and default behavior. ## Modifications ### Ascend Mamba State HiCache IO - Keep the synchronous Torch implementation as the default. - Add an optional runtime mode: ```bash SGLANG_NPU_HICACHE_MAMBA_IO=sync|async ``` - `sync`: existing Torch fallback (default); - `async`: require the native operator and fail fast at startup when it is unavailable. - Support conv and temporal state components, including conv-only models. - Accept physically dense transposed/NEXTN state payloads without requiring a logical contiguous layout. - Check native operator availability in SGLang before selecting the async path. - Keep tensor layout, dtype, pinned-memory, pitch, layer-range, and index validation in the native operator as the single transfer-safety boundary. - Dispatch the native path from the same component-level copy helpers used by the CUDA/ROCm implementation: per-layer PF->LF for Host-to-Device loads and all-layer LF->PF for Device-to-Host backups. Temporal and conv components retain their existing outer-loop structure instead of bypassing it through a monolithic Ascend-only fast path. - Preserve the helpers as `@staticmethod` functions with their original signatures. The H2D native entry consumes the current-layer Device view already passed as `dst`; no full `device_layers` argument or additional IO backend name is introduced. `kernel_ascend` remains the sole Ascend backend, with `sync|async` selecting its implementation. - Keep CUDA/ROCm HiCache paths unchanged. ### FIA + HiCache Compatibility - Retain contiguous all-layer Ascend K/V backing tensors. - Continue exposing per-layer views to FIA so graph capture does not retain the full tensor. - Let HiCache explicitly retrieve the contiguous backing tensors for `kernel_ascend` transfers. - Allocate the backing tensors inside the existing KV-cache memory-saver region so the allocation lifecycle remains unchanged. ### Tests - Add unit coverage for explicit sync/async selection, operator fail-fast, and conv-only states. - Cover component-level async dispatch and preservation of the synchronous Torch fallback. - Cover transposed dense layouts and invalid payload rejection in the native operator tests. - Extend staged write-back dispatch coverage for the Ascend Mamba path. ## Dependency The optional native path uses: - https://github.com/sgl-project/sgl-kernel-npu/pull/626 SGLang remains functional without that PR because `sync` is the default. Explicit `async` mode requires the matching native operator. ## Validation Latest component-level integration update (2026-08-12): - `python -m py_compile` passed for the modified SGLang files. - `git diff --check` passed. - The matching component-level native entries were loaded and exercised by the NPU microbenchmark reported in the kernel PR. - The current Codex environment does not expose an NPU, so the device-dependent unit file was not re-run here; the failure occurs during NPU availability detection before test collection. Earlier NPU unit and end-to-end results are retained below. The subsequent interface cleanup that preserves the static helper signatures passed Python compilation and `git diff --check`. Its current-layer Device-view entry still requires a fresh NPU build/unit/end-to-end run; the performance numbers below were collected before this interface-only cleanup and are not relabelled as a new measurement. - `test_npu_mamba_async_layout.py`: 4/4 passed. - `test_hicache_staged_write_back_dispatch.py`: 13 passed, 1 expected skip. - Native kernel tests: 9/9 passed on NPU. - Python compilation and `git diff --check` passed. Qwen3.6 AIME26 deterministic concurrent HiCache validation: - Round 1: cold build, 10 requests. - Round 2: 10/10 Device/L1 hits. - Round 3 after L1 pressure: all 10 requests hit cache (5 Device + 5 Host in the concurrent run). - Round 2 and Round 3 scores were both 1.0. Prewarmed 20-request, one-token-completion transfer A/B: - Torch fallback mean request latency: 641.15 ms. - Native async mean request latency: 505.01 ms. - Observed request-level speedup: 1.27x. Both variants recorded 20/20 Host/L2 hits. This transfer-heavy workload keeps generation to one token so that the host reload path is a material part of request latency. Repeated full end-to-end serving benchmark: - Five repeats per variant with rotated execution order. - Identical full-workload prewarming before measurement. - 16 prompts, 16K input tokens, 512 output tokens, concurrency 4. - `temperature=0`, fixed L1 capacity 486208, HiCache write-through with ratio 2. Mean results over five repeats: | Variant | Wall time | Mean TTFT | Mean TPOT | Mean E2E | Total TPS | | --- | ---: | ---: | ---: | ---: | ---: | | PR baseline | 70.930 s | 5243 ms | 10.948 ms | 10838 ms | 171.52 | | Adapted sync, FIA off | 70.878 s | 5111 ms | 11.144 ms | 10806 ms | 173.51 | | Adapted async, FIA off | 69.722 s | 5042 ms | 10.902 ms | 10612 ms | 175.75 | | FIA on, sync | 71.183 s | 5209 ms | 11.244 ms | 10955 ms | 171.20 | | FIA on, async | 71.836 s | 5384 ms | 10.802 ms | 10905 ms | 169.46 | Paired five-repeat comparisons: - The adapted synchronous path with FIA disabled is effectively neutral versus the clean PR baseline: wall-time difference +0.06%. - With FIA disabled, native async versus sync improves wall time by 1.60%, mean E2E by 1.77%, TTFT by 1.14%, TPOT by 2.14%, and throughput by 1.34%. - FIA is a compatibility path rather than a general performance optimization for this workload. The full FIA+async stack is 1.28% slower in wall time than the clean PR baseline, while TPOT improves by 1.33%. The isolated native operator benchmark is reported in the kernel PR. The gap between the isolated transfer and full serving gains is expected: Mamba state reload is only one part of long-context prefill and decode. ### Follow-up forced Host/L2 validation The five-variant comparison above is retained for completeness. A subsequent FIA-off A/B used stricter Host/L2 controls: - five repeats with alternating `sync -> async` / `async -> sync` order; - 16 prompts, 16K input tokens, 512 output tokens, concurrency 4; - PrefillDelayer disabled; - every measured variant/repeat recorded 16/16 pure Host hits, 261120 cached Host tokens, and zero Device hits. Mean Round-2 Host/L2 results: | Metric | Sync | Native async | Paired async improvement | | --- | ---: | ---: | ---: | | Wall time | 29.868 s | 28.884 s | 3.30% (3.14% median) | | Mean TTFT | 669.44 ms | 589.07 ms | 11.95% (12.94% median) | | Mean TPOT | 12.796 ms | 12.342 ms | 3.56% (3.40% median) | | Mean E2E | 7208.36 ms | 6895.76 ms | 4.34% (3.65% median) | | Output throughput | 274.32 tok/s | 283.72 tok/s | 3.42% (3.24% median) | All five paired wall-time samples favored the native path, with improvements from 2.06% to 5.21%. Because this follow-up removes the PrefillDelayer scheduling confounder and validates every Host/L2 hit, it is the more reliable result for the FIA-off forced-Host-reload path. It does not replace the FIA-on compatibility comparison above. ### Ascend Profile analysis (diagnostic only) A separate single-run Ascend Profile compared the Torch fallback and native path with FIA enabled. It used 16 prompts, 16K input tokens, 512 output tokens, concurrency 4, and verified 16/16 pure Host/L2 hits in the measured round. This run is used for attribution only; profiler overhead and the single sample make it unsuitable as a replacement for the repeated serving results above. - The identifiable Torch fallback materialization path accounted for about 1.884 s of Host API self time (`index_select`, `copy_`, `index_copy_`, and `_to_copy`). - Its state-specific device gather/scatter work was about 598.8 ms, but 93.2% of it overlapped those Host API intervals, so it must not be added again to the 1.884 s. - The native `transfer_state_dim_exchange` path used about 42.9 ms of Host API time. Its H2D and D2H DMA tasks totaled about 18.84 ms. - The timeline observed no overlap between those DMA tasks and model compute (including FIA) in this workload. The Profile therefore attributes the gain primarily to the layout-aware direct copy, which removes the fallback gather, temporary materialization, H2D Tensor creation, and device scatter. `aclrtMemcpy2dAsync` provides stream-ordered nonblocking submission, but this measurement does not claim a compute-overlap gain. Even perfect overlap of all currently visible native DMA would save at most about 18.84 ms, or 0.06% of the 31.64 s profiled async wall time. ## Checklist - [x] Synchronous fallback remains the default. - [x] Existing CUDA/ROCm paths are unchanged. - [x] SGLang checks native operator availability before enabling `async`. - [x] Native operator owns all transfer layout and bounds validation. - [x] Explicit `async` mode fails fast instead of silently changing behavior. - [x] Unit and NPU kernel validation pass. --- ### CI States Latest PR Test (Base): :x: [Run #31679442679](https://github.com/sgl-project/sglang/actions/runs/31679442679) Latest PR Test (Extra): :x: [Run #31679442484](https://github.com/sgl-project/sglang/actions/runs/31679442484) ",
"url": "https://github.com/sgl-project/sglang/pull/32500",
"createdAt": "2026-07-27T08:04:49Z",
"updatedAt": "2026-08-13T07:51:17Z",
"timestamp": "2026-08-13T07:51:17Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"hicache",
"npu"
],
"author": "qybnb",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32514",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(kv-events): Add component_types field to BlockStored for per-component placement tracking",
"text": "## Motivation KV-aware routers consume `BlockStored` / `BlockRemoved` to track which blocks live on which tier. On hybrid models the unified radix tree keeps several independent KV components on one node — full attention, sliding window, mamba state — and evicts and restores them independently. A `BlockStored` today only says \"this block is on GPU\". That is ambiguous once SWA has been tombstoned off an internal node while its full KV stays resident: the router still believes the block is fully reusable and routes a request that then has to recompute the window. This adds an opt-in per-component placement dimension so a consumer can tell what is actually resident at a given medium, following the same shape as #18205 which added the `medium` field. ## Modifications Gated behind `--enable-kv-events-component-types`, off by default. 1. **Schema.** A trailing `Optional[list[str]]` on `BlockStored` plus the `KV_COMPONENT_FULL/SWA/MAMBA` wire names. `BlockRemoved` stays whole-block — a removal always means the base (full) component left the tier, so there is nothing partial to express. 2. **Plumbing.** The flag flows from `ServerArgs` through `CacheInitParams` into the cache builder, and is documented in the server arguments reference. The UMBP subscriber logs the component dimension but does not forward it; its external-KV API is keyed by `(hashes, tier)` only. 3. **Emission.** `KVCacheEventMixin` gains a `_component_types_for_page` hook defaulting to `None`; `UnifiedTreeCore` overrides it and reports residency off live tree state, anchoring the single per-leaf mamba state to the leaf's last page. 4. **Tests.** 20 unit tests in `TestUnifiedRadixCacheComponentPlacementEvents`, plus wire-layout tests in `test_kv_events.py`. Three ordering constraints shaped the implementation and are worth flagging for review: - *Gains defer, losses do not.* SWA and Mamba attach device values through barrier-applied actions (`SWARebuild`, `MambaEvictExcessPathStates`), so a node's component set is not final until those apply. Inserts collect nodes in `_InsertWalkState` and emit one snapshot per node in the TAIL step; evictions are final when the component is dropped and emit immediately. - *Emission is root-first.* SWA splits the fresh leaf at commit time, so nodes are noted in neither tree nor creation order. `_record_store_event` seeds a node's lazily computed hash chain off its parent's hash, so announcing a child first would publish it as a root block under a hash no other code path reproduces. - *Partial eviction restates, never removes.* When SWA or Mamba is tombstoned while full survives the block has not left the tier, so we restate the smaller component set. Whole-block teardowns (`_demote`, host-leaf eviction) keep emitting `BlockRemoved` and never restate. `commit_load_back` also moves its emit after every component restores, unioning each component's `nodes_to_load`; emitting on full's list alone reported a promoted block as full-only. ## Subscriber compatibility `BlockStored` is a msgspec `array_like` struct, so every field is always encoded positionally — an unset `component_types` is a trailing `nil`, not an omitted slot (pinned by `test_component_types_is_the_trailing_slot`). With the flag **off** that `nil` is exactly what a positional decoder reads for an absent optional, so subscribers are unaffected; verified against both in-tree decoders, sgl-router (which drains unknown trailing fields, `experimental/sgl-router/src/policies/kv_events/wire.rs:420`) and dynamo's kv-router. With the flag **on**, the slot holds a list. sgl-router still ignores it. dynamo's kv-router, however, shares one positional layout with vLLM in which that slot is `lora_name` (a string), so it rejects the whole event batch. This is exactly why the feature is opt-in, and enabling it against a dynamo subscriber is not supported today. Aligning the two representations — dynamo already models this as one event per KV-cache group tagged with `kv_cache_spec_kind` — is a larger cross-project change and deliberately out of scope here. Only component-aware caches honour the flag; the mixin hook returns `None` elsewhere, so `radix_cache`, `hiradix_cache`, `swa_radix_cache`, `mamba_radix_cache` and `hi_mamba_radix_cache` are untouched. ## Accuracy Tests Not applicable — observability only. This touches no kernel, no model forward path, and no cache eviction or placement decision; it changes what is reported about placement, never the placement itself. Unit tests run locally across page_size 1/4/16/64/128 and the FULL, FULL+SWA, FULL+MAMBA and FULL+SWA+MAMBA configurations (`test_unified_radix_cache_unittest.py`, `test_kv_events.py`, and the radix/hiradix/swa/mamba mem_cache units): 1113 passed, 1030 skipped (config-conditional), 140 subtests passed. `test_unified_radix_cache_unittest.py` is one of the 12 files currently deferred at module level by #31812 during the ServerArgs config-namespace migration, so those cases will not run in CI until that skip is lifted upstream; the run above was done locally with the module skip removed. The wire-layout tests live in `test_kv_events.py`, which is not deferred and does run in CI. ## Speed Tests and Profiling Not applicable. With the flag off the added cost is one boolean check per store event. With it on, a per-page component scan bounded by the component count (at most 3) plus one ancestor walk per insert flush. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31725107771](https://github.com/sgl-project/sglang/actions/runs/31725107771) Latest PR Test (Extra): :hourglass_flowing_sand: [Run #31725107413](https://github.com/sgl-project/sglang/actions/runs/31725107413) ",
"url": "https://github.com/sgl-project/sglang/pull/32514",
"createdAt": "2026-07-27T10:35:52Z",
"updatedAt": "2026-08-13T17:31:59Z",
"timestamp": "2026-08-13T17:31:59Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"documentation",
"run-ci",
"run-ci-extra"
],
"author": "isytwu",
"state": "open",
"assignees": [
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32523",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(load-reporter): add push-based engine load reporting",
"text": "## Motivation Load-aware routing needs fresh, serving-stack-authoritative signals. Polling worker endpoints adds fan-out, creates inconsistent views, and can miss short-lived queue changes across data-parallel ranks. This PR adds the serving-side (Worker-process) Load Reporter only. It continuously pushes per-DP-rank load snapshots to external Routers over long-lived bidirectional gRPC streams. Router-side ingestion, discovery, retries, and the Prefill throughput extension are intentionally split into follow-up PRs so each layer can be reviewed independently. ## Modifications ### Architecture ```mermaid flowchart LR Scheduler[\"Scheduler per-DP-rank load snapshots\"] Source[\"Per-mode snapshot source manager get_loads / Router SHM reader\"] Sampler[\"Single-flight sampler coalesced refresh\"] Store[\"Validated latest-wins store\"] Builder[\"LoadReport builder status + sequence id\"] Service[\"Worker gRPC service --load-reporter-port\"] Router[\"Router dials in drives Monitor stream\"] Scheduler --> Source --> Sampler --> Store --> Builder --> Service --> Router ``` ### Protocol and configuration - The Worker hosts `sglang.router.loadmonitor.v1.LoadMonitorService.Monitor(stream RouterFrame) returns (stream WorkerFrame)` on a fixed, opt-in `--load-reporter-port`; the external Router dials in and drives the stream (`register` → ack → bounded first sample → periodic reports). Stable field numbers and enum values are preserved. - `RouterFrame` = `register | update_config | keep_alive | stop`, with `register` required first; `WorkerFrame` = `registered | report | error`. Re-registering the same `router_id` replaces its old session; different `router_id`s coexist. - Package `grpcio` and `protobuf` through the optional `load-reporter` extra across all supported Python wheel variants. - Keep reporting opt-in so normal SGLang installations do not gain mandatory runtime dependencies. ### Reporter runtime - Validate complete per-DP-rank snapshots and atomically publish immutable latest-wins views. - Use a single-flight sampler so request completions and timer refreshes never create concurrent `get_loads()` calls. - Each session has a capacity-1 latest-wins response queue so a slow Router never accumulates history; lease TTL with keep-alive renewal, mid-stream config updates, a bounded initial-sample timeout, and idempotent shutdown. - Isolate sampling and transport failures from the inference request path. ### Tokenizer integration - Single-tokenizer HTTP and native gRPC modes start one reporter runtime in the FastAPI lifespan (native gRPC reuses the same process and listener). - Multi-tokenizer mode: `MultiTokenizerRouter` is the sole runtime owner; HTTP workers send coalesced IPC refresh messages (50 ms window) instead of opening duplicate streams, and elastic scale changes update the expected DP-rank set. - Standalone SMG gRPC starts the reporter at request-manager readiness and shadows only the current instance's bound `generate_request`; older servicers without the readiness hook fail loudly when the reporter is enabled. - The embedded offline Engine API does not own the reporter; cleanup runs in the HTTP-server lifespan, the SMG gRPC-server finally path, and `MultiTokenizerRouter.close()`. ### Tests and documentation - Unit coverage: decorator contract (static + bound method, identity-safe restore), proto contract, session scheduling/lease/backpressure, service handshake and error frames, composition-root lifecycle, standalone SMG wiring, and IPC round-trip/coalescing. - Registered GPU E2E tests with a real `grpc.aio` fake Router cover single-owner, multi-owner (multiple HTTP workers create only one reporting owner), and standalone SMG reporting. - Document architecture, protocol constraints, dependency setup, timing, error handling, and shutdown behavior. ### Design advantages - **Fresh:** millisecond-level push cadence captures fast queue and capacity changes. - **Efficient:** persistent bidi streams avoid Router-side polling fan-out and connection churn. - **Topology-safe:** reports preserve DP-rank detail instead of hiding hot ranks behind worker averages. - **Resilient:** validation, sequence allocation, session replacement, leases, latest-wins backpressure, and bounded shutdown prevent stale tasks from accumulating. - **Low-impact:** request paths only emit lightweight refresh hints; sampling and network I/O stay in background tasks. - **Extensible:** the stable protocol and immutable snapshots provide a clean foundation for the separately reviewed Router monitor and future metrics. ## Accuracy Tests N/A for model accuracy. This PR does not change model forward code, kernels, sampling, or generated outputs. Functional validation: - Registered CPU tests cover the reporter protocol contract, service behavior, runtime sessions, lifecycle, standalone SMG wiring, and IPC coalescing. - Registered GPU E2E tests drive a real `grpc.aio` fake Router against single-owner, multi-owner, and standalone SMG modes. - Generated protobuf/gRPC bindings import successfully and expose the expected `sglang.router.loadmonitor.v1` service. ## Speed Tests and Profiling N/A for inference throughput benchmarking. This is an opt-in control-plane feature and does not modify model execution. The design minimizes runtime overhead by reusing gRPC connections, sampling already-published snapshots, coalescing refresh events, and keeping network I/O off the inference request path. No inference-speed improvement is claimed. Local CPU-only reporter micro-benchmark (fake snapshot sources, real `grpc.aio` Router client) covers all three wirings — single-tokenizer manager, multi-tokenizer Router/SHM, and gRPC request manager — at DP ranks 1/8/32 and 100 ms / 50 ms report intervals: - **Request-path hook overhead is negligible:** < 2.5 µs added per request lifecycle in every mode and rank count (baseline 0.3–1.2 µs). - **Report cadence is accurate:** interval error p50 ≤ 0.5 ms and p99 ≤ 2.4 ms at both intervals (≤ 4.7 % of interval at p99). - **First report is fast:** 2.3–4.1 ms across all cells, far below the 1 s bounded-initial-sample limit. - **Request activity keeps reports fresh:** with 200 req-end/s, `stale %` stays 0 and snapshot age stays ~39–42 ms at both intervals. - **Hints coalesce:** 200 req-end events/s drive only 12–27 samples/s — not one sample or report per request. - **STALE semantics hold:** the status flips once the drift accumulates past the threshold (1004 ms vs 500 ms threshold at 50 ms/sample drift; 154 ms vs 100 ms threshold). - **Backpressure is bounded:** a slow Router drops ~80 % of reports via latest-wins with no RSS growth. - **Multi-tokenizer IPC:** per-notify cost is sub-microsecond; the coalesced worker→router forward latency is ~50 ms (the fixed coalesce window), the design bound for request-end-driven freshness in multi-worker mode. Local checks: - Full pre-commit run for all changed files: **passed**. - `git diff --check`: **passed**. - Generated protobuf/gRPC import and service-package validation: **passed**. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci`. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31685767342](https://github.com/sgl-project/sglang/actions/runs/31685767342) Latest PR Test (Extra): :x: [Run #31685767196](https://github.com/sgl-project/sglang/actions/runs/31685767196) ",
"url": "https://github.com/sgl-project/sglang/pull/32523",
"createdAt": "2026-07-27T11:52:53Z",
"updatedAt": "2026-08-13T09:16:29Z",
"timestamp": "2026-08-13T09:16:29Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"documentation",
"feature",
"dependencies",
"npu",
"run-ci"
],
"author": "Bakerjc-bgner",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32564",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[PD] Don't release KV pages while Mooncake transfers are in flight",
"text": "## The bug An abort or timeout _during a prefill→decode transfer_ frees the request's KV pages back to the allocator, but it **_does not cancel the transfer_**. The next request can then be allocated those same pages and have them overwritten by the earlier, still-in-flight write, causing the next request to hold the first request's KV, with no error raised anywhere. @ShangmingCai already anticipated this in #27372, which prevents the prefill *starting* new work after an abort and this PR implements [what he mentioned as a TODO](https://github.com/sgl-project/sglang/blob/7cae831e41c942f4514c2458c5840c6b5ac0dd81/python/sglang/srt/disaggregation/mooncake/conn.py#L1726-L1728), which is handling transfers that are already in flight. The bug's window is narrow: the transfer has to still be in flight *and* the pages have to be reallocated before it lands. But in our setup, we often abort requests that are too slow, so I believe we are hitting this scenario. The only symptom is a slightly wrong answer to our users. **No cost on the happy-path:** no extra collective and no wait unless a request has failed. Only ~995 lines of implementation code, the rest is tests and doc strings. ## Reproduction script [This script](https://gist.github.com/KastanDay/a78f9ba98b3a255c2a0f94768a2522c2) reproduces the bug on current `main` (CPU only, no GPU): ``` [T1] transfer worker is inside a write to decode page 0xdead0000 [T2] request aborted while that write is still in flight [T3] scheduler was told: Failed RESULT: REPRODUCED ``` Reporting a terminal state is what lets the allocator hand those pages out, so `Failed` at [T3] is the bug. Asserting the *ordering* makes it deterministic. With this PR, [T3] reports `Transferring` and terminal arrives only after the write returns. ## The core change No new states and no new message types. `KVPoll` keeps its five states (`Failed`, `Bootstrapping`, `WaitingForInput`, `Transferring`, `Success`); what changes is **which one the scheduler is told, and when**: - **Before:** the first failure reported `Failed` immediately. A terminal state is what authorizes the allocator to hand the request's pages out — so pages could be reused while a transfer was still writing to them. - **After:** until every rank proves the pages are idle, the poll path reports `Transferring` — a state the scheduler already knows to wait on. `Failed` arrives only with proof. (Skipped entirely when nothing failed, so the happy path is unchanged.) Correspondingly, **`ABORT_ACK` changes meaning: from \"I heard your abort\" to \"your pages are safe to reuse.\"** Prefill now sends it only after that room's transfer work has drained, and it must echo a per-request nonce. Each bootstrap room gets a lease-counted lifetime. Transfer work holds a lease; a failed request closes the room; pages are released once it is closed with zero leases and, on decode, once every peer has acknowledged. ## Required for the barrier to hold Fixed these three paths from bypassing the abort: * The lease would drop early. `send_kvcache` returns on the first failing future, but `Future.cancel()` is a no-op once a future is running — so siblings keep writing while the worker reports the chunk done. * HiCache bypassed the gate. A failed restore releases pages regardless of poll state, including while the barrier is holding. * Room identity is ambiguous. `bootstrap_room` is recycled, so a queued chunk can lease the wrong request's lifetime. ## Tests - **Integrated 3-node H200 validation passed.** With two TP8 prefill and one TP8 decode (24 H200s, `STRICT` policy), 192/192 immediate victim reuse probes succeeded after 192 in-flight donor aborts. There were zero transport, semantic, hard, or critical failures; zero foreign canaries; a decode transfer-queue peak of 20; and zero model restarts. [Raw summary, matrix, telemetry, and the admission-control disclosure](https://gist.github.com/KastanDay/a2346f10d3a3e67a7f5870a5d8eb8b21). ```bash python -m pytest -q test/registered/unit/disaggregation/test_transfer_ownership_barrier.py ``` 71 tests named by property. Every fix has a regression test verified to fail with that fix reverted. `unit/disaggregation` 72 → 143 passed; `unit/managers` + `unit/mem_cache` unchanged apart from additions. ## Related PRs - **PP is unvalidated.** The added collectives cover TP and CP. #31869 looks complementary. - Overlaps #29984 (socket serialization) and #31144 (non-blocking sends). I deliberately used the same function names and placements so the PR diffs collapse and merge cleanly. Glad for this to land as a completion of #27372. @AsterWang identified this race first in #24580 but I think my solution is a bit safer. --- ### CI States Latest PR Test (Base): :x: [Run #31566610945](https://github.com/sgl-project/sglang/actions/runs/31566610945) Latest PR Test (Extra): :white_check_mark: [Run #31620191653](https://github.com/sgl-project/sglang/actions/runs/31620191653) ",
"url": "https://github.com/sgl-project/sglang/pull/32564",
"createdAt": "2026-07-27T18:59:18Z",
"updatedAt": "2026-08-13T17:55:42Z",
"timestamp": "2026-08-13T17:55:42Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"high priority",
"hicache"
],
"author": "KastanDay",
"state": "open",
"assignees": [
"ShangmingCai"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:32577",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] DeepSeek-V4: add aiter fused mHC post+pre with cross-layer boundary dispatch",
"text": "## Summary Add a HIP/aiter fused `mhc_post` + next-layer `mhc_pre` path for DeepSeek-V4 on gfx95, dispatched across the attention/MoE boundary alongside the existing TileLang and Triton fused paths. On MI355X (gfx950) this is a **+0.8%–1.8% output-throughput win across concurrency 4–64 with accuracy preserved**. ## What changed - **`python/sglang/srt/models/deepseek_common/amd/deepseek_v4_fused_mhc.py`** - `try_aiter_fused_mhc_post_pre()`: wraps `aiter.ops.mhc.mhc_fused_post_pre`. The aiter kernel self-gates fused-vs-unfused per arch/token-count, so no token threshold is duplicated here; import/runtime failures fall back and warn once. - `try_mhc_fused_post_pre_boundary()`: aiter → Triton dispatch. The two kernels expect opposite fn orientations (Triton transposed, aiter native). - `is_cross_layer_mhc_fusion_enabled()` / `_is_production_mhc_enabled()`: enable cross-layer fusion when either the TileLang path or the aiter gfx95 path is available. `_is_fused_mhc_post_pre_enabled()` is consolidated here (previously duplicated in `deepseek_v4.py`). - **`python/sglang/srt/models/deepseek_v4.py`**: route the attn-side, FFN-side, and `op_mhc_post_attn_pre_mlp` boundaries through `_apply_mhc_post_pre_boundary()`, falling back to the unfused `hc_post` + `hc_pre` sequence when no fused kernel fires. In the operation/batch-overlap path (DP attention) `state` is a `_StateDict` (attribute access, no `__getitem__`), so read it via `state.`. - **`test/registered/unit/models/test_deepseek_v4_amd_fused_mhc.py`**: gating/dispatch unit tests plus a gfx95-gated numerical test asserting `next_residual` (bit-exact) and `layer_input`/`post_mix` (bf16 tolerance) vs the unfused reference. ## Validation Model: `deepseek-ai/DeepSeek-V4-Pro`. Hardware: MI355X (gfx950), TP4. Image: `rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260724`. Baseline = image tree; Patched = image tree + this patch (only 2 model files differ). Env: `SGLANG_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton`. ### Unit test ``` $ python3 -m pytest test/registered/unit/models/test_deepseek_v4_amd_fused_mhc.py -q 8 passed ``` ### Accuracy (GSM8K, 1319 questions × 3 runs) | run | baseline | patched | |----:|:--------:|:-------:| | 1 | 0.942 | 0.947 | | 2 | 0.943 | 0.943 | | 3 | 0.937 | 0.944 | | **avg** | **0.941** | **0.945** | Invalid = 0.000 for all runs. Accuracy preserved (within run-to-run noise). ### Performance (ISL=8192, OSL=1024, output token throughput) | concurrency | baseline (tok/s) | patched (tok/s) | Δ | baseline TPOT (ms) | patched TPOT (ms) | |------------:|:----------------:|:---------------:|:-----:|:---:|:---:| | 4 | 216.38 | 219.66 | **+1.5%** | 17.507 | 17.242 | | 8 | 367.65 | 372.98 | **+1.4%** | 20.715 | 20.423 | | 16 | 568.64 | 579.04 | **+1.8%** | 26.460 | 25.960 | | 32 | 792.79 | 800.70 | **+1.0%** | 38.165 | 37.808 | | 64 | 1033.97 | 1042.04 | **+0.8%** | 58.854 | 58.412 | ### Commands used ```bash # Accuracy python3 benchmark/gsm8k/bench_sglang.py --num-questions 2000 --parallel 1200 # Performance (per concurrency C in 4 8 16 32 64) python3 benchmark_serving.py --model deepseek-ai/DeepSeek-V4-Pro --backend sglang \\ --dataset-name random --random-input-len 8192 --random-output-len 1024 \\ --random-range-ratio 0.8 --num-prompts $((C*10)) --max-concurrency C --ignore-eos ``` ## Test plan / Checklist - [x] New unit test `test_deepseek_v4_amd_fused_mhc.py` passes (8 passed; gfx95 numerical). - [x] GSM8K accuracy preserved vs baseline (0.945 vs 0.941, invalid 0.000). - [x] Output throughput improves +0.8%–1.8% at concurrency 4–64 (TP4, 8k/1k), TPOT reduced. - [x] Falls back to unfused hc_post + hc_pre when no fused kernel fires (non-gfx95 / import failure). --- ### CI States Latest PR Test (Base): :x: [Run #31664569839](https://github.com/sgl-project/sglang/actions/runs/31664569839) Latest PR Test (Extra): :x: [Run #31674828414](https://github.com/sgl-project/sglang/actions/runs/31674828414) ",
"url": "https://github.com/sgl-project/sglang/pull/32577",
"createdAt": "2026-07-27T22:20:04Z",
"updatedAt": "2026-08-13T06:43:01Z",
"timestamp": "2026-08-13T06:43:01Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"amd",
"deepseek",
"run-ci"
],
"author": "karverma-amd",
"state": "open",
"assignees": [
"kkHuang-amd",
"1am9trash",
"RolaoDenthu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32593",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Kernel] Enable Helion backend for Kimi Delta-Attention",
"text": "## Motivation Add an opt-in Helion backend (e.g. used for Inkling [silu_mul fusion](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/ops/moe/inkling_moe.py)) for Kimi Delta Attention packed decode and prefill while preserving the existing default backends, public contracts, state mutations, and fallback behavior. The decode kernel delivers a **1.1x speedup** over Triton default. The prefill kernel delivers a **1.6x speedup** over Triton default. For Kimi-Linear-48B-A3B-Instruct (TP=2, 2xGB200), we observe geomean **5.04%-5.27%** improvement in tok/s with **4.06%-5.22%** reduction in TTFT compared to default kernels. This enables helion==1.4.0 as an optional dependency that is required only for the helion backend, matching what is done for some of the other KDA backends. ## Modifications - Add Helion implementations for packed recurrent decode and fixed/packed-varlen prefill. - Support FP32, BF16, and FP16 recurrent states, padded cache indices, bounded gates, intermediate states, and an opt-in Newton-Schulz research path. - Register helion for KDA decode and prefill selection: ```bash sglang serve MODEL_PATH --linear-attn-backend helion ``` Phases can also be selected independently: ```bash sglang serve MODEL_PATH \\ --linear-attn-decode-backend helion \\ --linear-attn-prefill-backend triton ``` - Keep generic decode, speculative verification, and ReplaySSM on the existing Triton fallback. - Add registered CUDA kernel tests, CPU dispatcher tests, benchmark coverage, and backend documentation. - Use fixed Helion configurations and static shapes, with no runtime autotuning on the model hot path. ## Accuracy Tests - Decode comparisons against packed Triton: - FP32 state: maximum output difference: 7.63e-6; maximum state difference: 2.24e-8. - BF16 state: maximum output difference: 7.63e-6; maximum state difference: 1.22e-4. - Prefill comparisons against recurrent Triton: - Maximum output difference: 7.32e-4. - Maximum state difference: 4.90e-3. - Covered fixed and packed-varlen prefill, FP32/BF16/FP16 state, padded `-1` cache indices, partial chunks, safe gates with `lower_bound=-5.0`, and stable sub-chunk normalization. - End-to-end Kimi-Linear GSM8K smoke test with Helion decode passed with score 0.915. ## Speed Tests and Profiling Measured on NVIDIA 2xGB200 GPUs. End-to-End benchmarks: - Kimi-Linear-48B-A3B-Instruct - TP=2 - input length 8192, batches 1/4/16, and output lengths 64/256. - Geomeans cover all six shapes for each state dtype. - We compare against default prefill/decode for KDA. | State dtype | Activation dtype | Overall throughput gain | TTFT reduction | TPOT reduction | |---|---|---:|---:|---:| | FP32 | BF16 | 5.04% | 5.22% | 4.83% | | BF16 | BF16 | 5.27% | 4.06% | 5.68% | Microbenchmark summary: | Benchmark | State dtype | Activation dtype | Baseline | Geomean speedup | |---|---|---|---|---:| | Decode | FP32 | FP32 | Packed Triton | 1.134x | | Decode | BF16 | BF16 | Packed Triton | 1.067x | | Decode | FP32 | BF16 | Packed Triton | 1.118x | | Decode | BF16 | BF16 | FlashInfer | 1.393x | | Prefill | FP32 | BF16 | Triton | 1.603x | | Prefill | BF16 | BF16 | Triton | 1.595x | H = 12, lower_bound=-5.0 | Benchmark | State dtype | Activation dtype | Baseline | Geomean speedup | |---|---|---|---|---:| | Decode | FP32 | BF16 | Packed Triton | 1.552x | | Decode | FP32 | BF16 | CUDA decode | 0.721x | | Decode | BF16 | BF16 | Packed Triton | 1.130x | | Prefill | FP32 | BF16 | Triton | 1.385x | | Prefill | BF16 | BF16 | Triton | 1.390x | | Prefill| FP32 | BF16 | Triton ReplaySSM | 1.707x | | Prefill | BF16 | BF16 | Triton ReplaySSM | 1.225x | Currently Helion decode under performs the custom cuda KDA decode kernel, as the CUDA kernel fuses convolution, recurrence, and RMSNorm. A follow up PR will evaluate Helion under the same fusions. Note that the`bench_kda_prefill_cutedsl.py` script precomputes CuTe metadata and scratch buffers outside the timed region and bypasses the production adapters, so this is not a super faithful representation of performance (helion has a 0.86x speedup compared to cute on this benchmark). If we benchmark the entire KDA wrapper, we observe: | Benchmark | State dtype | Activation dtype | Baseline | Geomean speedup | |---|---|---|---|---:| | Helion | FP32 | BF16 | CuTe DSL | 2.031x | ## Benchmark details and results Packed decode ```bash for dtype in float32 bfloat16; do for heads in 16 32; do python benchmark/bench_linear_attention/bench_kda_decode.py \\ --dtype \"${dtype}\" \\ --num-q-heads \"${heads}\" \\ --num-v-heads \"${heads}\" done done ``` | Activation | State | H | B | Packed Triton (us) | Helion (us) | Speedup | |---|---|---:|---:|---:|---:|---:| | FP32 | FP32 | 16 | 1 | 31.2 | 28.3 | 1.102x | | FP32 | FP32 | 16 | 4 | 31.2 | 28.7 | 1.087x | | FP32 | FP32 | 16 | 8 | 32.0 | 28.6 | 1.119x | | FP32 | FP32 | 16 | 16 | 31.8 | 29.7 | 1.071x | | FP32 | FP32 | 16 | 32 | 31.6 | 29.2 | 1.082x | | FP32 | FP32 | 16 | 64 | 31.3 | 29.5 | 1.061x | | FP32 | FP32 | 16 | 128 | 57.7 | 47.1 | 1.225x | | FP32 | FP32 | 16 | 256 | 106.3 | 86.6 | 1.227x | | FP32 | FP32 | 32 | 1 | 31.9 | 28.9 | 1.104x | | FP32 | FP32 | 32 | 4 | 32.3 | 29.3 | 1.102x | | FP32 | FP32 | 32 | 8 | 32.3 | 29.4 | 1.099x | | FP32 | FP32 | 32 | 16 | 31.4 | 28.0 | 1.121x | | FP32 | FP32 | 32 | 32 | 31.8 | 28.8 | 1.104x | | FP32 | FP32 | 32 | 64 | 57.5 | 47.6 | 1.208x | | FP32 | FP32 | 32 | 128 | 106.3 | 87.2 | 1.219x | | FP32 | FP32 | 32 | 256 | 205.8 | 165.5 | 1.244x | | BF16 | BF16 | 16 | 1 | 33.3 | 30.9 | 1.078x | | BF16 | BF16 | 16 | 4 | 31.5 | 29.8 | 1.057x | | BF16 | BF16 | 16 | 8 | 33.8 | 31.5 | 1.073x | | BF16 | BF16 | 16 | 16 | 33.7 | 30.7 | 1.098x | | BF16 | BF16 | 16 | 32 | 34.0 | 31.2 | 1.090x | | BF16 | BF16 | 16 | 64 | 33.1 | 30.4 | 1.089x | | BF16 | BF16 | 16 | 128 | 32.5 | 30.0 | 1.083x | | BF16 | BF16 | 16 | 256 | 54.1 | 51.9 | 1.042x | | BF16 | BF16 | 32 | 1 | 33.9 | 31.0 | 1.094x | | BF16 | BF16 | 32 | 4 | 34.3 | 30.0 | 1.143x | | BF16 | BF16 | 32 | 8 | 31.9 | 29.0 | 1.100x | | BF16 | BF16 | 32 | 16 | 32.8 | 30.3 | 1.083x | | BF16 | BF16 | 32 | 32 | 33.1 | 30.0 | 1.103x | | BF16 | BF16 | 32 | 64 | 32.8 | 31.1 | 1.055x | | BF16 | BF16 | 32 | 128 | 52.0 | 53.5 | 0.972x | | BF16 | BF16 | 32 | 256 | 92.8 | 98.9 | 0.938x | | BF16 | FP32 | 16 | 1 | 34.9 | 31.8 | 1.097x | | BF16 | FP32 | 16 | 4 | 35.1 | 32.2 | 1.090x | | BF16 | FP32 | 16 | 8 | 35.2 | 32.2 | 1.093x | | BF16 | FP32 | 16 | 16 | 33.7 | 31.6 | 1.066x | | BF16 | FP32 | 16 | 32 | 34.5 | 31.6 | 1.092x | | BF16 | FP32 | 16 | 64 | 34.7 | 31.6 | 1.098x | | BF16 | FP32 | 16 | 128 | 54.4 | 45.5 | 1.196x | | BF16 | FP32 | 16 | 256 | 97.1 | 84.4 | 1.150x | | BF16 | FP32 | 32 | 1 | 33.8 | 30.3 | 1.116x | | BF16 | FP32 | 32 | 4 | 30.9 | 27.9 | 1.108x | | BF16 | FP32 | 32 | 8 | 31.0 | 28.6 | 1.084x | | BF16 | FP32 | 32 | 16 | 34.7 | 30.9 | 1.123x | | BF16 | FP32 | 32 | 32 | 32.6 | 28.8 | 1.132x | | BF16 | FP32 | 32 | 64 | 54.4 | 45.4 | 1.198x | | BF16 | FP32 | 32 | 128 | 95.8 | 84.0 | 1.140x | | BF16 | FP32 | 32 | 256 | 179.4 | 160.1 | 1.121x | FlashInfer decode ```bash for heads in 16 32; do python benchmark/bench_linear_attention/bench_kda_flashinfer_mtp.py \\ --task decode \\ --num-q-heads \"${heads}\" \\ --num-v-heads \"${heads}\" done ``` | H | B | FlashInfer (us) | Helion (us) | FI / Helion | Helion max diff | |---:|---:|---:|---:|---:|---:| | 16 | 1 | 94.9 | 67.3 | 1.410x | 0.00e+00 | | 16 | 4 | 108.2 | 67.5 | 1.603x | 1.91e-06 | | 16 | 16 | 93.2 | 69.1 | 1.349x | 1.91e-06 | | 16 | 32 | 89.8 | 64.2 | 1.399x | 1.91e-06 | | 16 | 64 | 98.9 | 72.9 | 1.357x | 1.91e-06 | | 16 | 128 | 92.4 | 75.6 | 1.222x | 3.81e-06 | | 32 | 1 | 92.9 | 66.1 | 1.405x | 1.49e-08 | | 32 | 4 | 93.7 | 67.0 | 1.399x | 1.91e-06 | | 32 | 16 | 92.8 | 67.8 | 1.369x | 1.91e-06 | | 32 | 32 | 91.3 | 66.3 | 1.377x | 3.81e-06 | | 32 | 64 | 90.8 | 66.7 | 1.361x | 3.81e-06 | | 32 | 128 | 113.7 | 76.1 | 1.494x | 7.63e-06 | Prefill ```bash for heads in 16 32; do python benchmark/bench_linear_attention/bench_kda_prefill_cutedsl.py \\ --num-heads \"${heads}\" done ``` | State | H | T | Triton ms | Helion ms | Triton / Helion | |---|---:|---:|---:|---:|---:| | FP32 | 16 | 512 | 0.067 | 0.042 | 1.595x | | FP32 | 16 | 1024 | 0.099 | 0.060 | 1.650x | | FP32 | 16 | 2048 | 0.174 | 0.109 | 1.596x | | FP32 | 16 | 4096 | 0.344 | 0.205 | 1.678x | | FP32 | 16 | 8192 | 0.673 | 0.384 | 1.753x | | FP32 | 32 | 512 | 0.079 | 0.055 | 1.436x | | FP32 | 32 | 1024 | 0.144 | 0.094 | 1.532x | | FP32 | 32 | 2048 | 0.270 | 0.174 | 1.552x | | FP32 | 32 | 4096 | 0.522 | 0.322 | 1.621x | | FP32 | 32 | 8192 | 1.008 | 0.617 | 1.634x | | BF16 | 16 | 512 | 0.067 | 0.042 | 1.595x | | BF16 | 16 | 1024 | 0.097 | 0.060 | 1.617x | | BF16 | 16 | 2048 | 0.173 | 0.109 | 1.587x | | BF16 | 16 | 4096 | 0.341 | 0.204 | 1.672x | | BF16 | 16 | 8192 | 0.667 | 0.383 | 1.741x | | BF16 | 32 | 512 | 0.079 | 0.055 | 1.436x | | BF16 | 32 | 1024 | 0.145 | 0.094 | 1.543x | | BF16 | 32 | 2048 | 0.271 | 0.175 | 1.549x | | BF16 | 32 | 4096 | 0.523 | 0.325 | 1.609x | | BF16 | 32 | 8192 | 1.009 | 0.625 | 1.614x | H=12 packed decode and packed-varlen prefill These CUDA-graph measurements use K3's local shape (H=12, K=V=128), BF16 activations, and the bounded-gate contract (`lower_bound=-5.0`). Prefill uses one packed sequence and includes raw-gate activation, Q/K normalization, the complete prefill pipeline, and the indexed final-state update. | Operation | Activation | State | B or T | Triton | Helion | Speedup | Output diff | State diff | |---|---|---|---:|---:|---:|---:|---:|---:| | Decode (us) | BF16 | FP32 | B=1 | 4.139 | 2.334 | 1.773x | 2.33e-10 | 7.45e-9 | | Decode (us) | BF16 | FP32 | B=4 | 4.582 | 2.701 | 1.697x | 4.77e-7 | 1.12e-8 | | Decode (us) | BF16 | FP32 | B=16 | 8.667 | 4.596 | 1.886x | 4.77e-7 | 1.12e-8 | | Decode (us) | BF16 | FP32 | B=32 | 10.043 | 7.009 | 1.433x | 1.91e-6 | 1.12e-8 | | Decode (us) | BF16 | FP32 | B=64 | 14.443 | 11.584 | 1.247x | 2.38e-7 | 1.49e-8 | | Decode (us) | BF16 | FP32 | B=128 | 40.967 | 29.679 | 1.380x | 1.91e-6 | 1.49e-8 | | Decode (us) | BF16 | BF16 | B=1 | 3.320 | 2.823 | 1.176x | 0.00e+0 | 1.53e-5 | | Decode (us) | BF16 | BF16 | B=4 | 3.492 | 3.008 | 1.161x | 0.00e+0 | 6.10e-5 | | Decode (us) | BF16 | BF16 | B=16 | 4.450 | 4.008 | 1.110x | 2.98e-8 | 1.22e-4 | | Decode (us) | BF16 | BF16 | B=32 | 5.904 | 5.452 | 1.083x | 9.54e-7 | 3.05e-5 | | Decode (us) | BF16 | BF16 | B=64 | 9.802 | 8.799 | 1.114x | 9.54e-7 | 1.22e-4 | | Decode (us) | BF16 | BF16 | B=128 | 16.619 | 14.583 | 1.140x | 3.81e-6 | 1.22e-4 | | Prefill (ms) | BF16 | FP32 | T=512 | 0.0681 | 0.0507 | 1.342x | 2.44e-4 | 2.88e-4 | | Prefill (ms) | BF16 | FP32 | T=1024 | 0.0967 | 0.0690 | 1.401x | 2.44e-4 | 1.94e-4 | | Prefill (ms) | BF16 | FP32 | T=2048 | 0.1538 | 0.1159 | 1.327x | 4.88e-4 | 1.69e-4 | | Prefill (ms) | BF16 | FP32 | T=4096 | 0.2991 | 0.2167 | 1.380x | 4.88e-4 | 8.77e-5 | | Prefill (ms) | BF16 | FP32 | T=8192 | 0.6026 | 0.4072 | 1.480x | 7.32e-4 | 2.40e-4 | | Prefill (ms) | BF16 | BF16 | T=512 | 0.0691 | 0.0504 | 1.371x | 2.44e-4 | 4.88e-4 | | Prefill (ms) | BF16 | BF16 | T=1024 | 0.0953 | 0.0684 | 1.393x | 2.44e-4 | 4.88e-4 | | Prefill (ms) | BF16 | BF16 | T=2048 | 0.1529 | 0.1153 | 1.325x | 4.88e-4 | 2.44e-4 | | Prefill (ms) | BF16 | BF16 | T=4096 | 0.2980 | 0.2150 | 1.386x | 4.88e-4 | 2.44e-4 | | Prefill (ms) | BF16 | BF16 | T=8192 | 0.5991 | 0.4059 | 1.476x | 7.32e-4 | 2.44e-4 | All outputs and states were finite. H=12 unbounded ReplaySSM decode These CUDA-graph measurements compare the unbounded Triton and Helion ReplaySSM implementations with L=16. Timings average all 16 ring positions, including the natural flush, and use the all-zero force-flush buffer supplied by CUDA graph replay. | Activation | State | H | B | Triton ReplaySSM (us) | Helion ReplaySSM (us) | Speedup | Output diff | State diff | |---|---|---:|---:|---:|---:|---:|---:|---:| | BF16 | FP32 | 12 | 1 | 9.384 | 4.540 | 2.067x | 1.22e-4 | 2.93e-3 | | BF16 | FP32 | 12 | 4 | 9.537 | 4.755 | 2.006x | 2.44e-4 | 4.61e-3 | | BF16 | FP32 | 12 | 16 | 10.740 | 6.034 | 1.780x | 2.44e-4 | 6.38e-3 | | BF16 | FP32 | 12 | 64 | 26.915 | 20.046 | 1.343x | 4.88e-4 | 7.15e-3 | | BF16 | FP32 | 12 | 128 | 55.264 | 37.753 | 1.464x | 4.88e-4 | 7.49e-3 | | BF16 | BF16 | 12 | 1 | 8.841 | 6.131 | 1.442x | 1.91e-6 | 3.05e-5 | | BF16 | BF16 | 12 | 4 | 9.120 | 6.796 | 1.342x | 3.05e-5 | 9.77e-4 | | BF16 | BF16 | 12 | 16 | 9.741 | 7.344 | 1.326x | 6.10e-5 | 1.95e-3 | | BF16 | BF16 | 12 | 64 | 23.268 | 20.498 | 1.135x | 6.10e-5 | 1.95e-3 | | BF16 | BF16 | 12 | 128 | 37.628 | 39.709 | 0.948x | 1.22e-4 | 1.95e-3 | All outputs and states were finite. Standalone full-wrapper Helion versus CuTe DSL The standalone harness [kda_prefill_production_benchmark.py] (https://gist.github.com/ethche/2fcd809cbb6bd28c48579d08995ba2bb) dispatches both implementations through `KDAKernelDispatcher.extend`. The timed boundary includes Q/K normalization, raw-gate activation, packed-sequence metadata, the complete prefill pipeline, and indexed state-pool updates. This is a direct backend comparison; the pre-staged core benchmark above is included only as kernel-level context. ```bash python kda_prefill_production_benchmark.py \\ --sglang-repo \"$PWD\" \\ --state-dtype float32 \\ --seq-lens 512 1024 2048 4096 8192 ``` The standalone harness dispatches both implementations through `KDAKernelDispatcher.extend`. The timed boundary includes Q/K normalization, raw-gate activation, packed-sequence metadata, the complete prefill pipeline, and indexed state-pool updates. This is a direct backend comparison; the pre-staged core benchmark above is included only as kernel-level context. ```bash python kda_prefill_production_benchmark.py \\ --sglang-repo \"$PWD\" \\ --state-dtype float32 \\ --seq-lens 512 1024 2048 4096 8192 ``` The 50-shape geomean is **2.031x** (uniform: **2.046x**, ragged: **2.008x**). All correctness checks passed. Errors are reported as maximum absolute error / relative L2 error. | B | H | Layout | Lengths | Tokens | Helion ms | CuTe DSL ms | CuTe / Helion | Output error | State error | |---:|---:|:---|---:|---:|---:|---:|---:|---:|---:| | 1 | 16 | uniform | 512 | 512 | 0.291 | 0.612 | 2.106x | 1.10e-03/5.50e-03 | 1.03e-02/4.52e-03 | | 1 | 16 | uniform | 1024 | 1024 | 0.313 | 0.652 | 2.080x | 1.46e-03/5.61e-03 | 1.08e-02/4.58e-03 | | 1 | 16 | uniform | 2048 | 2048 | 0.331 | 0.665 | 2.008x | 1.46e-03/5.65e-03 | 1.22e-02/4.60e-03 | | 1 | 16 | uniform | 4096 | 4096 | 0.331 | 0.679 | 2.049x | 1.46e-03/5.66e-03 | 1.33e-02/4.60e-03 | | 1 | 16 | uniform | 8192 | 8192 | 0.584 | 0.885 | 1.516x | 1.46e-03/5.68e-03 | 1.18e-02/4.51e-03 | | 1 | 32 | uniform | 512 | 512 | 0.316 | 0.663 | 2.095x | 1.22e-03/5.48e-03 | 1.23e-02/4.48e-03 | | 1 | 32 | uniform | 1024 | 1024 | 0.331 | 0.698 | 2.107x | 1.46e-03/5.56e-03 | 1.30e-02/4.46e-03 | | 1 | 32 | uniform | 2048 | 2048 | 0.333 | 0.685 | 2.055x | 1.46e-03/5.60e-03 | 1.12e-02/4.48e-03 | | 1 | 32 | uniform | 4096 | 4096 | 0.451 | 0.808 | 1.793x | 1.46e-03/5.62e-03 | 1.05e-02/4.46e-03 | | 1 | 32 | uniform | 8192 | 8192 | 0.848 | 1.493 | 1.761x | 1.59e-03/5.62e-03 | 1.06e-02/4.46e-03 | | 4 | 16 | uniform | 512 | 2048 | 0.336 | 0.704 | 2.098x | 1.46e-03/5.51e-03 | 1.16e-02/4.48e-03 | | 4 | 16 | ragged | 269-755 | 2048 | 0.339 | 0.713 | 2.101x | 1.46e-03/5.53e-03 | 1.11e-02/4.47e-03 | | 4 | 16 | uniform | 1024 | 4096 | 0.329 | 0.698 | 2.120x | 1.46e-03/5.59e-03 | 1.18e-02/4.54e-03 | | 4 | 16 | ragged | 525-1523 | 4096 | 0.335 | 0.676 | 2.020x | 1.46e-03/5.60e-03 | 1.21e-02/4.49e-03 | | 4 | 16 | uniform | 2048 | 8192 | 0.380 | 0.768 | 2.024x | 1.95e-03/5.65e-03 | 1.27e-02/4.49e-03 | | 4 | 16 | ragged | 1037-3059 | 8192 | 0.417 | 0.791 | 1.895x | 1.71e-03/5.65e-03 | 1.44e-02/4.51e-03 | | 4 | 16 | uniform | 4096 | 16384 | 0.712 | 1.416 | 1.989x | 1.46e-03/5.66e-03 | 1.31e-02/4.48e-03 | | 4 | 16 | ragged | 2061-6131 | 16384 | 0.782 | 1.450 | 1.853x | 1.46e-03/5.66e-03 | 1.10e-02/4.43e-03 | | 4 | 16 | uniform | 8192 | 32768 | 1.365 | 2.743 | 2.010x | 1.95e-03/5.67e-03 | 1.33e-02/4.51e-03 | | 4 | 16 | ragged | 4109-12275 | 32768 | 1.501 | 2.829 | 1.885x | 1.95e-03/5.67e-03 | 1.19e-02/4.51e-03 | | 4 | 32 | uniform | 512 | 2048 | 0.309 | 0.661 | 2.144x | 1.22e-03/5.47e-03 | 1.17e-02/4.44e-03 | | 4 | 32 | ragged | 269-755 | 2048 | 0.325 | 0.679 | 2.087x | 1.46e-03/5.49e-03 | 1.23e-02/4.39e-03 | | 4 | 32 | uniform | 1024 | 4096 | 0.357 | 0.756 | 2.117x | 1.46e-03/5.56e-03 | 1.26e-02/4.47e-03 | | 4 | 32 | ragged | 525-1523 | 4096 | 0.389 | 0.764 | 1.963x | 1.95e-03/5.56e-03 | 1.19e-02/4.46e-03 | | 4 | 32 | uniform | 2048 | 8192 | 0.668 | 1.393 | 2.087x | 1.59e-03/5.60e-03 | 1.11e-02/4.45e-03 | | 4 | 32 | ragged | 1037-3059 | 8192 | 0.726 | 1.412 | 1.946x | 1.46e-03/5.60e-03 | 1.18e-02/4.45e-03 | | 4 | 32 | uniform | 4096 | 16384 | 1.273 | 2.701 | 2.122x | 2.44e-03/5.61e-03 | 1.12e-02/4.43e-03 | | 4 | 32 | ragged | 2061-6131 | 16384 | 1.384 | 2.736 | 1.977x | 1.95e-03/5.61e-03 | 1.24e-02/4.42e-03 | | 4 | 32 | uniform | 8192 | 32768 | 2.489 | 5.270 | 2.117x | 1.95e-03/5.63e-03 | 1.36e-02/4.46e-03 | | 4 | 32 | ragged | 4109-12275 | 32768 | 2.709 | 5.342 | 1.972x | 3.17e-03/5.62e-03 | 1.31e-02/4.44e-03 | | 16 | 16 | uniform | 512 | 8192 | 0.360 | 0.759 | 2.110x | 1.46e-03/5.52e-03 | 1.26e-02/4.50e-03 | | 16 | 16 | ragged | 269-755 | 8192 | 0.384 | 0.771 | 2.008x | 1.95e-03/5.51e-03 | 1.31e-02/4.46e-03 | | 16 | 16 | uniform | 1024 | 16384 | 0.668 | 1.397 | 2.091x | 1.46e-03/5.60e-03 | 1.33e-02/4.53e-03 | | 16 | 16 | ragged | 525-1523 | 16384 | 0.709 | 1.410 | 1.990x | 1.46e-03/5.59e-03 | 1.20e-02/4.47e-03 | | 16 | 16 | uniform | 2048 | 32768 | 1.273 | 2.702 | 2.123x | 1.95e-03/5.64e-03 | 1.28e-02/4.51e-03 | | 16 | 16 | ragged | 1037-3059 | 32768 | 1.344 | 2.727 | 2.029x | 1.46e-03/5.64e-03 | 1.33e-02/4.48e-03 | | 16 | 16 | uniform | 4096 | 65536 | 2.483 | 5.263 | 2.119x | 1.95e-03/5.66e-03 | 1.21e-02/4.51e-03 | | 16 | 16 | ragged | 2061-6131 | 65536 | 2.616 | 5.314 | 2.031x | 1.95e-03/5.66e-03 | 1.24e-02/4.48e-03 | | 16 | 16 | uniform | 8192 | 131072 | 4.912 | 10.436 | 2.124x | 4.64e-03/5.67e-03 | 1.36e-02/4.53e-03 | | 16 | 16 | ragged | 4109-12275 | 131072 | 5.166 | 10.538 | 2.040x | 4.64e-03/5.67e-03 | 1.28e-02/4.52e-03 | | 16 | 32 | uniform | 512 | 8192 | 0.679 | 1.406 | 2.070x | 1.46e-03/5.48e-03 | 1.36e-02/4.45e-03 | | 16 | 32 | ragged | 269-755 | 8192 | 0.707 | 1.420 | 2.010x | 1.46e-03/5.48e-03 | 1.31e-02/4.39e-03 | | 16 | 32 | uniform | 1024 | 16384 | 1.286 | 2.714 | 2.111x | 2.44e-03/5.56e-03 | 1.30e-02/4.44e-03 | | 16 | 32 | ragged | 525-1523 | 16384 | 1.319 | 2.729 | 2.068x | 1.46e-03/5.55e-03 | 1.28e-02/4.42e-03 | | 16 | 32 | uniform | 2048 | 32768 | 2.495 | 5.286 | 2.118x | 1.95e-03/5.60e-03 | 1.21e-02/4.47e-03 | | 16 | 32 | ragged | 1037-3059 | 32768 | 2.546 | 5.306 | 2.084x | 3.17e-03/5.60e-03 | 1.59e-02/4.44e-03 | | 16 | 32 | uniform | 4096 | 65536 | 4.927 | 10.451 | 2.121x | 3.20e-03/5.61e-03 | 1.33e-02/4.46e-03 | | 16 | 32 | ragged | 2061-6131 | 65536 | 5.003 | 10.557 | 2.110x | 3.20e-03/5.61e-03 | 1.29e-02/4.44e-03 | | 16 | 32 | uniform | 8192 | 131072 | 9.794 | 20.954 | 2.139x | 3.04e-03/5.63e-03 | 1.49e-02/4.46e-03 | | 16 | 32 | ragged | 4109-12275 | 131072 | 9.911 | 21.004 | 2.119x | 4.47e-03/5.63e-03 | 1.30e-02/4.45e-03 | End-to-End Benchmarks Each cell has 16 measurements from two process runs with eight repeats. The comparison uses the default backend versus combined Helion. ```bash python -m sglang.benchmark.one_batch \\ --model-path moonshotai/Kimi-Linear-48B-A3B-Instruct \\ --trust-remote-code \\ --tp-size 2 \\ --mamba-ssm-dtype STATE_DTYPE \\ --disable-custom-all-reduce \\ --mem-fraction-static 0.85 \\ --max-running-requests 128 \\ --max-mamba-cache-size 640 \\ --context-length 16384 \\ --batch-size 1 4 16 \\ --input-len 8192 8192 8192 8192 8192 8192 8192 8192 \\ --output-len 64 256 \\ [--linear-attn-backend helion] ``` | State | B | Out | TTFT reduction | TPOT reduction | Throughput gain | |---|---:|---:|---:|---:|---:| | FP32 | 1 | 64 | 9.18% | 6.59% | 7.50% | | FP32 | 1 | 256 | 8.94% | 6.92% | 7.91% | | FP32 | 4 | 64 | 3.86% | 6.24% | 4.92% | | FP32 | 4 | 256 | 3.75% | 5.81% | 5.69% | | FP32 | 16 | 64 | 2.62% | 1.50% | 2.22% | | FP32 | 16 | 256 | 2.73% | 1.74% | 2.15% | | BF16 | 1 | 64 | 6.05% | 7.12% | 7.50% | | BF16 | 1 | 256 | 5.93% | 6.76% | 7.32% | | BF16 | 4 | 64 | 3.57% | 4.76% | 4.62% | | BF16 | 4 | 256 | 3.47% | 4.35% | 4.41% | | BF16 | 16 | 64 | 2.57% | 6.41% | 3.55% | | BF16 | 16 | 256 | 2.72% | 4.67% | 4.28% | #### Raw medians Timing and throughput cells show default / Helion. | State | B | Out | TTFT ms | TPOT ms | Overall tok/s | |---|---:|---:|---:|---:|---:| | FP32 | 1 | 64 | 79.77 / 72.45 | 4.428 / 4.136 | 22974.9 / 24699.0 | | FP32 | 1 | 256 | 79.01 / 71.95 | 4.384 / 4.080 | 7026.6 / 7582.2 | | FP32 | 4 | 64 | 221.10 / 212.55 | 5.047 / 4.732 | 61129.8 / 64139.4 | | FP32 | 4 | 256 | 220.93 / 212.65 | 4.972 / 4.683 | 22653.8 / 23943.5 | | FP32 | 16 | 64 | 862.22 / 839.64 | 6.697 / 6.597 | 102869.5 / 105148.6 | | FP32 | 16 | 256 | 863.30 / 839.75 | 6.819 / 6.700 | 51952.8 / 53070.3 | | BF16 | 1 | 64 | 75.90 / 71.32 | 4.432 / 4.117 | 23227.2 / 24968.2 | | BF16 | 1 | 256 | 75.78 / 71.28 | 4.371 / 4.075 | 7076.8 / 7594.9 | | BF16 | 4 | 64 | 220.41 / 212.55 | 4.987 / 4.749 | 61512.2 / 64356.3 | | BF16 | 4 | 256 | 220.39 / 212.73 | 4.937 / 4.723 | 22794.5 / 23800.4 | | BF16 | 16 | 64 | 863.14 / 840.96 | 6.769 / 6.335 | 102568.1 / 106206.2 | | BF16 | 16 | 256 | 862.77 / 839.32 | 6.717 / 6.404 | 52351.7 / 54593.2 | ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci`. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31555369047](https://github.com/sgl-project/sglang/actions/runs/31555369047) Latest PR Test (Extra): :x: [Run #31555368882](https://github.com/sgl-project/sglang/actions/runs/31555368882) ",
"url": "https://github.com/sgl-project/sglang/pull/32593",
"createdAt": "2026-07-28T01:43:48Z",
"updatedAt": "2026-08-12T19:17:14Z",
"timestamp": "2026-08-12T19:17:14Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"documentation",
"quant",
"dependencies",
"Multi-modal",
"blackwell",
"npu",
"run-ci",
"jit-kernel"
],
"author": "ethche",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32597",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support streaming session on NPU",
"text": " ## Motivation Support streaming session on NPU. ## Modifications Modify streaming_session.py, when page size>1, align KV cache in page size. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31353728791](https://github.com/sgl-project/sglang/actions/runs/31353728791) Latest PR Test (Extra): :x: [Run #31353728693](https://github.com/sgl-project/sglang/actions/runs/31353728693) ",
"url": "https://github.com/sgl-project/sglang/pull/32597",
"createdAt": "2026-07-28T02:43:20Z",
"updatedAt": "2026-08-13T10:50:59Z",
"timestamp": "2026-08-13T10:50:59Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "sigama-w",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32598",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[UT][NPU] Add npu unit test for ascend_gdn_backend and ascend_hybrid_linear_attn_backend",
"text": " ## Motivation Add npu unit test for ascend_gdn_backend and ascend_hybrid_linear_attn_backend ## Modifications Add test\\registered\\npu\\attention\\test_ascend_gdn_backend.py and test\\registered\\npu\\attention\\test_ascend_hybrid_linear_attn_backend.py ## Accuracy Tests N/A ## Speed Tests and Profiling N/A ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31586352689](https://github.com/sgl-project/sglang/actions/runs/31586352689) Latest PR Test (Extra): :x: [Run #31586352470](https://github.com/sgl-project/sglang/actions/runs/31586352470) ",
"url": "https://github.com/sgl-project/sglang/pull/32598",
"createdAt": "2026-07-28T02:49:44Z",
"updatedAt": "2026-08-12T17:28:20Z",
"timestamp": "2026-08-12T17:28:20Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"npu",
"run-ci"
],
"author": "luz3633",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:32611",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix transcription & audio-understanding for ASR/audio/speech models",
"text": "## Motivation Fix transcription for decoder-only speech/audio models that were already present Adds /v1/audio/transcriptions support for three decoder-only speech/audio LMs, so they serve real ASR instead of falling back to the Whisper adapter: - Qwen/Qwen2-Audio-7B-Instruct - zai-org/GLM-ASR-Nano-2512 - ibm-granite/granite-speech-3.3-8b ## Modifications **New transcription adapters.** Added glmasr.py, qwen2_audio.py, granite_speech.py (registered as GlmAsr / Qwen2Audio / GraniteSpeech). Each disables language detection and implements build_sampling_params (emitting temperature + a duration-scaled max_new_tokens: max(floor, int(duration_s * 15)), so long clips aren't silently truncated while short clips use a per-model floor) and build_verbose_response (no timestamp segments; \"auto\" default language). Without these, resolve_adapter() falls back to the Whisper adapter and injects Whisper-only params (_detect_language, forced language/task prefix) these models don't support. **Transcription-endpoint prompt handling.** The /v1/audio/transcriptions endpoint sends text=\"\", which carries no audio placeholder for the encoder features to fill. Added a per-processor _build_transcription_prompt() to qwen_audio.py / glmasr.py, and an audio branch + _build_audio_prompt() to transformers_auto.py (Granite), that render the model's audio placeholder via apply_chat_template when the request text is empty — fixing the empty-prompt failure while leaving the chat path unchanged. Also added audio_token_index to the generic processor's audio-token lookup. **Model-side audio-feature handling.** Rewrote GlmAsrForConditionalGeneration.get_audio_feature to reshape encoder output per 30s window and keep only the valid embeddings per window (derived from input_features_mask via the conv-subsampling + merge_factor downsampling), mirroring HF's get_audio_features. This keeps the emitted embedding count aligned with the inserted audio-placeholder tokens for variable-length / multi-window clips. Forced truncation=True for the Qwen2-Audio processor so clips longer than 30s are capped to the audio tower's fixed 3000-mel window instead of raising. **base_processor.py.** Added GraniteSpeechProcessor to the audio=-kwarg allowlist (its HF processor expects audio=, not the deprecated audios=). **Tests.** Added test/registered/unit/entrypoints/openai/test_transcription_adapters.py (CPU CI): adapter resolution per architecture + Whisper fallback, the max_new_tokens floor/duration-scaling boundary, and the verbose-response contract. --- ### CI States Latest PR Test (Base): :x: [Run #30954721199](https://github.com/sgl-project/sglang/actions/runs/30954721199) Latest PR Test (Extra): :x: [Run #30954720541](https://github.com/sgl-project/sglang/actions/runs/30954720541) ",
"url": "https://github.com/sgl-project/sglang/pull/32611",
"createdAt": "2026-07-28T04:18:58Z",
"updatedAt": "2026-08-13T07:31:21Z",
"timestamp": "2026-08-13T07:31:21Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"npu",
"jit-kernel"
],
"author": "SKRohit",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32630",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD][Not-Merge]: torch fallbacks for top-k/top-p renorm so DSPARK/DFLASH works on ROCm",
"text": "## Problem `build_dflash_verify_target_probs()` calls `top_k_renorm_prob()` and `top_p_renorm_prob()` unconditionally, but `sgl_kernel` only exports them under CUDA/MUSA. On ROCm both are `None`, so DSPARK/DFLASH speculative decoding dies with: ``` TypeError: 'NoneType' object is not callable ``` as soon as a request sets `top_p < 1` **or** `top_k > 1`. That includes any client inheriting sampling defaults from a model's `generation_config`, so in practice the server serves for a few minutes and then dies on the first such request. Fixes #32569 (reported on 8x MI350X; AMD confirmed it as \"a valid issue on ROCm\"). ## Fix Both ops are plain probability renormalisation with exact, portable torch equivalents, so the `else` branch now defines them instead of setting them to `None`. Deliberately minimal: - `tree_speculative_sampling_target_only` has no cheap equivalent and **stays `None`** - `_DFLASH_SAMPLING_VERIFY_AVAILABLE` stays `False`, so **no additional code path is enabled** by this change - no new imports (`torch` is already imported in this module) One edge case worth noting: `top_p <= 0` would mask every token and produce an all-zero row, so the most likely token is always kept. ## Verification Hardware: 8x MI350X (gfx950), ROCm 7.2.1, image `lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727`, Kimi-K3 MXFP4, TP8, DSPARK with a `block_size=7` draft model. **Reproduced the crash, then confirmed the fix** with requests that previously killed the server: - `temperature=0.7, top_p=0.8` → OK - `temperature=0.8, top_k=20` → OK **Semantics checked** against hand-computed expectations (top-k = 1/2/V, top-p = 0/0.8/1.0, batched mixed k/p, row sums == 1, scalar and tensor inputs). **Throughput on that box** (SGLang's own `bench_serving.py`, 1k in / 300 out, `--random-range-ratio 1.0 --seed 42 --flush-cache`, median of 3 runs): | | DSPARK off | DSPARK on (this fix) | |---|---|---| | single stream | 37.5 tok/s | **149–167 tok/s** | | TPOT | 26.1 ms | **~5 ms** | | output @ conc 128 | 1000.9 tok/s | **1606–1644 tok/s** | | total @ conc 128 | 4417 tok/s | **7090–7256 tok/s** | | median TTFT @ conc 8 | 717 ms | **490 ms** | Also verified unaffected: needle-in-a-haystack retrieval up to 641,779 prompt tokens, tool calling, and vision input. No restarts over several hours of load. I did not add a unit test since the fallbacks only activate off-CUDA and CI presumably runs on NVIDIA - happy to add one under `test/srt/` if you'd prefer. --- ### CI States Latest PR Test (Base): :x: [Run #30342976482](https://github.com/sgl-project/sglang/actions/runs/30342976482) Latest PR Test (Extra): :white_check_mark: [Run #30396990531](https://github.com/sgl-project/sglang/actions/runs/30396990531) ",
"url": "https://github.com/sgl-project/sglang/pull/32630",
"createdAt": "2026-07-28T08:35:02Z",
"updatedAt": "2026-08-12T17:47:58Z",
"timestamp": "2026-08-12T17:47:58Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"amd",
"DO NOT MERGE"
],
"author": "stefanskiasan",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32637",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Optimize delayed sample and mrope position computation",
"text": " ## Motivation Improve inference performance by reducing M-RoPE position computation overhead and improving the sampling stage of the overlap decode pipeline. ## Modifications 1. Optimize M-RoPE position computation with a vectorized decode path for common cases while preserving the fallback path for precomputed multimodal positions. 2. Enable delayed sampling for general overlap decode through SGLANG_ENABLE_DELAY_SAMPLE, allowing sampling overhead to overlap with model execution. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 3. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 4. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 5. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31557221230](https://github.com/sgl-project/sglang/actions/runs/31557221230) Latest PR Test (Extra): :x: [Run #31557221071](https://github.com/sgl-project/sglang/actions/runs/31557221071) ",
"url": "https://github.com/sgl-project/sglang/pull/32637",
"createdAt": "2026-07-28T09:25:57Z",
"updatedAt": "2026-08-13T11:15:47Z",
"timestamp": "2026-08-13T11:15:47Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "Estrella-xx",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32673",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts",
"text": "# [Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts ## Motivation In [arXiv:2607.21535](https://arxiv.org/abs/2607.21535) we show that a built-in MTP draft head's full-attention KV read grows with the prefix until it dominates the decode step, and that restricting the *draft's* attention to a sink plus recent window cuts the per-decode-step cost by +28% to +44% at 1M context across three architecture families at near-unchanged acceptance, with the target still verifying every token at full attention. The same idea motivated `--speculative-draft-window-size` in #24664, but that flag reaches the drafter through the model file (`llama_eagle3.py` sets `sliding_window_size` per layer) and the backend's per-layer SWA path, so it applies to `LlamaForCausalLMEagle3` checkpoints. This PR applies the window one level down, in the shared draft-decode index builder, so the built-in EAGLE / MTP (NEXTN) draft-decode path gets it on the Triton and FlashInfer draft backends with no model-file change and no second attention wrapper — the draft simply gets a shorter KV list. Only the windowing mechanism is upstreamed here; the paper's ring-buffer reclaim of the unread draft KV is possible follow-up work. ## Modifications 1. `generate_draft_decode_kv_indices` gains `window_size` and `sink_size` `tl.constexpr` params, both defaulting to `0`. With `window_size > 0` a draft-decode step gathers `[first sink_size tokens] + [most recent window_size tokens]` instead of the whole prefix, then appends the draft tree tokens unwindowed as before. `window_size == 0` collapses every added op to the identity, so the default path and the kernel's other callers (AITER, MLA) are unchanged. 2. `--speculative-draft-window-size` is now also honored by the built-in EAGLE / MTP draft-decode path on the Triton and FlashInfer draft backends. `spec_utils.resolve_draft_decode_window()` is the single mapping from server args to `(window_size, sink_size)`; both `*MultiStepDraftBackend`s call it once in `__init__`. 3. New `--speculative-draft-sink-size` for the attention sink, since a pure recent window drops the initial tokens attention reliably parks on. The window keeps DFLASH's unit and anchor — KV entries, counted back from the end of the prefix — so a step reads `min(seq_len, window + sink)` prefix entries, which at the default `sink=0` is `min(seq_len, window)`, the same expression as `_compute_compact_draft_seq_lens`, with no `-1`. Tree tokens are never capped, so the existing `window_size >= num_draft_tokens` check still holds. 4. A draft model with a sliding window of its own keeps it, and this windowing stays off. That covers an MTP head with SWA in its config and a `LlamaForCausalLMEagle3` drafter, whose window comes from this same flag. One KV list is shared by every draft layer, so the index builder cannot express a per-layer window, and a checkpoint's own window should win over a server flag either way. We warn only when the resulting window differs from the requested one, so the EAGLE-3 case stays quiet. CUDA graphs are safe: the per-request kept length rides in `kv_indptr`, exactly like the variable-seqlen path the graph already replays. Same launch grid, no host sync, no dynamic shapes. ## Speed Tests and Profiling Qwen3.6-35B-A3B bf16, one B200, `trtllm_mha` target, EAGLE `steps=6 topk=1 draft_tokens=7`, `W=4032 S=64` (4096 prefix entries per draft step), batch 1, RULER `niah_multiquery_enum`, 512 output tokens. `window` vs `native` on the same draft backend; the target is identical in every arm. `T_iter` is per-iteration wall time — the mechanism, without acceptance mixed in — and TPOT is `T_iter / AL`. | context | draft backend | TPOT native → window | speedup | T_iter native → window | AL native → window | |---|---|---|---|---|---| | 32K | triton | 3.895 → 3.068 ms | 1.27× | 12.67 → 10.05 ms (−21%) | 3.25 → 3.28 | | 64K | triton | 4.414 → 3.264 ms | 1.35× | 19.06 → 13.48 ms (−29%) | 4.32 → 4.13 | | 128K | triton | 7.047 → 4.568 ms | 1.54× | 31.65 → 20.02 ms (−37%) | 4.49 → 4.38 | | 256K | triton | 10.448 → 6.093 ms | 1.71× | 57.35 → 33.45 ms (−42%) | 5.49 → 5.49 | | 32K | flashinfer | 2.199 → 2.142 ms | 1.03× | 7.20 → 7.01 ms (−3%) | 3.28 → 3.28 | | 64K | flashinfer | 1.728 → 1.738 ms | 0.99× | 7.46 → 7.18 ms (−4%) | 4.32 → 4.13 | | 128K | flashinfer | 1.799 → 1.696 ms | 1.06× | 8.08 → 7.44 ms (−8%) | 4.49 → 4.38 | | 256K | flashinfer | 1.735 → 1.499 ms | 1.16× | 9.36 → 8.05 ms (−14%) | 5.39 → 5.37 | Dense (no speculation) TPOT at those four contexts, for reference: 2.997 / 3.158 / 3.320 / 3.698 ms. - `T_iter` drops at every point on both backends and the drop grows with context, which is the mechanism doing what it should: the draft's read is capped while native's grows. - TPOT also depends on acceptance, so the 64K FlashInfer point is a wash — a 4% per-iteration saving against a 4% acceptance dip. The window is a clear win on FlashInfer from ~128K up and at every point on Triton, whose full-prefix draft read is far more expensive to begin with. - Acceptance never beats native in this sweep. It does past 512K in the paper, but that needs RoPE extension beyond this checkpoint's 256K native context, so we leave it out of the PR's numbers. ## Testing `test/registered/unit/spec/test_draft_decode_window.py` — 13 tests / 28 subtests, 16 s on one GPU. Every expectation comes from a closed-form oracle over `seq_lens`, never from the kernel's own packing, so a bug can't be mirrored on both sides. - `window_size == 0` reproduces the full-KV read plan, offsets and gathered slots, for single- and multi-request batches at `topk` 1 and 2. This is the off-by-default regression guard. - Windowed `kv_indptr` lengths are `min(seq_len, sink + window) + tree`, and the gather really is `[sink] + [recent] + [tree]`, parametrized past 256K/512K/1M. - Tree drafts (`topk > 1`): each `(request, branch)` slot gets its own windowed base list plus its own branch at the oracle offset. Covered at `page_size == 1` and on the `page_size > 1 and topk > 1` paged branch, where tree tokens are read from the uncapped prefix end but written at the capped offset. - `resolve_draft_decode_window`, including the own-window opt-out and exactly when it warns, plus a CPU-only guard that both draft backends resolve through it. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#benchmark-the-speed). - [x] Follow the SGLang code style guidance. --- ### CI States Latest PR Test (Base): :x: [Run #31621192148](https://github.com/sgl-project/sglang/actions/runs/31621192148) Latest PR Test (Extra): :x: [Run #31621191599](https://github.com/sgl-project/sglang/actions/runs/31621191599) ",
"url": "https://github.com/sgl-project/sglang/pull/32673",
"createdAt": "2026-07-28T18:51:04Z",
"updatedAt": "2026-08-12T17:45:55Z",
"timestamp": "2026-08-12T17:45:55Z",
"metrics": {
"reactions": 1,
"comments": 10
},
"labels": [
"speculative-decoding",
"run-ci",
"jit-kernel"
],
"author": "avalliappan-nvidia",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:32741",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] MXFP4 KV Cache Decode for DSV4 on Hopper",
"text": "## Summary MXFP4 KV cache for DeepSeek V4 on Hopper (SM90/H20), end to end: codec + memory pool + fused decode attention kernel + backend integration. The decode kernel is a JIT-compiled port of the FlashMLA three-stage split-KV design (scheduler metadata kernel + persistent WGMMA main kernel + combine kernel), covering all 44 layers (SWA + C4/C128 + attn_sink) in one call. The PR is self-contained: the kernel and its vendored dependencies live under `python/sglang/kernels/jit/` and compile at first use via `load_jit` — no sgl-kernel wheel changes required. ## MXFP4 Layout (368 B/token) ``` [224 B packed E2M1 | 14 B E8M0 + 2 B pad | 128 B BF16 RoPE] ``` - E2M1 (4-bit FP: 1S+2E+1M) nope data, block-size 32 (14 groups) - E8M0 8-bit exponent-only scale per group (dequantized in-register via `bits << 23`) - BF16 RoPE (unquantized) - Row-major contiguous, no page tail padding; no global scale needed (unlike NVFP4) - 16-byte-aligned row (368 B = 16×23) enables 128-bit loads; the NVFP4 reference's 380 B is only 4-byte aligned Memory capacity gain vs FP8: ~+37% full-token capacity (bytes/token 7705 → 5609; the NVFP4 reference gains +31.75%). ## Decode Kernel (`kernels/jit/csrc/mxfp4_dsv4_decode_sm90/`) FlashMLA-style fused split-KV decode for MXFP4: 1. **Scheduler metadata kernel** — 1 CTA × 1 thread computes per-SM split assignments with device-side length clamping (CUDA-graph safe: runs inside the captured graph, re-executes with replayed top-k lengths) 2. **Persistent main kernel** — 384 threads (3 warp groups: producer dequant→SMEM, QK+PV local, PV remote), WGMMA 64×64×16 / 64×256×16; E8M0 block-32 dequant via bit-shift, E2M1 via LUT+PRMT (4 instructions per 16 elements) 3. **Combine kernel** — merges split-K partials with PDL (programmatic dependent launch) Entry point: `entry.cuh` (TVM-FFI single translation unit, explicit instantiations for h_q 64/128). Python wrapper: `kernels/ops/attention/mxfp4_dsv4_decode_sm90.py` (FlashMLASchedMeta, per-geometry scratch reuse, C++-side stream resolution via `TVMFFIEnvGetStream`). ### Vendored dependencies (provenance) | Component | Source | |---|---| | `params.h`, `defines.h`, `utils.h`, `flashmla_utils.h`, `kerutils/`, `combine` | FlashMLA upstream @ 05e26647 | | `config.h`, `dequant.h`, `layout.h`, `components/helpers.h`, splitkv skeleton | SGLang reference PR #31269 (NVFP4) | | `splitkv_mla.cuh` (MXFP4 variant), `entry.cuh` | This PR | CUTLASS/CUTE headers come from the JIT build's `extra_dependencies=[\"cutlass\"]` (pinned cutlass @ 147f5673, matching the reference PR). `kerutils` includes were converted to relative paths during vendoring. ## Changes - **Codec** (`srt/layers/attention/dsv4/mxfp4_k_cache.py`): quantize/dequant for the 368 B row (Triton), paged variants - **Compressor** (`compressor_v2.py`): `_forward_mxfp4` covering all 44 layers (C0/C4/C128), decode-plan boundary semantics - **Memory pool**: parameterized enablement via `--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4` (env alias with deprecation warning); pool sizing/accounting unified - **Startup guards**: HiSparse + MXFP4 combination rejected at init (incompatible sparse layout); DeepseekV4 + fp4_e2m1 on non-SM90 fails fast with a clear error - **Backend** (`deepseek_v4_backend.py`): `_forward_mxfp4_decode_flashmla` — one fused call per layer; scheduler metadata shared per geometry across layers, re-created during CUDA-graph capture (top-k lengths are replayed device inputs) - **Tests**: `test/registered/kernels/ops/attention/test_mxfp4_dsv4_decode.py` (17 kernel cases: C0/C4/C128 × flash/profiling geometries, graph replay + length clamping, invalid contracts, scheduler-refresh regressions) + `test_mxfp4_k_cache.py` (5 codec cases) + `test_dsv4_hisparse_pool.py` (3 pool-guard cases) + `test_dsv4_compress_write_pad.py` (3 restored cases); benchmark in `test/registered/kernels/benchmark/attention/bench_mxfp4_dsv4_decode.py` - **E2E**: `test/registered/kernels/e2e/test_mxfp4_dsv4_e2e.py` — 8-GPU H200 server + gsm8k, exercising pool configurator → compressor → CUDA-graph capture (CI stage extra-b) - Removed the first-generation standalone JIT kernel (`mxfp4_decode.cuh` + `_forward_mxfp4_decode`) superseded by the FlashMLA port ## Validation - Kernel SASS was byte-identical to the AOT (cmake) build of the same sources at port time; the subsequent refactors (V32 path removal, aligned 128-bit row loads) are instruction-level changes covered by the same unit suite - Unit: 28/28 kernel + codec + pool-guard tests pass; 8-GPU e2e integration test passes locally - End-to-end on 8×H20 (TP8, `DeepSeek-V4-Flash`), FP8 (`--kv-cache-dtype fp8_e4m3`) vs MXFP4 (`--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4`), identical server flags otherwise: | Metric | FP8 baseline | MXFP4 (this PR) | Δ | |---|---|---|---| | Decode throughput (server-side, 4k×256, conc 32) | 1013 tok/s | 980 tok/s | −3.3% | | GSM8K (1319, greedy, max_tokens 1024; truncated samples excluded) | 96.89% | 96.89% | 0.0pt | | AIME25 (480 = 30×16, greedy, max_tokens 32768; 73 truncated samples excluded from both) | 99.57% pass@1 | 99.78% pass@1 | +0.2pt | ## Known gaps / follow-ups - Small-batch (b≤8) kernel-level latency vs FP8: +8–12% (kernel-inherent scheduling granularity; end-to-end TPOT already within 5%) - Prefill-path throughput: −6.8% vs FP8 (out of scope for this PR; triton-based MXFP4 prefill) - Deep-context + high-concurrency decode (64 concurrent, 8k ctx, no rolling prefill): decode TPOT +17% vs FP8 (33.1 vs 28.3 ms); end-to-end 8k→8k throughput on par (1677 vs 1652 tok/s). One full AIME25 eval run showed −44% throughput vs FP8, but four controlled reruns (5-question replica, rolling short-prompt, 8k×8k fixed) could not reproduce it and measured MXFP4 at parity or faster — attributed to environment, not the kernel. - sm_90a-only (Hopper); no SM100 support (non-SM90 + fp4_e2m1 is rejected at startup) - HiSparse (sparse KV cache) not supported with MXFP4 — rejected at startup ## References - PR #31269 — NVFP4 KV cache for DSV4 in sgl-kernel (reference implementation for the FlashMLA three-stage decode) - FlashMLA upstream @ 05e26647 — scheduler/combine kernels and kerutils - This PR takes the jit_kernel approach (JIT CUDA via TVM-FFI) instead of AOT CUTLASS in sgl-kernel, keeping the PR self-contained --- **Status**: Ready for review — kernel + integration complete, end-to-end validated on 8×H20. --- ### CI States Latest PR Test (Base): :x: [Run #31670942627](https://github.com/sgl-project/sglang/actions/runs/31670942627) Latest PR Test (Extra): :x: [Run #31670942449](https://github.com/sgl-project/sglang/actions/runs/31670942449) ",
"url": "https://github.com/sgl-project/sglang/pull/32741",
"createdAt": "2026-07-29T07:39:49Z",
"updatedAt": "2026-08-13T16:39:47Z",
"timestamp": "2026-08-13T16:39:47Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"quant",
"deepseek",
"jit-kernel"
],
"author": "TobyMint",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32746",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix][AMD] MoRI EP: drop record_stream in TBO dispatch/combine (HSA out-of-resources)",
"text": "## Motivation MoRI EP prefill with `--enable-two-batch-overlap` on MI355X aborts with `HSA_STATUS_ERROR_OUT_OF_RESOURCES`. The traceback blames whichever kernel needed scratch at that moment (for us, aiter `per_1x32_mx_quant_hip`), which is a victim, not the cause. The MoRI EP TBO path calls `record_stream(comm_stream)` on every dispatch/combine tensor. That parks each block in the caching allocator's deferred-free list until a comm-stream event retires, and with `async_finish` the compute stream never blocks on comm — so the deferred frees accumulate across layers. `reserved` grows far past the live set until allocations fail, the allocator dumps its cache via `release_cached_blocks()`, and eventually ROCr cannot find scratch and aborts the queue. On one DeepSeek-V4 rank (TP8/EP8/DP8, TBO on) a `hipMalloc`/`hipFree` replay showed **283.89 GiB of 288 GiB** at the abort. The non-EP DP TBO path already fixed exactly this — see `_TBO_PERSIST_BUF` in `layers/dp_attention.py`: > per-layer `torch.empty` churn ... kept alive by the comm-stream record_stream ballooned `reserved` to ~270GB and tripped HSA_STATUS_ERROR_OUT_OF_RESOURCES ... even though the live (allocated) working set was only ~10GB. The EP path never got the equivalent treatment. ## Modifications `python/sglang/srt/layers/moe/token_dispatcher/moriep.py` only: - Remove the four `record_stream(comm_stream)` calls in `_MoriEPDispatcherImplNormal` (`_dispatch_core` inputs/outputs, `_combine_core` inputs/output). - Add `_CROSS_STREAM_SAFETY_NOTE` documenting the failure mode and the safety argument. Unlike the DP path, `record_stream` is *removable* here rather than needing a keep-alive replacement: `dispatch_b`/`combine_b` launch on the comm stream and then run `compute_stream.wait_event(done_event)` in the same call, with no TBO yield between, so existing event ordering already covers both directions — inputs stay referenced by that frame until after the wait, and outputs are gated by the `previous_event` / `wait_stream(compute_stream)` before the next comm launch. Scope: only the dual-stream branch, which exists solely under TBO (`enable_dual_stream = is_tbo_enabled()`). Non-TBO MoRI and `_MoriEPDispatcherImplLowLatency` are unaffected. ## Accuracy Tests Not run — no numerics change, only when the allocator may recycle a block. Would appreciate a reviewer with MoRI EP CI access running the TBO accuracy job; I can run gsm8k on 2×MI355X if preferred. ## Speed Tests and Profiling Not benchmarked. Expected neutral to slightly positive — `record_stream` is off the critical path, and removing it should drop the `release_cached_blocks()` stalls (two within 40 s in the trace above). ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). (black, isort, `ruff --select=F401,F821,UP037`, codespell clean; full-ruleset ruff findings unchanged from base.) - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). (Needs a multi-GPU MoRI EP TBO prefill to surface; suggestions for a cheaper check welcome.) - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No user-facing change; rationale is in-code.) - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). (See above.) - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31560569469](https://github.com/sgl-project/sglang/actions/runs/31560569469) Latest PR Test (Extra): :x: [Run #31560569402](https://github.com/sgl-project/sglang/actions/runs/31560569402) ",
"url": "https://github.com/sgl-project/sglang/pull/32746",
"createdAt": "2026-07-29T08:05:49Z",
"updatedAt": "2026-08-13T02:11:41Z",
"timestamp": "2026-08-13T02:11:41Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"amd",
"run-ci"
],
"author": "TianDi101",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32754",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Enable gfx1250 Support",
"text": "## Motivation AMD gfx1250 is a new GPU architecture that differs substantially from the existing gfx95x and gfx942 targets SGLang already supports. Several components require gfx1250-specific paths due to architectural differences: - **No MFMA/WMMA FP4 scale instructions** (`V_WMMA_SCALE_F32_32X16X128_F4`): the existing CK/ASM `fused_moe` and fp4 dense GEMM paths produce garbage or crash. - **Triton `tl.dot(fp8, fp8)` miscompiles** for contraction dim ≥ 128 on gfx1250 (verified correct at K=64, broken at K≥128), requiring bf16 upcasting in the MLA attention Triton kernels. - **`llvm.amdgcn.raw.buffer.*` buffer instructions are absent** on RDNA4 (gfx12): the `quick_all_reduce` kernel would fail to compile. - **aiter CK-backed kernels (rope, QK RMSNorm, etc.) do not build** against the gfx1250 ROCm 7.15 image's composable_kernel. - **TileLang CK-backed buffer addressing does not compile** on gfx1250, blocking the Sinkhorn/MHC path. - **`num_stages > 1` Triton pipelines miscompile** on gfx1250, causing NaN in block-FP8 matmuls. - **ROCm SDK is pip-installed** (not at `/opt/rocm`) on this new image, requiring build and runtime path fixups for AITER, TileLang, MORI, and SGLang itself. This PR brings up gfx1250 support for DSv4, DSR1, and GPT-OSS-120B models with MXFP4/FP8 quantization. ## Modifications ### Infrastructure & Docker (`docker/rocm.Dockerfile`, `scripts/ci/amd/amd_ci_install_dependency.sh`, `python/pyproject_other.toml`) - Added a new `gfx1250-rocm7_15` Docker build stage starting from `ubuntu:24.04` with pip-installed ROCm 7.15 SDK, PyTorch 2.11, and Triton 3.7. Includes path and symlink workarounds for a pip-based (non-`/opt/rocm`) ROCm SDK. - Added `GPU_ARCH_LIST_ARG` build-arg and made `PYTORCH_ROCM_ARCH` configurable; per-arch `TRITON_COMMIT_DEFAULT` env vars. - Added gfx1250-specific AITER build path (`ENABLE_CK=0`, pip `--no-build-isolation`) and MORI build path using `CMAKE_PREFIX_PATH` for the ROCm SDK. - Added gfx1250-aware TileLang `ROCM_PATH` override during CMake build. - Updated Triton rebuild logic to preserve the installed version string (workaround for ROCm 7.15 / [rocm-systems#7643](https://github.com/ROCm/rocm-systems/issues/7643)). - Renamed `SETUPTOOLS_SCM_PRETEND_VERSION` → `SGLANG_VERSION` to avoid leaking SGLang's nightly version into AITER's version. - Added `compressed-tensors==0.16.0` pin to `runtime_common` (supports torch ≥ 2.10) and dropped the old per-`srt_hip` `==0.15.0` pin. - CI install script: ROCm-version-aware AITER rebuild (`pip --no-build-isolation` for ROCm ≥ 7.15), MORI `CMAKE_PREFIX_PATH` fix, and MORI commit extraction validation. ### Core utility (`python/sglang/srt/utils/common.py`) - Added `is_gfx1250_supported()` (cached) to detect AMD RDNA4 by `gcnArchName`. - Extended `mxfp_supported()` to include `\"gfx1250\"`. ### sgl-kernel (`sgl-kernel/setup_rocm.py`, `sgl-kernel/csrc/allreduce/quick_all_reduce_base.h`) - Added `gfx1250` to the allowed arch list in `setup_rocm.py`; gfx1250 shares the `E4M3` FP8 type and the 128KB dynamic smem budget with gfx950. - Guarded `llvm.amdgcn.raw.buffer.load/store.v4i32` declarations behind `#if !defined(__gfx1250__)`, providing stub no-ops for RDNA4 where those intrinsics don't exist. ### Attention kernels (`python/sglang/kernels/ops/attention/`) - **`decode_attention.py`**: Do not downcast `q` to `fp8` before the QK dot; instead upcast K to `q`'s dtype (`bf16`) for the dot product, avoiding the gfx1250 fp8 dot bug at K≥128. Also keep softmax weights `p` in fp32 for the P·V accumulation (recovers GSM8K accuracy ~0.82 → ~0.92 on gfx1250 R1). - **`extend_attention.py`**: Same fp8 upcast fix and fp32 P·V fix for the prefill extend/unified kernels. - **`dsv4/unified_kv_kernels/paged_decode.py`**: On gfx1250, dispatch DSv4 sparse decode attention to `aiter.ops.triton.attention.pa_decode_sparse.pa_decode_sparse` (the triton path) instead of the custom Triton implementation. - **`dsv4/unified_kv_kernels/paged_prefill.py`**: Disable the OPUS (gfx950 ASM) prefill path on gfx1250. ### MoE gate JIT kernel (`python/sglang/kernels/jit/csrc/moe/moe_fused_gate.cuh`) - Widened `__shfl_down_sync` mask from `0xFFFFFFFF` to a 64-bit `0xFFFFFFFFFFFFFFFFULL` on gfx1250 (which requires a 64-bit sync mask), while staying 32-bit on all other arches. ### Layernorm (`python/sglang/kernels/ops/layernorm/mhc.py`, `python/sglang/srt/layers/layernorm.py`) - On gfx1250, use aiter's Triton RMSNorm and `rmsnorm2d_fwd_with_add` instead of the CK-backed aiter equivalents (which do not build). - Added pure-torch (`_hc_split_sinkhorn_torch`) and Triton (`_hc_split_sinkhorn_triton`) implementations of the Sinkhorn MHC kernel, dispatched on gfx1250 since TileLang's CK-backed addressing does not compile there. ### Quantization – FP8 (`python/sglang/kernels/ops/quantization/fp8_kernel.py`, `python/sglang/srt/layers/quantization/fp8.py`, `python/sglang/srt/layers/quantization/fp8_utils.py`) - **`fp8_kernel.py`**: Added `_w8a8_block_fp8_matmul_gfx1250`, a gfx1250-specific block-FP8 matmul that: (1) upcasts fp8 operands to bf16 before `tl.dot`, (2) uses only `num_stages=1` (no SW pipelining), and (3) avoids modulo-wrap index tricks that miscompile on gfx1250. Selected automatically when `_is_gfx1250` is set. - **`fp8_utils.py`**: Force `_use_aiter = False` on gfx1250 (CK/MFMA-dependent aiter quant/GEMM kernels cannot run), directing gfx1250 to the pure-Triton block-FP8 path. Extended `triton_w8a8_block_fp8_linear` to accept pre-quantized `(input, input_scale)` from gfx1250's fused quant ops (skipping re-quantization). - **`fp8.py`**: On gfx1250, use `moe_shuffle_scale` (n32k4 layout) instead of the gfx950 `shuffle_scale` (e8m0 layout) for FP8 MoE weight scales. Respect the `AITER_FORCE_A8W4` flag to select the triton a8w4 kernel shuffle layout. ### Quantization – MXFP4 Quark schemes (`python/sglang/srt/layers/quantization/quark/schemes/`) - **`quark_w4a4_mxfp4.py`**: On gfx1250 with `AITER_FORCE_A8W4=1`, dequantize FP4 weights to bf16 at load time (since no dense fp4 GEMM is available) and run a plain bf16 linear forward. Includes a pure-Python MXFP4 e2m1 → bf16 dequantization LUT. - **`quark_w4a4_mxfp4_moe.py`**: On gfx1250, use `moe_shuffle_scale` (n32k4 layout) for W4A4 MoE weight scales (instead of `e8m0_shuffle`). Shuffle weights with the (16,16) aiter layout when `AITER_FORCE_A8W4` is set. - **`quark_w4a8_mxfp4_moe.py`**: Added a dedicated gfx1250 W4A8 MoE path (`_use_gfx1250_w4a8`): bypasses the aiter CK/ASM fused_moe entirely; uses aiter's triton `moe_gemm_a8w4` kernel with a contraction-major, gate/up-interleaved, unshuffled weight layout. Weight preparation (`_process_weights_gfx1250`) and forward (`aiter_w4a8_gfx1250_forward` in new file `aiter_mxfp4_w4a8_moe.py`) are implemented separately from the gfx950 path. ### New file: `python/sglang/srt/layers/moe/fused_moe_triton/aiter_mxfp4_w4a8_moe.py` - Implements MXFP4 W4A8 GPT-OSS MoE forward for gfx1250 via `moe_gemm_a8w4`: routing, manual activation gather (in-kernel TDM gather disabled on gfx1250), static-FP8 downcast, two-GEMM (gate_up + down) execution with fused SwiGLU, and scatter back to token order. ### RoPE (`python/sglang/srt/layers/rotary_embedding/factory.py`) - Fall back from aiter's CK rope kernels to SGLang's native rope on gfx1250 (aiter rope does not build on this arch). ### Communicator / fused quant (`python/sglang/srt/layers/communicator.py`) - On gfx1250, use `fused_rms_fp8_group_quant` (triton, group_size=hidden_size = per-token) instead of aiter's CK `rmsnorm_quant` / `add_rmsnorm_quant` kernels. ### Sampler (`python/sglang/srt/layers/sampler.py`) - Disable aiter's greedy sample kernel on gfx1250 (not supported), falling back to `torch.argmax`. ### Capture mode (`python/sglang/srt/model_executor/runner_utils/capture_mode.py`) - Disable `torch.compile` during CUDA graph capture on gfx1250 (compilation + capture together triggers issues on this arch). ### MLA attention (`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`, new `triton_qk_rmsnorm.py`) - On gfx1250 (`AITER_FORCE_A8W4=1`), substitute the aiter `fused_qk_rmsnorm` (CK-backed, doesn't build) with a self-contained Triton RMSNorm or pure-torch fallback from the new `triton_qk_rmsnorm.py`. - Fixed the RoPE skip condition to check membership in `FORWARD_ABSORB_CORE_ATTENTION_BACKENDS` (not a string equality check), preventing an incorrect empty-K bug on non-aiter backends. ### DSv4 decoder layer (`python/sglang/srt/models/deepseek_v4.py`, `deepseek_common/utils.py`, `amd/deepseek_v4_fused_mhc.py`) - Extended all `_is_gfx95_supported` guards that gate MXFP4 fused quant, MQA q/k-norm, post-attention layernorm fused rmsnorm, and the fused MHC post/pre path to also include `_is_gfx1250_supported`. - On gfx1250, the fused MHC path is enabled by `SGLANG_OPT_FUSE_MHC_POST_PRE` alone (no TileLang dependency), and the aiter Triton `try_fused_hc_post_pre` is used at all token batch sizes (no M-threshold cap, since there is no TileLang fallback). - gfx1250 does not skip the post-layernorm step after the Triton MHC kernel (since `norm_fused=False` for the aiter path). ### C4 attention indexer (`python/sglang/srt/layers/attention/dsv4/indexer.py`) - Added `_use_torch_fn` guard so the `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH` env var correctly prevents the 1D c4sl unsqueeze on gfx1250. ### Server args (`python/sglang/srt/server_args.py`) - Track `_mem_fraction_static_set_by_user`: when the user explicitly sets `--mem-fraction-static`, skip the automatic 0.85 scaling for `aiter` backend with long contexts (instead, emit a warning so the user knows what is happening). ## Accuracy Tests Passed E2E accuracy tests of DSv4, DSR1, and GPT-OSS-120B. GPT-OSS-120B > AITER_FORCE_A8W4=1 ENABLE_CK=0 SGLANG_USE_AITER=1 SGLANG_USE_AITER_MOE_GU_ITLV=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 python3 -m sglang.launch_server --model-path /dockerx/model/gpt-oss-120b-w-mxfp4-a-fp8/ --trust-remote-code --tp 1 --prefill-attention-backend triton --decode-attention-backend aiter --max-running-requests 1024 --mem-fraction-static 0.9 --disable-radix-cache --page-size 64 DSR1 > ENABLE_CK=0 SGLANG_USE_AITER=1 AITER_FORCE_A8W4=1 AITER_GROUPED_FORCE_SPLIT_K1=1 SGLANG_MOE_SHUFFLE_GFX1250=1 ROCM_QUICK_REDUCE_QUANTIZATION=NONE SGLANG_AITER_FP8_PREFILL_ATTN=0 SGLANG_AITER_MLA_PERSIST=0 SGLANG_INT4_WEIGHT=0 SGLANG_MOE_PADDING=1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 SGLANG_USE_ROCM700A=1 AITER_GROUPED_CONTIGUOUS_TOKEN_THRESHOLD=16 python3 -m sglang.launch_server --model-path /dockerx/model/DeepSeek-R1-0528-MXFP4 --tensor-parallel-size 1 --trust-remote-code --host 0.0.0.0 --port 8000 --mem-fraction-static 0.90 --chunked-prefill-size 16384 --attention-backend triton --max-running-requests 128 --kv-cache-dtype auto --page-size 64 DSv4-Flash > SGLANG_DEFAULT_THINKING=1 AITER_GROUPED_CONTIGUOUS_TOKEN_THRESHOLD=16 SGLANG_DSV4_REASONING_EFFORT=max SGLANG_USE_ROCM700A=0 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton AITER_BF16_FP8_MOE_BOUND=0 AITER_FORCE_A8W4=1 SGLANG_USE_AITER_MOE_GU_ITLV=0 SGLANG_OPT_FUSE_MHC_POST_PRE=0 ENABLE_CK=0 SGLANG_USE_AITER=1 AITER_GROUPED_FORCE_SPLIT_K1=1 python3 -m sglang.launch_server --model-path /dockerx/model/DeepSeek-V4-Flash --trust-remote-code --tp 1 --attention-backend dsv4 --page-size 256 --mem-fraction-static 0.60 --swa-full-tokens-ratio 0.15 --disable-shared-experts-fusion --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --chunked-prefill-size 8192 --cuda-graph-max-bs 256 --max-running-requests 256 --disable-radix-cache --kv-cache-dtype fp8_e4m3 ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31350355796](https://github.com/sgl-project/sglang/actions/runs/31350355796) Latest PR Test (Extra): :x: [Run #31350355703](https://github.com/sgl-project/sglang/actions/runs/31350355703) ",
"url": "https://github.com/sgl-project/sglang/pull/32754",
"createdAt": "2026-07-29T08:50:52Z",
"updatedAt": "2026-08-13T16:04:54Z",
"timestamp": "2026-08-13T16:04:54Z",
"metrics": {
"reactions": 1,
"comments": 12
},
"labels": [
"high priority",
"amd",
"dependencies",
"deepseek",
"sgl-kernel",
"run-ci",
"jit-kernel",
"bypass-fastfail"
],
"author": "akao-amd",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32755",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] Occupancy tuning for DSA indexer fp8-quant Q kernel",
"text": "Optimize fused_q_indexer_rope_hadamard_quant scheduling: 8 warps/block (256 threads) + lane-0-only weights_out write. The math path (RoPE / 128-pt Hadamard / fp8 quant) is unchanged, so output is bitwise-identical for both the V4 (kRopeFirst=false, kHadamard=true) and V3.2 (kRopeFirst=true, kHadamard=false) template configs. Block size / cap default to 8 / 16 (tuned for sm_100 / B200), overridable via -DQ_BLOCK_SIZE / -DQ_MIN_BLOCKS_PER_SM. ## Motivation `fused_q_indexer_rope_hadamard_quant` (the DSA C4 indexer fp8-quant Q kernel, shared by the V4 rope-hadamard path and the V3.2/GLM rope-first path) is latency-bound with low occupancy: the baseline launches 4-warp blocks (128 threads, `__launch_bounds__(128,16)`), which run the schedulers at only ~38% achieved warp occupancy. NCU shows the top stall is `long_scoreboard` (warps waiting on global loads) with compute pipelines under 35% utilized — so the lever is scheduling (more warps in flight to hide the load latency), not the math. ## Modifications Pure launch-config changes to the quant kernel; the math path (RoPE, 128-pt Hadamard, dynamic fp8-e4m3 quant, weight scaling) is **untouched**: 1. **8 warps/block (256 threads) + `__launch_bounds__(256,16)`** — the kernel gains `kNumWarps` / `kMinBlocksPerSM` template params (defaulting to 8 / 16), doubling the warps resident per block so more work is in flight to cover the `long_scoreboard` stall. Occupancy goes ~38% → ~86%. Overridable at compile time via `-DQ_BLOCK_SIZE` / `-DQ_MIN_BLOCKS_PER_SM`. 2. **lane-0-only `weights_out` write** — the scale/weight are warp-uniform, so the other 31 same-address stores were pure waste. Each (token, head) row is a fully self-contained warp work-item (no cross-row state), so which SM / warp / order runs it does not change its 128 output bits. Output is therefore **bitwise-identical** to the previous kernel for both template configs — V4 (`kRopeFirst=false, kHadamard=true`) and V3.2/GLM (`kRopeFirst=true, kHadamard=false`). > Note: an earlier revision of this PR also added a single-wave grid cap + > persistent grid-stride loop. Ablation (below) showed the CTA change carries > essentially all of the speedup, while the persistent path was perf-neutral at > mid batch and ~3% *slower* at B≥2048 (extra loop bookkeeping once occupancy is > saturated). It has been dropped; this PR is now just the two changes above. ## Accuracy Tests `test/registered/kernels/ops/attention/test_dsv4_indexer_quant.py` — checks both template paths against a torch reference (dequantized q within fp8-e4m3 precision; `weights_out` to atol/rtol 1e-3), plus a strided-weight test. Batch sizes span small (latency-bound) through large (occupancy-saturated). ``` 19 passed ``` (V4: B in {1,8,64,256,512,2048} x {int32,int64}; V3.2: same batches; strided weight.) Additionally verified byte-exact against the pre-change kernel across B in {1,8,64,128,256,512,1024,2048,4096,8192,16384} for both configs: q_fp8 0 bytes differ, weights_out 0 elements differ, all finite. ## Speed Tests and Profiling NCU pure-kernel time on B200 (sm_100), interleaved baseline/candidate to cancel clock drift: | B | baseline (ns) | this PR (ns) | ratio | |-------:|--------------:|-------------:|:-----:| | 1 | 3104 | 3216 | 1.04 | | 8 | 3440 | 3360 | 0.98 | | 64 | 3968 | 3792 | 0.96 | | 128 | 5184 | 4400 | 0.85 | | 256 | 7392 | 6544 | 0.89 | | 512 | 11616 | 10176 | 0.88 | | 1024 | 20208 | 17312 | 0.86 | | 2048 | 37552 | 30784 | 0.82 | | 4096 | 71952 | 57344 | 0.80 | | 8192 | 141072 | 110512 | 0.78 | | 16384 | 279296 | 216352 | 0.78 | Small batch (≤64) is launch-bound and stays at parity (the grid can't fill the SMs); the benefit appears once work fills the GPU and grows to ~22% at large batch. Wall-clock cross-check (CUDA-event HOT) matches: B=256 0.95, B=2048 0.81, B=16384 0.79 — not a profiler artifact. **CTA-size sweep** (why 256 threads), ncu duration / achieved occupancy / registers-per-thread: | config | B=256 dur / occ / regs | B=2048 dur / occ / regs | |------------|------------------------|-------------------------| | 4w / 128t | 8704ns / 66% / 21 | 38896ns / 53% / 21 | | **8w / 256t** | **7728ns / 79% / 21** | **32320ns / 87% / 21** | | 12w / 384t | 7936ns / 77% / 21 | 32912ns / 81% / 21 | | 16w / 512t | 7792ns / 80% / 21 | 32880ns / 87% / 21 | 256 threads is the knee: registers stay at 21 (not register-bound; the limiter is the resident-block cap), occupancy is already saturated, and 384/512 give no further speedup. ## Checklist - [✓] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [✓] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [✓] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [✓] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31592945179](https://github.com/sgl-project/sglang/actions/runs/31592945179) Latest PR Test (Extra): :white_check_mark: [Run #31592944948](https://github.com/sgl-project/sglang/actions/runs/31592944948) ",
"url": "https://github.com/sgl-project/sglang/pull/32755",
"createdAt": "2026-07-29T08:59:02Z",
"updatedAt": "2026-08-13T08:18:06Z",
"timestamp": "2026-08-13T08:18:06Z",
"metrics": {
"reactions": 0,
"comments": 31
},
"labels": [
"quant",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "Rainchar9119",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32798",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "DFLASH support added for XPU",
"text": "This PR add support for XPU with DFLASH algo in speculative decoding. JH...! --- ### CI States Latest PR Test (Base): :x: [Run #30608529938](https://github.com/sgl-project/sglang/actions/runs/30608529938) Latest PR Test (Extra): :x: [Run #30608529765](https://github.com/sgl-project/sglang/actions/runs/30608529765) ",
"url": "https://github.com/sgl-project/sglang/pull/32798",
"createdAt": "2026-07-29T18:09:28Z",
"updatedAt": "2026-08-13T06:24:31Z",
"timestamp": "2026-08-13T06:24:31Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"speculative-decoding"
],
"author": "ANSHUMAN87",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32882",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Accept int64 top-k IDs in FlashInfer routed MoE packer",
"text": "[by Codex] ## Summary - accept both `torch.int32` and `torch.int64` router IDs in `PackTopkIds` - keep the existing in-kernel conversion to int32, avoiding a temporary cast allocation during CUDA-graph capture - add focused coverage for both input dtypes, multiple shapes, exact reference parity, CUDA-graph capture/replay, and invalid input This fixes a Qwen3 NVFP4 startup crash on Blackwell when torch compile uses the native top-k path: ```text AssertionError: topk_ids must be int32, got torch.int64 ``` ## Regression history This is a recent regression introduced by #25702. Before #25702, `_pack_topk_for_flashinfer_routed` explicitly converted `topk_ids` with `.to(torch.int32)`. The PR replaced that helper with `PackTopkIds.execute`, whose Python wrapper asserted that IDs were already int32. The Triton kernel itself has always converted loaded IDs with `ids.to(tl.int32)`, so the wrapper rejected an input the kernel could process. The strict packer originally came from #23882, where its initial caller supplied int32 IDs. The incompatibility became a regression only when #25702 reused it for the general FlashInfer routed-MoE paths. ## Before and after Reproduced with `lmsysorg/sglang:dev-cu13` on NVIDIA B200 (`torch==2.11.0+cu130`, CUDA 13.0): Before: ```text int32_eager=PASS int64_eager=FAIL: AssertionError: topk_ids must be int32, got torch.int64 ``` After: ```text int32_eager=PASS int64_eager=PASS int64_cuda_graph=PASS ``` The patched output is bit-for-bit identical to `PackTopkIds.vanilla`. ## Why testing did not catch it PR #25702 validated the routed MoE backend and reran the Blackwell model test, but that test did not enable torch compile. The normal CUDA top-k implementation returns int32 IDs, so it satisfied the new assertion. With compile enabled for a batch-size-1 graph, `TopK.enter_torch_compile` switches to `forward_native`; native `torch.topk` returns int64 indices. There was no direct unit test defining `PackTopkIds`' accepted input dtypes, so this contract mismatch escaped both the end-to-end and unit suites. - Covered: routed FlashInfer MoE with the normal CUDA top-k path (int32) - Missed: routed FlashInfer MoE + torch compile/native top-k (int64) - Remediation in this PR: direct kernel-contract tests for int32 and int64, including int64 CUDA-graph capture/replay ## Validation ```text python3 -m pytest -q test_pack_topk_ids.py ......... [100%] 9 passed in 14.55s ``` The focused reproducer also passed eager and CUDA-graph execution on B200. ## Checklist - [x] Added regression tests - [x] Verified the failure before the change on B200 - [x] Verified eager and CUDA-graph behavior after the change on B200 - [x] Ran repository pre-commit checks cc @trevor-m for visibility on the #25702 regression analysis. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31579462931](https://github.com/sgl-project/sglang/actions/runs/31579462931) Latest PR Test (Extra): :x: [Run #31579462668](https://github.com/sgl-project/sglang/actions/runs/31579462668) ",
"url": "https://github.com/sgl-project/sglang/pull/32882",
"createdAt": "2026-07-30T04:01:03Z",
"updatedAt": "2026-08-13T06:01:11Z",
"timestamp": "2026-08-13T06:01:11Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "nvpohanh",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32902",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Fix Llama 4 FA3 local attention with paged KV cache",
"text": "[by Codex] ## Motivation Llama 4 Scout with FlashAttention 3 crashes during warmup on H100 when the KV cache uses pages larger than one token. The same workload succeeds with `--page-size 1`. This is a long-standing paged-local-attention bug rather than a new 0.5.7 regression: the explicit page-size-16 reproducer fails on 0.5.6.post2, 0.5.7, and current `main`. Related to #7936. ## Reproduction Environment: - 8x NVIDIA H100 80GB HBM3 - Llama 4 Scout 17B-16E Instruct - Tensor parallel size 8 - FlashAttention 3 - Dummy weights Run: ```bash export CUDA_LAUNCH_BLOCKING=1 python -m sglang.bench_one_batch \\ --model-path \\ --disable-radix-cache \\ --context-length 2048 \\ --attention-backend fa3 \\ --load-format dummy \\ --batch-size 128 \\ --tp-size 8 \\ --input-len 1000 \\ --output-len 1000 \\ --page-size 16 ``` ### Before The run reaches warmup, then all TP ranks fail in the FA3 forward launch: ```text Warmup ... CUDA error (.../hopper/flash_fwd_launch_template.h:209): an illegal memory access was encountered ``` Changing only `--page-size 16` to `--page-size 1` completes warmup and the benchmark. ## Root cause There are two related issues in eager local-attention metadata construction: 1. `_maybe_init_local_attn_metadata` passes the token-granular `metadata.page_table` to `make_local_attention_virtual_batches`. The helper and FlashAttention consume page IDs. At page size 16, physical token locations are therefore interpreted as physical page IDs and can address beyond the allocated KV cache. 2. The shared helper normally reduces the requested attention chunk to a page-aligned value based on the current sequence length. For this 1,000-token prompt, that turns Llama 4's model-defined 8,192-token boundary into 992 and creates a spurious final 8-token virtual batch. Page size 1 masks both problems because token and page indices are identical, and the chunk reduction does not leave a page-sized remainder. ## Changes - Convert eager FlashAttention token locations to a page-granular block table before creating local virtual batches: ```python page_table = page_table[:, :: self.page_size] // self.page_size ``` - Let FlashAttention local-metadata construction explicitly preserve the model-defined attention chunk in eager execution, CUDA graph capture, and CUDA graph replay. - Keep the shared helper's existing chunk-normalization behavior as the default for callers that do not opt in. - Add regression coverage for token-to-page conversion, Llama 4's 8,192-token boundary, and the unchanged default helper behavior. ## Why other attention backends are unaffected - The token-to-page conversion is inside `FlashAttentionBackend`'s local-attention metadata path. Other backend classes build their own metadata and do not execute this code. - `make_local_attention_virtual_batches` is shared with the XPU backend, so the new `preserve_attn_chunk_size` option defaults to `False`. XPU does not pass the option and therefore retains the previous normalization and output layout. A regression test covers this default path. - The new option is enabled only at the three `FlashAttentionBackend` call sites. No attention kernel, global KV-cache layout, or metadata builder for Triton, FlashInfer, TensorRT-LLM, or other backends is changed. - Within FlashAttention, models without local attention return before this metadata path, and page size 1 skips the token-to-page conversion. ## After The exact page-size-16 command above completes both warmup and the full 1,000-token decode benchmark: ```text Warmup ... Prefill. latency: 3.44482 s, throughput: 37157.24 token/s Decode. median latency: 0.01917 s, median throughput: 6675.84 token/s Total. latency: 4.091 s, throughput: 32285.56 token/s Benchmark ... Prefill. latency: 1.79242 s, throughput: 71411.78 token/s Decode. median latency: 0.01957 s, median throughput: 6539.82 token/s Total. latency: 21.433 s, throughput: 11943.97 token/s ``` ## Validation - Reproduced the FA3 illegal memory access on: - SGLang 0.5.6.post2, page size 16 - SGLang 0.5.7, page size 16 - Current `main`, page size 16 - Verified the page-size-1 control passes. - Verified patched current source passes with page size 16 and output lengths 128 and 1,000. - Focused regression tests passed on the original fix; the review follow-up's opt-in and compatibility paths passed isolated validation against the actual helper source. - Full repository validation after the review follow-up: ```bash BLACK_NUM_WORKERS=1 SKIP=no-commit-to-branch \\ pre-commit run --all-files --show-diff-on-failure ``` All hooks passed. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31579432381](https://github.com/sgl-project/sglang/actions/runs/31579432381) Latest PR Test (Extra): :x: [Run #31579432056](https://github.com/sgl-project/sglang/actions/runs/31579432056) ",
"url": "https://github.com/sgl-project/sglang/pull/32902",
"createdAt": "2026-07-30T08:01:07Z",
"updatedAt": "2026-08-13T06:49:42Z",
"timestamp": "2026-08-13T06:49:42Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "nvpohanh",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32926",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Don't request the unused softmax LSE in the AITER diffusion backend",
"text": "## Motivation `AITerImpl.forward` requests the softmax LSE and then discards it on the same line via `output, _ =`. Minor cleanup: ask for the output only. Ring attention is the only consumer of the LSE, and it is already restricted to the FA and SageAttention backends, so nothing reads this value today. ## Modifications `return_lse=True` -> `False` in the BF16 path. `aiter.flash_attn_func` returns a bare tensor when `return_lse=False` (and a tuple otherwise), so the unpacking is adjusted to match. ## Accuracy Tests Output is bitwise identical before/after on MI350X (gfx950), Wan2.2 shape `[1, 75600, 5, 128]` bf16 non-causal: `torch.equal(out, ref) == True`, max abs diff `0.0`. ## Benchmarking and Profiling No measurable perf change ## Checklist - [x] Format the code with pre-commit - [x] Verified output is unchanged --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31719178932](https://github.com/sgl-project/sglang/actions/runs/31719178932) Latest PR Test (Extra): :x: [Run #31719178985](https://github.com/sgl-project/sglang/actions/runs/31719178985) ",
"url": "https://github.com/sgl-project/sglang/pull/32926",
"createdAt": "2026-07-30T12:08:11Z",
"updatedAt": "2026-08-13T16:36:02Z",
"timestamp": "2026-08-13T16:36:02Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"diffusion"
],
"author": "sushildubey171",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32941",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[minimax m3][npu]Adaptation of Minimax M3(w8a8) for NPU platforms [1/2]",
"text": " ## Motivation Adaptation of Minimax M3 for NPU platforms ## Modifications **1、Define the complete attention implementation for MiniMax-M3 on the NPU platform, with deep optimization based on NPU‑specific features. 2、Implement speculative inference adaptation for MiniMax‑M3 based on the Eagle3 draft model. 3、Fix the W8A8 weight loading method for MiniMax‑M3. 4、Fix the memory allocation issue in CUDA graph where the number of predicted tokens is not considered when CUDA graph is enabled.** 5、Complete implementation of fuseep based on SwiGLU‑OAI (work in progress). ## Accuracy Tests Accuracy based on 1,000 data samples from GSM8K： ## Speed Tests and Profiling Results for 16K input with 50 concurrent requests ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31661553050](https://github.com/sgl-project/sglang/actions/runs/31661553050) Latest PR Test (Extra): :x: [Run #31661553025](https://github.com/sgl-project/sglang/actions/runs/31661553025) ",
"url": "https://github.com/sgl-project/sglang/pull/32941",
"createdAt": "2026-07-30T14:41:08Z",
"updatedAt": "2026-08-13T03:23:14Z",
"timestamp": "2026-08-13T03:23:14Z",
"metrics": {
"reactions": 0,
"comments": 16
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"lora",
"Multi-modal",
"deepseek",
"speculative-decoding",
"sgl-kernel",
"blackwell",
"npu",
"run-ci",
"diffusion",
"apple-silicon",
"jit-kernel"
],
"author": "vstone-w",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32944",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MoE] Fuse swiglu moe up gemm epilogue",
"text": "## Motivation **What the MoE activation costs today.** The triton fused-MoE path runs three steps per layer: an up-GEMM that writes `intermediate_cache1` at the full gate+up width `N`, a standalone `silu_and_mul` that reads that buffer and writes `intermediate_cache2` at width `N/2`, and a down-GEMM. The middle step is pure data movement dressed as compute — it round-trips the entire intermediate tensor through HBM to apply an elementwise function whose inputs the up-GEMM already had sitting in registers. **Why that is worth removing at small batch.** At batch 1 a MoE layer's GEMMs degenerate into `M=4..8` slivers, so the step is bound by weight bandwidth and by the intrinsic latency floor of several hundred tiny kernels rather than by arithmetic. In that regime an extra launch plus a full round trip of the intermediate buffer, paid once per MoE layer per decode step, is a visible fraction of the step. It is also a link in the dependency chain — `GEMM1 -> act -> GEMM2` — that no amount of overlap can shorten while the activation stands between the two GEMMs. **Why it cannot simply be folded in.** The activation needs `gate[i]` and `up[i]` together, but the checkpoint stores W13 as two contiguous halves, so those two values come out of output columns `i` and `i + N/2` — different tiles of the up-GEMM. A tile holding `gate[i]` does not hold `up[i]`, and vice versa. **The fix.** Permute the rows of W13 once at load so gate and up of the same intermediate channel become adjacent columns. Each GEMM output column is an independent dot product of the input with one row of W13, so reordering rows can only reorder columns — the permute is value-neutral. After it, every `(gate[i], up[i])` pair lands in the *same* output tile as an even/odd column pair, the epilogue applies `silu(gate) * up` in-register, and only the half-width result is stored. `intermediate_cache1` and the standalone activation launch disappear, and the chain shortens to `GEMM1 -> GEMM2`. ## Modifications - `python/sglang/kernels/ops/moe/fused_moe_triton_kernels.py`: `FUSE_SWIGLU` constexpr on `fused_moe_kernel`, plus the epilogue that splits the accumulator into even/odd column pairs, applies the activation, and stores the half-width result. `invoke_fused_moe_kernel` gains a `fuse_swiglu` argument and asserts the output flavors it is incompatible with. - `python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py`: allocates the half-width buffer as the up-GEMM destination, skips the standalone activation, and threads the flag through `fused_experts` / `fused_experts_impl` / `_fused_moe_kernel_sequence`. - `python/sglang/srt/layers/moe/moe_runner/triton.py`: `TritonMoeQuantInfo.fuse_swiglu_interleaved` carries the layout fact from the weight owner to both `fused_experts` entry points. - `python/sglang/srt/layers/quantization/unquant.py`: `_maybe_interleave_w13_for_fused_swiglu` performs the load-time permute, per expert so the gather temporary stays at one expert's slice. - `python/sglang/srt/environ.py`: `SGLANG_OPT_FUSE_SWIGLU_INTERLEAVED`, default off. ### Bit parity is instruction-level, not approximate The kernel being replaced is compiled with `-use_fast_math`, so matching it means matching the instructions it emits. The epilogue uses `tl.inline_asm_elementwise` to do exactly that: `mul + ex2.approx.ftz` for `__expf`, `div.approx.ftz` for the division, and `mul.ftz.f32` for the product, with a single final rounding to bf16. The accumulator is already `compute_type` at that point, which matches the reference's bf16 store to `intermediate_cache1` followed by a reload. One detail is easy to get wrong and invisible from the call site: the reference instantiates its activation functor at `float`, so silu never lands in bf16 before the multiply. An implementation that rounds silu to bf16 first double-rounds and diverges on a large fraction of inputs. The first version of this change did exactly that, and only an element-wise bit comparison caught it. ### Why this is opt-in The permuted layout is understood by the fused epilogue and by nothing else, so every consumer that reads W13 — or the pre-activation buffer — in the checkpoint's halves layout has to be excluded at load rather than trapped later. The gate requires CUDA, the plain triton runner, standard a2a, no bias, bf16, and gated silu without `alpha` / `clamp_limit` / `swiglu_limit`, and additionally excludes: - **LoRA.** `_add_lora_gate_up_delta` adds its delta into the full-width pre-activation buffer this path eliminates, and indexes gate/up as contiguous halves of it — both assumptions break. Excluding it at load matters more than asserting at run time: permuting first and asserting later would turn `--enable-lora` into a crash after warmup instead of a clean fallback. The virtual-experts LoRA path is unaffected, since it consumes `lora_a` / `lora_b` and never the base W13. - **EPLB**, which rearranges experts by copying checkpoint-layout weights into the parameter. - Runtime weight updates, for the same reason; this one is documented on the env var rather than detected. Two further guards fail loudly rather than silently: `_fused_moe_kernel_sequence` asserts the call carries no hooks and no quantization, and `invoke_fused_moe_kernel` asserts that none of `fuse_add_to_output` / `mask_output` / `fuse_sum_all_reduce` / `lora_preserve_base` are set — those address `C` at the full width `N` and would write past the row end of the half-width buffer. Filtered (`-1`) expert blocks skip the store entirely for the same reason. Under the unfused path those rows are zero-filled at full width; here the row is only `N/2` wide, so an `N`-wide zero store would run into the next token's data. The rows are never read, because the down-GEMM CTA for such a block early-exits before loading its input. ## Accuracy Tests New: `test/registered/kernels/ops/moe/test_fused_swiglu_epilogue.py`, registered `base-b-kernel-unit` / `1-gpu-large`. Both cases assert **bitwise** equality; a tolerance would defeat their purpose, since the failure mode this change risks is a plausible-looking few-ULP drift that no `assert_close` would flag. - `test_fused_matches_unfused_bitwise` drives the real `fused_experts` entry point twice — halves layout with the standalone activation, then interleaved layout with the epilogue — and requires the final MoE output to be bit-identical. Shapes cover a bs=1 decode row and a ragged token count that forces the `BLOCK_SIZE_M` tail. It also covers the filtered-expert claim: if those unwritten rows were ever read, the comparison would fail immediately. - `test_epilogue_matches_silu_and_mul_bitwise` isolates the epilogue arithmetic against `silu_and_mul` over random values plus the saturating tails and signed zeros. This is the guard against a future \"simplification\" to `tl.sigmoid` or an early bf16 cast. All three pass on H20-3e (sm90). End-to-end on `Kimi-Linear-48B-A3B-Instruct` (256 experts, `moe_intermediate_size` 1024, bf16, TP4, H20-3e): six greedy prompts at 128 new tokens each, served once with the flag off and once with it on. The two output sets are **byte-identical** (equal md5). The fused run logs `Interleaved w13 gate/up: ...` on all four TP ranks and the baseline run logs it zero times, so the comparison is not vacuously passing on an inactive fast path. ## Speed Tests and Profiling Removed, per MoE layer per forward: | | before | after | |---|---|---| | kernel launches | up-GEMM, activation, down-GEMM | up-GEMM, down-GEMM | | intermediate traffic | `intermediate_cache1` written at width `N`, read back at width `N` | none | | dependency chain | `GEMM1 -> act -> GEMM2` | `GEMM1 -> GEMM2` | Measured on `Kimi-Linear-48B-A3B-Instruct`, TP4 on H20-3e (sm90), `bench_serving --dataset-name random --num-prompts 24 --random-input-len 512 --random-output-len 256 --max-concurrency 1`, four runs per configuration: | | mean TPOT (4 runs) | output throughput | |---|---|---| | baseline | 2.49 / 2.49 / 2.49 / 2.49 ms | 362.2 - 368.6 tok/s | | fused | **2.46 / 2.46 / 2.46 / 2.46 ms** | 364.5 - 372.4 tok/s | **-1.2% mean TPOT**, with the two sets fully separated — no overlap across four runs each. Throughput moves the same way but its run-to-run spread (~1.8%) is wider than the effect, so only the TPOT column supports a conclusion here. The win scales with how launch-bound the step is, so it is largest at small batch and shrinks toward zero once the GEMMs grow big enough to dominate. sm90 with a 27-layer model at 512/256 is a conservative setting for it; the shape this was originally built for is batch-1 decode on Blackwell with a deeper MoE stack, which is not measured here. That asymmetry is the other reason this ships opt-in rather than on by default. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31598793624](https://github.com/sgl-project/sglang/actions/runs/31598793624) Latest PR Test (Extra): :white_check_mark: [Run #31598793191](https://github.com/sgl-project/sglang/actions/runs/31598793191) ",
"url": "https://github.com/sgl-project/sglang/pull/32944",
"createdAt": "2026-07-30T15:08:08Z",
"updatedAt": "2026-08-13T15:24:53Z",
"timestamp": "2026-08-13T15:24:53Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"quant",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "yuan-luo",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32947",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "GLM5.2 DSA Indexer Fallback",
"text": "## Motivation `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1` is meant to be the escape hatch that runs the FP8 MQA-logits indexer without DeepGEMM. Today it only covers the **dsv4** indexer path (`dsv4/metadata.py`, `dsv4/indexer.py`). GLM 5.2 (`GlmMoeDsaForCausalLM`, derived from `DeepseekV2ForCausalLM`) runs the **dsa** indexer instead, which calls `deep_gemm.get_paged_mqa_logits_metadata` / `fp8_paged_mqa_logits` / `fp8_mqa_logits` unconditionally. The env var is therefore a no-op for GLM 5.2, and DeepGEMM stays a hard requirement. This PR makes the flag effective on the dsa path so GLM 5.2 can be functionally validated on hosts where DeepGEMM is unavailable or where the DeepGEMM MQA-logits kernels are not usable. ## Modifications **1. Route the dsa indexer's DeepGEMM entry points through resolvers** (`dsa/utils.py`, consumed by `dsa_backend.py` and `dsa/dsa_indexer.py`): | resolver | `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1` | default | |---|---|---| | `resolve_paged_mqa_logits_metadata_fn()` | `sglang.kernels.ops.attention.dsv4.get_paged_mqa_logits_metadata` (JIT) | `deep_gemm.get_paged_mqa_logits_metadata` | | `resolve_fp8_paged_mqa_logits_fn()` | `_fp8_paged_mqa_logits_torch` → existing `dsv4.indexer.fp8_paged_mqa_logits_torch` | `deep_gemm.fp8_paged_mqa_logits` | | `resolve_fp8_mqa_logits_fn()` | `_fp8_mqa_logits_torch` (new) | `deep_gemm.fp8_mqa_logits` | The paged and metadata variants reuse existing kernels; only the prefill path needed a new pure-torch implementation. It accumulates per-head `relu(q·k)` weighted by the gate and applies the fp8 scale, iterating over heads to keep the `[num_q, num_kv]` intermediate small. Out-of-range positions are filled with `0.0`, matching the already-validated `dsv4.indexer.fp8_paged_mqa_logits_torch` reference. All four call sites pass `clean_logits=False`, which is the only mode implemented (asserted). **2. Remove the last hard DeepGEMM dependency on this path.** `dsa_backend.py` imported `deep_gemm` at module scope solely for `get_num_sms()`, so the module could not even be loaded without DeepGEMM installed. Added `resolve_num_sms()`, which reads `torch.cuda.get_device_properties(...).multi_processor_count` on the fallback path, and dropped the import. Both consumers only use the value to shape the schedule-metadata buffer (`new_empty(num_sm + 1, 2)`), so the raw device SM count is an acceptable stand-in. `configure_deep_gemm_num_sms` (`dsa_indexer.py:488/575`) needs no change: it yields immediately when `not ENABLE_JIT_DEEPGEMM`, and that flag is already `False` when the DeepGEMM import fails. No behavior change when the env var is unset. ## Accuracy Tests Resolver wiring verified in both directions: - **Env var set, DeepGEMM blocked from the import system** (a `sys.meta_path` finder raising `ImportError` for `deep_gemm*`, simulating a host without it): all three dsa modules import successfully and every resolver resolves — `resolve_num_sms()` → 152, metadata → JIT kernel, paged/prefill → the torch implementations. - **Env var unset:** all resolvers return the `deep_gemm` callables and `resolve_num_sms()` matches `deep_gemm.get_num_sms()` (152 == 152). ## Speed Tests and Profiling Not applicable — this is a correctness/portability fallback, not a performance path. `_fp8_mqa_logits_torch` loops over heads in Python with `torch.mm`, and `_pad_heads_for_deep_gemm` pads to 32 heads when the model has fewer, so several of those matmuls are on zero-weight padding. The default DeepGEMM path is unchanged. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31630938791](https://github.com/sgl-project/sglang/actions/runs/31630938791) Latest PR Test (Extra): :x: [Run #31630938531](https://github.com/sgl-project/sglang/actions/runs/31630938531) ",
"url": "https://github.com/sgl-project/sglang/pull/32947",
"createdAt": "2026-07-30T17:15:50Z",
"updatedAt": "2026-08-13T08:26:35Z",
"timestamp": "2026-08-13T08:26:35Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"feature",
"run-ci",
"GLM"
],
"author": "wenscarl",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32963",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NVIDIA][comm] Merge EP+MoE-TP post-experts all-reduces into one _TP reduction",
"text": "## Problem statement With `--tp-size 4 --ep-size 2`, the post-experts reduction requires two all-reduces over orthogonal groups (`_MOE_EP` then `_MOE_TP`). The allreduce fusion gate was skipping *both* once `fuse_mlp_allreduce` was published, then Observed as GSM8k accuracy 0.012 on `nvidia/DeepSeek-V4-Flash-NVFP4` with silently returning under-reduced activations with no error. Observed as GSM8k accuracy 0.012 on `nvidia/DeepSeek-V4-Flash-NVFP4` with the fused residual+LN kernel reduced over only `_MOE_EP` (2 of 4 peers) — silently returning under-reduced activations with no error ## Motivation Under hybrid MoE EP+TP (`moe_ep_size > 1` **and** `moe_tp_size > 1`), allreduce fusion silently produces under-reduced activations. There is no error — the model just returns garbage. Reproduced on `main`, 4×GB200: ```bash python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B \\ --tp-size 4 --ep-size 2 --host 0.0.0.0 --port 30000 ``` ```bash curl -X POST http://localhost:30000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"Qwen/Qwen3-30B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Where is capital city of France? Answer in one short word.\" } ], \"chat_template_kwargs\": { \"enable_thinking\": false }, \"max_tokens\": 8, \"temperature\": 0 }' ``` The completion is garbage unless `--enforce-disable-flashinfer-allreduce-fusion` is passed. `--ep-size 4` and `--ep-size 1` are both fine. ### Root cause With `--tp-size 4 --ep-size 2`, the post-experts reduction spans **two** disjoint groups (`Qwen3MoeSparseMoeBlock.forward_normal`): ``` moe_expert_parallel_all_reduce -> _MOE_EP moe_tensor_model_parallel_all_reduce -> _MOE_TP ``` `_MOE_EP × _MOE_TP` is an orthogonal decomposition of the TP world (`moe_tp_size = tp_size // moe_ep_size // moe_dp_size`), so both reductions are needed to cover all 4 ranks. When the decoder publishes `fuse_mlp_allreduce`, `should_skip_post_experts_all_reduce()` skips **both** of them — it keys off `should_skip_mlp_all_reduce()`, which does not distinguish the EP path from the TP path. The reduction is then supposed to be absorbed by the next layer's fused residual+RMSNorm, but that kernel reduces over a **single** group: the MoE fusion workspace, which `ensure_workspace_initialized()` rendezvouses on `_MOE_EP` when `moe_ep_size > 1`. Net effect: the MoE output is reduced over 2 of the 4 required ranks. `--ep-size 4` (`moe_tp_size == 1`) and `--ep-size 1` (`moe_ep_size == 1`) are unaffected, because there a single group already covers every peer. This is pre-existing on `main` and independent of any in-flight FlashInfer work. It was surfaced while reviewing #30700 (thanks @mmangkad for the hybrid EP+TP question that led here). ### Graph illustration ## Why hybrid EP+TP specifically With `--tp-size 4 --ep-size 2`, `moe_tp_size = 4 // 2 // 1 = 2`, and the MoE output needs reducing over two orthogonal subgroups that together tile the TP world: ``` rank: 0 1 2 3 ep_rank: 0 0 1 1 <- which expert shard it holds tp_rank: 0 1 0 1 <- which FFN shard it holds _MOE_EP: {0,2} {1,3} _MOE_TP: {0,1} {2,3} ``` output needs reducing over two orthogonal subgroups that together tile the TP world: ``` rank: 0 1 2 3 ep_rank: 0 0 1 1 <- which expert shard it holds tp_rank: 0 1 0 1 <- which FFN shard it holds _MOE_EP: {0,2} {1,3} _MOE_TP: {0,1} {2,3} ``` Either reduction alone covers 2 of the 4 ranks; both are required. ``` layer N (MoE output) | layer N+1 ----------------------------------------------|------------------------ main experts --> [EP skipped] [TP skipped] -----|--> fused AR + RMSNorm | workspace group = _MOE_EP | sums {0,2} only -> WRONG this PR experts --> AR(_MOE_EP) --> AR(_MOE_TP) ----|--> RMSNorm {0,2} {0,1} | all 4 ranks summed -> ok ``` Publishing `fuse_mlp_allreduce` makes `should_skip_post_experts_all_reduce()` drop **both** reductions, because it keys off that flag without distinguishing the EP path from the TP path. The next layer's fused residual+LN then reduces over a single group — the MoE fusion workspace, which `ensure_workspace_initialized()` rendezvouses on `_MOE_EP` when `moe_ep_size > 1`. Half the peers never contribute, and the result is wrong without any error. No other configuration breaks, because no other configuration has two distinct subgroups: ``` ep_size=1 -> moe_tp_size == tp_size, _MOE_TP *is* _TP (parallel_state.py:2483) ep_size=4 -> moe_ep_size == tp_size, _MOE_EP *is* _TP (parallel_state.py:2454) ep_size=2 -> neither aliases _TP -> the only broken case ## Modifications `LayerCommunicator.should_fuse_mlp_allreduce_with_next_layer()` now returns `False` when `moe_ep_size > 1 and moe_tp_size > 1`. Fusion is only valid when one group covers every peer of the reduction, which is not the case here. Both post-experts all-reduces then run inline as usual. The guard sits before the backend-specific checks, so it covers the aiter fusion branch too — that path reduces over a single group for the same reason. Attention-side fusion is untouched: it uses the attention-TP workspace (`use_attn_tp_group=True`) whose group spans the full TP world, and it is gated on `apply_flashinfer_allreduce_fusion` alone, not on `fuse_mlp_allreduce`. New unit test `test/registered/unit/layers/test_layer_communicator_fusion_gate.py` (CPU, 3 cases): hybrid EP+TP must not fuse; pure-TP and pure-EP must still fuse. The hybrid case fails on pre-fix code and passes on the fix; the other two guard against the gate degrading to always-off. After fix: ``` {\"id\":\"0dc05777d22f4efdbbba2ba896d4d196\",\"object\":\"chat.completion\",\"created\":1785440815,\"model\":\"nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"Paris\",\"reasoning_content\":null,\"tool_calls\":null},\"logprobs\":null,\"finish_reason\":\"stop\",\"matched_stop\":151645}],\"usage\":{\"prompt_tokens\":25,\"total_tokens\":27,\"completion_tokens\":2,\"prompt_tokens_details\":null,\"reasoning_tokens\":0},\"metadata\":{\"weight_version\":\"default\"}} ``` ## Accuracy Tests - `--tp-size 4 --ep-size 2` on 4×GB200 with Qwen3-30B-A3B: garbage before, correct completions after, **without** `--enforce-disable-flashinfer-allreduce-fusion`. - `--ep-size 4` and `--ep-size 1` re-checked for no regression. - `python -m pytest test/registered/unit/layers/test_layer_communicator_fusion_gate.py` — 3 passed. GSM8k: ``` on GB200 python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1316 --parallel 1316 reference: (--tp-size 4 --ep-size 4) Accuracy: 0.743 Invalid: 0.001 Latency: 18.692 s Output throughput: 17213.786 token/s before fix: (--tp-size 4 --ep-size 2) Accuracy: 0.012 Invalid: 0.014 Latency: 37.922 s Output throughput: 17453.860 token/s after fix: (--tp-size 4 --ep-size 2) Accuracy: 0.777 Invalid: 0.003 Latency: 17.508 s Output throughput: 16609.659 token/s ``` Also support DSV4: ``` python -m sglang.launch_server --trust-remote-code --model-path nvidia/DeepSeek-V4-Flash-NVFP4 --ep-size 2 --tp-size 4 --enable-dp-attention --chunked-prefill-size 4096 --disable-flashinfer-autotune --swa-full-tokens-ratio 0.1 --host 0.0.0.0 --port 30000 --moe-runner-backend flashinfer_trtllm_routed --max-running-requests 1024 --mem-fraction-static 0.8 ``` gsm8k: ``` python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1316 --parallel 1316 Accuracy: 0.954 Invalid: 0.000 Latency: 90.583 s ``` --- ### CI States Latest PR Test (Base): :x: [Run #31630301030](https://github.com/sgl-project/sglang/actions/runs/31630301030) Latest PR Test (Extra): :x: [Run #31630300806](https://github.com/sgl-project/sglang/actions/runs/31630300806) ",
"url": "https://github.com/sgl-project/sglang/pull/32963",
"createdAt": "2026-07-30T19:40:02Z",
"updatedAt": "2026-08-13T07:53:27Z",
"timestamp": "2026-08-13T07:53:27Z",
"metrics": {
"reactions": 0,
"comments": 22
},
"labels": [
"bug",
"deepseek",
"run-ci"
],
"author": "wenscarl",
"state": "open",
"assignees": [
"mmangkad"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32982",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "docs: sync LMSYS SGLang blog cards",
"text": "Automated sync of the LMSYS SGLang blog cards in `docs_new/index.mdx`. 🤖 Generated with GitHub Actions --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #30594717790](https://github.com/sgl-project/sglang/actions/runs/30594717790) Latest PR Test (Extra): :x: [Run #30594717633](https://github.com/sgl-project/sglang/actions/runs/30594717633) ",
"url": "https://github.com/sgl-project/sglang/pull/32982",
"createdAt": "2026-07-31T00:51:05Z",
"updatedAt": "2026-08-13T12:24:12Z",
"timestamp": "2026-08-13T12:24:12Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"documentation"
],
"author": "sglang-bot",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32991",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(attention): add architecture-owned SM12x FA4 kernels",
"text": "## Motivation SGLang currently routes SM12x FA4 through an SM80-derived forward path. That path uses the older mainloop and does not expose the SM12x TMA dataflow, paged KV, SplitKV, or relative/shared-bias support. This PR adds an architecture-owned SM12x implementation and keeps its kernel, scheduler, and runtime policy under `sglang.kernels.ops.attention.fa4_sm120`. ## Modifications - Add TMA-based SM12x forward and paged-decode kernels with warp MMA. - Add persistent scheduling and runtime-SM-count-aware tile selection instead of assuming a fixed GPU size. - Support BF16/FP16, MHA/GQA/MQA, dense and variable-length attention, paged KV, SplitKV, global/local/noncausal masks, attention sinks, packed GQA, and relative/shared bias with shearing. - Add SM12x-owned host policy, launch-plan caching, preallocated-output paths, CUDA Graph-safe decode plans, and TVM-FFI launch fast paths. - Route SM12x through a narrow architecture dispatch bridge while leaving SM80/SM9x/SM10x kernel ownership unchanged. - Replace the previous SM80-derived SM120 fallback and add comprehensive SM12x regression tests. ## Accuracy Tests The focused SM12x test file passes in full: ```text test_flash_attention_4_sm120.py: 53 passed ``` Coverage includes dense/varlen attention, BF16/FP16, MHA/GQA/MQA, asymmetric Q/V dimensions, packed and unpacked GQA, paged KV, SplitKV, attention sinks, relative/shared bias, preallocated output, CUDA Graph replay, PDL, and specialization-cache ordering. The 684 measured prefill SKU-case pairs all passed comparison against the SGLang Triton baseline: | GPU | Cases | Maximum absolute difference | | --- | ---: | ---: | | RTX PRO 6000 Blackwell | 228 | 0.0078125 | | RTX PRO 5000 Blackwell | 228 | 0.0078125 | | GB10 (SM121) | 228 | 0.015625 | ## Speed Tests and Profiling ### Prefill The common three-way matrix contains 228 matched cases per GPU: BF16/FP16, MHA/GQA/MQA, head dimensions 32/64/96/128/192->128/256, sequence lengths 128/512/2048/6144, dense and ragged varlen inputs, and causal-global, causal-local-with-sink, and noncausal-global modes. Each case uses five warmups followed by three shuffled trials with 20 CUDA event samples per trial. PR and SM80-fallback runs are each paired with the same byte-identical Triton implementation. `PR / SM80` below is the ratio of those paired speedups, which normalizes process-level clock drift. | GPU | SMs | PR / Triton | Wins | SM80 fallback / Triton | Wins | PR / SM80 fallback | Wins | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | RTX PRO 6000 Blackwell | 188 | 3.027x | 228/228 | 1.274x | 112/228 | 2.377x | 222/228 | | RTX PRO 5000 Blackwell | 110 | 3.311x | 228/228 | 1.574x | 150/228 | 2.104x | 224/228 | | GB10 (SM121) | 48 | 3.610x | 228/228 | 2.087x | 171/228 | 1.730x | 205/228 | Across all 684 equally weighted SKU-case pairs, the PR is **3.308x over Triton with 684/684 wins**, and **2.053x over the SM80-derived fallback with 651/684 wins**.  Representative absolute latency: | GPU | Case | PR | Triton | SM80 fallback | | --- | --- | ---: | ---: | ---: | | RTX PRO 6000 | MHA, BF16, HD64, L128 | 22.66 us | 32.22 us | 60.85 us | | RTX PRO 6000 | local+sink GQA, BF16, HD128, L2048 | 31.12 us | 161.34 us | 84.64 us | | RTX PRO 6000 | MHA, BF16, HD256, L6144 | 456.29 us | 2588.77 us | 571.95 us | | RTX PRO 5000 | MHA, BF16, HD64, L128 | 14.64 us | 21.10 us | 33.42 us | | RTX PRO 5000 | local+sink GQA, BF16, HD128, L2048 | 30.37 us | 198.30 us | 91.81 us | | RTX PRO 5000 | MHA, BF16, HD256, L6144 | 745.12 us | 3487.39 us | 827.04 us | | GB10 (SM121) | MHA, BF16, HD64, L128 | 18.14 us | 24.93 us | 38.37 us | | GB10 (SM121) | local+sink GQA, BF16, HD128, L2048 | 66.75 us | 437.20 us | 158.91 us | | GB10 (SM121) | MHA, BF16, HD256, L6144 | 2108.42 us | 13977.18 us | 1823.94 us | The few PR/fallback losses are concentrated in long ragged HD256 and dense noncausal GQA, most visibly on the 48-SM device. Every measured PR case still beats Triton. Theoretical useful-compute SOL and logical-memory roofline ratios: | GPU | Compute SOL median / max | Logical-memory ratio median / max | | --- | ---: | ---: | | RTX PRO 6000 Blackwell | 14.2% / 72.4% | 10.8% / 83.3% | | RTX PRO 5000 Blackwell | 35.6% / 82.0% | 14.2% / 80.0% | | GB10 (SM121) | 34.0% / 74.3% | 40.8% / 165.1% | The logical-memory ratio uses one compulsory Q/K/V read and one output write against peak DRAM bandwidth. Repeated resident tensors can be served from cache, so values above 100% are not literal measured DRAM SOL. ### Paged decode Paged decode covers 108 cases per GPU across MHA/GQA/MQA, head dimensions 64/128/256, batch sizes 1/8/32, KV lengths 128/512/2048/8192, and page size 64. CUDA Graph replay is the kernel-focused comparison; eager and synchronized wall timings include host overhead. | GPU | Graph PR / Triton | Graph wins | Eager PR / Triton | Wall PR / Triton | | --- | ---: | ---: | ---: | ---: | | RTX PRO 6000 Blackwell | 1.311x | 92/108 | 1.432x | 1.317x | | RTX PRO 5000 Blackwell | 1.317x | 96/108 | 1.424x | 1.286x | | GB10 (SM121) | 1.214x | 90/108 | 1.377x | 1.307x | The SM80-derived fallback explicitly rejects paged KV and SplitKV, so it is reported as unsupported rather than assigned a synthetic timing. The fallback also does not support the relative/shared-bias path. ## Checklist - [x] Format changed code with the repository `prek` hooks. - [x] Add focused SM12x unit and regression tests. - [x] No user-facing documentation update is required for this internal attention-kernel implementation. - [x] Provide accuracy and speed benchmark results. - [x] Follow the SGLang code-style guidance. --- ### CI States Latest PR Test (Base): :x: [Run #31651905677](https://github.com/sgl-project/sglang/actions/runs/31651905677) Latest PR Test (Extra): :white_check_mark: [Run #31651905587](https://github.com/sgl-project/sglang/actions/runs/31651905587) ",
"url": "https://github.com/sgl-project/sglang/pull/32991",
"createdAt": "2026-07-31T02:10:01Z",
"updatedAt": "2026-08-13T06:58:43Z",
"timestamp": "2026-08-13T06:58:43Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "lucifer1004",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33005",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[FP8][MoE] Honor UE8M0 activation scales in Triton MoE",
"text": "## Motivation DeepSeek-V4 checkpoint metadata declares `scale_fmt: ue8m0` for its FP8 path. The official DSV4 reference applies this policy when quantizing activations before every expert Linear, so both FC1 and FC2 require power-of-two activation scales. The Triton MoE path previously dropped that policy and used continuous `amax / 448` FP32 scales instead. ## Test Made Triton MoE recognize `use_scale_ue8m0` and apply power-of-two activation scales to both FC1 and FC2, then ran the following model-level A/B evaluation. - Machine: 8×NVIDIA B200 - Model: `DeepSeek-V4-Flash` - Runtime: `lmsysorg/sglang:latest` (`sha256:7b6a35df9839fd593a94a1eaee82d7777f472225d9f3ad1f8a2e0cb2bd1785d0`) - Parallelism: TP8 / EP1 - Expert path: the checkpoint's packed FP4 expert weights were converted once to FP8 at load time with `SGLANG_DSV4_FP4_DEQUANT=1`, allowing the Triton FP8 MoE path to be evaluated - MoE: `--moe-a2a-backend none`; runtime logs confirmed the Triton MoE implementation | Benchmark / protocol | Baseline (`c039e1a7`) | PR (`47f3c98c`) | Delta | |---|---:|---:|---:| | GSM8K, temperature 0, 1 run | 1266 / 1319 (95.9818%) | 1264 / 1319 (95.8302%) | -0.1516 pp | | GPQA Diamond, temperature 1, top-p 1, max thinking, 10 paired samples/question | 1676 / 1980 (84.6465%) | 1672 / 1980 (84.4444%) | -0.2020 pp | --- ### CI States Latest PR Test (Base): :x: [Run #31609723050](https://github.com/sgl-project/sglang/actions/runs/31609723050) Latest PR Test (Extra): :x: [Run #31609722764](https://github.com/sgl-project/sglang/actions/runs/31609722764) ",
"url": "https://github.com/sgl-project/sglang/pull/33005",
"timestamp": "2026-08-12T16:15:28Z",
"metrics": {
"reactions": 0,
"comments": 14
},
"labels": [
"deepseek",
"jit-kernel"
],
"author": "kaixih",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:33006",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(dsa): use FlashInfer fused top-k for packed PAGED rows",
"text": "## Motivation @humansand - [sgl-project/sglang#32490](https://github.com/sgl-project/sglang/pull/32490) added an SGL-kernel fallback because FlashInfer previously used one `row_starts` tensor for both score and page-table windows. - [flashinfer-ai/flashinfer#4169](https://github.com/flashinfer-ai/flashinfer/pull/4169) adds `page_table_row_starts`, allowing independent score and page-table origins. It is available in [FlashInfer v0.6.17rc1](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.17rc1). - The rebased PR is validated against the official FlashInfer v0.6.17rc1 CUDA 13 artifacts. It is waiting for final v0.6.17 and the aligned SGLang Python, Docker, and runtime pins. ## Modifications - Remove the packed-PAGED FlashInfer-to-SGL-kernel fallback. - Pass batch-global `row_starts` as the score-window origin and request-local `page_table_row_starts` as the page-table origin. - Preserve `row_to_batch`, `dsa_graph_safe=True`, and the SGL-only top-k-v2 policy. - Extend the fused backend-equivalence test to cover shifted packed-PAGED rows and verify both FlashInfer start tensors. This PR intentionally does not change FlashInfer dependency pins. Upstream main currently pins `0.6.15.post1`. ## Accuracy Tests Static validation at rebased head `e074a275de1290cc5f284ee16a90644d5119cd74`: ```text git diff --check python3 -m compileall -q python/sglang/srt/layers/attention/dsa/dsa_topk_backend.py test/registered/kernels/ops/attention/test_dsa_indexer.py pre-commit run --all-files ``` All passed. B200 runtime validation used: - Image: `lmsysorg/sglang:nightly-dev-cu13-20260806-ae5f8c94` - Image digest: `sha256:96cbac6a4c834f8233873b19df6f0e025658d53fb81b8baa510bc9cd3ea62b95` - `flashinfer-python==0.6.17rc1` - `flashinfer-cubin==0.6.17rc1` - `flashinfer-jit-cache==0.6.17rc1+cu130` SGLang was installed from the exact rebased PR head with `--no-deps`. Focused DSA top-k coverage: ```bash python3 -m pytest -vv --tb=short \\ test/registered/kernels/ops/attention/test_dsa_indexer.py::TestDSAIndexer::test_topk_unfused_backends_valid_selection \\ test/registered/kernels/ops/attention/test_dsa_indexer.py::TestDSAIndexer::test_topk_fused_backends_equivalence \\ test/registered/kernels/ops/attention/test_dsa_indexer.py::TestDSAIndexer::test_topk_v2_respects_topk_backend ``` ```text 3 passed, 15 warnings, 30 subtests passed in 14.34s ``` A CUDA graph smoke test captured and replayed the packed-PAGED FlashInfer path, then mutated the scores and shifted every global `row_starts` origin before replaying again: ```text capture/replay passed updated-input replay passed ``` No model-level accuracy benchmark was run. ## Speed Tests and Profiling Not run. This change selects the configured FlashInfer fused transform instead of the temporary SGL-kernel fallback; no performance claim is made. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No user-facing API change.) - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31669056967](https://github.com/sgl-project/sglang/actions/runs/31669056967) Latest PR Test (Extra): :x: [Run #31669056828](https://github.com/sgl-project/sglang/actions/runs/31669056828) ",
"url": "https://github.com/sgl-project/sglang/pull/33006",
"createdAt": "2026-07-31T03:40:14Z",
"updatedAt": "2026-08-13T09:47:23Z",
"timestamp": "2026-08-13T09:47:23Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "zianglih",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33030",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] add Ascend 950 (Atlas A5) backend paths for DeepSeek-V4",
"text": "## Motivation Ascend 950 (Atlas A5) is a new-generation NPU whose numeric capabilities differ substantially from the currently supported 910B/910C (Atlas A3) parts: it natively supports FP8-e4m3 KV cache, E8M0-scaled MXFP8 GEMM, and MXFP4 grouped matmul, and the corresponding sparse-attention / KV-compression kernels take different signatures and buffer layouts. As a result DeepSeek-V4 cannot currently be brought up on A5 from community `main` at all. This PR upstreams an A5 adaptation that has been developed and validated on an internal branch (`A5_dsv4_acc`), rebuilt as a thin overlay on top of `main`. Every A5 code path is placed behind a new `is_npu_before_atlas_a5()` device gate, so **910B/910C behavior on this branch is byte-for-byte the same as `main`**, and CUDA/ROCm paths are untouched. ## Modifications ### Device gate - `utils/common.py`: new `is_npu_before_atlas_a5()` — `lru_cache`d, returns `True` on Ascend NPUs older than Atlas A5. Queries `torch_npu.npu.current_device()` rather than device 0, so a rank pinned to another die does not create an ACL context on the wrong one. ### Attention / KV cache (DSV4) - `npu/attention/ascend_dsv4_backend.py`: on A5, dispatch sparse attention to `npu_kv_quant_sparse_attn_sharedkv{,_metadata}` (FP8 KV) instead of `npu_sparse_attn_sharedkv{,_metadata}`, passing `kv_quant_mode` / `tile_size` / `rope_head_dim`. Adds the FP8 Q path and the fused `indexer_compress_epilog` write; the lightning-indexer KV dtype becomes `float8` on A5 and stays `int8` on A3. `_forward_full` passes `cmp_ratio` only on A5 — the pre-A5 op signature is untouched. - `npu/dsv4/dsv4_memory_pool.py`: A5 KV buffers are allocated as `float8_e4m3fn` with a packed per-token row (nope + 2×rope + per-64 scale, padded to 128B) via the new `a5_packed_kv_dim`; indexer buffers become `float8_e4m3fn` / `float32` instead of `int8` / `float16`. Both `set_swa_buffer` and `set_compress_buffer` write through a shared `_write_a5_packed_kv()` helper backed by `kv_compress_epilog`. - `npu/attention/ascend_backend.py`: fix hybrid-SWA `block_tables_swa` construction to slice `req_to_token` with a stride instead of materializing the full `[bs, max_len]` gather and then striding the result. Behavior-preserving, applies to all Ascend parts. ### Quantization - **New** `npu/quantization/fp4_moe_methods.py` (624 lines): `NPUW4A4Fp4MoEMethod` plus the W4A4-MXFP expert kernels — prefill/decode fused-expert entry points, MXFP4 scale reshaping for the NPU layout, SwiGLU limit application, and the grouped-matmul chain (`w4a4_mxfp_gmm_npu` / `_w4a4_mxfp_gmm` / `_w4a8_mxfp_gmm`). Three dispatch-format handlers are provided: `..._ascend_tp`, `..._deepep`, and `npu_apply_without_routing_weights_w4a4_mxfp`. The AscendTP handler owns only the grouped-matmul chain, because `AscendTPDispatcher` already performs `npu_moe_init_routing_v2` on dispatch and `npu_moe_finalize_routing` on combine. - `npu/quantization/linear_method_npu.py`: new `npu_w8a8_block_fp8_linear()` — an MXFP8 GEMM built on `npu_dynamic_mx_quant` + `npu_quant_matmul` with `float8_e8m0fnu` scales. - `layers/quantization/fp8_utils.py`: `_dispatch_auto_backend()` selects `npu_w8a8_block_fp8_linear` on A5 only (NPU imports are lazy so non-NPU builds are unaffected). - `layers/quantization/fp8.py`: on A5, `process_weights_after_loading_block_quant` reinterprets DeepSeek's `[128, 128]` block-FP8 fp32 scales as E8M0 exponent bytes — extract the exponent field, expand it over the block it covered, pair it, and transpose both weight and scale K-major. DSV4's `wo_a` is additionally batched into the `[G, D, R]` / `[G, D/64, R, 2]` layout that `npu_transpose_quant_batchmatmul` expects. Also routes FP4 experts to `NPUW4A4Fp4MoEMethod` on NPU. ### Model - `models/deepseek_v4.py`: `_use_npu_a5_mxfp8_wo_a()` decides whether `wo_a` is allocated as an FP8 parameter; `MQALayer.__init__` and `load_weights` now share that single decision, so the load-side dequant and the allocated parameter dtype cannot disagree. Adds the A5 `npu_transpose_quant_batchmatmul` forward branch, gates the batched `npu_hc_post` on `is_npu_before_atlas_a5()`, and reinterprets FP4 E8M0 scales as `uint8` in the expert weight loader. - `models/deepseek_v2.py`: DSV4's non-hash `MoEGate` layers run the router GEMM in fp32 on NPU, mirroring what the CUDA path already does via `linear_bf16_fp32`. These layers route on near-degenerate logits and the top-k boundary flips in bf16. ## Accuracy Tests ```plaintext ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 128 Benchmark duration (s): 215.64 Total input tokens: 1048576 Total input text tokens: 1048576 Total generated tokens: 131072 Total generated tokens (retokenized): 131072 Request throughput (req/s): 0.59 Input token throughput (tok/s): 4862.58 Output token throughput (tok/s): 607.82 Peak output token throughput (tok/s): 3622.00 Peak concurrent requests: 84 Total token throughput (tok/s): 5470.41 Concurrency: 55.60 Accept length: 2.74 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 93662.17 Median E2E Latency (ms): 94025.97 P90 E2E Latency (ms): 143925.22 P95 E2E Latency (ms): 144095.34 P99 E2E Latency (ms): 146977.56 ---------------Time to First Token---------------- Mean TTFT (ms): 13803.26 Median TTFT (ms): 9604.62 P90 TTFT (ms): 28401.46 P95 TTFT (ms): 35824.40 P99 TTFT (ms): 39820.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 78.06 Median TPOT (ms): 72.07 P90 TPOT (ms): 120.75 P95 TPOT (ms): 128.16 P99 TPOT (ms): 135.14 ---------------Inter-Token Latency---------------- Mean ITL (ms): 78.06 Median ITL (ms): 18.21 P90 ITL (ms): 27.23 P95 ITL (ms): 28.67 P99 ITL (ms): 2281.07 Max ITL (ms): 11796.57 ``` ## Benchmarking and Profiling ```plaintext 2026-08-05 03:51:55 - evalscope - INFO: Benchmark aime26 evaluation finished. 2026-08-05 03:51:55 - evalscope - INFO: Running[eval] 100%| 1/1 [Elapsed: 59:38 < Remaining: 00:00, 3578.99s/benchmark] Running[eval]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [59:38<00:00, 3578.99s/benchmark] 2026-08-05 03:51:55 - evalscope - INFO: Overall report table: ┌─────────┬───────────┬──────────┬──────────┬───────┬─────────┬─────────┐ │ Model │ Dataset │ Metric │ Subset │ Num │ Score │ Cat.0 │ ├─────────┼───────────┼──────────┼──────────┼───────┼─────────┼─────────┤ │ dsv │ aime26 │ mean_acc │ default │ 30 │ 0.9667 │ default │ └─────────┴───────────┴──────────┴──────────┴───────┴─────────┴─────────┘ ``` ## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation / docstrings / example guides as outlined in [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#write-documentations). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as outlined in [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR. --- ### CI States Latest PR Test (Base): :x: [Run #31682486675](https://github.com/sgl-project/sglang/actions/runs/31682486675) Latest PR Test (Extra): :x: [Run #31682486234](https://github.com/sgl-project/sglang/actions/runs/31682486234) ",
"url": "https://github.com/sgl-project/sglang/pull/33030",
"createdAt": "2026-07-31T06:49:22Z",
"updatedAt": "2026-08-13T08:33:00Z",
"timestamp": "2026-08-13T08:33:00Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"quant",
"deepseek",
"npu",
"run-ci"
],
"author": "AndyLi429",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33040",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[minimax m3][npu]Adaptation of Minimax M3(w8a8) for NPU platforms [2/2]",
"text": " ## Motivation Adaptation of Minimax M3 for NPU platforms ## Modifications 1、Define the complete attention implementation for MiniMax-M3 on the NPU platform, with deep optimization based on NPU‑specific features. 2、Implement speculative inference adaptation for MiniMax‑M3 based on the Eagle3 draft model. 3、Fix the W8A8 weight loading method for MiniMax‑M3. 4、Fix the memory allocation issue in CUDA graph where the number of predicted tokens is not considered when CUDA graph is enabled. **5、Complete implementation of fuseep based on SwiGLU‑OAI (THIS PR).** previous PR: https://github.com/sgl-project/sglang/pull/32941 relative kernel PR: https://github.com/sgl-project/sgl-kernel-npu/pull/629 ## Accuracy Tests ## Speed Tests and Profiling Results for 16K input with 58 concurrent requests ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31704989069](https://github.com/sgl-project/sglang/actions/runs/31704989069) Latest PR Test (Extra): :x: [Run #31704988802](https://github.com/sgl-project/sglang/actions/runs/31704988802) ",
"url": "https://github.com/sgl-project/sglang/pull/33040",
"createdAt": "2026-07-31T07:47:34Z",
"updatedAt": "2026-08-13T13:28:08Z",
"timestamp": "2026-08-13T13:28:08Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci"
],
"author": "vstone-w",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33048",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Hold references to fire-and-forget tasks in disaggregation",
"text": "## Motivation Five `asyncio.create_task()` calls in `srt/disaggregation` throw the returned task away. The event loop only keeps a **weak** reference to a running task, so one whose sole reference was the `create_task()` expression can be garbage collected before it finishes — [the asyncio docs call this out explicitly](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task): > Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn't referenced elsewhere may get garbage collected at any time, even before it's done. `MMEncoder` in `encode_server.py` already handles this correctly — `self.background_tasks` at line 346, used at 1060, 2022 and 2069. This PR applies that same pattern to the classes in the package that miss it. | site | task | what a collection costs | |---|---|---| | `encode_server.py:2929` `DPDispatcher.start` | `_result_listener` | dispatcher stops consuming worker results | | `encode_server.py:2930` | `_worker_watchdog` | dead ranks are never detected | | `encode_server.py:2931` | `_cleanup_stale_mappings` | stale mooncake mappings accumulate | | `common/conn.py:1721` `CommonKVBootstrapServer._run_server` | `_cleanup_expired_entries` | `room_to_dp_rank` grows without bound | | `encode_receiver.py:1668` `MMReceiverBase` | the encode dispatch | the encode never happens; the paired `_recv_mm_data` just times out | The first four are process-lifetime loops, which is the worst case for this bug — nothing else ever holds them. `encode_receiver.py` deserves a note: the code creates the encode task and then awaits `_recv_mm_data` on a socket, so the task is awaited *indirectly*, never through the task object. A local binding would only cover the happy path — if `wait_for` raises `TimeoutError` the frame unwinds while the encode may still be in flight — so this uses the same instance-level set as the rest. ## Modifications - `DPDispatcher`: add `self.background_tasks`, register the three loops from `start()`. - `CommonKVBootstrapServer`: add `self._background_tasks`, register the cleanup loop. - `MMReceiverBase`: add `self._background_tasks`, register the encode dispatch. Each uses `set.discard` in the done callback so a double-callback can't raise. No control flow, scheduling order or awaited behaviour changes — the tasks were fire-and-forget before and remain so; they simply can no longer be collected while still running. ## Accuracy Tests Not applicable — no kernel or model forward code is touched. ## Speed Tests and Profiling Not applicable. The added work is one `set.add` per task and one `discard` on completion. ## Checklist - [x] Format your code according to the pre-commit config — run with the pinned versions: ```console $ isort --check-only # isort 7.0.0 $ black --check # black 26.1.0 All done! ✨ 🍰 ✨ 3 files would be left unchanged. $ ruff check --select=F401,F821,UP037 # ruff 0.15.1 All checks passed! ``` - [ ] Add unit tests — I did not add one, and would rather say why than leave the box quietly unticked: the failure mode is a garbage-collection race, so a test has to force a GC at a chosen moment and assert a task did *not* vanish, which is flaky by construction. All five call sites also need a live dispatcher / bootstrap server / receiver to reach. Happy to add one if you have a shape in mind. - [x] Update documentation — not applicable. - [ ] Accuracy and speed benchmarks — not applicable, see above. - [x] Follow the SGLang code style guidance. Found with an AST scan for `create_task` / `ensure_future` results discarded as bare expression statements; after this change that scan reports zero remaining sites in `srt/disaggregation`. --- ### CI States Latest PR Test (Base): :x: [Run #31452866731](https://github.com/sgl-project/sglang/actions/runs/31452866731) Latest PR Test (Extra): :x: [Run #31452866525](https://github.com/sgl-project/sglang/actions/runs/31452866525) ",
"url": "https://github.com/sgl-project/sglang/pull/33048",
"createdAt": "2026-07-31T09:10:47Z",
"updatedAt": "2026-08-13T06:29:33Z",
"timestamp": "2026-08-13T06:29:33Z",
"metrics": {
"reactions": 0,
"comments": 11
},
"labels": [
"run-ci"
],
"author": "noron12234",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33057",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(xpu): enable compressed-tensors FP8 W8A8 on XPU (RedHatAI FP8-dynamic models)",
"text": "## Motivation compressed-tensors FP8 W8A8 quantized models (e.g. RedHatAI's `*-FP8-dynamic` family — `Apertus-8B-Instruct-2509-FP8-dynamic`, `Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic`, `granite-4.0-h-small-FP8-dynamic`, `NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic`) currently fail to serve on XPU. The very first failure happens at layer-construction time, before any kernel dispatch: ``` AssertionError: Torch not compiled with CUDA enabled ``` Fixing that surfaces two further, distinct XPU-specific issues before the model actually serves and generates correctly. ## Modifications - `compressed_tensors.py` (`CompressedTensorsConfig`): `_check_scheme_supported` unconditionally called `torch.cuda.get_device_capability()`, which is CUDA-only. Only `CompressedTensorsW8A8Fp8` has a real XPU-usable kernel path today (via the existing device-agnostic `_apply_fallback_scaled_mm` in `fp8_utils.py`, not a missing native kernel); every other capability-gated scheme (NVFP4/cutlass, Marlin-based W8A16Fp8, WNA16, ...) is explicitly rejected with a clean `RuntimeError` on XPU instead of crashing on the CUDA-only call. - `fp8_kernel.py`: the non-CUDA/non-MUSA branch never imported `sgl_per_token_quant_fp8`/`sgl_per_tensor_quant_fp8` for XPU, even though sgl-kernel-xpu registers both natively — this caused a `NameError` on the first forward pass once the above crash was fixed. - `fp8_utils.py` (`apply_fp8_linear`): the activation-quant padding decision (`num_token_padding=17`, meant to make `torch._scaled_mm` more performant on CUDA for small batches) was only skipped when `cutlass_fp8_supported` (CUDA-only). sgl-kernel-xpu's per-token quant kernel requires `output_q` to exactly match `input`'s shape and doesn't support padded output, raising a shape-mismatch `RuntimeError` whenever a forward pass had fewer than 17 rows (e.g. small prefill/decode steps). None of these fixes depend on a new kernel — compressed-tensors FP8 W8A8 on XPU runs through sglang's existing, device-agnostic `_apply_fallback_scaled_mm` fallback (`torch._scaled_mm` with identity scale + a manual rowwise/colwise scale multiply) once these three CUDA-only assumptions are corrected. ## Accuracy Tests Verified locally end-to-end in a docker container (XPU backend, Intel Arc Pro/BMG) with the patched sglang installed in place of the image's baked-in copy: - `RedHatAI/Apertus-8B-Instruct-2509-FP8-dynamic` (TP=1, `--attention-backend intel_xpu`) serves and evaluates gsm8k (5-shot) correctly: - `exact_match,flexible-extract = 0.5891 ± 0.0136` - `strict-match = 0.5390 ± 0.0137` - Regression check that the shared code paths touched here don't affect other backends/schemes: - All three changes are gated behind `if _is_xpu:` / `elif _is_xpu:` branches that are new no-ops on every other platform; `cutlass_fp8_supported`-driven CUDA behavior is unchanged (verified by re-reading every other call site of the touched functions — `scaled_fp8_quant`'s other callers in `fp8_utils.py` are gated by `if _is_cuda:`/`if _is_hip:`, never reaching the changed XPU branch on other devices). - No currently-passing XPU model can regress from these changes: any compressed-tensors quantized model on XPU previously hit the `_check_scheme_supported` crash unconditionally at layer-construction time (before this PR), so none could have been passing beforehand. ## Speed Tests and Profiling Not applicable — these are correctness/compatibility fixes for an existing fallback code path (`_apply_fallback_scaled_mm`), no new kernel or scheduling changes. The activation-quant padding skip on XPU has a minor performance implication (an extra kernel-launch on small batches that would otherwise reuse a padded buffer), but padding was never functional on XPU before this PR (it crashed), so there's no regression relative to current behavior. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31668474881](https://github.com/sgl-project/sglang/actions/runs/31668474881) Latest PR Test (Extra): :x: [Run #31668474677](https://github.com/sgl-project/sglang/actions/runs/31668474677) ",
"url": "https://github.com/sgl-project/sglang/pull/33057",
"createdAt": "2026-07-31T10:16:17Z",
"updatedAt": "2026-08-13T04:54:52Z",
"timestamp": "2026-08-13T04:54:52Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"jit-kernel"
],
"author": "vshekhawat-hlab",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33062",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[qwen3.5] perf: Skip the GDN qkvzba split kernel on single-token forwards",
"text": "## Motivation On Qwen3.5 `fused_qkvzba_split_reshape_cat_contiguous_kernel` is a bit-exact identity copy. The checkpoint stores `in_proj_qkv` and `in_proj_z` separately, so `MergedColumnParallelLinear` already emits `[all_q | all_k | all_v | all_z]` and every load/store offset in the kernel maps to itself. Its four outputs are plain column slices: ```python mixed_qkv = qkvz[:, :qkv_dim] z = qkvz[:, qkv_dim:].view(-1, num_v_heads, head_v) b, a = ba[:, :num_v_heads], ba[:, num_v_heads:] ``` That is one wasted launch in 45 of Qwen3.5-397B's 60 layers, and at bs=1 decode every kernel is launch-bound, so it costs 2.46us of pure serialization each. ## Modifications - `triton_gdn_fused_proj.py`: add `qkvzba_split_is_pure_view()` and `qkvzba_split_reshape_cat_contiguous_views()`. The kernel is untouched. - `qwen3_5.py`: dispatch to the views when the guard holds. - Unit test asserting `torch.equal` against the kernel. The `shape[0] == 1` guard is load-bearing, not conservatism: with more than one row the slices are strided and `z.reshape(-1, head_v)` inserts a compaction copy, trading the saved launch for a new one. Nothing downstream needed changing — the Triton `causal_conv1d_update` already takes `x.stride()`, and FlashInfer's GDN decode marks q/k/v/a/b layout-dynamic. Only the contiguous variant is an identity. Qwen3-Next's interleaved `fused_qkvzba_split_reshape_cat` is a real permutation and is untouched. ## Correctness The unit test asserts the view outputs are `torch.equal` to the kernel's and equally contiguous, across 4 TP shapes x 2 dtypes, with the guard rejecting the multi-row and non-contiguous cases. GSM8K 5-shot, full 1319 questions, `sglang.test.few_shot_gsm8k --parallel 128`, temperature 0, `--max-new-tokens 4096`: | arm | GSM8K | |---|---:| | main | 0.822 | | this PR | **0.814** | Run-to-run spread on this harness is ~2pp, larger than the gap between arms. The eval runs at 128-way concurrency where this PR's fast path never fires, so it shows the build serves correctly; the unit test is what establishes bit-exactness. ## Performance Qwen3.5-397B-A17B-FP8, 4xB200, TP4, `trtllm_mha` + `flashinfer_trtllm` MoE, fp8 weights + fp8 KV cache, bs=1, input-len 8192. **-110us per decode step, 2.01% of a 5.490ms step.** 45 launches removed; the GDN linear-attention layer span drops 2.45us, x45 such layers. | | main | this PR | |---|---:|---:| | kernels per decode step | 1656 | 1611 | | GDN layer span | 77.85us | 75.39us | | full-attn layer span | 91.10us | 91.81us | --- ### CI States Latest PR Test (Base): :x: [Run #31687259332](https://github.com/sgl-project/sglang/actions/runs/31687259332) Latest PR Test (Extra): :x: [Run #31687258719](https://github.com/sgl-project/sglang/actions/runs/31687258719) ",
"url": "https://github.com/sgl-project/sglang/pull/33062",
"createdAt": "2026-07-31T11:07:21Z",
"updatedAt": "2026-08-13T13:27:22Z",
"timestamp": "2026-08-13T13:27:22Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"jit-kernel"
],
"author": "mattteochen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33068",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Fuse quantized in_proj layers in Qwen3.5",
"text": "## Motivation In Qwen3.5 GDN layers the input projection is issued as two separate GEMMs, `in_proj_qkvz` and `in_proj_ba`. Both read the same activation, so on checkpoints that quantize both to the *same* scheme they can be served by one wider GEMM. Keeping them apart costs an extra kernel launch, an extra activation-quantization pass and an extra epilogue per GDN layer per decode step — overhead that is a significant share of decode time at low-to-mid concurrency. This became actionable with the AMD Qwen3.5 MXFP4+AttnFP8 \"V2\" line, which quantizes `in_proj_ba` to FP8; on V2 all four shards resolve to one scheme and the merge is eligible. ## Modifications - **`qwen3_5.py`**: add `create_qkvzba_proj()`, which builds a single `in_proj_qkvzba` from the four shards (`qkv`, `z`, `b`, `a`) via `MergedColumnParallelLinear`, with N padded 10304 → 10368 for alignment. Gated by `SGLANG_GDN_FUSE_QKVZBA` (default **off**). The merge is attempted only when all four checkpoint shards resolve to the same quantization scheme; a mismatch raises `ValueError`, which is caught and falls back to the existing separate projections. Mixed-precision checkpoints are therefore untouched — on V1, where `in_proj_a/b` stay bf16, the model loads exactly as before. - **`triton_gdn_fused_proj.py`**: parameterize the split/reshape/cat kernel by `QKVZ_STRIDE` / `BA_STRIDE` rather than deriving the row stride from logical widths, so it can read column slices of a wider, padded projection. - No `communicator.py` change, and `_forward_input_proj_fused_quant_amd` is untouched: the merged GEMM is dispatched at the top of `_forward_input_proj`, where `_select_fused_ar_input_for_linear` handles a plain tensor and a fused-AR tuple alike, so one branch covers both entry paths. - **test**: cover the fused split/reshape/cat kernel both for dense inputs and for column slices of a wider projection, which is what the merge produces. ## Accuracy Tests gsm8k 5-shot, **full 1319** questions, `lm_eval` `local-chat-completions` with `--apply_chat_template` against `/v1/chat/completions`, server run with `--reasoning-parser qwen3` and `--default-chat-template-kwargs '{\"enable_thinking\": false}'`. Checkpoint: AMD Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 (PR#4 revision), 2× MI355X, TP=2. | `SGLANG_GDN_FUSE_QKVZBA` | strict-match | flexible-extract | | --- | --- | --- | | `0` (separate, today's behaviour) | 0.9712 ± 0.0046 | 0.9712 ± 0.0046 | | `1` (merged) | **0.9773 ± 0.0041** | **0.9765 ± 0.0042** | The merged arm is 0.6 pp higher, well inside the ±0.4–0.5 pp standard error of each measurement, i.e. the two arms are statistically indistinguishable. No accuracy regression. Note for anyone reproducing: Qwen3.5 V2 is a thinking model. Grading it on `/v1/completions` scores the raw `` text and yields a spurious ~0.82–0.83 for *both* arms; the chat + thinking-off recipe above is the one that reflects real quality. ## Speed Tests and Profiling `bench_serving`, random dataset, OSL 1024, `--random-range-ratio 0.8`, `--num-prompts 10×cc`, `request_rate=inf` with `--max-concurrency`, seed 0. Same checkpoint/hardware as above. Server: `--attention-backend aiter --kv-cache-dtype fp8_e4m3 --page-size 16 --chunked-prefill-size 8192 --disable-radix-cache --enable-aiter-allreduce-fusion --enable-dense-fp8` (effective `mem_fraction_static` 0.578, `max_running_requests` 636); the 1k regime adds `--enable-mixed-chunk`. TPOT figures are medians. **8k context (ISL 8192, OSL 1024)** | cc | prompts | TPOT separate (ms) | TPOT merged (ms) | ΔTPOT | total tok/s separate | merged | Δ | | --- | --- | --- | --- | --- | --- | --- | --- | | 4 | 40 | 8.84 | 8.60 | **−2.71%** | 3845.6 | 3954.6 | +2.83% | | 8 | 80 | 11.72 | 11.38 | **−2.90%** | 5782.4 | 5964.7 | +3.15% | | 16 | 160 | 16.57 | 16.15 | **−2.53%** | 8299.6 | 8511.3 | +2.55% | | 32 | 320 | 25.34 | 24.78 | **−2.21%** | 10930.6 | 11136.7 | +1.89% | | 64 | 640 | 40.49 | 39.92 | **−1.41%** | 13903.8 | 14095.0 | +1.38% | **1k context (ISL 1024, OSL 1024)** | cc | prompts | TPOT separate (ms) | TPOT merged (ms) | ΔTPOT | total tok/s separate | merged | Δ | | --- | --- | --- | --- | --- | --- | --- | --- | | 4 | 40 | 8.06 | 7.85 | **−2.61%** | 950.1 | 976.2 | +2.75% | | 8 | 80 | 10.11 | 9.80 | **−3.07%** | 1532.4 | 1579.4 | +3.07% | | 16 | 160 | 13.06 | 12.72 | **−2.60%** | 2409.1 | 2475.6 | +2.76% | | 32 | 320 | 17.91 | 17.52 | **−2.18%** | 3498.6 | 3571.2 | +2.07% | | 64 | 640 | 24.81 | 24.51 | **−1.21%** | 5065.3 | 5132.7 | +1.33% | All 20 cells (2 regimes × 5 concurrencies × 2 arms) favour the merged arm on both TPOT and throughput; tok/s columns are total input+output throughput, and output-only moves by the same percentage. The gain is largest at low-to-mid concurrency, where the per-GEMM launch and epilogue overhead removed by the merge is the biggest share of decode time, and shrinks as concurrency grows and the GEMMs become compute-bound. An independent earlier repeat of this sweep reproduced the same deltas within a few tenths of a percent. ### Absolute numbers vs the previous V2 baseline For context beyond the A/B, the same sweep against our July 28 V2 baseline (aiter #4017+#3986 on the published `Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2`), run with an identical harness and identical server settings. **Medians**, matching how that baseline was recorded: | ISL | cc | out tok/s/gpu | tok/s | TTFT (ms) | TPOT (ms) | ITL (ms) | | --- | --- | --- | --- | --- | --- | --- | | 1024 | 4 | 234 → **243** (+3.8%) | 940 → **976** | 95.9 → **92.2** | 8.1 → **7.8** | 7.9 → **7.6** | | 1024 | 8 | 383 → **396** (+3.5%) | 1527 → **1579** | 99.0 → **95.6** | 10.1 → **9.8** | 9.6 → **9.3** | | 1024 | 16 | 596 → **616** (+3.3%) | 2398 → **2476** | 103.4 → **100.3** | 13.1 → **12.7** | 12.0 → **11.7** | | 1024 | 32 | 866 → **894** (+3.3%) | 3459 → **3571** | 112.0 → **106.9** | 18.1 → **17.5** | 16.0 → **15.6** | | 1024 | 64 | 1240 → **1283** (+3.5%) | 4962 → **5133** | 144.7 → **122.2** | 25.1 → **24.5** | 21.1 → **20.9** | | 8192 | 4 | 214 → **220** (+2.8%) | 3840 → **3955** | 293.8 → **291.4** | 8.9 → **8.6** | 8.0 → **7.7** | | 8192 | 8 | 326 → **335** (+2.8%) | 5804 → **5965** | 294.4 → **287.8** | 11.7 → **11.4** | 9.7 → **9.4** | | 8192 | 16 | 460 → **472** (+2.5%) | 8294 → **8511** | 308.4 → **300.4** | 16.6 → **16.1** | 12.2 → **11.9** | | 8192 | 32 | 612 → **623** (+1.7%) | 10942 → **11137** | 320.4 → **314.4** | 25.2 → **24.8** | 16.2 → **16.0** | | 8192 | 64 | 775 → **782** (+0.9%) | 13970 → **14095** | 343.9 → **337.4** | 40.2 → **39.9** | 21.9 → **21.7** | Every cell improves on every metric. Note these are medians; under `request_rate=inf` the initial burst gives mean TTFT a long tail (at 8k/cc64 the baseline's mean TTFT is 1358 ms against a median of 344 ms), so mean and median must not be compared across runs. **Merge engagement was verified, not assumed.** Nothing logs the merge, so both harnesses use the KV-cache token count as a signature: the padded merged projection carries ~12 MB more weights per rank, so at a fixed `mem-fraction-static` the merged arm has slightly less room for KV. The perf sweep reported 4,505,008 → 4,495,728 tokens and the accuracy run 11,774,352 → 11,765,072 — the same −9,280 delta in two independently configured runs. Identical counts would have meant the flag never took effect. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31711897904](https://github.com/sgl-project/sglang/actions/runs/31711897904) Latest PR Test (Extra): :x: [Run #31711897564](https://github.com/sgl-project/sglang/actions/runs/31711897564) ",
"url": "https://github.com/sgl-project/sglang/pull/33068",
"createdAt": "2026-07-31T11:56:29Z",
"updatedAt": "2026-08-13T14:48:20Z",
"timestamp": "2026-08-13T14:48:20Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"amd",
"run-ci",
"jit-kernel"
],
"author": "mqhc2020",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33165",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] DeepSeek-V4 MI355X: eliminate bpreshuffle fp8-scale relayout copy in dense w8a8 linear",
"text": "## Summary On MI355X (gfx950) the CK bpreshuffle w8a8 blockscale GEMM consumes the per-group activation scale in **column-major** `[num_groups, tokens]` layout. `aiter_w8a8_block_fp8_linear` quantizes the activation row-major and then relays the scale out with `materialize_bpreshuffle_fp8_scale` = `.t().contiguous().t()` — a real relayout **copy per dense w8a8 GEMM** (the MLA q/kv/o projections and MoE), one of the larger ELEWISE deltas in the DeepSeek-V4 MI355X decode breakdown. `aiter_per1x128_quant` already supports `transpose_scale=True`, which emits the exact `[num_groups, tokens]` byte-order directly. This PR asks for that and reinterprets the strides with a zero-copy view, removing the copy on the fresh-quant path. ## What changed - **`python/sglang/srt/layers/quantization/fp8_utils.py`** — in the fresh-quant branch of `aiter_w8a8_block_fp8_linear`, pass `transpose_scale=emit_bpreshuffle_scale` to `aiter_per1x128_quant` and consume the result via the shared `view_aiter_fused_rms_transposed_fp8_scale` helper (the `as_strided((1, M))` metadata repair that landed on main in #31727) instead of `materialize_bpreshuffle_fp8_scale`. Guarded `M >= 2`; the M==1 single-row case keeps the materialize path, which for a contiguous `[1, G]` input is a **no-op view** anyway (the `[1, G]` and `[G, 1]` byte orders coincide, so `.t().contiguous().t()` shares storage and launches no copy kernel). Bit-identical output; removes one relayout copy on the M>=2 path. The pre-existing Triton row-major-view branch is routed through the same helper too, so the transposed-scale repair lives in one documented place rather than as duplicated inline `as_strided` expressions. ## Tests - **CPU** (`test_fp8_bpreshuffle_scale.py`) — the shared-helper stride/no-copy contract, and the `M == 1` materialize fallback: values intact on the natural `(G, 1)` stride (the no-op-view layout, not `(1, 1)`), sharing storage. - **MI35X GPU** (`test_fp8_bpreshuffle_dense_linear_mi35x.py`, `stage-b-test-1-gpu-small-amd-mi35x`) — exercises the **real** kernels: (1) `aiter_per1x128_quant` with `transpose_scale` True vs False, asserting identical quantized bytes, equal scale values, the `(1, M)` stride (M>=2) / `(G, 1)` no-op (M==1), and zero-copy sharing; (2) end-to-end `aiter_w8a8_block_fp8_linear` new path vs the original path (forced by patching quant→row-major + relayout→materialize), asserting **bit-identical** GEMM output over M ∈ {1, 2, 8, 16} (M==1 exercises the materialize fallback), and **spying the GEMM dispatch to assert the CK `gemm_a8w8_blockscale_bpreshuffle` path is taken (not Triton)** so a future tuned-shape-list change can't silently void the coverage. This is the direct real-path evidence the fabricated-layout CPU test could not provide. ## Validation Model: `deepseek-ai/DeepSeek-V4-Pro`. Hardware: MI355X (gfx950), TP8. Image: `rocm/sgl-dev:v0.5.16-rocm720-mi35x-20260731`. Baseline = image tree; Patched = image tree + this patch. Env: `SGLANG_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton`. **Correctness:** GEMM output bit-identical vs the materialize path (`gemm_a8w8_blockscale_bpreshuffle`, max-abs-diff 0.0). ### Accuracy (GSM8K, 1319 questions × 2 runs) | run | baseline | patched | |----:|:--------:|:-------:| | 1 | 0.942 | 0.941 | | 2 | 0.939 | 0.947 | Accuracy preserved (within run-to-run noise); the bit-identical correctness check above is the primary guarantee. ### Perf (random 8k/1k, `--random-range-ratio 1.0`, seed 42, num-prompts = conc×10) — preliminary **Single-shot, one run per arm — preliminary, not an established speedup.** The Δ below (≈ 0.9–2.0%) is not yet backed by repeated A/B runs, so I'm presenting it as directional rather than a confirmed win; the trend (larger at lower concurrency, i.e. more launch/latency-bound) is consistent with removing a per-GEMM relayout copy but isn't proven by these numbers alone. The correctness/no-regression guarantee is the **bit-identical GEMM output** check above. Happy to add repeated-run A/B (mean/p50/variance) if a throughput claim is wanted. | conc | baseline TPOT (ms) | patched TPOT (ms) | ΔTPOT | baseline total tok/s | patched total tok/s | Δtot | |-----:|:------------------:|:-----------------:|:-----:|:--------------------:|:-------------------:|:----:| | 16 | 19.48 | 19.10 | −2.0% | 6500.2 | 6613.0 | +1.7% | | 32 | 24.63 | 24.22 | −1.7% | 9685.3 | 9821.3 | +1.4% | | 64 | 35.03 | 34.60 | −1.2% | 12819.0 | 12935.7 | +0.9% | Coherent, zero dropped requests. --- ### CI States Latest PR Test (Base): :x: [Run #31675718292](https://github.com/sgl-project/sglang/actions/runs/31675718292) Latest PR Test (Extra): :x: [Run #31675718006](https://github.com/sgl-project/sglang/actions/runs/31675718006) ",
"url": "https://github.com/sgl-project/sglang/pull/33165",
"createdAt": "2026-08-01T02:32:54Z",
"updatedAt": "2026-08-13T09:35:06Z",
"timestamp": "2026-08-13T09:35:06Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"amd",
"deepseek",
"run-ci"
],
"author": "karverma-amd",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33166",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] DeepSeek-V4 MI355X: eliminate bpreshuffle fp8-scale copies at producer sites (MoE down, MLA o_proj bmm)",
"text": "## Summary Follow-up to the dense-linear bpreshuffle scale no-copy. Several DeepSeek-V4 sites **pre-quantize** an activation and hand a `(fp8, scale)` tuple to a downstream Linear; those scales are emitted row-major and then relaid out with `materialize_bpreshuffle_fp8_scale` — a relayout **copy per site, per layer** on MI355X (gfx950). This PR eliminates those copies for the producer quant kernels that honor `transpose_scale`, via a zero-copy `torch.as_strided` reinterpret. ## What changed - **`python/sglang/srt/layers/quantization/fp8_utils.py`** — one producer-neutral `view_aiter_fused_rms_transposed_fp8_scale` (+ `_tuple`) helper: the zero-copy stride reinterpret of a scale a quant kernel already emitted with `transpose_scale=True`. This **consolidates** the identical `view_aiter_fused_rms_transposed_fp8_scale` that landed on main in #31727 with this PR's producer helper (both were `as_strided(scale, scale.shape, (1, scale.shape[0]))`) into a single operation, keeping the established #31727 name for consistency. Also adds `emit_transposed_bpreshuffle_scale(m, on_bpreshuffle_gfx95=...)`, centralizing the shared `M >= 2` emit gate. - **`python/sglang/srt/models/deepseek_v4.py`** — fused-RMS producer repointed to the consolidated helper. - **`python/sglang/srt/models/deepseek_v2.py`** — MLP down-proj input producer (`fused_clamp_act_mul`): request `transpose_scale=True` and reinterpret instead of copy (gated `M>=2`). - **`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_rocm.py`** — o_proj bmm producers ×2 in `rocm_absorb_v_bmm` (`fused_flatten_fp8_group_quant`): same treatment. (ROCm MLA now dispatches here after the recent `forward_mla_rocm.py` split; the optimization was moved off `forward_mla.py`, which is back to matching upstream, so it lands on the actual MI355X call path.) Sites fed by `fused_rms_fp8_group_quant` follow the same helper now (the #31727 fix made that kernel honor `transpose_scale`). ## Tests - **CPU** (`test_fp8_bpreshuffle_scale.py`) — helper-level stride/no-copy contract, and the centralized `emit_transposed_bpreshuffle_scale` gate incl. the `M == 1` materialize fallback. - **MI35X GPU** (`test_fp8_bpreshuffle_producer_mi35x.py`, `stage-b-test-1-gpu-small-amd-mi35x`) — invokes **both real producers** (`fused_clamp_act_mul`, `fused_flatten_fp8_group_quant`) with `transpose_scale=True` + no-copy vs `transpose_scale=False` + materialize, asserting bit-identical quantized output, equal scale values, the `(1, M)` column-major stride, and zero-copy storage sharing (M ∈ {1, 2, 8, 16}; M==1 exercises the materialize fallback). This is the direct producer-level equivalence evidence — the invariant the optimization relies on, checked on the real kernels rather than a fabricated layout. ## Validation Model: `deepseek-ai/DeepSeek-V4-Pro`. Hardware: MI355X (gfx950), TP8. Image: `rocm/sgl-dev:v0.5.16-rocm720-mi35x-20260731`. Env: `SGLANG_USE_AITER=1 AITER_BF16_FP8_MOE_BOUND=0 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton`. ### Accuracy (GSM8K, 1319 questions × 2 runs) | run | baseline | patched | |----:|:--------:|:-------:| | 1 | 0.942 | 0.933 | | 2 | 0.939 | 0.931 | The patched runs sit ~0.8–0.9 pt below baseline. With only two runs per arm this is consistent with, but does not by itself establish, DeepSeek-V4 fp4-MoE run-to-run noise. The correctness guarantee for this PR comes instead from the **producer-level bit-exact test** above: the converted kernels emit identical quantized output and scale values on both paths, so the reroute cannot change the model result. (Happy to add more GSM8K reps if useful.) ### Perf (random 8k/1k, `--random-range-ratio 1.0`, seed 42) — preliminary **Single-shot, one run per arm — treat as preliminary, not an established speedup.** The Δ below (≈ ±0.4–0.8%) is within DeepSeek-V4 decode run-to-run noise at these concurrencies and is not backed by repeated A/B runs, so I'm not claiming it as a confirmed win. The correctness/no-regression guarantee for this PR is the **producer-level bit-exact test** above (identical quantized output + scale values on both paths); the value here is eliminating a per-site/per-layer relayout copy, which these numbers are consistent with but do not by themselves prove. I can add repeated-run A/B (mean/p50/variance) if a throughput claim is wanted. | conc | baseline TPOT (ms) | patched TPOT (ms) | ΔTPOT | baseline total tok/s | patched total tok/s | Δtot | |-----:|:------------------:|:-----------------:|:-----:|:--------------------:|:-------------------:|:----:| | 16 | 19.48 | 19.32 | −0.8% | 6500.2 | 6549.1 | +0.8% | | 32 | 24.63 | 24.50 | −0.5% | 9685.3 | 9728.3 | +0.4% | | 64 | 35.03 | 34.79 | −0.7% | 12819.0 | 12898.1 | +0.6% | Measured under pure TP8. (No DP-attention measurement yet — the earlier \"larger under DP-attention\" note is removed until that's actually measured.) Coherent, zero dropped requests. Applies on top of the dense-linear PR (different `fp8_utils` region) or standalone. --- ### CI States Latest PR Test (Base): :x: [Run #31675631374](https://github.com/sgl-project/sglang/actions/runs/31675631374) Latest PR Test (Extra): :x: [Run #31675631181](https://github.com/sgl-project/sglang/actions/runs/31675631181) ",
"url": "https://github.com/sgl-project/sglang/pull/33166",
"createdAt": "2026-08-01T02:33:04Z",
"updatedAt": "2026-08-13T09:17:19Z",
"timestamp": "2026-08-13T09:17:19Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"amd",
"deepseek",
"run-ci"
],
"author": "karverma-amd",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33198",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Fix causal sliding-window attention in MiMo audio tokenizer",
"text": "## Motivation MiMo-V2.5 declares its audio tokenizer encoder as causal and uses hybrid attention: causal sliding-window attention `(128, 0)` alternating with causal full attention. After the TP-aware `VisionAttention` refactor in [#31343](https://github.com/sgl-project/sglang/pull/31343), `AudioEncoderAttention` continued storing `causal` and `window_size`, but these settings were not forwarded to the selected attention backend: - attention remained non-causal; - sliding-window layers were executed as full attention because `VisionAttention` only enabled windowing for attention-sink layers. This allowed audio prefix representations to depend on future audio frames. Consequently, encoding a short audio clip produced different codec tokens from encoding the same clip as the prefix of a longer audio sequence, severely degrading long-audio ASR quality. ## Modifications - Add optional causal-attention support to `VisionAttention`, keeping `causal=False` as the default to preserve existing vision-model behavior. - Preserve causal sliding-window attention independently of attention sinks. - Forward causal/window settings through supported multimodal attention backends: - FA3 - FA4 - SDPA - Triton - FlashInfer cuDNN - AITER - AMX - Intel XPU - Forward `window_size`, `causal`, and attention sinks through FA4 for parity with FA3. - Raise explicit errors for backend/window combinations that are not supported instead of silently executing incorrect full attention. - Configure MiMo `AudioEncoderAttention` with the checkpoint's causal setting. - Add CPU regression tests covering: - MiMo causal/window configuration; - FA3 and FA4 argument forwarding; - FA4 attention sinks; - Triton causal forwarding; - SDPA causal sliding-window behavior. ## Accuracy Tests A causal prefix-invariance test compares codec tokens produced from: - a 6-second audio clip; - the same 6-second prefix within a 15-second audio clip. The last four codec rows are excluded to account for convolution boundary effects. Before this fix: - all-channel codec equality: `6.9048%` - complete codec-row equality: `0.0000%` - first-quantizer equality: `19.0476%` After this fix: - all-channel codec equality: `100.0000%` - complete codec-row equality: `100.0000%` - first-quantizer equality: `100.0000%` The causal/window-aware reference implementation also produces `100.0000%` equality. Additional validation: - MiMo audio attention unit test: passed - FA3/FA4/Triton/SDPA targeted unit tests: passed - all pre-commit hooks: passed ## Speed Tests and Profiling No dedicated performance benchmark was run. This change does not introduce a new kernel. It forwards the intended causal/window settings through the existing attention backends. Sliding-window layers attend to fewer positions than the previously incorrect full-attention execution, so no performance regression is expected for those layers. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (N/A: no user-facing API or configuration changes.) - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31683807164](https://github.com/sgl-project/sglang/actions/runs/31683807164) Latest PR Test (Extra): :x: [Run #31683806987](https://github.com/sgl-project/sglang/actions/runs/31683806987) ",
"url": "https://github.com/sgl-project/sglang/pull/33198",
"createdAt": "2026-08-01T11:31:14Z",
"updatedAt": "2026-08-13T08:50:34Z",
"timestamp": "2026-08-13T08:50:34Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"Multi-modal"
],
"author": "CJack812",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33216",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "kernel: port CUTLASS fp8_scaled_mm to JIT and expand SM120 M tiles",
"text": "## Motivation `fp8_scaled_mm` is the last large CUTLASS GEMM still in the AOT wheel — the blockwise FP8 GEMM already moved to `sglang.kernels.jit`. This brings the per-row/per-column one across too. While porting: SM120 ran a single `128x128x128` tile for every M, which wastes most of a 128-row tile on decode-shaped batches. It now buckets on M. ## Modifications The AOT kernel compiled SM89/SM90/SM100/SM120 into one wheel and picked between them at runtime with `getSMVersion()`. A JIT module is built for the device it runs on, so the entry header branches on the `SGL_CUDA_ARCH` the JIT build already defines; only that arch's templates are instantiated, and an unsupported arch is a compile error. - `kernels/jit/csrc/gemm/fp8_per_tensor/` — the four arch paths plus the bias epilogue. SM100 and SM120 share one gemm struct; SM90 keeps its tuned swap-AB table verbatim; SM89 stays separate as CUTLASS 2.x. - `kernels/ops/gemm/fp8_per_tensor_gemm.py` — wrapper, arch resolution, `torch.compile`-safe custom op. - `fp8_utils.py` — `apply_fp8_linear` routes here. - Deleted `csrc/gemm/fp8_gemm_kernel.cu`, its SM90 dispatch header, the torch binding, the CMake entry and the `sgl_kernel` export (~1.7k lines). SM120 now buckets on M (16 / 32 / 256 / default). The sub-128 buckets run pingpong because the cooperative kernel requires tile M >= 128, and the two smallest need an explicit `EpilogueTile` since `EpilogueTileAuto` sizes for the 128-wide tiles. ## Accuracy Checked against an fp32 reference across both output dtypes, with and without bias, on both sides of every bucket edge, plus the SM90 swap-AB shape list carried over from the AOT test. Also diffed against the AOT kernel over 300 shape/dtype/bias combinations — never materially further from the reference. SM89/SM90/SM100 are compile-verified only; I have no Ada/Hopper/B200 to run them on. ## Benchmarks RTX PRO 6000 Blackwell (SM120), cold-L2 CUPTI under CUDA graph, over FP8 attention-projection shapes of two LLM checkpoints (hidden 6656 and 5120) at TP1 and TP2. **vs the AOT kernel this replaces**, geomean over 14 shapes: | M | 1 | 8 | 16 | 32 | 64 | 128 | 256 | >256 | |---|---|---|---|---|---|---|---|---| | speedup | 1.77x | 1.71x | 1.64x | 1.59x | 1.47x | 1.35x | 1.26x | 1.00x | 1.55x geomean for M<=256, 1.00x above it where the dispatch falls through to the same tile as before. Faster on 225/280 points; the three regressions are 4-6% at exactly M=256. **vs autotuned `flashinfer.bmm_fp8`** — 0.97x geomean overall, 1.04x for M in 17..256. Note `bmm_fp8` takes scalar scales while this kernel reads `[M]`/`[N]` scale vectors, so the comparison is mildly unfavourable to CUTLASS but matches the real `apply_fp8_linear` path. Narrow N (128/256) is the weak spot at 0.62-0.68x: `ceil(N/64)` gives only 2-4 CTAs against 188 SMs. Tile N=16 does not build, tile N=32 helps there but regresses wider N, and split-K is unreachable since sub-128 tile M forces pingpong which rejects the stream-K scheduler. Left as-is and documented in the source. ## Checklist - [x] Format with `pre-commit run --all-files` - [x] Accuracy tests under `test/registered/jit/` --- ### CI States Latest PR Test (Base): :x: [Run #31699327508](https://github.com/sgl-project/sglang/actions/runs/31699327508) Latest PR Test (Extra): :x: [Run #31699327316](https://github.com/sgl-project/sglang/actions/runs/31699327316) ",
"url": "https://github.com/sgl-project/sglang/pull/33216",
"createdAt": "2026-08-01T18:19:21Z",
"updatedAt": "2026-08-13T14:00:35Z",
"timestamp": "2026-08-13T14:00:35Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"sgl-kernel",
"run-ci",
"jit-kernel"
],
"author": "b8zhong",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33249",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MoonEP] BF16 PoC integration for Kimi-K3",
"text": "It is opened as a draft for partial task of #32607's Integrate MoonEP, not as a final production-ready merge request. ### What is included - Distinct `moonep` MoE A2A backend recognition. - MoonEP dispatch/combine data contracts and process-wide buffer facade. - BF16 contiguous expert layout and correctness-first expert segment runner. - Eager runtime path: dispatch -> weight prefetch -> BF16 compute -> combine. - Distributed validation script: `scripts/moonep/validate_moonep_bf16_poc.py`. ### Validation - 4x H100 SXM (`NV6`) was insufficient: CUDA multicast unsupported. - 8x H100 SXM (`NV18`) passed BF16 validation on 8 ranks with `global_ok=true`, `max_abs_err=0.0`, `relative_err=0.0`. - `docker-ncu-benchmark` preflight was run. Suitable Vast.ai H100 NVSwitch hosts were container instances without Docker/NVIDIA Container Toolkit; host NCU counters were restricted (`ERR_NVGPUCTRPERM`), so no NCU performance claims are made. ### Known limitations - BF16/unquantized only. - Eager-only; CUDA graph disabled. - Correctness-first Python segment loop, not final grouped GEMM kernel. - PoC global expert-row storage is memory-heavy; Kimi-K3 production needs sharded symmetric-memory expert ownership. - Requires NVSwitch/multicast-capable hosts (8x H100/H200/B200 class). Multi-RTX 5090 without NVLink/multicast is only useful for build/Docker smoke. ### Notes Local agent workflow files, scratch artifacts, and spec notes are intentionally omitted from this upstream PR diff. Fork PoC PR: wirybeaver/sglang#17 --- ### CI States Latest PR Test (Base): :x: [Run #31653766427](https://github.com/sgl-project/sglang/actions/runs/31653766427) Latest PR Test (Extra): :x: [Run #31653766396](https://github.com/sgl-project/sglang/actions/runs/31653766396) ",
"url": "https://github.com/sgl-project/sglang/pull/33249",
"createdAt": "2026-08-02T05:55:08Z",
"updatedAt": "2026-08-13T00:15:29Z",
"timestamp": "2026-08-13T00:15:29Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"documentation"
],
"author": "wirybeaver",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33290",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD]Fuse Q/K L2 normalization in Qwen3.5 GDN",
"text": "Reduce GDN prefill launches for benchmarked BF16/FP32 GPU shapes while preserving the original path on unsupported platforms, dtypes, layouts, and launch-bound wide shapes. ## Motivation The Triton GDN prefill path normalizes Q and K with two separate `l2norm_fwd` launches. Qwen3.5 GDN uses equal Q/K head layouts, so both normalizations can be computed in one Triton launch. This change reduces GDN prefill launch overhead on AMD GPUs while preserving the existing two-kernel implementation as the fallback. ## Modifications - Add `fused_l2norm_qk` to normalize compatible Q/K tensors in one Triton launch. - Preserve runtime token-count behavior with `do_not_specialize` to avoid recompilation by sequence length. - Enable automatic fused routing only when: - the platform is AMD; - execution is in the GDN `chunk_gated_delta_rule` path; - Q/K are GPU BF16/FP32 tensors with identical shape and contiguous layout. - Preserve two existing `l2norm_fwd` calls for: - non-AMD platforms; - asymmetric Q/K row layouts; - non-contiguous or otherwise incompatible tensors. - unbenchmarked head dimensions larger than 512; - the launch-bound `head_dim=512, rows<32` corner. - Add AMD/CUDA correctness tests, explicit fused/fallback dispatch tests, and a focused benchmark. No KDA, Mamba, full-attention, CuTe DSL, FlashInfer, CPU, NPU, or XPU path is changed. ## Accuracy Tests ```bash PYTHONPATH=python python3 -m pytest -q \\ test/registered/kernels/ops/attention/test_fused_gdn_l2norm.py \\ test/registered/attention/test_chunk_gated_delta_rule.py ``` ## Speed Tests and Profiling Representative equal-head Qwen3.5 GDN shapes (`head_dim=128`): | Tokens | Local heads | Fused (us) | Separate (us) | Latency reduction | |---:|---:|---:|---:|---:| | 16 | 2 | 3.34 | 5.32 | 37.2% | | 16 | 8 | 3.41 | 5.30 | 35.7% | | 257 | 8 | 4.87 | 6.50 | 25.1% | | 1024 | 2 | 4.94 | 6.49 | 23.9% | | 1024 | 8 | 10.73 | 13.35 | 19.6% | | 1024 | 16 | 17.67 | 21.02 | 16.0% | Across the tested Qwen3.5 equal-head `D=128` cases, fused normalization reduced median latency by approximately **16–38%**. Expanded BF16 sweep: - tokens: 1–16384; - local heads: 1–64; - head dimensions: 64/128/256/512; - 135 total shape combinations. | Head dim | Auto-enabled shapes | Minimum speedup | Median speedup | Maximum speedup | |---:|---:|---:|---:|---:| | 64 | 27 | 1.13× | 1.19× | 1.48× | | 128 | 63 | 1.11× | 1.15× | 1.48× | | 256 | 27 | 1.06× | 1.14× | 1.52× | | 512 (`rows>=32`) | 14 | 1.03× | 1.09× | 1.14× | | Model | Parallelism | Scenario | ISL | OSL | Concurrency | Requests per cycle | Metric | Paired mean improvement | |---|---:|---|---:|---:|---:|---:|---|---:| | 27B | TP1 | Short-prefill online latency | 256 | 8 | 1 | 64 | Mean TTFT | +0.79% | | 27B | TP1 | Long-prefill online latency | 8192 | 8 | 1 | 32 | Mean TTFT | +0.016% | | 27B | TP1 | Concurrent serving throughput | 4096 | 128 | 64 | 64 | Input throughput | +0.19% | | 397B | TP8 | Short-prefill online latency | 256 | 8 | 1 | 64 | Mean TTFT | +0.99% | | 397B | TP8 | Long-prefill online latency | 8192 | 8 | 1 | 64 | Mean TTFT | +0.048% | | 397B | TP8 | Concurrent serving throughput | 4096 | 128 | 64 | 64 | Input throughput | +0.013% | ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31572323599](https://github.com/sgl-project/sglang/actions/runs/31572323599) Latest PR Test (Extra): :x: [Run #31572323444](https://github.com/sgl-project/sglang/actions/runs/31572323444) ",
"url": "https://github.com/sgl-project/sglang/pull/33290",
"createdAt": "2026-08-02T17:37:50Z",
"updatedAt": "2026-08-13T01:20:04Z",
"timestamp": "2026-08-13T01:20:04Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "IzacharyI",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33301",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Enable DSpark speculative decoding on ROCm",
"text": "## Motivation DSpark block-diffusion speculative decoding was silently falling back to greedy decoding on AMD ROCm. The sampler kernels (`top_k_renorm_prob`, `top_k_top_p_sampling`, etc.) were only imported when `is_cuda() or is_musa()`, so on ROCm the import was skipped and DSpark produced greedy output with no error or warning. ## Modifications `python/sglang/srt/speculative/dflash_utils.py`: Add `is_hip()` to the sampler kernel import guard so the kernels are loaded on ROCm. ## Accuracy Tests DSpark accept rate and output quality on Kimi-K3 verified correct on MI355X (gfx950) after this fix. Previously accept rate was 0% (greedy fallback); after fix matches CUDA behavior with expected accept rates. ## Speed Tests and Profiling N/A — correctness fix. DSpark throughput gains (~3x) now apply on ROCm identically to CUDA. ## Repro Steps **Hardware:** AMD MI355X (gfx950), ROCm 7.x **Unit test (verifies sgl_kernel ops available on ROCm):** ```bash python test/manual/test_amd_dspark_rocm.py ``` **Verify DSpark is active (non-zero accept rate) on ROCm:** ```bash python -m sglang.launch_server \\ --model --tp 8 \\ --speculative-algorithm dspark \\ --speculative-dspark-block-size 7 \\ --trust-remote-code & sleep 60 # wait for server # Before this fix: DSpark accept rate = 0% (greedy) # After this fix: accept rate > 0%, matching CUDA behavior # Check server logs for accept_rate ``` ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #30775063151](https://github.com/sgl-project/sglang/actions/runs/30775063151) Latest PR Test (Extra): :x: [Run #30775063087](https://github.com/sgl-project/sglang/actions/runs/30775063087) ",
"url": "https://github.com/sgl-project/sglang/pull/33301",
"createdAt": "2026-08-03T00:41:17Z",
"updatedAt": "2026-08-12T17:35:48Z",
"timestamp": "2026-08-12T17:35:48Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [],
"author": "nehaprakriya",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33313",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] DeepSeek-V4: route decode wo_a bf16 batched matmul to aiter batched_gemm_bf16",
"text": "# [AMD] DeepSeek-V4: route decode wo_a bf16 batched matmul to aiter `batched_gemm_bf16` ## Summary On the DeepSeek-V4 ROCm decode path, the MLA output-absorb (`wo_a`) bf16 GEMM runs `torch.einsum(\"tgd,grd->tgr\", o, wo_a)`, which dispatches to a **rocBLAS/Tensile `Cijk_*` batched GEMM** — the single largest kernel in the DSV4 decode attention region (~30 µs/step at TP8). aiter ships a tuned `batched_gemm_bf16` for exactly this shape, and the reference ATOM stack uses it for the same op (~25 µs). This PR routes the einsum to `batched_gemm_bf16` on gfx95, behind a new opt-in env flag, with an einsum fallback. ## Motivation Attributing the DSV4 decode attention region kernel-by-kernel (torch-profiler External-id + enclosing python stack) shows the `Cijk_Alik_Bljk_*_MT16x16x1024` Tensile kernel is launched by `aten::bmm` from the `wo_a` bf16 einsum. The same logical op in the reference stack is aiter's `_batched_gemm_bf16_kernel`. Routing sglang to that kernel closes an attention-region gap and matches the reference kernel choice. ## What changed - `deepseek_v4.py`: new module helper `_apply_wo_a_bf16_matmul(o, wo_a, is_decode)`. When `SGLANG_OPT_USE_AITER_BATCHED_GEMM` is set, the global `SGLANG_USE_AITER` switch is on, the platform is HIP/gfx95, **and the forward is decode**, it calls aiter `batched_gemm_bf16` (`Y[i] = X[i] @ W[i]^T`): `XQ = o.transpose(0,1)` `[G,T,D]`, `WQ = wo_a` `[G,R,D]` → `[G,T,R]` → transpose back to `[T,G,R]`. Otherwise (or on any failure) it falls back to the original einsum. The `wo_a` else-branch now calls this helper instead of `torch.einsum` inline, passing `forward_batch.forward_mode.is_decode()`. - Restricted to **decode** — the kernel was benchmarked/validated for decode; prefill keeps the einsum. - Gated on the global `SGLANG_USE_AITER` switch, so the reroute cannot activate when aiter is disabled even if the opt-in flag is set. - **One-shot fallback** — the first aiter import/kernel failure disables the reroute for the process and logs once, instead of retrying and re-logging on every layer/token on the decode critical path. - `environ.py`: new `SGLANG_OPT_USE_AITER_BATCHED_GEMM = EnvBool(False)`. No behavior change unless the flag is set. CUDA, non-gfx95, and prefill paths are untouched. ## Validation (MI355X, DeepSeek-V4-Pro, TP8) **Numerics** — model-free bit-check of `batched_gemm_bf16` vs the einsum across `T/G/D/R` shapes: max relative error ≤ 5e-4 (bf16 rounding; mostly exactly 0.0). **Accuracy** — 5-shot GSM8K, 1319 questions, `--parallel 1319`, cuda-graph on: einsum (flag off) 0.952 / aiter (flag on) 0.946 — same 0.946–0.952 fp4-MoE run-to-run band, no regression. **Coherence** — server generates correctly with the flag on (`capital of France is` → \"Paris. The capital of Germany is Berlin.\"). **Kernel swap (trace-confirmed)** — decode profile, flag off vs on: - off: `Cijk_Alik_Bljk_*_MT16x16x1024` 2.04 ms ×244 - on: `_batched_gemm_bf16_kernel_*` 2.07 ms ×244 (aiter), `Cijk` drops 248→4 launches **Profiling** Before After **Throughput** (pure-TP eager, output tok/s, single-shot): | conc | baseline (einsum) | +flag (aiter) | Δ | |------|-------------------|---------------|-----| | 4 | 31.5 | 31.2 | −1% (noise) | | 32 | 257.3 | 266.2 | **+3.5%** | | 64 | 516.2 | 532.2 | **+3.1%** | The win scales with decode batch (attention on the critical path in pure-TP); under DP-attention / high concurrency it is expected to be ~neutral (comm-bound). ## How to enable ``` SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 ``` Requires `SGLANG_USE_AITER=1` on gfx95 (MI355X). Applies to the decode path only. ## Checklist - [x] Numerically validated (bf16-equivalent vs einsum) - [x] Coherent end-to-end on MI355X TP8 - [x] GSM8K accuracy preserved - [x] Opt-in (default off); CUDA / non-gfx95 / prefill unchanged - [ ] Multi-rep + DP-attention A/B (single-shot pure-TP numbers above) --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31552829437](https://github.com/sgl-project/sglang/actions/runs/31552829437) Latest PR Test (Extra): :x: [Run #31674829342](https://github.com/sgl-project/sglang/actions/runs/31674829342) ",
"url": "https://github.com/sgl-project/sglang/pull/33313",
"createdAt": "2026-08-03T02:27:25Z",
"updatedAt": "2026-08-13T06:43:01Z",
"timestamp": "2026-08-13T06:43:01Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"amd",
"deepseek",
"run-ci"
],
"author": "karverma-amd",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33316",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "docs: note OpenAI client base_url for multi-model gateways",
"text": "## Summary The OpenAI client docs already show pointing `base_url` at a local SGLang server. This PR adds a short tip that the same `base_url` pattern works with any OpenAI-compatible multi-model gateway when you are not self-hosting SGLang, using [DaoXE](https://daoxe.com) (`https://api.daoxe.com/v1`) as one concrete example. Docs only — no runtime behavior changes. Updated: - Merged with `main`, which moved this file from `docs_new/docs/` to `docs/docs/`. The branch is now up to date and conflict-free. - Removed the `utm_*` tracking parameters that were previously on the example link; it is now a plain `https://daoxe.com`. ## Test plan - [x] No conflicts with `main` - [x] Tip landed in the renamed `docs/docs/basic_usage/` path - [x] Local SGLang example unchanged - [x] Link contains no tracking parameters - [x] Tip is clearly optional --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31626133904](https://github.com/sgl-project/sglang/actions/runs/31626133904) Latest PR Test (Extra): :x: [Run #31626133648](https://github.com/sgl-project/sglang/actions/runs/31626133648) ",
"url": "https://github.com/sgl-project/sglang/pull/33316",
"createdAt": "2026-08-03T02:35:49Z",
"updatedAt": "2026-08-12T18:10:43Z",
"timestamp": "2026-08-12T18:10:43Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"documentation"
],
"author": "seven7763",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33318",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Use SYCL kernels for topk_transform_512 on XPU",
"text": "## Motivation Route `topk_transform_512` and `topk_transform_512_v2` to the native SYCL kernels on XPU. The SYCL implementations are added in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) and exposed as `torch.ops.sgl_kernel.topk_transform_512{,_v2}`. Without this dispatch, XPU runs fall through to the JIT CUDA path and fail to load, or drop to a significantly slower vectorized PyTorch fallback. ## Modifications - `python/sglang/kernels/ops/attention/dsv4/topk.py`: - `topk_transform_512(...)`: added an `elif scores.device.type == \"xpu\"` branch that calls `torch.ops.sgl_kernel.topk_transform_512(...)`, before the existing generic JIT path. - `topk_transform_512_v2(...)`: added an early-return branch for `scores.device.type == \"xpu\"` that calls `torch.ops.sgl_kernel.topk_transform_512_v2(...)`. On XPU the `metadata` argument is accepted for API parity but ignored by the kernel. ## Accuracy Tests Correctness of the XPU dispatch is verified against the vectorized PyTorch reference at the kernel level in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) $ pytest tests/test_topk_transform.py -k topk_transform_512 ============ 450 passed, 196 deselected in 164.22s (0:02:44) ============ ## Speed Tests and Profiling Speedup of the XPU dispatch is compared against the vectorized PyTorch reference at the kernel level in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) **topk_transform_512: SYCL vs Torch** | bs | topk | seq_len | page_size | sycl_ms | torch_ms | speedup | |-----:|-------:|----------:|------------:|----------:|-----------:|----------:| | 132 | 512 | 4096 | 64 | 0.150384 | 0.373984 | 2.487 | | 132 | 1024 | 4096 | 64 | 0.146692 | 0.307580 | 2.097 | | 132 | 512 | 16384 | 64 | 0.373464 | 1.723852 | 4.616 | | 132 | 1024 | 16384 | 64 | 0.383864 | 1.741480 | 4.537 | | 132 | 512 | 65536 | 256 | 1.354756 | 6.357572 | 4.693 | | 132 | 1024 | 65536 | 256 | 1.356524 | 6.405880 | 4.722 | | 256 | 512 | 16384 | 64 | 0.659724 | 3.232632 | 4.900 | | 256 | 1024 | 16384 | 64 | 0.667940 | 3.279172 | 4.909 | | 4096 | 512 | 4096 | 64 | 3.748472 | 5.637736 | 1.504 | | 4096 | 1024 | 4096 | 64 | 3.671408 | 6.628648 | 1.805 | | 4096 | 512 | 16384 | 64 | 9.758008 | 49.686104 | 5.092 | | 4096 | 1024 | 16384 | 64 | 9.875788 | 50.707020 | 5.134 | **`topk_transform_512_v2`: SYCL vs Torch** | bs | topk | seq_len | page_size | sycl_ms | torch_ms | speedup | |-----:|-------:|----------:|------------:|----------:|-----------:|----------:| | 132 | 512 | 4096 | 64 | 0.150540 | 0.308152 | 2.047 | | 132 | 1024 | 4096 | 64 | 0.147264 | 0.304356 | 2.067 | | 132 | 2048 | 4096 | 64 | 0.108524 | 0.322608 | 2.973 | | 132 | 512 | 16384 | 64 | 0.374972 | 1.723956 | 4.598 | | 132 | 1024 | 16384 | 64 | 0.384748 | 1.750684 | 4.550 | | 132 | 2048 | 16384 | 64 | 0.399412 | 1.785212 | 4.470 | | 132 | 512 | 65536 | 256 | 1.352936 | 6.369012 | 4.708 | | 132 | 1024 | 65536 | 256 | 1.356732 | 6.406712 | 4.722 | | 132 | 2048 | 65536 | 256 | 1.359332 | 6.461468 | 4.753 | | 256 | 512 | 16384 | 64 | 0.660400 | 3.227588 | 4.887 | | 256 | 1024 | 16384 | 64 | 0.668148 | 3.285464 | 4.917 | | 256 | 2048 | 16384 | 64 | 0.697112 | 3.342768 | 4.795 | | 4096 | 512 | 4096 | 64 | 3.750656 | 5.645016 | 1.505 | | 4096 | 1024 | 4096 | 64 | 3.671252 | 6.634992 | 1.807 | | 4096 | 2048 | 4096 | 64 | 2.630264 | 8.526544 | 3.242 | | 4096 | 512 | 16384 | 64 | 9.760660 | 49.661300 | 5.088 | | 4096 | 1024 | 16384 | 64 | 9.876672 | 50.683152 | 5.132 | | 4096 | 2048 | 16384 | 64 | 10.285704 | 52.551252 | 5.109 | --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31368297417](https://github.com/sgl-project/sglang/actions/runs/31368297417) Latest PR Test (Extra): :x: [Run #31368297269](https://github.com/sgl-project/sglang/actions/runs/31368297269) ",
"url": "https://github.com/sgl-project/sglang/pull/33318",
"createdAt": "2026-08-03T02:41:45Z",
"updatedAt": "2026-08-13T06:26:24Z",
"timestamp": "2026-08-13T06:26:24Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "cyxlily",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33354",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5",
"text": " ## Motivation To use a fused GDN kernel provided by sgl-kernel-xpu for better performance than what the existing triton kernels in SGLang produce. ## Modifications - Added a subclass of `GDNAttnBackend` for XPU, which does some checks and calls the fused GDN kernel. - Added a simple dispatch mechanism for the fused GDN kernel on XPU with existing paths unchanged. ## Accuracy Tests Qwen3.5-9B, GSM8K, 200 samples Fused GDN kernel from sgl-kernel-xpu - Accuracy: 0.910 - Invalid: 0.000 Triton GDN in SGLang - Accuracy: 0.900 - Invalid: 0.000 ## Speed Tests and Profiling Qwen3.5-9B on a Intel Arc BMG60 machine ``` python -I -m sglang.bench_one_batch --batch-size 1 --input 4096 --output 1024 --model Qwen/Qwen3.5-9B --tp 1 --device xpu --mem-fraction-static 0.9 --attention-backend intel_xpu --disable-radix-cache --profile --profile-stage all --profile-activities CPU XPU --chunked-prefill-size -1 --max-prefill-tokens 20480 --max-total-tokens 20480 ``` Latency | Old | New -- | -- | -- Prefill E2E | 0.937 s | 0.808 s Prefill GDN | 165 ms | 59.17 ms Decode E2E | 0.0404 s | 0.0402 ms Decode GDN | 0.522 ms | 0.452 ms ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31660770933](https://github.com/sgl-project/sglang/actions/runs/31660770933) Latest PR Test (Extra): :x: [Run #31660770766](https://github.com/sgl-project/sglang/actions/runs/31660770766) ",
"url": "https://github.com/sgl-project/sglang/pull/33354",
"createdAt": "2026-08-03T06:33:14Z",
"updatedAt": "2026-08-13T03:30:04Z",
"timestamp": "2026-08-13T03:30:04Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "Xia-Weiwen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33366",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU][Diffusion] Enable MiniMax H3 on XPU platforms",
"text": "This PR adds key changes to cover the enabling for Minimax H3 model specific on XPU platforms, including - Mapping to XPU attention backend with optimized kernel from sgl-kernel-xpu - Necessary device dispatching/autocast codes - Runtime XPU memory D2H/H2D fix Tested serving pipelines include: - TP + layer_wise offload for both encoder/dit/vae ( similar as cookbook on cuda 5090: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3#3-serve-minimax-h3, but preferring no offload (only encoder) when device memory is enough, i.e., x4 or x8) Next plans (as follow-up PRs, for further optimizations). - Apply more fusions (like QKnorm_RoPE and silu_and_mul, to be on par with cuda fusion pathes) - More serving pipelines/parallelism to be explored and evaluated. - XPU FA backend with non-contiguous support Tested example: - T2VA example from cookbook: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3#4-generate-video-and-audio Result: https://github.com/user-attachments/assets/09e086a7-95ae-489b-bb9b-5e48553eca91 --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #30969623195](https://github.com/sgl-project/sglang/actions/runs/30969623195) Latest PR Test (Extra): :white_check_mark: [Run #31564550674](https://github.com/sgl-project/sglang/actions/runs/31564550674) ",
"url": "https://github.com/sgl-project/sglang/pull/33366",
"createdAt": "2026-08-03T08:07:44Z",
"updatedAt": "2026-08-13T06:13:31Z",
"timestamp": "2026-08-13T06:13:31Z",
"metrics": {
"reactions": 0,
"comments": 15
},
"labels": [
"intel",
"xpu",
"run-ci",
"diffusion",
"run-ci-extra"
],
"author": "jianan-gu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33370",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] Add process-local in-memory KV indexer and Router integration",
"text": "## Motivation This PR supersedes #32662 with a smaller, dependency-free milestone for the KV Indexer proposed in #31458. The Indexer is an experimental metadata-only service that tracks which SGLang worker may reuse each content-addressed KV-cache block. KV data remains owned by the workers. The end-to-end path is: ```text SGLang worker └─ ZMQ component-aware KV events └─ kv-indexer-bridge └─ gRPC └─ process-local in-memory indexer └─ MatchExternalKvPrefix └─ experimental sgl-router ``` This revision removes Redis, Dragonfly, and Redis Cluster from the architecture. Placement metadata is held in one Indexer process and is intentionally soft state. That makes the core event → index → prefix-query path smaller, easier to run, and directly testable without an external database. Related: - RFC: #31458 - Previous KV Indexer PR: #32662 - KV-event correctness fixes: #32537, #32514 ## Scope and failure model The process-local backend is protected by an `RwLock` and bounded request work. It supports concurrent reads and serialized placement mutations in one server process. It is not a distributed or persistent store. - Run exactly one Indexer process for a deployment. - Restarting the Indexer loses all placement metadata. - There is no missed-event replay, restart fencing, worker liveness TTL, or cross-process replication in this milestone. - A disconnected Bridge reconnects transport but cannot recover events emitted while it was absent. A batch the Indexer sheds, or refuses as oversized, costs those events and a reconnect rather than terminating the Bridge. - A prefix hit only improves worker choice, so the Router degrades rather than failing when the Indexer cannot answer. An empty result, a rejection from either admission bound, a query that exceeds its deadline, and an unreachable Indexer all become no signal: the Router selects by minimum active load and logs at `WARN`. An Indexer outage therefore costs cache affinity rather than availability. Only a rejected RPC — the Router and the Indexer disagreeing on the request contract — still fails the request with `503`, because degrading would hide that disagreement from every request. An endpoint the Router could never dial is rejected at startup instead of failing every later query. The Router's local radix tree is used only when no Indexer endpoint is configured. These constraints are explicit. This PR establishes a validated single-process soft-state baseline; persistence and recovery belong in follow-up work. ## What changed ### Process-local memory backend - Replace the Redis backend with an in-process component-aware placement map. - Track `(block hash, worker, tier)` placement snapshots and worker cache specs. - Support FULL, SWA, and MAMBA reusable-prefix semantics. - Bound prefix scanning and request hash counts to keep work predictable. - Remove Redis features, scripts, configuration, and integration requirements. ### Admission and resource protection - Bound concurrent prefix queries in the Indexer. - Shed queries whose caller deadline has already expired before backend work. - Bound concurrent external Indexer queries in the Router. - Add configurable Router timeout and max-inflight controls. - Rate-limit rejection logs with doubling cadence to avoid log storms. ### Experimental Router integration ```bash sgl-router \\ --model-id \\ --tokenizer-path \\ --worker-urls \\ --policy cache_aware_zmq \\ --kv-indexer-endpoint http://:50061 \\ --kv-indexer-query-timeout-ms 100 \\ --kv-indexer-query-max-inflight 32 ``` For each eligible request, the Router hashes normalized request tokens into the same ordered block chain, calls `MatchExternalKvPrefix`, intersects returned addresses with current routing candidates, and selects among the best reusable prefix holders using active load. ### Component-aware prefix matching The bridge accepts the gated `component_types` field from SGLang KV events and stores a component snapshot for each placement. A versioned `WorkerCacheSpec` describes which components and tiers are reusable. - **FULL** must be present contiguously on every matched block. - **SWA** must cover the configured trailing window at the candidate boundary, or form an unbroken run from the prompt head. - **MAMBA** must be present on the candidate boundary block. Component-aware placements without a valid worker cache spec fail closed. Legacy whole-block events continue to use contiguous-prefix matching. ### Crate layout `sgl-kv-indexer` lives inside the Router workspace, at `experimental/sgl-router/sgl-kv-indexer`, as a workspace member. Both crates share one lockfile, one toolchain, and one CI workflow. Membership also means Cargo unifies features across the workspace, so the Indexer pins the same narrow `zeromq` feature set as the Router (`tokio-runtime`, `tcp-transport`): adding it neither enlarges the Router binary nor widens its license surface. ## Validation ### Current process-local backend: two machines, 15 Workers One host runs a single Indexer (`prefix_query_max_inflight=32`) and the experimental Router (`--policy cache_aware_zmq`, `--kv-indexer-query-timeout-ms 100`). Two further hosts run 8 + 7 GPUs, giving 15 independent Workers each with its own Bridge, page size 64. Both component profiles were driven end to end: FULL+SWA (`gpt-oss-20b-bf16`) and FULL+MAMBA (`Qwen3.5-27B-FP8`). This is a Worker-density claim, not a host-count claim. #### Placement and prefix correctness | check | FULL+SWA | FULL+MAMBA | | --- | --- | --- | | expected blocks == Unified Tree cached blocks == Indexer prefix result | 15/15 | 15/15 | | unique prefix routed by the Router to its exact holder | 15/15 | 15/15 | | shared prefix routed to a holder from the shared set | pass | pass | Prefix sizes differ per Worker by construction (SWA 104–118 blocks, MAMBA 13–41). FULL+SWA routing was verified before and after churn. #### Event replay against indexed state The raw ZMQ event stream from both hosts (`BlockStored` / `BlockRemoved` / `AllBlocksCleared`, including `component_types`) was replayed independently and compared with the Indexer's `MatchExternalKv` snapshot, hash by hash and mask by mask. | scenario | stored events | removals | AllBlocksCleared | distinct blocks | Workers exactly equal | | --- | --- | --- | --- | --- | --- | | FULL+SWA, seed and selective flush | 3,135 | 0 | 24 | 1,665 | 15/15 | | FULL+SWA, after 15 min churn | 101,475 | 2,353 | 54 | 28,416 | 15/15 | | FULL+MAMBA, seed | 405 | 0 | 15 | 405 | 15/15 | | FULL+MAMBA, after 5 min churn | 268,603 | 4,286 | 54 | 97,511 | 15/15 | The churn rows cover eviction: caches were driven past capacity, so blocks were removed and later re-stored. The MAMBA seed row also matched the expected mask distribution, exactly one FULL|MAMBA checkpoint block per sequence and FULL only elsewhere. #### Cache churn Concurrent generation, prefix reuse and capacity-exceeding requests across all 15 Workers, starting from a fresh `AllBlocksCleared` so both sides share one origin. | profile | duration | requests | fresh / reuse / evicting | | --- | --- | --- | --- | | FULL+SWA | 15 min, 30 threads | 815 | 387 / 246 / 182 | | FULL+MAMBA | 5 min, 15 threads | 3,984 | 1,626 / 1,380 / 978 | Workers run `max_running_requests=1`, so the SWA churn saturated them and 111 client calls hit their read timeout; correctness is judged from the event stream, not from client success. #### Selective flush Flushing one Worker, then all 8 Workers of one host: flushed placements dropped to zero, untouched Workers kept their exact block counts, and the replayed event state still equalled the Indexer state for 15/15 Workers. Verified for both profiles. #### MAMBA checkpoint boundary Four probes per Worker, 15/15 passing: a query ending on the checkpoint block returns the full prefix with the correct sole holder; including the trailing partial page returns the same; one block short of the checkpoint returns 0; the first block alone returns 0. Sweeping all truncation points for a 17-block prefix, cuts 1–16 return 0 and cuts 17–18 return 17. #### Query capacity and admission Capacity was measured against the Indexer alone, with no GPUs: 15 synthetic Workers holding 118-block FULL+SWA prefix chains, loaded through 15 real Bridge processes, then queried by an increasing number of independent client processes (16 threads each, co-located, 224-core host). Sweeping the process count is what separates a driver limit from the server's: queries from a single Python process are capped by its own interpreter, so one process cannot measure this server. | client processes | in-flight callers | accepted qps | p50 ms | p99 ms | `RESOURCE_EXHAUSTED` | | --- | --- | --- | --- | --- | --- | | 1 | 16 | 1,554 | 7.6 | 41.9 | 0 | | 2 | 32 | 3,441 | 6.8 | 38.5 | 0 | | 4 | 64 | 7,559 | 5.6 | 37.6 | 0 | | 8 | 128 | 19,422 | 4.5 | 28.7 | 0 | | 16 | 256 | 83,347 | 2.8 | 7.7 | 0 | | 32 | 512 | 158,810 | 3.0 | 8.2 | 0 | | 48 | 768 | 217,604 | 3.2 | 9.1 | 12 | | 64 | 1,024 | 241,816 | 3.8 | 11.4 | 350 | | 96 | 1,536 | 244,557 | 4.8 | 17.7 | 609,498 | One Indexer process serves roughly 245k prefix queries per second at this query size of 118 blocks. Latency falls as clients are added rather than rising, up to the point of saturation, which is what confirms the low-process rows measure the driver. Capacity depends on how many blocks a query carries. Each size was driven to its own saturation point against 1024-block chains (the scan is capped at 2,048 blocks): | blocks per query | approx tokens | peak accepted qps | at client processes | p50 ms | p99 ms | | --- | --- | --- | --- | --- | --- | | 1 | 64 | 444,302 | 128 | 3.6 | 14.1 | | 8 | 512 | 412,621 | 96 | 3.1 | 11.6 | | 32 | 2,048 | 364,360 | 128 | 4.4 | 14.9 | | 118 | 7,552 | 248,681 | 64 | 3.7 | 11.0 | | 256 | 16,384 | 184,295 | 64 | 5.0 | 14.6 | | 512 | 32,768 | 124,032 | 64 | 7.4 | 21.1 | | 1,024 | 65,536 | 74,443 | 64 | 12.1 | 36.5 | Each size needed its own client load to saturate, hence the differing process counts. State here holds 1,024 blocks per Worker rather than the 118 of the sweep above, which is why the 118-block row differs from it slightly. The relationship is strongly sublinear: a 1,024x larger request costs about 6x more. A least-squares fit over these rows gives 2.5 us of fixed cost per request plus 10.8 ns per block, matching every row within 10%. Most of that per-block cost is not the tree walk. A query whose first hash is absent exits after reading one block, and at saturation it beats a fully matching query of the same length by 6.8% at 118 blocks (265,516 vs 248,681 qps) and 13.5% at 1,024 (84,461 vs 74,443). So the scan accounts for 6-12% of per-query cost, around 1.6-2.2 ns per block, and the remainder is moving and decoding the hash list, which the proto carries as `repeated string`. Two consequences: cache hit rate changes the Indexer's load only modestly, and request encoding is the lever if this ever needs to be cheaper. **The inflight bound rejects as designed.** With `prefix_query_max_inflight=32`, rejections begin at ~768 concurrent callers and grow from there. At 1,536 callers the excess is shed while accepted throughput holds near 245k qps and p99 stays at 17.7 ms, so the bound caps the tail instead of letting queueing delay grow without limit. **Deadline shedding is demonstrated.** 5,119 of 5,120 requests carrying a 1 ms deadline were shed as `DEADLINE_EXCEEDED` before backend work; one completed before expiry. Shedding protects the Indexer, not the caller: like an admission rejection, a shed query reaches the Router as no signal, so the request is still routed, by minimum active load. **Concurrent event ingestion is free at realistic event rates.** With 15 Bridges applying ~150 event batches/s, saturated 118-block load returned 249,491 qps against a 248,681 qps idle baseline, so applies and prefix queries do not contend measurably. Taken together: because one process absorbs between 74k and 444k queries/s depending on prompt length, and does not reject below ~700 concurrent in-flight queries, neither protection is expected to engage at realistic Router query rates. Both bound the tail, not the normal path. **Router under Worker saturation.** With every Worker at capacity during churn, 16 concurrent end-to-end chat requests all exceeded the 30 s client timeout. The Indexer saw zero errors in the same window and routing returned to 15/15 once churn stopped, so this is Worker saturation rather than Router or Indexer behaviour. This is the scale claim for the process-local backend in this PR. A current 4/8-machine memory-backend result is intentionally not claimed. ### Event-pipeline evidence from the previous Redis-backed build Before the backend replacement, the same component-event schema, Bridge decode path, Unified Tree semantics, and prefix rules were validated at larger scale: | deployment | phase | workers | Tree = Indexer = expected | raw event replay = indexed state | | --- | --- | ---: | ---: | ---: | | 4 machines | FULL+MAMBA | 32 | 32/32 | 32/32 | | 4 machines | FULL+SWA | 32 | 32/32 | 32/32 | | 8 machines | FULL+MAMBA | 63 | 63/63 | 63/63 | | 8 machines | FULL+SWA | 62 | 62/62 | 62/62 | The 4-machine run also completed 15 minutes of churn with 4,224 checkpointed Tree/Indexer alignment checks and zero failures. These results support event production, Bridge decoding, component replacement, and holder attribution at scale; they are not presented as 4/8-machine validation of the new memory backend. ### Code and build gates - One workspace gate over both crates: format, Clippy with warnings denied, release build, and **650 tests** (99 Indexer, 551 Router). - `cargo deny` clean for advisories, bans, licenses, and sources. - Repository pre-commit checks passed. - Both Router images built from the final source: the release image's distroless entrypoint passed a `--help` smoke test, and the k8s e2e image runs `sgl-router` with its tokenizer fixture in place. ## Test coverage The branch includes: - memory-backend placement, tier, clear, and component-transition tests; - FULL, SWA, MAMBA, and legacy prefix-rule tests; - deadline, request-size, and max-inflight admission tests; - gRPC validation, wire-contract, and client endpoint-validation tests; - Bridge decoding and component-snapshot coverage; - Router policy, fallback, degradation, configuration, and concurrency tests; - process-local memory integration tests. ## Current limitations - **Component metadata depends on #32514, which is not yet merged.** The `component_types` field on `BlockStored` and the `--enable-kv-events-component-types` option come from that PR. - Metadata is process-local soft state and is lost on restart. - Exactly one Indexer process is supported per deployment. - No replay/reconciliation, liveness filtering, persistence, failover, or network-partition recovery is claimed. Bridge reconnect restores transport only; events missed while a Bridge was absent are not recovered, and this was not exercised. - `KV_INDEXER_WORKER_ADDRESS` must match the Router worker URL byte-for-byte. - Prefix scans are capped at 2,048 blocks (~131k tokens). A longer request is truncated rather than rejected, so it is matched on its first 2,048 blocks only. - Worker density was validated at 15 Workers against one Indexer; higher densities only on the previous Redis-backed build. ## Checklist - [x] Format and lint code according to repository conventions. - [x] Add unit, contract, integration, and Router tests. - [x] Update architecture, configuration, and limitation documentation. - [x] Validate the final process-local backend on two machines. - [x] Build and smoke-test the final Router Docker image. - [ ] Confirm the updated PR CI is green. cc: @isytwu @TianDi101 Made with [Cursor](https://cursor.com) --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31711597817](https://github.com/sgl-project/sglang/actions/runs/31711597817) Latest PR Test (Extra): :x: [Run #31711597980](https://github.com/sgl-project/sglang/actions/runs/31711597980) ",
"url": "https://github.com/sgl-project/sglang/pull/33370",
"createdAt": "2026-08-03T08:22:13Z",
"updatedAt": "2026-08-13T14:45:46Z",
"timestamp": "2026-08-13T14:45:46Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"documentation",
"high priority",
"dependencies"
],
"author": "wuyl1",
"state": "open",
"assignees": [
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33373",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU][DSV4] Fix MXFP4 expert scale dtype handling",
"text": "## Description Reinterpret MXFP4 expert scales as raw `uint8` E8M0 bytes for the Intel XPU fused MoE kernel. --- ### CI States Latest PR Test (Base): :x: [Run #31554105834](https://github.com/sgl-project/sglang/actions/runs/31554105834) Latest PR Test (Extra): :x: [Run #31554105718](https://github.com/sgl-project/sglang/actions/runs/31554105718) ",
"url": "https://github.com/sgl-project/sglang/pull/33373",
"createdAt": "2026-08-03T08:41:44Z",
"updatedAt": "2026-08-13T06:17:10Z",
"timestamp": "2026-08-13T06:17:10Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"intel",
"xpu",
"run-ci"
],
"author": "YangKai0616",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33395",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Speculative] Seed rejection-sampling draft proposals for deterministic inference",
"text": "## Motivation Classic EAGLE/NEXTN rejection sampling draws each draft proposal with `fast_sample()`. Although target sampling and verify-side rejection coins support request-local seeds, this proposal draw used the global PyTorch RNG (`exponential_()`), so `--enable-deterministic-inference` still changed outputs with batch shape and CUDA graph replay. This is the draft-worker follow-up documented in #30822. Before this change, Qwen3.5-9B NEXTN rejection sampling produced 12 unique outputs in 12 changing-batch trials with a fixed sampling seed. ## Modifications - Add an optional deterministic path to `fast_sample()` keyed by: - request sampling seed - absolute sequence position - draft step / MTP layer - vocabulary entry - Domain-separate draft proposal randomness from target sampling and verify-side rejection coins. - Convert MurmurHash32 output to open-interval uniforms with midpoint binning, then use exponential-race categorical sampling in float64. - Preserve the existing RNG-backed fast path when deterministic inference is disabled. - Plumb request seeds and positions through eager and CUDA-graph EAGLE/NEXTN workers. - Support both ordinary and multi-layer EAGLE, including the single-CUDA-graph multi-layer draft-extend path. - Keep deterministic rejection sampling guarded on non-CUDA-compatible devices because this implementation uses a Triton hash kernel. - Add sampler tests for replay, batch permutation/splitting, seed/position/step separation, hash endpoints, CUDA graph updates, and categorical distribution sanity. - Add dedicated deterministic E2E smoke tests without changing the existing unseeded rejection-sampling and chain-MTP test configurations. - Add a reproducible seeded/unseeded sampling microbenchmark. ## Accuracy Tests ### Regular EAGLE/NEXTN Qwen3.5-9B, TP2, NEXTN, rejection sampling, temperature 1.0, sampling seed 42: | Version | Changing-batch trials | Unique outputs | |---|---:|---:| | Before | 12 | 12 | | After | 12 | 1 | The deterministic harness also passed changing batch sizes, prefix batching, bitwise logprob comparison, and radix-cache consistency. Post-rebase validation on B200: ```text 7 passed, 16 warnings, 2 subtests passed Total samples: 6, Unique samples: 1 OK ``` The unit suite includes 100,000 distinct seeded draws from `[0.1, 0.2, 0.7]` and verifies that observed frequencies are within 0.01 absolute tolerance. ### Multi-layer EAGLE `stepfun-ai/Step-3.5-Flash`, TP8 on 8x B200, FA4, three MTP layers, single-CUDA-graph draft extend: ```text Total samples: 6, Unique samples: 1 MULTI_LAYER_EAGLE_REJECTION_SAMPLING_IS_DETERMINISTIC ``` ## Speed Tests and Profiling B200, vocabulary size 163,840, CUDA-graph replay: | Batch | Unseeded | Seeded | Slowdown | |---:|---:|---:|---:| | 1 | 31.9 us | 47.2 us | 1.48x | | 8 | 38.0 us | 63.5 us | 1.67x | | 32 | 54.5 us | 131.9 us | 2.42x | | 128 | 138.9 us | 544.1 us | 3.92x | Peak allocated memory for batch 128 and vocabulary 163,840: ```text Input probabilities: 80.00 MiB Baseline allocated: 80.00 MiB Peak allocated: 321.25 MiB Incremental seeded peak: 241.25 MiB ``` The deterministic mode is opt-in. A fused hash/exponential/argmax kernel can be considered separately if end-to-end profiling shows that the materialized hash and score matrices materially affect serving throughput. ## Checklist - [x] Format code and pass Ruff / `git diff --check`. - [x] Add unit and registered E2E tests. - [x] Preserve existing unseeded rejection-sampling and chain-MTP coverage. - [x] Provide accuracy, latency, and peak-memory measurements. - [x] Follow the SGLang code style guidance. - [x] No documentation update is needed; this fixes an existing flag combination without adding user-facing options. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31581112407](https://github.com/sgl-project/sglang/actions/runs/31581112407) Latest PR Test (Extra): :x: [Run #31581112329](https://github.com/sgl-project/sglang/actions/runs/31581112329) ",
"url": "https://github.com/sgl-project/sglang/pull/33395",
"createdAt": "2026-08-03T14:05:17Z",
"updatedAt": "2026-08-13T13:01:53Z",
"timestamp": "2026-08-13T13:01:53Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"bug",
"speculative-decoding",
"run-ci",
"deterministic"
],
"author": "ajit283",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33431",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] Skip padded state slots in the chunked GDN kernel",
"text": "_forward_metadata poisons padded request rows with state index -1, and the decode kernel already skips them (fused_recurrent: `if idx >= 0`). The chunked extend kernel had no such guard, so -1 reached `initial_state + index * stride_init_state` for both the initial-state load and the in-place final-state store, addressing before the state pool. boundary_check does not help: it validates against the declared block shape, not the allocation. Under breakable-CUDA-graph prefill with DP attention an idle DP rank replays an extend with an all-padded batch, so every row carries -1 and the kernel faults with cudaErrorIllegalAddress. Reproduced on Qwen3-Next-80B and Qwen3.5-397B on pristine main; a single request against --dp 4 is enough. No per-commit suite covers a linear-attn hybrid with DP attention, which is why this went unseen. Add a regression test in the existing base-b (1-GPU, per-commit) kernel suite; it fails without the guard. ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31642894656](https://github.com/sgl-project/sglang/actions/runs/31642894656) Latest PR Test (Extra): :white_check_mark: [Run #31642894462](https://github.com/sgl-project/sglang/actions/runs/31642894462) ",
"url": "https://github.com/sgl-project/sglang/pull/33431",
"createdAt": "2026-08-03T21:31:21Z",
"updatedAt": "2026-08-12T23:15:43Z",
"timestamp": "2026-08-12T23:15:43Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"jit-kernel",
"bypass-fastfail",
"run-ci-extra"
],
"author": "Oasis-Git",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33473",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] Batch PP write and load completion sync",
"text": " ## Motivation Reduce scheduler overhead for HiCache with pipeline parallelism by batching write and load completion-count synchronization into a single operation. ## Modifications - Synchronize [write_finish_count, load_finish_count] together for PP > 1. - Reuse the existing CP/TP reduction and PP propagation path. - Preserve the existing ACK processing, event synchronization, storage, and lock-release behavior. - Add unit coverage for PP leader and follower behavior. ## Benchmark We evaluated the change using a prefill-only shared-prefix workload with 3.5K, 32K, and 128K input lengths and one output token. The benchmark adaptively increased request rate and concurrency to search for the serving throughput limit. The clearest improvement was observed in the 32K high-pressure workload, where the maximum logical input throughput increased from approximately 494K to 677K tokens/s (+37%). ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31148776821](https://github.com/sgl-project/sglang/actions/runs/31148776821) Latest PR Test (Extra): :white_check_mark: [Run #31374174529](https://github.com/sgl-project/sglang/actions/runs/31374174529) ",
"url": "https://github.com/sgl-project/sglang/pull/33473",
"createdAt": "2026-08-04T03:51:40Z",
"updatedAt": "2026-08-13T12:04:14Z",
"timestamp": "2026-08-13T12:04:14Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"run-ci-extra"
],
"author": "luoroger37",
"state": "open",
"assignees": [
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33480",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Support prefill context parallel two batch overlap for DeepSeek V4",
"text": " ## Motivation ## Modifications ## Accuracy Tests config | GSM8K 1319 accuracy | -- | -- cp | 0.944 cp+tbo | 0.936 ## Speed Tests and Profiling I/O = 128k/1k conc | cp TPUT | tbo TPUT | ΔTPUT | cp TTFT | tbo TTFT | ΔTTFT | cp TPOT | tbo TPOT | ΔTPOT -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 2 | 9536 | 9424 | −1.2% | 4813 | 4417 | −8.2% | 22.25 | 22.97 | +3.2% 4 | 15534 | 15713 | +1.2% | 7663 | 7047 | −8.0% | 25.57 | 25.86 | +1.1% 8 | 22449 | 23349 | +4.0% | 13479 | 12258 | −9.1% | 32.51 | 31.96 | −1.7% 16 | 28333 | 29651 | +4.6% | 25237 | 22501 | −10.8% | 47.65 | 47.11 | −1.1% I/O = 50k/1, prefill only conc | cp TPUT | tbo TPUT | ΔTPUT | cp TTFT | tbo TTFT | ΔTTFT -- | -- | -- | -- | -- | -- | -- 2 | 44133 | 49299 | +11.7% | 2141 | 1941 | −9.3% 4 | 45040 | 49594 | +10.1% | 4042 | 3711 | −8.2% 8 | 45410 | 50311 | +10.8% | 8003 | 7266 | −9.2% 16 | 45536 | 51859 | +13.9% | 15683 | 13823 | −11.9% ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31569928864](https://github.com/sgl-project/sglang/actions/runs/31569928864) Latest PR Test (Extra): :x: [Run #31569928786](https://github.com/sgl-project/sglang/actions/runs/31569928786) ",
"url": "https://github.com/sgl-project/sglang/pull/33480",
"createdAt": "2026-08-04T04:49:43Z",
"updatedAt": "2026-08-13T14:33:09Z",
"timestamp": "2026-08-13T14:33:09Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"deepseek",
"run-ci",
"bypass-fastfail"
],
"author": "At1a8",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33502",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "add prefill graph support on cpu",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31692644323](https://github.com/sgl-project/sglang/actions/runs/31692644323) Latest PR Test (Extra): :x: [Run #31692643880](https://github.com/sgl-project/sglang/actions/runs/31692643880) ",
"url": "https://github.com/sgl-project/sglang/pull/33502",
"createdAt": "2026-08-04T06:55:57Z",
"updatedAt": "2026-08-13T10:48:05Z",
"timestamp": "2026-08-13T10:48:05Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"dependencies",
"sgl-kernel"
],
"author": "CaoE",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33514",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[UT][NPU]Add npu unit test for dsv4",
"text": " ## Motivation Add npu unit test for dsv4 ## Modifications Add test/registered/npu/dsv4/test_dsv4_allocator.py, test/registered/npu/dsv4/test_dsv4_common_hooks.py, test/registered/npu/dsv4/test_dsv4_memory_pool.py, test/registered/npu/dsv4/test_dsv4_req_to_token_pool.py, and test/registered/npu/dsv4/test_dsv4_rope.py ## Accuracy Tests N/A ## Speed Tests and Profiling N/A ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31662610231](https://github.com/sgl-project/sglang/actions/runs/31662610231) Latest PR Test (Extra): :x: [Run #31662609897](https://github.com/sgl-project/sglang/actions/runs/31662609897) ",
"url": "https://github.com/sgl-project/sglang/pull/33514",
"createdAt": "2026-08-04T08:29:54Z",
"updatedAt": "2026-08-13T03:02:00Z",
"timestamp": "2026-08-13T03:02:00Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "luz3633",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33554",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add new spec-dec support and quant recipe for Nano v3",
"text": "## Motivation Add support for Nemotron 3.5 including DFlash and DSpark spec-dec ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/33554",
"createdAt": "2026-08-04T13:09:55Z",
"updatedAt": "2026-08-13T17:58:06Z",
"timestamp": "2026-08-13T17:58:06Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"quant",
"speculative-decoding",
"new-model"
],
"author": "rystewart-nvidia",
"state": "open",
"assignees": [
"b8zhong"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33559",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[kernel] add triton moe TMA up support",
"text": "## Motivation Following PR #10567 (\"Opt fused triton moe: add tma for down proj kernel\"), which added TMA support for the down-projection (second MoE GEMM), this PR extends TMA support to the up-projection (first MoE GEMM, a.k.a. gate_up). Runtime couples up/down TMA via c_sorted = down_moe_use_tma. Since c_sorted is a tl.constexpr, different values produce different kernel binaries. The old tuning script always used c_sorted=False for up, which doesn't match runtime when down TMA is enabled. Solution: Two-Round Coupled Tuning When --enable-tune-up-tma is set: Round 1 (Down): Measure down no-TMA vs TMA for each config. Extract down_use_tma_map = {BLOCK_SIZE_M: bool}. Round 2 (Up): Look up c_sorted = down_use_tma_map[BLOCK_SIZE_M] from Round 1. Measure up no-TMA vs TMA with the correct c_sorted matching runtime. Final: Select best BLOCK_SIZE_M by up_time + down_time, output both configs with USE_TMA flags. Code Changes (1 file: tuning_fused_moe_triton_sep.py) benchmark_config: Added tune_round (\"both\"/\"down\"/\"up\") and down_use_tma_map params. Conditionally builds only the kernels needed for each round. tune: Two-round loop when enable_up_tma=True; single-round unchanged when False. benchmark: Two-step (down then up) for single-config path. Key Properties Default behavior (no --enable-tune-up-tma) is unchanged — only down TMA tuned, up c_sorted=False. Total kernels per config: 4 (2 in round 1 + 2 in round 2) ## Modifications 1. sglang/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py 2. sglang/python/sglang/srt/layers/moe/moe_runner/triton.py 3. sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py ## Accuracy Tests ## Speed Tests and Profiling Add --enable-tune-up-tma to tune script：sglang/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py. It uses the original tune‑down TMA behavior by default when the flag is not provided. baseline: python3 -m sglang.bench_serving --backend sglang --host localhost --port 25000 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --num-prompts 500 --random-output-len 1000 --warmup-requests 10 --random-input-len 20000 --max-concurrency 500 --tokenizer /ufs/models/Qwen3.5-397B-A17B-FP8/ --request-rate 0.8 --model /ufs/models/Qwen3.5-397B-A17B-FP8/ --random-range-ratio 1 use down TMA ohly: use up TMA + down TMA: results: As summarized from the table, combining up TMA + down TMA achieves better performance than using down TMA alone. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31573435275](https://github.com/sgl-project/sglang/actions/runs/31573435275) Latest PR Test (Extra): :x: [Run #31573435123](https://github.com/sgl-project/sglang/actions/runs/31573435123) ",
"url": "https://github.com/sgl-project/sglang/pull/33559",
"createdAt": "2026-08-04T14:00:15Z",
"updatedAt": "2026-08-13T05:29:33Z",
"timestamp": "2026-08-13T05:29:33Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"jit-kernel",
"bypass-fastfail",
"run-ci-extra"
],
"author": "xieminghe1",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33561",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Model] Support Ling-3.0-flash (BailingMoeV3)",
"text": "## Motivation Day-0 support for [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) (`BailingMoeV3ForCausalLM`, `model_type=bailing_hybrid`): a hybrid MoE architecture interleaving KDA linear attention (with a safe-gate lower bound) and MLA, with MTP (NEXTN) and DSPARK speculative decoding support. ## Modifications - `BailingMoeV3ForCausalLM` model (`bailing_hybrid` config): KDA + MLA hybrid layers, 512-expert MoE, and MTP (NEXTN). - DSPARK support for the KDA hybrid path: graph-folded draft proposal, fused KDA chain-verify kernel, and DSA token-to-KV pool sizing. - KDA safe gate propagation through the Triton decode/target-verify kernels and their call sites, matching fused verification. - `ling3` reasoning and tool-call parsers. - BF16, FP8, compressed-tensors INT4, and native MXFP4 checkpoint support on Hopper and Blackwell. - Exactness, fusion-gate, quantization, and parser tests. ## Validation All serving runs kept decode CUDA Graph enabled and used the default graph sizing. No run used `--disable-cuda-graph` or `--cuda-graph-max-bs-decode`. ### Quantized checkpoints Full GSM8K uses 1,319 questions. Performance uses 80 exact-length random requests at ISL/OSL 8192/1024 with concurrency 1 and 16; throughput includes input and output tokens per second per GPU. | GPU | format | GSM8K | stop rate | throughput/GPU (c1 / c16) | P50 TTFT ms (c1 / c16) | P50 TPOT ms (c1 / c16) | default graph range | |---|---|---:|---:|---:|---:|---:|---:| | H200 | INT4 | 95.30% | 100% | 1,159 / 7,184 | 205.96 / 1,891.74 | 3.68 / 8.16 | through batch 218 | | H200 | MXFP4 | 96.29% | 100% | 709 / 5,215 | 305.23 / 2,725.52 | 6.05 / 11.17 | through batch 227 | | B200 | INT4 | 96.74% | 99.70% | 482 / 1,344 | 641.41 / 6,202.62 | 8.72 / 48.01 | through batch 305 | | B200 | MXFP4 | 96.29% | 99.62% | 785 / 7,933 | 127.97 / 1,117.16 | 5.60 / 7.98 | through batch 314 | INT4 selects Marlin on H200 and the graph-compatible Triton WNA16 path on B200. MXFP4 routed experts use the native FlashInfer runner on both GPUs. The B200 MXFP4 checkpoint also contains block-FP8 dense/shared projections, which use the Triton FP8 GEMM backend to avoid an unsupported FlashInfer FP8 tactic without converting the routed experts. BF16 and FP8 were rerun on H200 and B200 after the quantization changes. All four combinations served with CUDA Graph enabled, completed generation, and passed the 20-example GSM8K regression gate with 100% stop rate. ### Speculative decoding Ling-3.0-flash BF16, 4x B200 TP4, `temperature=1.0 / top_p=0.95`, thinking on, full GSM8K, `max_tokens=8192`: | spec decoding | score | stop rate | error rate | |---|---:|---:|---:| | DSPARK | 96.66% | 99.77% | 0.00% | | MTP (NEXTN) | 96.44% | 99.62% | 0.00% | Non-stop finishes are long reasoning outputs reaching `max_tokens`, not repetition loops. ## Checklist - [x] Format code with pre-commit. - [x] Add unit tests and register them in CI. - [x] Validate serving correctness, stop rate, and accuracy on H200 and B200. - [x] Publish deployment recipes and benchmark results in #34363. --- ### CI States Latest PR Test (Base): :x: [Run #31694712753](https://github.com/sgl-project/sglang/actions/runs/31694712753) Latest PR Test (Extra): :x: [Run #31694712651](https://github.com/sgl-project/sglang/actions/runs/31694712651) ",
"url": "https://github.com/sgl-project/sglang/pull/33561",
"createdAt": "2026-08-04T14:07:18Z",
"updatedAt": "2026-08-13T12:58:49Z",
"timestamp": "2026-08-13T12:58:49Z",
"metrics": {
"reactions": 9,
"comments": 10
},
"labels": [
"documentation",
"quant",
"amd",
"deepseek",
"sgl-kernel",
"blackwell",
"npu",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "JustinTong0323",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33569",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] [Diffusion] Support MiniMax H3 on Ascend NPU's",
"text": "## Motivation Extend the native MiniMax-H3 pipeline introduced in [#33275](https://github.com/sgl-project/sglang/pull/33275) to Ascend NPU. MiniMax-H3 packs real multimodal tokens together with sequence-parallel alignment padding. Its attention path is therefore variable-length even when request batch size is one: the real-token segment and the alignment-padding segment must remain isolated. ## Scope and related work - [#33275](https://github.com/sgl-project/sglang/pull/33275) provides the upstream MiniMax-H3 model and pipeline implementation that this PR extends. - [#33366](https://github.com/sgl-project/sglang/pull/33366) is the related XPU enablement PR. This PR does not include XPU backend code; shared paths use `current_platform` so the Ascend changes remain isolated and existing CUDA/ROCm behavior is preserved. - Experimental NPU RoPE, SwiGLU, and gate kernel paths were removed after benchmarking (since they did not improve performance). The final PR keeps the attention and platform-integration changes required for MiniMax-H3 on Ascend. ## Modifications ### Ascend Flash Attention - Add `forward_varlen` using `torch.ops.npu.npu_fused_infer_attention_score` with the `TND` layout and actual packed-sequence lengths. - Preserve independent packed-sequence boundaries. - Repack strided Ulysses Q/K/V views with one allocation instead of three independent contiguous copies. - Preserve the segmented fallback for causal attention. ### Laser Attention - Enable `laser_attn` for MiniMax-H3. - Add variable-length handling so real tokens cannot attend alignment padding. - Run attention only for the real-token segment, retain padding rows for sequence-parallel shape alignment, and zero-fill their output instead of computing attention for them. - Preserve BF16 activation range when the FP16 Laser kernel is used by compensating the Q/K softmax scale and restoring the value/output scale. - Limit this BF16 compensation to MiniMax-H3's packed variable-length path; the existing dense Laser path keeps its previous behavior. ### NPU dependencies and media output - Install `attentions*.whl` alongside the `deep_ep` and `sgl_kernel_npu` wheels from the pinned `sgl-kernel-npu` release archive in both the NPU Docker image and NPU CI. - Scope the missing-`attentions` layer-normalization warning to NPU. Layer normalization falls back to its native implementation; explicitly selecting Laser Attention without the package raises an actionable import error. - Install `ffmpeg` in the NPU Docker and CI environments and fail MiniMax-H3 startup early when required media tools are absent, preventing the late output loss reported in [#33927](https://github.com/sgl-project/sglang/issues/33927). - Keep these dependency checks and warnings isolated from CUDA, ROCm, XPU, MUSA, and CPU. ### Ascend platform integration - Use `current_platform` for shared memory residency, RNG, autocast, and denoising dispatch. - Add NPU support to component offload/cache-release handling while preserving CUDA/ROCm behavior. - Keep Qwen3-VL position IDs and rotary embeddings on the execution device only on NPU. - Avoid enabling Ascend-specific behavior on XPU, MUSA, or CPU. - Add the validated eight-NPU TP2 + SP4 launch command and a four-NPU TP2 + SP2 variant to the MiniMax-H3 cookbook. Both commands list the benchmarked Cache-DiT configuration explicitly; the cookbook records 55.07 s and 103.57 s end-to-end latency respectively. ### FL2VA keyframe encoding - Register `MiniMaxH3Qwen3VLEncoder.encode_ids` as an FSDP2 forward method when text-encoder CPU offload is active. - Ensure the root FSDP hooks materialize non-layer parameters, including the Qwen3-VL vision `Conv3d` patch embedding, before image-conditioned encoding. - Prevent the `aten.convolution` mixed `torch.Tensor`/`DTensor` failure observed by FL2VA requests. - Keep the change platform-independent and limited to FSDP-wrapped encoders that expose `encode_ids`; non-FSDP execution is unchanged. ## Accuracy Tests Validated end-to-end 5-second MiniMax-H3 T2VA generation with both Ascend Flash Attention and Laser Attention on 8 Ascend NPUs. Image-conditioned FL2VA was also validated with Laser Attention after the FSDP2 custom-forward fix. Server startup: ```bash SGLANG_CACHE_DIT_ENABLED=true \\ SGLANG_CACHE_DIT_FN=2 \\ SGLANG_CACHE_DIT_BN=1 \\ SGLANG_CACHE_DIT_WARMUP=4 \\ SGLANG_CACHE_DIT_RDT=0.4 \\ SGLANG_CACHE_DIT_MC=4 \\ SGLANG_CACHE_DIT_TAYLORSEER=true \\ SGLANG_CACHE_DIT_TS_ORDER=2 \\ HCCL_BUFFSIZE=256 sglang serve \\ --model-path ./weights/MiniMax-H3/ \\ --model-type diffusion \\ --model-variant fl2va \\ --num-gpus 8 \\ --tp-size 2 \\ --sp-degree 4 \\ --attention-backend laser_attn \\ --port 30088 ``` For a four-NPU host, use TP2 + SP2: ```bash SGLANG_CACHE_DIT_ENABLED=true \\ SGLANG_CACHE_DIT_FN=2 \\ SGLANG_CACHE_DIT_BN=1 \\ SGLANG_CACHE_DIT_WARMUP=4 \\ SGLANG_CACHE_DIT_RDT=0.4 \\ SGLANG_CACHE_DIT_MC=4 \\ SGLANG_CACHE_DIT_TAYLORSEER=true \\ SGLANG_CACHE_DIT_TS_ORDER=2 \\ HCCL_BUFFSIZE=256 sglang serve \\ --model-path ./weights/MiniMax-H3/ \\ --model-type diffusion \\ --model-variant fl2va \\ --num-gpus 4 \\ --tp-size 2 \\ --sp-degree 2 \\ --attention-backend laser_attn \\ --port 30088 ``` Use `--attention-backend fa` to validate the Ascend Flash Attention path. T2VA request: ```bash curl -sS -X POST http://127.0.0.1:30088/v1/videos \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"MiniMaxAI/MiniMax-H3\", \"prompt\": \"At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out.\", \"seconds\": 5, \"task\": \"t2va\", \"conditions\": [], \"target\": { \"short_edge\": 768, \"aspect_ratio\": \"16:9\", \"duration_seconds\": 5.0 }, \"num_outputs_per_prompt\": 1, \"num_inference_steps\": 50, \"flow_shift\": 12.0, \"audio_flow_shift\": 3.0, \"seed\": 1101 }' ``` Result: https://github.com/user-attachments/assets/6fd97552-b6ce-446c-b8d2-c26f61f7c91f FL2VA request: ```bash curl -sS -X POST http://127.0.0.1:30088/v1/videos \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"MiniMaxAI/MiniMax-H3\", \"prompt\": \"The supplied frame continues with calm, natural motion and synchronized ambient sound.\", \"seconds\": 5, \"task\": \"fl2va\", \"conditions\": [ { \"type\": \"image\", \"uri\": \"./Forest.jpg\", \"role\": \"keyframe\", \"frame_index\": 0 } ], \"target\": { \"short_edge\": 768, \"aspect_ratio\": \"auto\", \"duration_seconds\": 5.0 }, \"num_outputs_per_prompt\": 1, \"num_inference_steps\": 50, \"flow_shift\": 12.0, \"audio_flow_shift\": 3.0, \"seed\": 2101 }' ``` Forest.jpg Result: https://github.com/user-attachments/assets/0cad1f72-86f4-45c2-bc70-5f8d4831ef97 ## Speed Tests and Profiling The Ascend FA path performs one packed `TND` attention operation using actual sequence lengths. For strided Ulysses tensors, Q/K/V are repacked through one allocation. Laser Attention computes only the real-token segment and zero-fills alignment-padding outputs, avoiding the previous short-segment fallback calculation. FA vs Laser Attention on T2VA: | Attention backend | Topology | End-to-end generation time | Speedup vs. 8-NPU FA | | --- | --- | ---: | ---: | | Ascend Flash Attention | 8 NPUs, TP2 + SP4 | 69.51 s | 1.00× | | Laser Attention | 8 NPUs, TP2 + SP4 | 55.07 s | 1.26× | | Laser Attention | 4 NPUs, TP2 + SP2 | 103.57 s | — | On the same eight-NPU topology, Laser Attention reduced end-to-end generation time by 20.8% compared with Ascend Flash Attention. The attention measurements use the same 5-second T2VA request at 1344×768, 124 frames, 24 FPS, 50 inference steps, and seed 1101. The launch commands list the measured Cache-DiT configuration explicitly: `FN=2`, `BN=1`, `WARMUP=4`, `RDT=0.4`, `MC=4`, TaylorSeer enabled, and TaylorSeer order 2. Experimental NPU RoPE, SwiGLU, and gate kernels were disabled; they are not present in the final PR. Laser Attention performance: T2VA: FL2VA: ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci`. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31495020724](https://github.com/sgl-project/sglang/actions/runs/31495020724) Latest PR Test (Extra): :x: [Run #31495020587](https://github.com/sgl-project/sglang/actions/runs/31495020587) ",
"url": "https://github.com/sgl-project/sglang/pull/33569",
"createdAt": "2026-08-04T15:43:02Z",
"updatedAt": "2026-08-13T07:51:02Z",
"timestamp": "2026-08-13T07:51:02Z",
"metrics": {
"reactions": 0,
"comments": 14
},
"labels": [
"documentation",
"npu",
"run-ci",
"diffusion"
],
"author": "OrangeRedeng",
"state": "open",
"assignees": [
"ping1jing2"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33572",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Cosmos3] Add cosmos3 Reasoner to llm only inference",
"text": " ## Motivation Cosmos3 Omni model has 2 transformer towers stacked on top of each other for video generation. First tower is understanding tower that is also trained for text/visual reasoning. Metrics are included in model card under Reasoning Benchmarks: https://huggingface.co/nvidia/Cosmos3-Nano#reasoning-benchmarks . This PR adds support for Reasoner only inference. ## Modifications Should be merged after: https://github.com/sgl-project/sglang/pull/31590 * Added new model type cosmos3_omni * Modified DefaultModelLoader with allow_patterns_overrides argument to allow reading Diffusers format checkpoints * Added Cosmos3 specific checkpoint key mappings * Extended Qwen3VL execution path checks with new model type (Cosmos3 Reason follows Qwen3VL architecture) ## Accuracy Tests Simple request to make sure model generates reasonable output, since rest of the implementation relies on Qwen3VL integration. ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31727450906](https://github.com/sgl-project/sglang/actions/runs/31727450906) Latest PR Test (Extra): :x: [Run #31727450532](https://github.com/sgl-project/sglang/actions/runs/31727450532) ",
"url": "https://github.com/sgl-project/sglang/pull/33572",
"createdAt": "2026-08-04T15:58:33Z",
"updatedAt": "2026-08-13T17:48:06Z",
"timestamp": "2026-08-13T17:48:06Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"diffusion"
],
"author": "Zhylkaaa",
"state": "open",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:33602",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [GLM5] Add opt-in PTPC FP8 projections on gfx950",
"text": "## Motivation The GLM-5.2 MXFP4 checkpoint keeps four attention projections in BF16: - fused `q_a + kv_a` - `q_b_proj` - `o_proj` This PR adds an opt-in FP8 path for those projections on gfx950. PTPC means per-token activation scaling and per-channel weight scaling. BF16 remains the default and fallback path. ## Modifications - Cache bpreshuffled FP8 projection weights while retaining the original BF16 parameters. - Use one flag, `SGLANG_DSA_FP8_PROJ_GEMM`, to switch all four projections between BF16 and PTPC FP8. - Reuse fused quantized inputs for `q_b_proj` and `o_proj` where available. - Keep fused `q_a + kv_a` on tuned Aiter BF16 for M≤512; use PTPC FP8 for M>512. - Limit the path to GLM-5.2 on gfx950 with Aiter enabled. ## Accuracy Tests GLM-5.2-MXFP4, TP4 MI355X: | Validation | Result | | --- | ---: | | PTPC GSM8K, 1319 examples | 0.933 | ## Speed Benchmarks GLM-5.2-MXFP4, TP4 MI355X, 8k input / 1k output. A. BF16 — `SGLANG_DSA_FP8_PROJ_GEMM=0`: | concurrency | TTFT (ms) | ITL (ms) | E2EL (ms) | output tok/s | | ---: | ---: | ---: | ---: | ---: | | 4 | 1027.49 | 11.773 | 13386.33 | 305.72 | | 8 | 1637.90 | 13.517 | 16432.02 | 498.29 | | 16 | 2915.32 | 16.130 | 21704.23 | 754.64 | | 32 | 5769.21 | 20.130 | 30926.24 | 1059.44 | | 64 | 10870.96 | 27.013 | 48250.20 | 1357.82 | B. PTPC FP8 — `SGLANG_DSA_FP8_PROJ_GEMM=1` (Δ vs BF16): | concurrency | TTFT (ms) | Δ | ITL (ms) | Δ | E2EL (ms) | Δ | output tok/s | Δ | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 4 | 972.22 | −5.38% | 11.820 | +0.40% (within noise) | 13387.67 | +0.01% | 305.76 | +0.01% | | 8 | 1615.05 | −1.40% | 13.560 | +0.32% (within noise) | 16423.83 | −0.05% | 498.43 | +0.03% | | 16 | 2877.66 | −1.29% | 15.917 | −1.32% | 21314.90 | −1.79% | 768.28 | +1.81% | | 32 | 5566.14 | −3.52% | 19.793 | −1.67% | 30263.03 | −2.14% | 1082.37 | +2.16% | | 64 | 10420.25 | −4.15% | 26.633 | −1.41% | 47426.84 | −1.71% | 1382.11 | +1.79% | **Baseline** = Image `rocm/sgl-dev:v0.5.16-rocm720-mi35x-20260804` plus these 6 PRs PR #30575: Triton sparse MLA prefill/decode PR #30715: fused indexer query Hadamard + FP8 quant PR #30808: gfx950 dense-MHA short-context fallback PR #31323: fused shared-expert append PR #31324: skip unnecessary dense-regime decode indexer work PR #33620: packed-MXFP4 absorbed K/V BMM enabled **This PR** = Baseline + the flag flip ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31623100648](https://github.com/sgl-project/sglang/actions/runs/31623100648) Latest PR Test (Extra): :x: [Run #31623100214](https://github.com/sgl-project/sglang/actions/runs/31623100214) ",
"url": "https://github.com/sgl-project/sglang/pull/33602",
"createdAt": "2026-08-04T21:06:40Z",
"updatedAt": "2026-08-12T17:34:22Z",
"timestamp": "2026-08-12T17:34:22Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"quant",
"deepseek"
],
"author": "Raiden-Makoto",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33604",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix Whisper transcription for audio over 30 seconds",
"text": "## Summary Whisper's feature extractor pads or truncates input to its fixed 3000-mel-frame (30-second) encoder window, so `/v1/audio/transcriptions` silently drops everything after 30 seconds when the serving layer sends a long file as one generation. This change: - splits long Whisper audio into contiguous chunks of at most 30 seconds; - places each seam at the quietest 100 ms RMS window within the final second of the chunk, matching the former vLLM speech-to-text behavior; - runs non-streaming chunks concurrently and stitches results in audio order; - streams chunks sequentially with one final finish frame; - offsets `verbose_json` segment timestamps by each chunk's start time; - preserves fused language auto-detection and reports the first chunk's detected language; - gates chunking through the transcription adapter, so Qwen3-ASR and other adapters remain unchanged. Each chunk receives an independent sampling-parameter dictionary because the multimodal processor consumes transcription-only keys while preparing decoder prompts. ## Validation - Added CPU unit coverage for energy-aware split points, contiguity, maximum duration, waveform preservation, short-audio behavior, ordered stitching, chunk failures and aborts, streaming, fused auto-detection, spaceless scripts, and verbose timestamp offsets. - Added GPU endpoint coverage using a 40-second clip whose speech begins after Whisper's 30-second encoder window: JSON transcription, `verbose_json` timestamps past 30 seconds, and streaming. - The audio splitter tests pass locally (5 tests). - Changed Python files pass AST, isort, Ruff, Black, codespell, CI registry, merge-conflict, and private-key checks; `compileall` and `git diff --check` pass. - Live validation with `openai/whisper-large-v3` on a 57.06-second file produced the complete transcript, including segments after 30 seconds, with one streaming stop frame. Split points and explicit-language output matched vLLM 0.11.0. --- ### CI States Latest PR Test (Base): :x: [Run #31727027933](https://github.com/sgl-project/sglang/actions/runs/31727027933) Latest PR Test (Extra): :hourglass_flowing_sand: [Run #31727027759](https://github.com/sgl-project/sglang/actions/runs/31727027759) ",
"url": "https://github.com/sgl-project/sglang/pull/33604",
"createdAt": "2026-08-04T21:15:55Z",
"updatedAt": "2026-08-13T17:54:10Z",
"timestamp": "2026-08-13T17:54:10Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"run-ci-extra"
],
"author": "shenxiul",
"state": "open",
"assignees": [
"JustinTong0323"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33614",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] Fix Dspark state divergence across TP rank",
"text": " ## Motivation Related to: https://github.com/sgl-project/sglang/issues/33289 To fix https://github.com/sgl-project/sglang/issues/33289 bug, upgrade NCCL to the newest version (2.30.7). When TP > 1, Dspark makes serveral sampling decisions on reach rank: 1. The draft Markov chain samples proposal tokens step by step (with in-graph philox noise introduced since https://github.com/sgl-project/sglang/pull/33298) 2. Target verify derives `correct_len` / `bonus` / `cap_trim_lens` 3. Prefill samples `next_token_ids` In order to maintain the speedup, SGLang skips cross-rank sync of sampled tokens by default (I found the comment here: https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/sampler.py#L496). So nothing forces these values to agree. Once one rank commits a different token or accept length, sequence lengths and KV state drift apart and a later collective deadlocks (demonstrated by fault injection in the Accuracy Tests section below). ## Modifications Broadcast rank 0's sampling decisions before they affect state. The idea follows the existing `verify_lens` handling in `dspark_planner.py` and the EAGLE proposals in https://github.com/sgl-project/sglang/issues/29003 and https://github.com/sgl-project/sglang/pull/31478: 1. Broadcast every Markov proposal token in **both** eager and graph-folded sampling. Sync only at the end of a draft block is too late because each step depends on the previous token. 2. Broadcast `correct_len`, `bonus`, and `cap_trim_lens` in the eager verify path and the captured verify epilogue, before finalization, token output, or KV commit. Separate `all_reduce(MIN)` calls could combine fields from different ranks. 3. Broadcast prefill `next_token_ids` before entering the draft path. `DsparkTpSync` in `dspark_tp.py` (newly added file) uses the TP group's PyNCCL communicator. Process-group collectives cannot be captured in CUDA graphs, and using the model's existing TP communicator preserves collective ordering across eager and captured execution. With DP attention, it uses `attn_tp_group`, matching `verify_lens_broadcast_group` in original `dspark_planner.py`. TP=1 is a no-op. The reason why I don't match RNG seeds is because ranks can still diverge if they consume different numbers of random values, and it does not cover non-RNG differences. In addition, for TP > 1, this add `gamma + 3` broadcasts of a `[bs]` tensor per decode step, all within the captured graphs. ## Accuracy Tests All tests below on following configs: - 2 DGX Sparks - TP=2 - DeepSeek-V4-Flash-0731 with 131k context - Decode CUDA graphs and folded sampling enabled - Add https://github.com/sgl-project/sglang/pull/33407 code. It fixes an unrelated bug for SM12x sparse-MLA topk dispatch crash at boot. Dspark + DeepSeek V4 Flash cannot start on DGX Sparks without it. - **NCCL 2.30.7** for SGLang's PyNCCL communicator via `SGLANG_NCCL_SO_PATH` (torch's bundled 2.28.9 wedges this workload's graph/eager mix with or without this patch; a separate problem from this fix) For accuracy test, we run`python3 -m sglang.test.few_shot_gsm8k --num-questions 200` (temp=0): accuracy 0.965, invalid 0.000. This is expected since there's no mathematical changes. To demonstrate the failure mode this patch addresses, we fault-injected a rank-local accept divergence into both builds: rank 1 lowers every accept length by one (`correct_len = torch.clamp(correct_len - 1, min=0)`, NCCL 2.30.7 in both _runs). **_Note: the fault reproduction code has been removed before submitting PR._** Without the fix, the server deadlocks before finishing startup warmup. With the fix, the same injection is harmless: rank 0's broadcast overwrites the divergent values, the server boots, and a temp=0 request returns the expected output. As a stability check on the fix itself (its broadcasts run inside the captured decode graphs), I sent 42 sequential chat requests with `ignore_eos=true` and `max_tokens=512` (30 at temp=1, then 12 at temp=0) and probed `/health_generate` after every request with a 180s client timeout, so a stuck collective surfaces as a failed request instead of a silent hang. As the result, all 42 returned HTTP 200 with `finish_reason=length`. 21504 tokens generated in total and mean acc_len during the temp=1 run was 2.86 (block_size=4). Same for temp=0, neither of them occur any error. ## Speed Tests and Profiling Test on 24 natural prompts, 256 output tokens, concurrency=1, temp=1, same setup as above: | Requests | Output throughput | TTFT mean/median/P90 (ms) | TPOT mean/median/P90 (ms) | |---|---|---|---| | 24 | 31.9 tok/s | 157.6 / 153.7 / 178.6 | 30.8 / 31.7 / 35.5 | Comparing with pre-fix TP baseline is impossible because it cannot survive under such load. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31302168114](https://github.com/sgl-project/sglang/actions/runs/31302168114) Latest PR Test (Extra): :x: [Run #31302167999](https://github.com/sgl-project/sglang/actions/runs/31302167999) ",
"url": "https://github.com/sgl-project/sglang/pull/33614",
"createdAt": "2026-08-04T23:11:47Z",
"updatedAt": "2026-08-13T08:16:31Z",
"timestamp": "2026-08-13T08:16:31Z",
"metrics": {
"reactions": 0,
"comments": 12
},
"labels": [],
"author": "JackZeng0208",
"state": "open",
"assignees": [
"hnyls2002"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33623",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Kimi K3] Fuse MLA gate projection into QKV-A GEMM",
"text": "Replaces #33521, which was automatically closed when its base branch was deleted. Rebasing to main as requested in the review. ## Motivation Kimi-K3 MLA computes QKV-A and the TP-local output gate from the same hidden states. They currently run as separate GEMMs. Fuse them to reduce projection cost while keeping the gate output TP-local. ## Modifications - Merge unquantized BF16/FP16 QKV-A and g_proj weights after loading. - Run one merged projection and stash the gate for the pre-o_proj gate multiply, preserving quantized and LoRA fallback paths. - Select tile_m=32 for the K3 merged [3648, 7168] fused-A shape so its 114 CTAs fit in one H200 wave. - Add CPU equivalence and SM90 CUDA Graph tests for M=1/8/16 at K3 TP8 dimensions. ## Accuracy Tests - H200 model unit test: 2/2 passed. - Merged QKV-A and gate outputs matched separate F.linear references. - Attention-layer A/B correctness passed for M=1/4/8/16 and context 512/4096. - Full pre-commit passed on the three changed files. ## Speed Tests and Profiling Single H200, BF16, FlashInfer MLA, CUDA Graph, K3 TP8 single-rank dimensions: | Decode M | Context 512 | Context 4096 | | ---: | ---: | ---: | | 1 | +4.47% | +2.90% | | 4 | +2.99% | +2.25% | | 8 | +1.28% | +1.01% | | 16 | +3.32% | +1.57% | The merged fused-A kernel dropped from 16.8-18.0 us to 14.9-15.1 us after the tile selection fix. ## Checklist - [x] Format the code with the project pre-commit hooks. - [x] Add unit tests for the new functionality. - [x] Provide accuracy and performance results. - [x] Keep quantized and LoRA paths on their existing fallback. --- ### CI States Latest PR Test (Base): :x: [Run #31550507014](https://github.com/sgl-project/sglang/actions/runs/31550507014) Latest PR Test (Extra): :x: [Run #31550506888](https://github.com/sgl-project/sglang/actions/runs/31550506888) ",
"url": "https://github.com/sgl-project/sglang/pull/33623",
"createdAt": "2026-08-05T00:23:24Z",
"updatedAt": "2026-08-13T05:40:28Z",
"timestamp": "2026-08-13T05:40:28Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"run-ci",
"jit-kernel",
"bypass-fastfail",
"run-ci-extra"
],
"author": "Hayden727",
"state": "closed",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33634",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] Add test for --dllm-fdfo",
"text": " ## Motivation This PR adds a test case to verify that the `--dllm-fdfo` (First-Done-First-Out) scheduling flag takes effect and improves performance for Diffusion LLM models on Ascend NPU backend. When enabled, FDFO prioritizes requests that complete their diffusion steps earlier, reducing waiting time and improving hardware utilization. ## Modifications - Add `TestLLaDA2MiniFDFO` test class in `test_ascend_llada2.py` - Benchmark LLaDA2.0-mini model with 128 prompts (3584 input tokens, 1024 output tokens) - Compare performance between `--dllm-fdfo` enabled and disabled (`--no-dllm-fdfo`) - Assert that enabling FDFO yields: - Lower mean TTFT (Time-To-First-Token) - Higher total throughput - Lower P99 TTFT - Register test with NPU CI pipelines (stage-b-test-2-npu-a3 and nightly-2-npu-a3) ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31663841896](https://github.com/sgl-project/sglang/actions/runs/31663841896) Latest PR Test (Extra): :x: [Run #31663841776](https://github.com/sgl-project/sglang/actions/runs/31663841776) ",
"url": "https://github.com/sgl-project/sglang/pull/33634",
"createdAt": "2026-08-05T03:00:21Z",
"updatedAt": "2026-08-13T03:25:23Z",
"timestamp": "2026-08-13T03:25:23Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"npu"
],
"author": "longxin9715",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33647",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[KDA] Add FlashInfer CAKE prefill and decode backends",
"text": "## Motivation This PR integrates exported FlashInfer CAKE KDA kernels into SGLang and adds an explicit `cake` backend for recurrent KDA prefill and decode: ```bash --mamba-ssm-dtype bfloat16 --linear-attn-backend cake ``` The shared flag selects CAKE for both phases. Existing per-phase flags remain authoritative when a mixed configuration is requested, and the verify backend remains independently selected. The serving-native packed decode API is being upstreamed in [flashinfer-ai/flashinfer#4378](https://github.com/flashinfer-ai/flashinfer/pull/4378). This PR depends on that PR landing and on SGLang consuming a FlashInfer revision containing it. H=12 prefill support is already available through [flashinfer-ai/flashinfer#4351](https://github.com/flashinfer-ai/flashinfer/pull/4351), and the overall work is tracked in [flashinfer-ai/flashinfer#4254](https://github.com/flashinfer-ai/flashinfer/issues/4254). ## Modifications - Add `cake` to the linear-attention backend choices and dispatcher. - Make `--linear-attn-backend cake` select CAKE for both prefill and decode, while preserving explicit per-phase overrides. - Integrate CAKE prefill through `flashinfer.kda.recurrent_kda`. - Integrate the Kimi-K3 T=1 serving hot path through the independent `flashinfer.packed_kda_decode` API. - Forward SGLang's post-convolution packed QKV, raw gate/beta logits, caller-owned state pool, and original int32 state indices directly. - Avoid Q/K/V splitting and copying, gate/beta preprocessing, and state gather/scatter on the covered packed-decode path. - Preserve `-1` CUDA-graph padding rows and update active state slots in place. - Log once when the serving-native packed CAKE route is actually exercised. - Bypass Kimi-K3's separate fused decode path when `cake` is explicitly selected, ensuring that the requested backend is exercised. ## Packed decode contract and fallback The CAKE packed path is selected only for the frozen FlashInfer contract: - `T=1`, `H=HV=12`, `K=V=128`; - BF16 packed QKV, raw gate/beta, output, and recurrent state; - FP32 `A_log[12]` and `dt_bias[1536]`; - packed QKV `[B, 4608]`; - state pool `[N, 12, 128, 128]` with compact inner dimensions and a positive valid outer slot stride; - contiguous CUDA int32 state indices `[B]`, with unique in-bounds active indices and `-1` for inactive graph rows; - `scale=1/sqrt(128)`, `lower_bound=-5`, and `1 <= B <= 65535`; - exact SM100 with CUDA >= 12.8 or SM103 with CUDA >= 12.9. Unsupported shapes, dtypes, layouts, architectures, GQA, a missing packed API, and ReplaySSM route through the existing Triton packed implementation. Triton errors are propagated rather than hidden. CAKE prefill similarly retains Triton for contracts requiring speculative or state tracking, intermediate-state returns, T=1-only batches, unsupported dimensions, or prefill CUDA-graph capture. Decode CUDA graphs are supported by the packed CAKE path. ## Validation Candidate revisions: - SGLang PR head: `a6e3452c1cf42ada75fbc301ec99439651d1dfd2`; - validated SGLang snapshot: `2c86a82dab900f4fb1589951e13a6cd8eb74bd68`; - identical tracked tree for both SGLang revisions: `a50bae03fbfb97711d5189fa778155ab82f7e095`; - FlashInfer performance commit: `89a5d3fcee8f8ddcff9da5547d095ffd73c1ef79`; - FlashInfer #4378 head after non-kernel review cleanup: `826ed5f2fdc301440aa1ec799066f2fa3dfb63d5`. GB300 / SM103 validation: ```text FlashInfer JIT specification tests: 19 passed FlashInfer packed KDA GPU tests: 26 passed SGLang CPU adapter/selector tests: 9 passed SGLang packed KDA GPU tests: 7 passed, 11 deselected Pre-commit on changed files: passed ``` The SGLang GPU tests cover B=1/8/64/128, the exact H=12 serving contract, non-contiguous packed-QKV outer strides, non-identity state slots, `-1` padding, padded state-pool outer strides, bitwise preservation of untouched slots, forwarding of original state/index pointers, GQA fallback, and CUDA Graph replay with changed inputs and indices through stable pointers. FlashInfer #4378 additionally validates B=1/8/16/31/32/64/128/256/512 on B200 and GB300, current-stream execution, production strides, a 512-step FP64 diagnostic, and synccheck/memcheck with zero errors on both architectures. ### Exported public API vs exact SGLang packed Triton Strict CUPTI, cold L2, 30 rounds per backend, same inputs, and alternating AB/BA order: | GPU | Direct B=1–512 geomean | CUDA Graph B=1–512 geomean | |---|---:|---:| | B200 / SM100 | `1.1058x` | `1.0863x` | | GB300 / SM103 | `1.0708x` | `1.0526x` | The complete per-batch tables, including retained low points, are in FlashInfer #4378. ## Same-configuration Kimi-K3 E2E Performance and accuracy use the same server launch configuration: - Kimi-K3 TP8/DCP8 on 8-GPU GB300 allocations; - BF16 recurrent state, HiCache, and full decode CUDA graphs; - identical server and workload arguments apart from the selected linear-attention backend and unavoidable port values; - random seed 42. This is a same-configuration cross-run comparison, not a current-source paired A/B in one allocation. The recorded Triton baseline used SGLang `0a3efffc2f7fe8e0898473d86320d006973fdd71` and FlashInfer `6b4ecd9d459caf389ce8c4a92bbb491214f2eac8` (Slurm `371396`). The CAKE run used the validated SGLang snapshot `2c86a82dab900f4fb1589951e13a6cd8eb74bd68` and FlashInfer `826ed5f2fdc301440aa1ec799066f2fa3dfb63d5` (Slurm `378402`). Both used the same GB300 SKU and the configuration above. ### Serving performance Workload: 128 requests, 128 input tokens, 512 requested output tokens, concurrency 64, greedy decoding, one warmup, cache flush before each measured run, and three repetitions per backend. | Metric | Triton runs | CAKE runs | Triton median | CAKE median | Speedup | |---|---|---|---:|---:|---:| | Decode output tok/s | `2008.276 / 1979.075 / 1959.839` | `1992.671 / 1985.439 / 1967.150` | `1979.075` | `1985.439` | `1.0032x` | | Total tok/s | `2510.345 / 2473.844 / 2449.798` | `2490.838 / 2481.799 / 2458.938` | `2473.844` | `2481.799` | `1.0032x` | | Median E2E latency (ms) | `16301.127 / 16542.684 / 16705.231` | `16432.781 / 16490.611 / 16645.532` | `16542.684` | `16490.611` | `1.0032x` | | Median TTFT (ms) | `650.728 / 805.163 / 741.787` | `666.114 / 748.696 / 808.828` | `741.787` | `748.696` | `0.9908x` | | Median TPOT (ms) | `30.656 / 30.806 / 30.934` | `30.794 / 30.814 / 30.987` | `30.806` | `30.814` | `0.9998x` | | Median ITL (ms) | `30.459 / 30.661 / 30.799` | `30.402 / 30.642 / 30.821` | `30.661` | `30.642` | `1.0006x` | All repetitions completed with zero request errors. The CAKE route log was checked only after real generation requests had exercised the server. All six runs completed 128/128 requests with the expected 16,384 input and 65,536 output tokens. The output-throughput median is nominally `1.0032x`, but the decode TPOT median is effectively flat (`0.9998x`). Given the cross-run method and this noise-scale delta, these data do not establish a material E2E speedup. ### GSM8K accuracy Protocol: 200 examples, five-shot completion API, greedy sampling, maximum 512 tokens, 128 client threads, and three repetitions against the same server process used for the performance measurements. | Triton runs | CAKE runs | Triton median | CAKE median | Delta | Gate | |---|---|---:|---:|---:|---| | `0.970 / 0.980 / 0.985` | `0.975 / 0.985 / 0.975` | `0.980` | `0.975` | `-0.005` | PASS | The quality gate requires a CAKE median of at least `0.97` and a median regression of no more than `0.01` relative to Triton. ## Checklist - [x] Add the serving-native packed CAKE decode route. - [x] Preserve explicit Triton fallbacks for unsupported contracts. - [x] Add targeted CPU and GPU correctness tests. - [x] Run pre-commit on all changed files. - [ ] Merge FlashInfer #4378. - [ ] Update SGLang's FlashInfer dependency to a revision containing #4378. - [x] Report same-configuration Kimi-K3 performance and GSM8K results. --- ### CI States Latest PR Test (Base): :x: [Run #31091488033](https://github.com/sgl-project/sglang/actions/runs/31091488033) Latest PR Test (Extra): :no_entry_sign: [Run #31224311301](https://github.com/sgl-project/sglang/actions/runs/31224311301) ",
"url": "https://github.com/sgl-project/sglang/pull/33647",
"createdAt": "2026-08-05T04:53:30Z",
"updatedAt": "2026-08-13T17:09:43Z",
"timestamp": "2026-08-13T17:09:43Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"high priority",
"kernel",
"release-highlight"
],
"author": "yyihuang",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33672",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSV4] Support raw-index output in TopK v2",
"text": "## Summary Allow TopK v2 to populate a raw-index output buffer. DSV4 sparse prefill internally allocates `c4_sparse_raw_indices` even when `--enable-return-indexer-topk` is disabled. The previous `raw_indices is None` gate therefore forced the InfX sparse-prefill path to fall back to TopK v1. When `--enable-return-indexer-topk` is enabled, the capture-only temporary buffer could additionally take precedence over the functional sparse-prefill buffer. This change makes the sparse-prefill buffer canonical, passes it through TopK v2, and lets the capturer copy the same result afterward. ## Changes - Pass the raw-output buffer to `topk_transform_512_v2`. - Prioritize `core_metadata.c4_sparse_raw_indices`; the capturer reads the same result afterward. - Add a regression test covering TopK v2 with sparse prefill and capture enabled. No CUDA kernel, ABI, metadata format, default, or HiSparse behavior changes. Related: #32771 contains the same buffer-ordering change; #33288 owns chunked-path metadata refresh. ## Testing - DSV4 indexer unit tests: **8 passed, 12 subtests passed** - TopK v2 GPU subset on GB300: **36 passed, 208 deselected** - `pre-commit run --all-files` A prior GB300 microbenchmark (8,192 queries, max C4 length 2,048, `topk=1024`, metadata planning excluded) measured **0.1023 ms** for v1 and **0.0646 ms** for v2 (**36.9% lower kernel latency**). --- ### CI States Latest PR Test (Base): :x: [Run #31662394592](https://github.com/sgl-project/sglang/actions/runs/31662394592) Latest PR Test (Extra): :x: [Run #31662394475](https://github.com/sgl-project/sglang/actions/runs/31662394475) ",
"url": "https://github.com/sgl-project/sglang/pull/33672",
"createdAt": "2026-08-05T07:48:14Z",
"updatedAt": "2026-08-13T04:44:04Z",
"timestamp": "2026-08-13T04:44:04Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "weireweire",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33673",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MiniMax-M3] Add MiniMax-M3 DSpark support",
"text": "## Summary This PR adds MiniMax-M3 DSpark speculative decoding support with MiniMax-M3 as the target model and MiniMax-M3-DSpark as the draft model. MiniMax-M3-DSpark is a draft model and cannot be served standalone. When it is paired with MiniMax-M3, several integration points are required across auxiliary hidden-state capture, VL wrapper forwarding, target-verify metadata, MiniMax sparse attention geometry, DP padding handling, and DFLASH/DSPARK config loading. This PR wires those pieces together and validates two-node TP16 serving for MiniMax-M3 + MiniMax-M3-DSpark. It also keeps the implementation compatible with the latest MiniMax-M3 NPU and EAGLE3 changes on `main`. ## Modifications ### MiniMax-M3 target model - Add `set_dspark_layers_to_capture` to `MiniMaxM3SparseForCausalLM`. - Reuse the existing layer-capture setup shared with EAGLE3 so DSpark and EAGLE3 mark `_is_layer_to_capture` consistently. - Require explicit DSpark layer IDs, since DSpark needs deterministic auxiliary hidden-state capture points. - Preserve compatibility with the latest MiniMax-M3 NPU and EAGLE3 changes from `main`. ### MiniMax-M3 VL wrapper - Add `set_dspark_layers_to_capture` to the VL wrapper. - Reuse the same capture setup as EAGLE3 for layer marking. - Forward `aux_hidden_states` to the logits processor when auxiliary capture is enabled. - Keep the tuple guard for warmup/idle forwards, where the inner language model may return a plain hidden-state tensor instead of `(hidden_states, aux_hidden_states)`. ### ForwardBatch TARGET_VERIFY metadata - Initialize TARGET_VERIFY `extend_*` metadata in `ForwardBatch` instead of synthesizing it inside the MiniMax sparse attention backend. - Assign zero-length verify rows for DP-padded requests so `sum(extend_seq_lens_cpu)` reflects the real non-padded verify token count. - Add `target_verify_q_cap()` to provide a capture-safe fixed query bound for CUDA Graph replay. - Add `build_uniform_target_verify_layout()` to build uniform target-verify layouts with both device tensors and host-side metadata. - Avoid CUDA Graph capture-time device-to-host sync and CPU-to-CUDA copies by carrying `verify_lens_cpu` and `total_verify_tokens`. ### MiniMax sparse attention - Consume TARGET_VERIFY geometry from `ForwardBatch`. - Support both ragged target-verify layouts and uniform CUDA Graph capture layouts through `build_ragged_target_verify_geometry`. - Resolve extend metadata through a shared path that returns `cu_seqlens`, cache sequence lengths, prefix lengths, and host-side per-request verify lengths. - Trim DP-padded `q` and `idx_q` rows before entering the sparse kernel. - Reject DP-padded TARGET_VERIFY requests when per-request verify lengths are unavailable, instead of falling back to `batch_size * verify_len`. - Keep the latest NPU sparse attention paths, metadata cache, lazy GPU sparse-kernel import, and EAGLE3-compatible behavior from `main`. ### DFLASH / DSPARK config handling - Add a fallback for reading `sliding_window` from the raw `config.json` when the loaded Transformers config exposes it as `None`. - Preserve MiniMax-M3-DSpark compatibility with DFLASH/DSPARK draft-model loading. ## Error cases fixed This PR fixes the following MiniMax-M3 + MiniMax-M3-DSpark serving failures: ```text ValueError: DFLASH sliding_attention layers require config.sliding_window. ``` ```text ValueError: Model MiniMaxM3SparseForConditionalGeneration implements neither set_dspark_layers_to_capture nor set_dflash_layers_to_capture, one of which is required for DFLASH/DSPARK. ``` ```text AttributeError: 'NoneType' object has no attribute 'device' ``` ```text ValueError: DFLASH target_hidden feature dim mismatch. Expected shape [N, 36864] (num_context_features=6, hidden_size=6144), but got shape=(..., 6144). ``` ```text RuntimeError: CUDA error: operation failed due to a previous error during capture. ``` It also fixes two target-verify correctness issues: - DP-padded TARGET_VERIFY batches no longer use `batch_size * verify_len` as the real sparse-kernel token count. - Ragged and uniform target-verify paths now use shared per-request geometry, so padded rows are trimmed before the MiniMax sparse kernel. ## Validation ### Static checks ```bash python3 -m py_compile \\ python/sglang/srt/layers/attention/minimax_sparse_backend.py python/sglang/srt/model_executor/forward_batch_info.py python/sglang/srt/speculative/dflash_utils.py python/sglang/srt/models/minimax_m3_vl.py python/sglang/srt/models/minimax_m3.py ``` ### Two-node TP16 serving Validated with MiniMax-M3 as the target model and MiniMax-M3-DSpark as the DSPARK draft model: ```bash python3 -m sglang.launch_server \\ --model-path /models/MiniMax-M3 \\ --served-model-name MiniMax-M3 \\ --reasoning-parser auto \\ --tool-call-parser auto \\ --trust-remote-code \\ --tp-size 16 \\ --nnodes 2 \\ --node-rank <0-or-1> \\ --dist-init-addr 192.168.1.81:20000 \\ --speculative-algorithm DSPARK \\ --speculative-draft-model-path /models/MiniMax-M3-DSpark \\ --speculative-dspark-block-size 8 \\ --chunked-prefill-size 8192 \\ --mem-fraction-static 0.80 \\ --dtype bfloat16 \\ --host 0.0.0.0 \\ --port 31000 ``` ### End-to-end serving results | Metric | Input 8K / Output 512 | Input 32K / Output 512 | |---|---:|---:| | Successful requests | 256 | 256 | | Request throughput | 0.14 req/s | 0.05 req/s | | Input token throughput | 1140.76 tok/s | 1518.18 tok/s | | Output token throughput | 73.01 tok/s | 24.29 tok/s | | Peak output throughput | 80.00 tok/s | 75.00 tok/s | | Total token throughput | 1213.76 tok/s | 1542.47 tok/s | | Effective concurrency | 15.76 | 15.91 | | DSpark accept length | 3.39 | 3.40 | | Mean E2E latency | 110538.41 ms | 335299.96 ms | | Median E2E latency | 97803.34 ms | 286638.45 ms | | P99 E2E latency | 266417.01 ms | 876994.82 ms | | Mean TTFT | 7546.60 ms | 54129.68 ms | | Median TTFT | 5068.49 ms | 42576.83 ms | | P99 TTFT | 64566.62 ms | 239261.61 ms | | Mean TPOT | 201.55 ms | 550.24 ms | | Median TPOT | 177.17 ms | 444.06 ms | | P99 TPOT | 511.15 ms | 1607.82 ms | ### Accuracy | Benchmark | Examples | Sampling | Repeats | Score | Latency | |---|---:|---|---:|---:|---:| | GSM8K | 1,314 | Default evaluation settings | 8 | 0.970 mean | 796.733 s mean | | GPQA | 32 | `temperature=1.0`, `top_p=0.95`, `max_tokens=128000` | 8 | 0.992 mean | 11,345.289 s mean | --- ### CI States Latest PR Test (Base): :x: [Run #31677831954](https://github.com/sgl-project/sglang/actions/runs/31677831954) Latest PR Test (Extra): :x: [Run #31677831868](https://github.com/sgl-project/sglang/actions/runs/31677831868) ",
"url": "https://github.com/sgl-project/sglang/pull/33673",
"createdAt": "2026-08-05T08:02:19Z",
"updatedAt": "2026-08-13T07:28:21Z",
"timestamp": "2026-08-13T07:28:21Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "shiyang814-cpu",
"state": "open",
"assignees": [
"hnyls2002"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33684",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Weight Cache] Support static DP/EP layouts",
"text": "## Motivation Weight-cache daemons and engines must construct identical model-parallel layouts before CUDA IPC can safely map a cached MoE shard. The existing daemon launcher rejected `dp_size > 1`, preventing static DP/EP deployments from using the daemon-backed weight cache. This change adds static DP/EP layout support to the CUDA IPC weight-cache path. It is the implementation foundation for the DP/EP roadmap in #33522. ## Modifications - Propagate static DP/EP, MoE DP, attention DP, MoE A2A, and DeepEP-mode configuration from the engine into weight-cache daemons. - Initialize daemon-side model-parallel and MoE state with the same static layout as the engine. - Extend `CacheConfig` with MoE DP/EP shard identity and layout fields, so daemon/client compatibility validation rejects incompatible shards. - Fix static Mooncake EP dispatch to use an all-active rank mask when Elastic EP state is absent. - Route Qwen3 MoE through the existing DeepEP-class backend predicate so Mooncake uses the EP execution path. - Add CPU coverage for MoE shard-rank calculation, extended cache fingerprints, and daemon DeepEP-mode forwarding. The implementation supports static layouts. Dynamic expert placement, EPLB, redundant experts, and recovery lifecycle ownership remain roadmap work. ## Accuracy Tests - `PYTHONPATH=python python -m pytest -q test/registered/unit/model_loader/test_weight_cache_protocol.py` - `29 passed` - Eight-GPU Qwen3-30B-A3B-FP8 daemon IPC serving with `TP=DP=EP=8`, Mooncake low-latency A2A, DP attention, and DP LM head. - 8 daemons exported IPC handles. - 8 schedulers logged `Loaded model via IPC`. - A deterministic completion generated `Paris` for `The capital of France is`. ## Speed Tests and Profiling N/A. This change establishes static DP/EP correctness and does not claim a new performance result. ## Checklist - [x] Format modified Python files with Black; CI will run the repository pre-commit hooks. - [x] Add unit tests according to [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31676640818](https://github.com/sgl-project/sglang/actions/runs/31676640818) Latest PR Test (Extra): :x: [Run #31676640548](https://github.com/sgl-project/sglang/actions/runs/31676640548) ",
"url": "https://github.com/sgl-project/sglang/pull/33684",
"createdAt": "2026-08-05T09:10:25Z",
"updatedAt": "2026-08-13T09:36:11Z",
"timestamp": "2026-08-13T09:36:11Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "UNIDY2002",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33685",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU CI] Reorganize test output/log directory structure with workflow context",
"text": " ## Motivation Restructure NPU CI output/log persistence so runs from different workflows are distinguishable by directory and re-runs never overwrite prior results, and migrate nightly single-node tests from a per-case matrix to per-suite jobs. ## Modifications - Reorganize outputs/logs under `{branch_label}-{create_time}-{run_id}-{run_attempt}/{workflow_name}/{test_type}/{tc_name}`; create_time is UTC+8 date + minute precision, recorded once at run start and shared across jobs, so re-runs (via run_attempt) never overwrite prior results. - Plumb branch_label/workflow_name/create_time through nightly-test-npu via a single `run_start_metadata` JSON, consumed by the single-node stage and multi-node e2e templates; whitelist branch_label characters for safe path use. - Replace the per-case `nightly-test-npu-e2e-single-node.yml` matrix with six per-suite jobs (`nightly-{perf,acc}-{2,4,16}-npu-a3`) that run `run_suite.py --nightly --continue-on-error --timeout-from-est-time`. - Extend `_npu-single-node-test-stage.yml` with a nightly mode: install test dependencies, upload metrics.json artifacts and back up plogs so nightly logs/results keep the structured layout. - Register cases to the new `nightly-acc-*` / `nightly-perf-*` suites and add them to `run_suite.py` NIGHTLY_SUITES (keeping existing base-c registrations for PR tests). - Add RUN_LABEL/TIMESTAMP and missing env vars to k8s templates; derive run_label from the persistence directory and persist metrics/plog under the same run prefix in the accuracy/performance utilities. ## Accuracy Tests N/A. CI configuration and output/log path changes only; model outputs are not affected. ## Speed Tests and Profiling N/A. CI configuration and output/log path changes only; inference speed is not affected. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31698149322](https://github.com/sgl-project/sglang/actions/runs/31698149322) Latest PR Test (Extra): :x: [Run #31698149326](https://github.com/sgl-project/sglang/actions/runs/31698149326) ",
"url": "https://github.com/sgl-project/sglang/pull/33685",
"createdAt": "2026-08-05T09:13:18Z",
"updatedAt": "2026-08-13T17:39:58Z",
"timestamp": "2026-08-13T17:39:58Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"deepseek",
"npu",
"run-ci"
],
"author": "pllimax",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33726",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(bcg): preserve Qwen3-VL DeepStack inputs during replay",
"text": "## Motivation Prefill breakable CUDA graph (BCG) replay should preserve the same model semantics as eager execution. Qwen3-VL passes intermediate visual features through `input_deepstack_embeds` and adds them at selected language-model layers. However, BCG did not copy this request-specific tensor into a stable graph input slot. As a result, normal BCG execution behaved like an explicit zero-DeepStack run rather than eager execution. Requests still completed successfully, making this a silent correctness issue. ## Reproduction Environment: - Model: `Qwen/Qwen3-VL-8B-Instruct` - Revision: `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b` - GPU: NVIDIA H200 - Dtype: BF16 - Decoding: greedy, `max_new_tokens=20` - Input: deterministic RGB vertical-stripe image The GPU test harness temporarily enabled Qwen3-VL for BCG testing. The experiment compares four arms: 1. `eager_normal` 2. `eager_zero_deepstack` 3. `bcg_normal` 4. `bcg_zero_deepstack` The zero-DeepStack arms replace `input_deepstack_embeds` with zeros immediately before the language-model forward call. Each result is formatted as: ```text (common_prefix_length, all_token_ids_equal, maximum_absolute_logprob_difference) ``` | Comparison | Before fix | After fix | |---|---:|---:| | `bcg_normal` vs `eager_normal` | `(7, False, 1.154)` | `(15, True, 0.071)` | | `bcg_normal` vs `bcg_zero_deepstack` | `(20, True, 0.000)` | `(7, False, 1.148)` | | `bcg_zero_deepstack` vs `eager_zero_deepstack` | `(20, True, 0.066)` | `(20, True, 0.066)` | | `eager_zero_deepstack` vs `eager_normal` | `(7, False, 1.138)` | `unchanged` | Before the fix, `bcg_normal` was identical to `bcg_zero_deepstack` and diverged from `eager_normal`. After the fix, `bcg_normal` matches `eager_normal` at the token level and correctly diverges from `bcg_zero_deepstack`. The zero-DeepStack control remains unchanged. Instrumentation also confirmed that `input_deepstack_embeds` was present and non-zero at the language-model entry before replay. The RGB image is used only as a deterministic semantic-equivalence test, not as a visual-quality benchmark. ## Modifications - Add an explicit `supports_bcg_deepstack_replay` capability to Qwen3-VL. - Add an optional stable `input_deepstack_embeds` buffer and graph slot. - Allocate the slot only for multimodal models that opt in and have a non-zero DeepStack width. - Pass the stable slot during BCG capture so the DeepStack additions are included in the graph. - Copy the request-specific DeepStack tensor into the slot before replay. - Clear missing and padded regions to prevent stale data from previous requests. - Add unit tests for capability gating, buffer allocation, slot registration, and inactive model paths. `input_embeds` cannot be reused because DeepStack has a different shape and is consumed at selected intermediate language-model layers. Current Qwen3.5 checkpoints have no active DeepStack embeddings, so the new replay path remains inactive for them. The BCG allowlist is unchanged. ## Accuracy Tests ```bash pytest \\ test/registered/unit/model_executor/test_deepstack_replay_slot.py \\ test/registered/unit/model_executor/test_cuda_graph_buffer_registry.py \\ test/registered/unit/model_executor/test_prefill_cuda_graph_runner.py \\ test/registered/unit/model_executor/test_prefill_cuda_graph_runner_helpers.py ``` Result: ```text 57 passed, 21 warnings, 2 subtests passed in 10.55s ``` This includes 11 new tests and 46 existing adjacent tests. The GPU four-arm validation results are shown in the Reproduction section. ## Speed Tests and Profiling This is a correctness fix, not a speed optimization. Models that do not activate DeepStack replay do not allocate the new buffer or perform the replay copy. For Qwen3-VL-8B with `max_num_tokens=8192`, `hidden_size=4096`, three DeepStack embeddings, and BF16, the maximum additional buffer is: ```text 8192 × 4096 × 3 × 2 bytes = 192 MiB ``` No replay-latency benchmark was performed because Qwen3-VL is not currently enabled in the production BCG allowlist. ## Checklist - [x] Format your code according to the [[Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit)](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [[Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests)](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [[Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations)](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). No user-facing API or configuration change. - [ ] Provide accuracy and speed benchmark results according to [[Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy)](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [[Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed)](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). Accuracy results and memory overhead are reported above; latency was not benchmarked. - [x] Follow the SGLang code style [[guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance)](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31674823658](https://github.com/sgl-project/sglang/actions/runs/31674823658) Latest PR Test (Extra): :x: [Run #31674823338](https://github.com/sgl-project/sglang/actions/runs/31674823338) ",
"url": "https://github.com/sgl-project/sglang/pull/33726",
"createdAt": "2026-08-05T15:24:02Z",
"updatedAt": "2026-08-13T09:28:57Z",
"timestamp": "2026-08-13T09:28:57Z",
"metrics": {
"reactions": 0,
"comments": 11
},
"labels": [
"run-ci"
],
"author": "bowenwan6",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33743",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MoE] Fix flashinfer TRT-LLM BF16 expert weight reload on refit",
"text": "## Motivation `process_weights_after_loading` rewrites BF16 MoE expert weights into the flashinfer TRT-LLM BlockMajorK layout whenever `use_flashinfer_trtllm_moe` is set, and that flag covers **both** `flashinfer_trtllm` and `flashinfer_trtllm_routed`. The inverse hook, however, was gated on `is_flashinfer_trtllm_routed()` alone. So with `--moe-runner-backend flashinfer_trtllm` the destination parameter stayed in block layout and the hot copy raised: ``` The size of tensor a (64) must match the size of tensor b (2048) at non-singleton dimension 2 ``` torch names the `copy_` destination first, so the 64 is ours: `block_k` is 128 bytes and the conversion runs on a `uint8` view, giving 128 / 2 bytes-per-bf16. Nobody opts into this — `flashinfer_trtllm` is auto-selected on sm100 for BF16 MoE models when `moe_runner_backend` is left at `\"auto\"`, so an RL weight refit on Blackwell hits it by default. Fixes #27787 ## Modifications 1. **Gate the inverse on the same flag that gates the transform** (`use_flashinfer_trtllm_moe`), so the two cannot drift apart again. 2. **The restore inverts the data, not just the shape.** RL callers batch weights, so one refit is many update RPCs; with a shape-only restore an earlier bucket's re-derive would block expert slots that a later bucket has not written yet a second time. 3. **New `repack_weights_after_hot_update` hook** to re-derive the layout once the copies are done. `update_weights_from_disk` and the checkpoint-engine IPC path already re-run `process_weights_after_loading`; `update_weights_from_tensor`, `update_weights_from_distributed` and the bucketed variants call `model.load_weights()` directly and do not. Widening the gate on its own would therefore only trade the loud copy failure for a parameter left in canonical layout while the kernel reads BlockMajorK. The re-derive is called from a `finally` on all four update paths, so a mid-update exception cannot leave parameters canonical while the kernel expects block layout. Files touched: `layers/quantization/unquant.py`, `layers/quantization/base_config.py` (no-op base hook), `model_executor/model_runner_components/weight_updater.py`, plus a new CPU unit test. Note on the rebase onto current main: the only conflict was add/add with #33905 (XPU MoE row-stride padding), which inserts helpers at the same module-level position. Both blocks are kept and there is no interaction — the XPU padding applies under an XPU default device, the BlockMajorK rewrite under flashinfer on sm100, and #33905 only touches `create_weights`, which this PR does not. ## Accuracy Tests Reproduced against the real code path on CPU with Qwen3-30B-A3B geometry (hidden 2048, moe_intermediate 768, tp=2 -> 384 per partition): ``` before: postprocess (4,32,768,64) -> restore no-op -> copy raises after: restore -> (4,768,2048) -> copy ok -> repack -> (4,32,768,64) ``` `test/registered/unit/layers/quantization/test_flashinfer_trtllm_bf16_moe_reload.py` covers this end to end and is registered as a CPU test (11 cases): the block layout is applied on cold load and changes the shape; the restore runs for the non-routed backend and *inverts* the layout rather than reshaping it; a reload reproduces the cold-load layout exactly, including through the bucketed update path; the repack is a no-op when nothing was reverted and when the backend is inactive; non-bijective permutations and unexpected element counts are rejected; and the `finally` re-derive still runs when `load_weights` raises. ## Speed Tests and Profiling Not applicable to the forward path — this PR only changes the weight-update path. The re-derive runs once per update RPC, after the copies land; no kernel or forward-pass code is touched. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31643837888](https://github.com/sgl-project/sglang/actions/runs/31643837888) Latest PR Test (Extra): :x: [Run #31643837810](https://github.com/sgl-project/sglang/actions/runs/31643837810) ",
"url": "https://github.com/sgl-project/sglang/pull/33743",
"createdAt": "2026-08-05T18:52:04Z",
"updatedAt": "2026-08-13T07:17:09Z",
"timestamp": "2026-08-13T07:17:09Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"bug",
"quant",
"run-ci"
],
"author": "Kh4L",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33778",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Avoid materializing GDN QKV tensors during target verification",
"text": "## Summary This PR removes redundant Q/K/V materialization from compatible GDN speculative target-verification paths. `causal_conv1d_update` already produces packed QKV. Previously, every GDN layer launched `fused_qkv_split_gdn_prefill_kernel` to copy that output into three contiguous tensors before Triton target verification. Triton accepts explicit token strides, so it can consume zero-copy `torch.split`/`view` tensors instead. The optimization is route-aware: - Triton and ReplaySSM target verification use strided Q/K/V views. - Prefill, FlashInfer, CuTeDSL, and other routes requiring contiguous inputs retain materialization. - Model arithmetic, recurrent-state updates, speculative acceptance, logits, and sampling are unchanged. This eliminates one memory-copy kernel launch per GDN layer on every compatible target-verification pass. ## Profile evidence Matched H200 CPU/GPU traces using Qwen3.5-4B, ReplaySSM, and NEXTN T=3: | Target-verification metric | Baseline | Patch | Change | |---|---:|---:|---:| | QKV-split launches/pass | 24 | 0 | **-100%** | | QKV-split GPU time/pass | 56.946 us | 0 | **-56.946 us** | | CUDA-graph kernels | 452 | 428 | **-24** | | CUDA-graph span | 8,132.684 us | 8,040.898 us | **-91.786 us / -1.129%** | The graph-span reduction's approximate 95% interval was **79.732-103.841 us**. ## Standardized GSM8K benchmark 3 alternating baseline/QKV pairs ran on the same H200 with fresh servers, seed 0, empty prefix cache, and a 30-second cooldown. The benchmark used its defaults: 200 questions, five shots, 512 maximum output tokens, temperature 0, and parallelism 64. ### Pair 1 | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,433.642 tok/s | 6,762.467 tok/s | **+5.111%** | | Latency | 5.379 s | 5.145 s | **-4.350%** | | Engine TPOT | 9.7374 ms | 9.4735 ms | **-2.710%** | ### Pair 2 | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,202.258 tok/s | 6,366.221 tok/s | **+2.644%** | | Latency | 5.470 s | 5.401 s | **-1.261%** | | Engine TPOT | 10.2534 ms | 9.7761 ms | **-4.655%** | ### Pair 3 | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,496.073 tok/s | 6,669.102 tok/s | **+2.664%** | | Latency | 5.270 s | 5.134 s | **-2.581%** | | Engine TPOT | 9.3372 ms | 9.4393 ms | **+1.093%** | ### Three-pair aggregate | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,377.324 tok/s | 6,599.263 tok/s | **+3.480%** | | Latency | 5.373 s | 5.227 s | **-2.723%** | | Engine TPOT | 9.7760 ms | 9.5630 ms | **-2.179%** | | Accuracy | 87.667% | 87.500% | -0.167 pp | ## Reproduction Server: ```bash CUDA_VISIBLE_DEVICES=0 \\ PYTHONPATH=/python \\ sglang serve /path/Qwen/Qwen3.5-4B/ \\ --port 30000 \\ --dtype bfloat16 \\ --language-only \\ --limit-mm-data-per-request '{\"image\":0,\"video\":0,\"audio\":0}' \\ --context-length 32768 \\ --mem-fraction-static 0.8 \\ --max-running-requests 64 \\ --linear-attn-decode-backend triton \\ --random-seed 0 \\ --speculative-algorithm NEXTN \\ --speculative-draft-model-path /path/Qwen/Qwen3.5-4B/ \\ --speculative-num-steps 3 \\ --speculative-eagle-topk 1 \\ --speculative-num-draft-tokens 4 \\ --enable-linear-replayssm-spec ``` Benchmark: ```bash python benchmark/gsm8k/bench_sglang.py \\ --data-path /data/gsm8k/test.jsonl ``` Dataset SHA-256: ```text 3730d312f6e3440559ace48831e51066acaca737f6eabec99bccb9e4b3c39d14 ``` ## Validation - Exact target-verification output and recurrent-state parity. - ReplaySSM ring/fold parity and routing coverage. - Contiguous-input behavior retained for unsupported routes. - 31 affected H200 tests passed, covering 46 parameterized cases. --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31523531400](https://github.com/sgl-project/sglang/actions/runs/31523531400) Latest PR Test (Extra): :x: [Run #31523531310](https://github.com/sgl-project/sglang/actions/runs/31523531310) ",
"url": "https://github.com/sgl-project/sglang/pull/33778",
"createdAt": "2026-08-06T01:06:36Z",
"updatedAt": "2026-08-13T17:03:51Z",
"timestamp": "2026-08-13T17:03:51Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "vedantjh2",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33793",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(glm5.2): restrict MoE weights to local PP layers",
"text": " ## Summary PR [#32022](https://github.com/sgl-project/sglang/pull/32022) previously fixed this issue for Qwen3.5, but the same risk exists in GLM5.2. - restrict DeepseekV2 routed MoE weight collection to the layers owned by the current pipeline-parallel rank - avoid accessing mlp on PPMissingLayer placeholders from other pipeline stages - match the local layer range (self.model.start_layer to self.model.end_layer) already used by the DeepseekV2 forward path ## Testing - python3 -m py_compile python/sglang/srt/models/deepseek_v2.py - git diff HEAD^ HEAD --check --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31348606625](https://github.com/sgl-project/sglang/actions/runs/31348606625) Latest PR Test (Extra): :x: [Run #31348606360](https://github.com/sgl-project/sglang/actions/runs/31348606360) ",
"url": "https://github.com/sgl-project/sglang/pull/33793",
"createdAt": "2026-08-06T02:33:40Z",
"updatedAt": "2026-08-12T19:32:09Z",
"timestamp": "2026-08-12T19:32:09Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"deepseek",
"run-ci"
],
"author": "plusbang",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33804",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Intel][XPU]Enable chunked prefill scnearios for XPU with UT",
"text": " ## Motivation The chunked-prefill scripted-runtime suite under `test/manual/chunked_prefill/` was written and validated on CUDA. Bringing it up on Intel XPU surfaced a set of genuine, device-agnostic defects plus a few test-harness assumptions that only happen to hold on CUDA. This PR fixes those so the chunked-prefill scenarios run on XPU, without weakening what the suite verifies on CUDA. Concretely, the following problems are addressed: 1. **Attention backends read a KV pool layer they may not own.** `triton_backend`, `aiter_backend`, and `wave_backend` all derived `v_head_dim` from `token_to_kv_pool.get_value_buffer(0)`. The pool indexes buffers as `layer_id - start_layer`, so under pipeline parallelism a non-first rank owns layers `[start_layer, end_layer)` and a query for global layer 0 resolves to a negative index — `IndexError` at backend construction. 2. **`continue_generation` never freed the allocator cache off CUDA.** The handler called `torch.cuda.memory_reserved()` / `torch.cuda.empty_cache()` directly. On a non-CUDA device those early-return (CUDA was never initialized), so nothing was freed while the log reported a successful reclaim. 3. **The PP dynamic-chunk-size predictor was silently disabled by JIT recompile spikes.** `ChunkSizePredictor.fit` skipped only sample 0, but JIT backends recompile at arbitrary points mid-sweep. On XPU/Triton, spikes of 172.8 ms @ `l=256` and 127.5 ms @ `l=192` against a ~25–44 ms baseline inverted the fitted curvature (`a = -1.01e-04`), which the existing coefficient validation rejects — so dynamic chunking turned itself off with no actionable signal. 4. **`--enable-hierarchical-cache` fails unintelligibly on XPU.** Both the `kernel` and `direct` hicache io backends resolve their movers from `sgl_kernel.kvcacheio`, which is CUDA/HIP-only and absent from the XPU wheel. The server died several minutes into startup with a bare `NameError: transfer_kv_all_layer_lf_pf is not defined` deep inside the host pool. 5. **Mistral-native checkpoint detection missed single-file HF checkpoints.** `_is_mistral_native_format` looked for `model-*.safetensors`, which does not match an unsharded `model.safetensors`. A checkpoint shipping both `consolidated*.safetensors` and a single `model.safetensors` was therefore misclassified as Mistral-native, loading Mistral-named weights into an HF-named architecture. 6. **Scripted-runtime harness assumptions.** Control POSTs are fire-and-forget, so a server-side rejection presented as an unrelated socket timeout; the kv-canary is CUDA-only and its startup crash cost the whole suite; rid reuse raced the TokenizerManager's release; and two drain/assert helpers were stricter than the states the scheduler legitimately reaches. ## Modifications ### Runtime fixes (device- and PP-correctness) - `python/sglang/srt/layers/attention/triton_backend.py`, `aiter_backend.py`, `wave_backend.py` — derive `v_head_dim` from `pool.get_value_buffer(pool.start_layer)`, i.e. this rank's first *local* layer, instead of hardcoding global layer 0. - `python/sglang/srt/managers/scheduler.py` — `continue_generation` now routes through `self.device_module.memory_reserved()` and `empty_device_cache(self.device_module)`, and the log line names the device. Adds the `empty_device_cache` import from `sglang.srt.utils.common`. - `python/sglang/srt/managers/scheduler_pp_mixin.py` — new `_drop_latency_outliers(seq_lens, latencies)` helper, applied inside `ChunkSizePredictor.fit` before building the design matrix. Filtering uses median / MAD (multiplier `_LATENCY_OUTLIER_MAD_MULTIPLE = 6.0`) rather than mean / stddev, so the spikes cannot inflate the very threshold meant to exclude them. It returns the inputs unchanged when MAD is degenerate (`<= 0`) or when filtering would leave fewer than 8 points, on the principle that a bad fit is caught by the existing coefficient validation whereas over-dropping would trade a diagnosable error for a silently wrong model. Dropped samples are logged with the median and MAD. - `python/sglang/srt/server_args.py` - `_resolve_layout_io_compatibility` rejects `--enable-hierarchical-cache` on XPU up front with a message that names the missing module and the workaround, instead of the late `NameError`. - `_is_mistral_native_format` HF-weight detection widened from `model-*.safetensors` to `model*.safetensors` (both the local-directory glob and the Hub-listing branch). The Hub branch additionally requires `\"/\" not in f` so nested files under subdirectories are not counted as top-level HF weights. The docstring is updated to match. ### Test-infrastructure fixes - `python/sglang/test/scripted_runtime/background_http_poster.py` — `post` now raises on HTTP >= 400 with status and a truncated body; failed background coroutines are recorded (under a lock) and drained via the new `take_failures()`. `_log_coro_exception` becomes an instance method so it can record. - `python/sglang/test/scripted_runtime/context/http_post.py` — `_http_post_and_await_recv_msg` catches `TimeoutError` and, when the poster has recorded failures, re-raises an `AssertionError` naming the server's actual complaint (e.g. duplicate rid) rather than letting the timeout implicate the transport. - `python/sglang/test/scripted_runtime/http_server.py` — new `_canary_launch_kwargs(device)` gated on `_CANARY_SUPPORTED_DEVICES = {\"cuda\"}`. On CUDA the canary settings are exactly as before (`kv_canary=\"raise\"`, `kv_canary_real_data=\"partial\"`, `kv_canary_sweep_interval=100`); elsewhere the canary is disabled with a warning, because `install_canary` raises during `ModelRunner.alloc_memory_pool` and kills the server before the dispatch-loop handshake, costing the full `LISTENER_ACCEPT_TIMEOUT_S`. `_spawn_server_process` resolves the device the same way `ServerArgs` does (`engine_kwargs.get(\"device\") or get_device()`) so explicit and auto-detected devices pick the same knobs. Chunked-prefill *scheduling* behavior remains fully exercised on non-CUDA devices; only KV-corruption checking is skipped. - `python/sglang/test/scripted_runtime_chunked_helpers.py` — new `run_until_finished_and_rid_released(handle)` plus `RID_RELEASE_SETTLE_STEPS = 40`. A finished req is dropped scheduler-side before the TokenizerManager clears its rid from `rid_to_state` (which happens when the output stream drains), so within that window a resubmit of the same rid is rejected as a duplicate. - `test/manual/chunked_prefill/test_scripted_multi_req.py` — `test_rid_reuse_after_finish` uses the new settle-aware helper. - `test/manual/chunked_prefill/test_scripted_pp.py` — `_drain_until_released` includes `lock_refs == 0` in its exit condition, matching `test_scripted_abort.py` / `test_scripted_regression.py` / `test_scripted_lifecycle.py`. The radix `lock_ref` is dropped one iteration after the KV pages, so the old condition could return while `lock_refs == 1` and the caller's `assert lock_refs == 0` then failed spuriously. - `test/manual/chunked_prefill/test_scripted_regression.py` — two abort-release assertions accept `req is None`, following the `req is None or ...` idiom already used in `test_scripted_abort.py` / `test_scripted_lifecycle.py`. A fully-released req is dropped from the scheduler entirely, which is strictly stronger than \"row released\"; the engine-wide `lock_ref` check is what actually proves nothing leaked and holds in either case. ## Accuracy Tests Not run, and not expected to be needed: no kernel or model-forward code is touched. The three attention-backend edits change only which pool layer is queried for the `v_head_dim` *shape* at construction time — on a single-rank (`pp_size=1`) setup `start_layer == 0`, so the value read is bit-identical to before; under PP the previous code raised rather than producing a different number. The remaining runtime changes affect allocator cache reclaim, a scheduling heuristic's curve fit, startup argument validation, and checkpoint-format detection. Validation was via the chunked-prefill scripted-runtime suite under `test/manual/chunked_prefill/` on Intel XPU, which asserts on scheduler state (KV pages, `lock_refs`, `req_pool_idx`, chunk counts) rather than on generated text. Please advise if maintainers want a GSM8K / MMLU run on CUDA to confirm no regression there. ## Speed Tests and Profiling No benchmark sweep was run. Inference-speed impact is expected to be nil-to-positive: - `_drop_latency_outliers` runs once per `ChunkSizePredictor.fit`, not per step, and is `O(n)` over a handful of profiled samples. Its effect is to *restore* dynamic chunking where recompile spikes previously disabled it — on the XPU/Triton profile above, filtering the two spike samples moved the fitted quadratic coefficient from `a = -1.01e-04` (rejected, dynamic chunking off) to `a = +1.30e-04` (accepted). - The attention-backend and `server_args` changes are startup-path only. - `continue_generation` now actually frees the allocator cache on non-CUDA devices; that is a real (intended) reclaim where previously it was a no-op, so the first steps after a resume may re-warm the allocator. Happy to add a `bench_serving` comparison if maintainers want the dynamic-chunk-size path quantified on a specific configuration. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. cc: @siju-samuel @rbabukv --- ### CI States Latest PR Test (Base): :x: [Run #31691658488](https://github.com/sgl-project/sglang/actions/runs/31691658488) Latest PR Test (Extra): :x: [Run #31691658316](https://github.com/sgl-project/sglang/actions/runs/31691658316) ",
"url": "https://github.com/sgl-project/sglang/pull/33804",
"createdAt": "2026-08-06T03:44:33Z",
"updatedAt": "2026-08-13T10:38:45Z",
"timestamp": "2026-08-13T10:38:45Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci"
],
"author": "AnuSajikumar6264",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33812",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Rainj me/rust server pd lb",
"text": " ## Motivation Refactor PD related code and embed the pd lb to decode worker. ## Modifications ## Accuracy Tests ```bash MOONCAKE_PROTOCOL=nvlink SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK SGLANG_ENABLE_METRICS_DEVICE_TIMER=1 SGLANG_RUST_SERVER=1 CUDA_VISIBLE_DEVICES=0 sglang serve --model-path Qwen/Qwen3-0.6B-FP8 --host 0.0.0.0 --port 30000 --trust-remote-code --tp-size 1 --mem-fraction-static 0.80 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --page-size 64 MOONCAKE_PROTOCOL=nvlink SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK SGLANG_ENABLE_METRICS_DEVICE_TIMER=1 SGLANG_RUST_SERVER=1 CUDA_VISIBLE_DEVICES=1 sglang serve --model-path Qwen/Qwen3-0.6B-FP8 --host 0.0.0.0 --port 30001 --trust-remote-code --tp-size 1 --mem-fraction-static 0.80 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --page-size 64 curl -X POST http://127.0.0.1:30001/prefill_workers -H 'Content-Type: application/json' -d '{\"url\": \"http://127.0.0.1:30000\"}' curl -kvvv http://127.0.0.1:30001/prefill_workers curl -N http://127.0.0.1:30001/generate -H \"Accept: text/event-stream\" -H \"Content-Type: application/json\" -d '{\"text\": [\"The capital of France is\", \"How to travel to New York from San Francisco\"], \"stream\":true, \"sampling_params\":[{\"temperature\": 0, \"max_new_tokens\": 20}, {\"temperature\": 1.0, \"max_new_tokens\": 200}]}' ``` ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31685638525](https://github.com/sgl-project/sglang/actions/runs/31685638525) Latest PR Test (Extra): :x: [Run #31685638264](https://github.com/sgl-project/sglang/actions/runs/31685638264) ",
"url": "https://github.com/sgl-project/sglang/pull/33812",
"createdAt": "2026-08-06T04:15:32Z",
"updatedAt": "2026-08-13T09:28:27Z",
"timestamp": "2026-08-13T09:28:27Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"dependencies",
"run-ci"
],
"author": "rainj-me",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33824",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Simulator] Add high-fidelity CPU-based inference simulator",
"text": "## Motivation This PR introduces SGLang Simulator, a CPU-based inference simulator for evaluating SGLang scheduling, latency, throughput, and prefix-cache behavior without loading model weights or executing model kernels. It is a current-main rework of the simulator prototype in [#22250](https://github.com/sgl-project/sglang/pull/22250), while retaining SGLang v0.5.16 as the tested compatibility floor. Large-scale inference configuration studies are expensive to run directly on accelerators. SGLang Simulator reuses the real SGLang scheduler, radix cache, hierarchical cache, and request lifecycle, while replacing model-forward execution with a latency predictor. It supports measured traces, synthetic workloads, and both in-process and HTTP serving workflows. ## What This PR Adds - Add `tools/sglang-simulator` as a monorepo component. - Reuse the SGLang scheduler, cache allocator, radix cache, hierarchical cache, and request lifecycle with dummy model weights and a minimal KV allocation. - Add three forward-latency predictors: - AIConfigurator; - an sklearn-compatible ML predictor; - batch-composition replay. - Support Random, ShareGPT, and timestamped Autobench workloads. - Support direct Python execution through `SGLangBenchmarkRunner`. - Support server and benchmark-client execution through the HTTP serving path. - Collect request, latency, throughput, prefix-cache, cache-tier, and predictor metrics. - Add reusable example configurations, workloads, tokenizer assets, and replay fixtures. - Add compatibility checks for SGLang launch and serving APIs. - Run the simulator regression suite from CPU PR CI when `tools/sglang-simulator` changes. ## Simulation Modes The simulator provides two execution modes: - `OFFLINE`: advances a logical clock without sleeping, intended for fast configuration and workload studies. - `BLOCKING`: sleeps for predicted forward latency and visible L2-to-L1 load latency, intended for serving-path and timing-semantics validation. Replay replaces model-forward latency only. CPU preprocessing and postprocessing overhead remain enabled by default. For `OFFLINE` runs, latency, throughput, and duration should be read from the server-side simulator metrics rather than benchmark-client wall-clock time. ## Accuracy Validation The following results compare simulator output with measured serving traces. Latency, throughput, and duration values are absolute percentage errors. Prefix reuse is reported as absolute percentage-point error. The `SGLang baseline` column identifies the version used by the real service that produced the measurement trace. | Model | SGLang baseline | Hardware | Predictor / coverage | TTFT | TPOT | ITL | Input throughput | Duration | Prefix reuse | |:---|:---:|:---:|:---|---:|---:|---:|---:|---:|---:| | Qwen3-8B | v0.5.9 | H20 | AIConfigurator; no-cache, L1, and L2 cases | 2.38–4.15% | 1.64–4.05% | 1.78–4.07% | 1.41–2.40% | 1.39–2.33% | 0.00–0.04 pp | | Qwen3-32B-FP8 | v0.5.9 | H20 | AIConfigurator; no-cache, L1, and L2 cases | 2.40–3.05% | 0.58–1.11% | 0.52–1.02% | 0.52–1.13% | 0.51–1.12% | 0.00–0.04 pp | | GLM-5.1-FP8 | v0.5.11 | GPU | ML; L1 and L2 cases | 5.27% | N/A | N/A | 1.11% | 1.08% | 0.028 pp | | DeepSeek-V4-Pro | v0.5.11 | GPU | ML; L1 and L2 cases | 6.12% | N/A | N/A | 0.23% | 0.22% | 0.048 pp | | DeepSeek-V4-Flash | v0.5.16 | GPU | ML; L1 case | 0.786–1.343% | N/A | N/A | 0.000212% | 0.000212% | 0.000 pp | TPOT and ITL are not applicable to the GLM-5.1-FP8, DeepSeek-V4-Pro, and DeepSeek-V4-Flash validation traces because they use one-token outputs(Prefill-Decode Disaggregation). Prefix-cache accuracy is particularly sensitive to `max_total_tokens`. This value determines simulated device KV-cache capacity and also affects hierarchical host-cache sizing and eviction behavior. For deployment-faithful comparisons, use the `max_total_num_tokens` value reported by the real SGLang server startup log as the simulator's `--max-total-tokens` value. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31567949472](https://github.com/sgl-project/sglang/actions/runs/31567949472) Latest PR Test (Extra): :x: [Run #31567949404](https://github.com/sgl-project/sglang/actions/runs/31567949404) ",
"url": "https://github.com/sgl-project/sglang/pull/33824",
"createdAt": "2026-08-06T06:24:32Z",
"updatedAt": "2026-08-13T06:24:23Z",
"timestamp": "2026-08-13T06:24:23Z",
"metrics": {
"reactions": 0,
"comments": 12
},
"labels": [
"documentation",
"dependencies",
"hicache",
"run-ci",
"run-ci-extra"
],
"author": "littlefatfat",
"state": "open",
"assignees": [
"ispobock",
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33827",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: make Cache-DiT actually cache on MiniMax-H3",
"text": "## Motivation Cache-DiT (DBCache) has been wired up for MiniMax-H3 for a while (`_CUSTOM_BLOCK_ADAPTER_SPECS` already maps `MiniMaxH3DiTModel`), but it never actually cached anything: enabling `SGLANG_CACHE_DIT_ENABLED=1` gave a 1.00x speedup and byte-identical output, with no error and no warning. ## Root cause `CachedBlocks_Pattern_3_4_5` retains the block-stack input **by reference** at two points, to measure the residuals that drive the hit/miss decision — once in `forward` for the Fn residual, and again in `call_Mn_blocks` for the Bn residual: ```python original_hidden_states = hidden_states hidden_states = self.call_Fn_blocks(hidden_states) Fn_hidden_states_residual = hidden_states - original_hidden_states ``` `MiniMaxH3DiTBlock.forward` applies its first gated residual with an in-place Triton kernel, which rewrites that same buffer. The snapshot aliases the mutated output, the residual is identically zero, and the similarity test degenerates to `0/0 = NaN`; `NaN < threshold` is always False, so the cache never hits — silently. Measured on upstream without the fix: **45 of 49 cache decisions produced a NaN residual diff, 0 hits**, output bit-identical to running with caching disabled. ## Fix A block must not overwrite the input Cache-DiT is holding. `MiniMaxH3DiTModel.set_cache_dit_input_preservation(enabled)` flips every block at once, and only the **first** gated residual of a block honours it — the second operates on a buffer the block just allocated, so it keeps the in-place fused kernel either way. `indexed_modulation.py` gains an out-of-place variant `indexed_gate_bf16` that shares the same `_indexed_gate_bf16_kernel` as the in-place `indexed_gate_bf16_`, with an added output-row stride, and still applies the shared `numerics.round_bf16_to_fp32`. Both variants execute the same kernel arithmetic; the preserved path costs one allocation rather than falling back to eager. `MiniMaxH3DenoisingStage` owns the lifecycle, so `cache_dit_integration.py` stays model-agnostic. Preservation is armed before delegating to the parent — `cache_dit.enable_cache` swaps `blocks` for a single `CachedBlocks` wrapper, after which the real blocks are unreachable — and disarmed when the parent turns out not to have mounted. If the mount raises, the stage unmounts first and disarms only once that succeeds; if the unmount itself fails it stays armed and logs, because disarming with the cache still attached is the silent case. Normal disable follows the same order. The stage arms whenever an H3 mode requests caching and caching is not already enabled, without predicting whether the parent will accept the mount. Predicting means mirroring the parent's gates, and a mirror can fail open if the parent relaxes one: it would then mount unpreserved blocks and reproduce the original silent failure. Arming and disarming only loops over the blocks setting a boolean, with no model execution in between, so a call that turns out not to mount never takes the out-of-place path; what preservation costs while caching is actually running is measured below. For the same reason the setter fails closed: if the H3 model cannot be reached it raises rather than skipping, because skipping is indistinguishable from the bug. ## Verification B200, T2VA 768p, 8.7s, 50 steps, seed 1101, default cache params (`Fn=1, Bn=0, warmup=4, RDT=0.24, MC=3`). | check | result | |---|---| | reproduces the bug, upstream without the fix | 45/49 NaN diffs, 0 hits; output md5-identical to caching disabled | | with the fix | 34/49 hits, 0 NaN | | this rework does not change the output | the uncached hash is unchanged across it, and the cached hash is unchanged across it | Merging main moved the baselines, so the md5 values quoted earlier in this PR no longer apply to the current head: | head | cache off | cache on | |---|---|---| | `462f9bc` — before the merge | `c9242e50…` | `f5326803…` | | `a0cf4c2ab` — after the merge, before this rework | `7fdffc7c…` | `48423260…` | | current head — after this rework | `7fdffc7c…` | `48423260…` | `a0cf4c2ab` is the merge commit itself, so the shift between the first two rows came in with the merge rather than from this branch — we checked that the merge carries no manual resolution. The rework then leaves both arms unchanged, and the cached value reproduced across two separate jobs on different nodes. Scope: this is one fixed case. It shows the implementations agree bit-for-bit there; it does not establish equivalence across other prompts, Fn/Bn settings, tasks, or parallel configurations. ### Latency and peak memory: all-block versus Fn/Mn-only A = the previous Fn/Mn-only implementation, B = this rework, both from frozen worktrees where B's parent commit is A, so the only delta is this change. Interleaved A1 → B1 → A2 → B2 in one job on one B200, same input, same cache params. | arm | generate | denoising stage | torch peak allocated | torch peak reserved | hits | NaN | |---|---|---|---|---|---|---| | A1 | 118.1s | 102.68s | 131279 MiB | 137346 MiB | 34/49 | 0 | | B1 | 115.3s | 101.82s | 131715 MiB | 137346 MiB | 34/49 | 0 | | A2 | 114.7s | 101.59s | 131279 MiB | 137346 MiB | 34/49 | 0 | | B2 | 114.9s | 101.64s | 131715 MiB | 137346 MiB | 34/49 | 0 | All four produce md5 `48423260a94f11c759df7281941ab8fc`, so the all-block rework did not change the output in this case. **Peak memory: +436 MiB allocated (+0.33%).** The delta reproduced exactly in both repeats — 131279 MiB in both A runs and 131715 MiB in both B runs. Peak reserved is identical across arms (137346 MiB), so the extra sits inside the same allocator pool, and no corresponding increase was observed in the 2-second `nvidia-smi` samples. We are reporting the number rather than a mechanism: one extra activation-sized live buffer is a plausible hypothesis, but 436 MiB does not match the hidden-state tensor size and we have not taken a memory snapshot to confirm it. **Latency: no regression distinguishable from run-to-run drift at n=2.** A1 is the slowest run on both metrics, and both B runs fall between A1 and A2 on both metrics. With the fixed A → B → A → B ordering, the difference between the arm means (1.1% on generate) is confounded with run order, and the three runs after A1 sit within about 0.5% of each other. This cannot establish equivalence or rule out a smaller effect. Scope: one prompt, one configuration, two repeats per arm, single node, A → B → A → B ordering (not fully counterbalanced against time drift). ### Sequence-parallel scaling Measured on 462f9bc, before the merge, each degree against its own uncached baseline. Merging main shifted both arms of the acceptance case, so these numbers describe the pre-merge tree; they are kept because the code path they exercise — Cache-DiT deciding on all-reduced means — is unchanged by the merge and by this rework. | config | end-to-end | speedup | denoising stage | stage speedup | SSIM (Y) | PSNR | |---|---|---|---|---|---|---| | 1 GPU | 332.2s → 116.9s | 2.84x | 307.4s → 103.8s | 2.96x | 0.804 | 25.35 dB | | SP=2 | 222.1s → 93.2s | 2.38x | 203.6s → 82.0s | 2.48x | 0.823 | 25.66 dB | | SP=4 | 107.5s → 44.1s | 2.44x | 90.6s → 33.5s | 2.70x | 0.832 | 25.98 dB | | SP=8 | 72.3s → 37.5s | 1.93x | 56.5s → 26.2s | 2.15x | 0.795 | 25.00 dB | SP is covered deliberately: with `sp_group` set, Cache-DiT installs `_patch_cache_dit_similarity` and decides on all-reduced means — a different code path. The bug reproduces there without the fix (45 NaN, output md5-identical to uncached) and the fix holds. The `ipc_a2a_sync` JIT extension does not build in our environment (CCCL header/compiler version check), so SGLang takes its fallback path for SP>=2. The latencies and ratios above therefore characterise that fallback environment and do not necessarily transfer to the optimized path. Method: ffmpeg decodes the H.264 MP4s and computes whole-clip SSIM/PSNR on the decoded frames, cached versus uncached, each configuration against its own baseline — outputs are not bit-comparable across parallel configurations, since TP/SP change reduction order. We report Y-plane SSIM and average PSNR. Because the frames went through an H.264 round trip rather than being scored with `cache-dit-metrics` directly, the absolute values are not directly comparable to Cache-DiT's documented thresholds. One sample per configuration; we do not infer a quality trend across SP degrees. --- ### CI States Latest PR Test (Base): :x: [Run #31712841426](https://github.com/sgl-project/sglang/actions/runs/31712841426) Latest PR Test (Extra): :x: [Run #31712841027](https://github.com/sgl-project/sglang/actions/runs/31712841027) ",
"url": "https://github.com/sgl-project/sglang/pull/33827",
"createdAt": "2026-08-06T06:46:56Z",
"updatedAt": "2026-08-13T16:39:59Z",
"timestamp": "2026-08-13T16:39:59Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"run-ci",
"diffusion",
"jit-kernel",
"run-ci-extra"
],
"author": "Elaine4CY",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33829",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Model] Complete dots.note.omni support with native encoders, video preprocessing, and MTP decoding",
"text": " ## Motivation Merge dots.note.omni model ## Modifications This PR completes the SGLang integration of dots.note.omni, including: - Native in-process vision and audio encoders - Train-consistent native video preprocessing - Full-sharing MTP/NextN speculative decoding - DP/TP/EP execution support, including overlap scheduling - Correct KV-cache sizing for the hybrid sliding-window draft model - DSA correctness support for dots3's tail-RoPE layout ## How dots.note.omni differs from existing models dots.note.omni does not fit several assumptions made by existing multimodal and speculative-decoding paths. ### Multimodal architecture Unlike models that reuse standard Hugging Face vision/audio encoders, dots.note.omni uses: - A custom MoE vision transformer - A Whisper-derived audio encoder with a model-specific convolutional frontend and token-length calculation - Model-specific image and audio placeholder expansion - **A native video pipeline that jointly samples and interleaves frames, timestamps, and audio segments under a token budget** The video representation must match the training-time flattening algorithm. Applying a generic uniform-frame video processor changes the modality ordering, sampling strategy, and token allocation, which causes an inference/training mismatch. ### Language-model attention (**Introduce dots_hybrid_attn_backend**) The dots3 language model combines: - MLA - Full-attention and sliding-window layers with different attention geometry - Optional DSA indexing on full-attention layers - Head-wise or element-wise attention gates - **RoPE dimensions placed at the tail of the DSA index key, unlike the leading-RoPE layout assumed by the existing fused DSA kernels** (maybe changed as same as existing?) Therefore, its full-attention and SWA layers cannot always share the same attention path or KV-cache accounting. ### MTP structure the standard single-layer EAGLE/NextN execution path with an ordinary SWA-capacity, MLA-based draft layer. ## Model-specific changes ### Native vision and audio towers This PR migrates the dots.note.omni encoders into SGLang: - Adds the dots MoE ViT implementation - Adds the dots audio model and Whisper-derived speech encoder - Uses SGLang FA3 and fused MoE kernels - Loads encoder weights directly from the model directory - Removes runtime imports and compatibility shims for mm_encoder_server - Honors --language-only, allowing encoder/LLM disaggregation and text-only deployments without loading the multimodal towers ### Train-consistent video preprocessing This PR vendors the required dots video flattening pipeline and integrates it with the OpenAI chat API. The processor: - Accepts a native video_url - Decodes and processes remote video payloads entirely in memory - Uses the tokenizer loaded from the served model - Produces interleaved timestamp, image, and audio content matching training - Uses a deterministic seed derived from the video and question - Accounts for max_new_tokens when reserving the context budget - Falls back to visual-only processing if audio would exceed the configured token budget - Records per-request preprocessing latency The following request-scoped controls are added: | Field | Purpose | |-------|---------| | seq | Total sequence budget used by the video flattener | | output_reserve | Explicit output-token reservation | | audio_cap | Maximum fraction of the input budget assigned to audio | | audio_sr | Audio sample rate | | k_mode | Deterministic evaluation/sampling mode | These options are request-scoped because evaluation jobs with different context budgets may share one server. Current native-video limitations: - One video per request - A native video cannot currently be mixed with separate image or audio inputs in the same request ### Full-sharing MTP/NextN A dedicated Dot3NoteForCausalLMNextN draft model is added. For draft-model loading, the dots3 configuration is rewritten before attention shapes and KV-cache geometry are derived: - The draft architecture becomes Dot3NoteForCausalLMNextN - Exactly one recursively shared MTP layer is exposed - The layer uses the dots3 SWA attention geometry - A checkpoint-provided MTP embedding is retained when present - The target LM head is shared with the draft model Target verification and draft extension use the paged, absorbed SWA-MLA path. The expanded SWA-MHA path remains limited to prefill, where its logical-tail metadata is available. ## Shared SGLang changes and why they are needed The changes below are implemented in shared infrastructure because the underlying correctness conditions are not specific to one model name. ### 1. Hybrid SWA draft KV-cache accounting The existing hybrid pool configurator charged all EAGLE/NextN draft layers as full-attention layers. That is incorrect for dots3 because its shared draft layer is an ordinary SWA layer with: - SWA latent-vector geometry - SWA token capacity rather than full-attention capacity The speculative auxiliary configuration now records how many draft layers are SWA layers. The hybrid pool configurator separately accounts for: - Full-capacity full-attention draft layers - Ordinary SWA draft layers using the target SWA capacity - Full-capacity SWA-shaped draft layers, such as the existing Inkling case This prevents both under-allocation and unnecessary loss of target KV-cache capacity. The logic is generic for hybrid-SWA models using EAGLE-family draft models. ### 2. DP dummy-row normalization for overlap MTP With DP attention and overlap scheduling, an idle rank must still participate in MLP and attention collectives. SGLang creates dummy rows for this purpose. For speculative decoding, attention metadata may be planned before DP synchronization adds those rows. This previously left the following structures with inconsistent request dimensions: - GPU and CPU extend-length metadata - Prefix-length metadata - Draft cache locations - Positions and hidden states - Dense attention page tables - Draft logits returned to the speculative worker This PR normalizes speculative dummy rows before draft buffers are allocated, keeps the CPU and GPU metadata aligned, preserves the normalized output rows on idle ranks, and exposes a backend hook to refresh row-dependent attention metadata after DP padding. The backend hook defaults to a no-op. Only backends whose metadata can be safely rebuilt opt in, so pre-planned sparse/DSA schedules remain untouched. ### 3. Attention page-table normalization A shared helper is added to align a pre-planned page table with a DP-normalized batch. It truncates extra rows or appends zero-valued dummy rows. Both the dots hybrid backend and the FA3 SWA-MLA fallback use this helper. This avoids model-name checks and makes the invariant explicit at the attention-backend boundary. ### 4. FA3 SWA-MLA fallback dots3's MTP layer is both SWA-shaped and MLA-based. Target verification and draft extension already have a paged latent KV view, while the existing expanded SWA-MHA path is designed for prefill. FA3 therefore gains an absorbed SWA-MLA fallback that consumes the SWA page table and latent KV cache directly. This is exposed as a backend capability and selected by the model only when available. ### 5. DSA tail-RoPE correctness The existing fused DSA indexer kernels assume that RoPE dimensions are at the front of the index vector. dots3 places them at the tail. The fusion is now enabled only when rope_dim_at_front is true. Tail-RoPE models use the generic path until the fused kernels support this layout explicitly. The K-only rotary path is also fixed to avoid passing aliased query and key views to an in-place rotary implementation, which could rotate or corrupt the index key twice during early chunked prefill. These are general DSA correctness fixes rather than dots3-specific branches. ### 6. OpenAI request and internal request plumbing The dots video controls and the unrendered user question are propagated through: - ChatCompletionRequest - OpenAI chat serving - GenerateReqInput - The dots multimodal processor The original question is required because the training-consistent adapter uses it for deterministic sampling and cache-key construction. It cannot reliably be recovered from the rendered chat prompt, which also contains template, role, and system tokens. The new fields retain defaults and are only consumed by the dots native-video path. ### 7. Encoder/LLM disaggregation validation Dot3NoteForCausalLM is added to the architectures allowed to use --encoder-only and --language-only. The model itself skips vision and audio tower construction in language-only mode. This avoids consuming memory needed by the language model and enables the standard encoder-disaggregation deployment pattern. ## Tests The PR adds coverage for the shared correctness changes: - DSA tail-RoPE disables the incompatible fused indexer path - The DSA K-only rotary path does not alias query and key inputs - Draft-extend dummy requests keep CPU and GPU length metadata aligned - Idle speculative batches retain normalized dummy logits and hidden states - Hybrid-SWA draft layers use SWA geometry and capacity during KV-cache sizing An EP8/TP8/DP8 launch script is also included for dots.note.omni with: - DeepEP - NEXTN speculative decoding - FA3 draft attention - Optional overlap scheduling - Optional language-only mode - Optional DSA and speculative-decoding switches ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31571249264](https://github.com/sgl-project/sglang/actions/runs/31571249264) Latest PR Test (Extra): :x: [Run #31571249179](https://github.com/sgl-project/sglang/actions/runs/31571249179) ",
"url": "https://github.com/sgl-project/sglang/pull/33829",
"createdAt": "2026-08-06T06:56:30Z",
"updatedAt": "2026-08-13T04:17:39Z",
"timestamp": "2026-08-13T04:17:39Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"documentation",
"Multi-modal",
"deepseek",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "jianfei-wangg",
"state": "open",
"assignees": [
"alphabetc1",
"yhyang201"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33831",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[speculative] DSpark: support the mask-filling draft convention (verify width = gamma)",
"text": "## Motivation DSpark currently assumes the autoregressive draft-block convention: a gamma-row block emits gamma drafts and the verify window is gamma + 1 rows. DFlash-style mask-filling draft heads use a different block layout — slot 0 carries the anchor token and slots 1..gamma-1 are the predictions — so a gamma-row block emits gamma-1 real drafts and the faithful verify width is gamma (= the config block_size). Serving such a head under the AR convention appends a junk token sampled from the clamped last slot and mis-sizes every width-derived buffer (CG capture widths, ragged verify layout, committed-token rail). ## Modifications The convention is declared by the draft checkpoint config (`dspark_mask_filling: true`), parsed into `DSparkDraftConfig.mask_filling`. Builtin AR heads are unaffected: the flag defaults off and the verify width stays gamma + 1. A config declaring the convention names an out-of-tree head, so builtin head-name validation is skipped for it (the implementation is validated at draft-model build). Width choke points updated: - `dspark_config`: mask-filling heads hard-error on `speculative_num_draft_tokens != block_size`; `read_draft_checkpoint_is_mask_filling` helper. - `speculative_hook`: faithful verify width at the arg-validation site. - `spec_info.get_num_tokens_per_req_for_target_verify`: draft block rows == verify width at the CG/buffer sizing choke point; `set_dspark_mask_filling_convention` is called at server-args resolution AND at `DSparkWorkerV2` init (worker processes may not re-run arg hooks). - `dspark_planner`: `DSparkScheduleConfig` gains `verify_width`; the ragged layout must match the runtime verify width (one row wider makes the host committed-token stream read neighbor-row metadata as token ids at bs>1). - `dspark_verify`: `TargetVerifyExecutor` derives the drafts width as `verify_num_draft_tokens - 1` (identical for legacy AR heads). - `dspark_draft`: the folded sampler takes the gamma-1 draft view for mask-filling heads (duck-typed `markov_head.mask_filling`). - `dspark_worker_v2`: assert drafts+1 == verify width at the verify-ids joint. - `kernels/ops dspark_verify_window`: `compact_verify_ids` splits `block_w` (anchor stride over `draft_block_ids`) from `draft_w` (drafts stride); both shapes covered. ## Checklist - Default behavior for existing DSpark drafts is unchanged (flag off => gamma + 1 everywhere; drafts-width derivation is identity for AR heads). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31081163775](https://github.com/sgl-project/sglang/actions/runs/31081163775) Latest PR Test (Extra): :x: [Run #31081163154](https://github.com/sgl-project/sglang/actions/runs/31081163154) ",
"url": "https://github.com/sgl-project/sglang/pull/33831",
"createdAt": "2026-08-06T07:30:06Z",
"updatedAt": "2026-08-13T06:16:28Z",
"timestamp": "2026-08-13T06:16:28Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"speculative-decoding",
"jit-kernel"
],
"author": "shenxiul",
"state": "open",
"assignees": [
"Qiaolin-Yu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33834",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSV4] Support dsv4 decode TBO with deepep backend",
"text": " ## Motivation When deploying deepseek v4 with large-scale EP, cross-node communication latency is non-negligible and TBO can hide the latency with sub-batch computation. Thus we support dsv4 decode two-batch-overlap with deepep backend only. Moreover, if we want to apply hisparse on dsv4 to maximize the available memory and maximize the decode throughput, swap-in IO latency is introduced and non-negligible as well (#28874). Thus tbo can also be used to hide the io latency. ## Modifications - Add deepseek v4 decode OpStrategy. - **C4 and C128 has different stage strategy**. - In C4, indexer and swap-in (in hisparse) becomes significant and they are designed to be overlapped by dispatch. Attention, Wo, op_gate are overlapped by combine. - In C128, there's no indexer and swap-in, so attention is overlapped by dispatch and the rest are overlapped by combine. The main reason for this layerwise strategy is to balance the workload that dispatch and combine overlap. ## Accuracy Tests ## Speed Tests and Profiling ### Benchmark result We tested Deepseek V4 Pro on 2*B200 with EP16 deployment. And we use fake-prefill benchmark to test the performance of decode only. The result may be affected by benchmark itself. When the concurrency increases to 6782, the benchmark spends lots of time on receiving and processing the stream responses. Hence, we record and present the per-gpu throughput in the log file as well. ### Per-GPU throughput from log file We record the per-rank throughput (median) for every per-rank batchsize from the log file, and make comparison between TBO and non-TBO, which is shown below. **At what batchsize, will TBO win non-TBO？** The answer is 320. Applying TBO will lead to huge performance degradation when bs is small, e.g. bs=60 on each dp rank. **Only if bs>320 on each dp rank, TBO has around 5~10% performance improvement compared with non-TBO baseline**. ### Profile We profile the case where bs=424 on each dp rank. **Hisparse only. Each decode step costs 135ms.** **Hisparse + tbo. Each decode step costs 124ms. 8% performance gains on large batchsize.** ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31479832051](https://github.com/sgl-project/sglang/actions/runs/31479832051) Latest PR Test (Extra): :x: [Run #31479831902](https://github.com/sgl-project/sglang/actions/runs/31479831902) ",
"url": "https://github.com/sgl-project/sglang/pull/33834",
"createdAt": "2026-08-06T07:38:10Z",
"updatedAt": "2026-08-13T06:26:51Z",
"timestamp": "2026-08-13T06:26:51Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"deepseek"
],
"author": "SYChen123",
"state": "open",
"assignees": [
"Fridge003",
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33857",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] Skip trivial DSV4 nonpaged indexer logits",
"text": "## Motivation SGL Top-K v1/v2 directly emits sequential indices when a row has no more candidates than `index_topk`, so those logits are never read. The DSV4 eager nonpaged path still computed them with DeepGEMM. ## Modifications - Encode trivial DeepGEMM rows as empty `[ks, ks)` ranges. - Preserve the original lengths used by KV gathering and Top-K. - Apply the optimization only to SGL Top-K; other backends keep full ranges. - Add one focused `<=/> index_topk` boundary assertion to the existing plan test. This targets the current eager `NonPagedIndexerPlan`; unlike #25400, it does not compact rows or add a host-side decision. ## Validation - Nonpaged indexer unit test: 7 passed, 12 subtests passed. - SGL Top-K v1/v2 GPU checks: mixed and all-trivial cases preserve selected indices. - Full pre-commit passed. ## Performance GB300, 8K prefill, `index_topk=1024`, 80 samples: - MQA + Top-K: `0.14361 -> 0.13467 ms` (`-6.23%`) --- ### CI States Latest PR Test (Base): :x: [Run #31718371894](https://github.com/sgl-project/sglang/actions/runs/31718371894) Latest PR Test (Extra): :x: [Run #31718371557](https://github.com/sgl-project/sglang/actions/runs/31718371557) ",
"url": "https://github.com/sgl-project/sglang/pull/33857",
"createdAt": "2026-08-06T09:06:23Z",
"updatedAt": "2026-08-13T17:06:50Z",
"timestamp": "2026-08-13T17:06:50Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "weireweire",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33863",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] PP Support PD + DSpark",
"text": " ## Motivation Fixes #32750. The existing PD + DSpark path works without Pipeline Parallelism because the prefill worker captures all required target hidden states, injects draft KV locally, and transfers target KV + draft KV to the decode worker. With PP enabled, the captured target hidden states are distributed across PP stages. Each stage only owns a subset of the features required by the DSpark context projection. Injecting draft KV independently on each stage is not equivalent to the non-PP computation because RMSNorm must be applied after the full context projection is accumulated. ## Design This PR keeps the existing PD data plane unchanged: ```text Prefill -> target KV + draft KV -> Decode ``` Hidden states are not transferred across the PD boundary. Each prefill PP stage computes its local pre-norm context contribution: ```text ctx_i = h_i @ W_i.T ``` The accumulated context is carried through PP proxy tensors: ```text ctx_acc = recv_ctx_acc + ctx_i ``` The last PP stage performs: ```text ctx_norm = RMSNorm(ctx_acc) K, V = KVProj(ctx_norm) ``` and writes draft KV into the prefill-side draft KV pool. This preserves the original non-PP computation: ```text concat(h1, ..., hk) @ W_fc.T == sum_i(h_i @ W_i.T) ``` while keeping the additional PP payload fixed at `[num_tokens, hidden_size]`. ## Changes ### PP-aware target capture - Capture only target layers owned by the current PP rank. - Carry local captures through `dspark_aux_hidden_states`. - See [`deepseek_v4.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4.py#L2455-L2467) and [`kimi_k3.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/kimi_k3.py#L2483-L2525). ### Fixed-size PP context accumulation - Each rank computes its local projection contribution. - `dspark_ctx_acc` remains `[num_tokens, hidden_size]` across PP hops. - Raw `dspark_aux_hidden_states` is removed after local projection. - RMSNorm is applied once on the final PP rank. - See [`dspark_worker_v2.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/speculative/dspark_components/dspark_worker_v2.py#L693-L726). ### DSV4 block-FP8 column slicing - Slice matching FP8 weight columns and scale blocks. - Support canonical FP8 scales and DeepGEMM packed UE8M0 scales. - Avoid repeated zero-padded full-K GEMMs for cross-rank target features. - See [`deepseek_v4_dspark.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4_dspark.py#L70-L147). Unsupported quantization backends retain the full-K fallback in [`project_target_hidden_partial()`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4_dspark.py#L803-L846). ### Single-owner fast path When all target layers belong to the final PP rank, that rank uses the existing full projection directly. Non-owner DSV4 ranks use lifecycle-only skeleton models: - No draft stages or heads. - No checkpoint weight loading. - No draft KV pool. - No planner, proposer, or injector. - Distributed lifecycle remains aligned across ranks. See [`dspark_config.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/speculative/dspark_components/dspark_config.py#L41-L73) and [`deepseek_v4_dspark.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4_dspark.py#L678-L708). ### Draft KV injection and transfer - Add projected-context KV injection on the final PP rank. - Apply normalization, WKV projection, KV normalization, and RoPE before writing draft SWA KV. - Assign target KV global layer IDs. - Use the `1_000_000+` namespace for draft KV entries. - Only the final PP rank transfers draft KV; every PP rank transfers its local target KV. See [`dspark_kv_inject.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/speculative/dspark_components/dspark_kv_inject.py#L77-L159), [`utils.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/disaggregation/utils.py#L965-L1001), and [`prefill.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/disaggregation/prefill.py#L197-L257). ### PP bootstrap and transfer consensus - Ordered intersection for successful bootstrap/transfer requests. - Ordered union for failures. - Any PP-rank failure becomes a global failure. - KV is not released until the local transfer reaches a terminal state. See [`scheduler_pp_mixin.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/managers/scheduler_pp_mixin.py#L861-L914) and [`prefill.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/disaggregation/prefill.py#L857-L938). ## Supported Scope - Supported: `PP + DSPARK + PD prefill` - Unsupported: `PP + DSPARK decode` - Unsupported: `PP + DSPARK unified serving` - Unsupported: `DSPARK + context parallelism` ## Why `scheduler_pp_mixin.py` Is Modified The last PP stage performs additional work because it injects and transfers draft KV, while the other PP stages only transfer their local target KV. This creates asymmetric transfer completion times. The previous PP scheduler logic treated successful and failed transfers as one unordered set. It could also admit requests based only on `KVPoll.WaitingForInput`, before confirming that every PP stage had sufficient local metadata credits. Under sustained load, these behaviors could cause: - different PP stages to admit requests in different orders; - early KV release on stages that completed before the last PP stage; - inflight queue divergence after a transfer failure; - PP send/receive or proxy-tensor scheduling desynchronization. The scheduler changes make the control plane deterministic: - bootstrap readiness is probed without reserving resources; - request order is preserved during PP intersection and union; - transfer success requires intersection across all PP stages; - transfer failure is propagated as a union across all PP stages; - release occurs only after the last PP stage has completed both target KV and draft KV transfer; - DSpark draft input is reconstructed after PP output processing; - PP output tokens are moved to the batch device before creating the next draft state. The scheduler does not transfer hidden states or draft KV itself. Its role is to keep admission, completion, and release decisions consistent across PP stages despite the extra work performed by the last stage. ## fixes state-component matching This patch fixes state-component matching for PD transfer when PP ranks expose different state component layouts. Previously, state components were matched positionally: ```text prefill state[i] -> decode state[i] ``` That is not always valid for DSV4 under PP. Some state components are optional per PP stage. For example, `C128_STATE` only exists on PP ranks that own C128 layers. When the final PP rank sends draft SWA but does not own any C128 layer, the source and destination layouts can diverge: ```text Prefill final PP rank: state[0] = target SWA state[1] = draft SWA Decode rank: state[0] = target SWA state[1] = target C128_STATE state[2] = draft SWA ``` With positional matching, `prefill state[1]` would incorrectly map to `decode state[1]`, so draft SWA could be written into the target C128 state buffer. The fix is to include `state_types` in the decode registration metadata and match state components by `(StateType, occurrence)` instead of list index: ```text prefill SWA #1 -> decode SWA #1 prefill SWA #2 -> decode SWA #2 ``` This keeps the existing behavior when layouts are already aligned, while correctly handling missing optional components such as `C128_STATE`. The change is applied to both Mooncake and NIXL transfer paths. For mixed-version compatibility, if the peer does not provide `state_types`, the code falls back to the old positional mapping. This does not change the actual KV/state data layout or copy logic. It only changes how the destination state component is selected. ## Accuracy Tests ``` SGLANG_PP_LAYER_PARTITION=\"6,5,6,5,6,5,5,5\" SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 NCCL_SOCKET_IFNAME=eth0 NCCL_IB_DISABLE=0 SGLANG_DSV4_FP4_EXPERTS=1 GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path /data02/models/DeepSeek-V4-Flash-0731 --host 0.0.0.0 --port 30000 --trust-remote-code --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.9 --max-running-requests 64 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --pp-size 8 --attention-backend dsv4 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --disable-overlap-schedule --disable-piecewise-cuda-graph --disaggregation-transfer-backend mooncake --enable-metrics --moe-runner-backend flashinfer_mxfp4 --speculative-algo DSPARK --disaggregation-mode prefill --disaggregation-ib-device \"mlx5_1,mlx5_2,mlx5_3,mlx5_4\" --disable-flashinfer-autotune --disable-shared-experts-fusion SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_JIT_DEEPGEMM_PRECOMPILE=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=1 SGLANG_OPT_USE_TILELANG_MHC_PRE=1 GLOO_SOCKET_IFNAME=eth0 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8 sglang serve --trust-remote-code --model-path /data02/models/DeepSeek-V4-Flash-0731 --tp 8 --dp-size 8 --enable-dp-attention --cuda-graph-max-bs 32 --max-running-requests 256 --enable-metrics --host 0.0.0.0 --port 30000 --mem-fraction-static 0.85 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --moe-runner-backend flashinfer_mxfp4 --disaggregation-mode decode --disaggregation-ib-device \"mlx5_1,mlx5_2,mlx5_3,mlx5_4\" --speculative-algo DSPARK --tokenizer-worker-num 8 --enable-dp-lm-head --load-balance-method round_robin --swa-full-tokens-ratio 0.8 --disable-shared-experts-fusion ``` Hardward | MMLU | GSM8K | QPQA | aime25 repeats 16 -- | -- | -- | -- | -- H20 | 0.878 | 0.945 | 0.961 | 98.33% +/- 2.11% (SEM 0.53%) ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31662550711](https://github.com/sgl-project/sglang/actions/runs/31662550711) Latest PR Test (Extra): :x: [Run #31662550618](https://github.com/sgl-project/sglang/actions/runs/31662550618) ",
"url": "https://github.com/sgl-project/sglang/pull/33863",
"createdAt": "2026-08-06T09:42:11Z",
"updatedAt": "2026-08-13T06:49:11Z",
"timestamp": "2026-08-13T06:49:11Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"Multi-modal",
"deepseek",
"speculative-decoding",
"hicache",
"sgl-kernel",
"blackwell",
"diffusion",
"model-gateway",
"mthreads",
"jit-kernel"
],
"author": "zhangxiaolei123456",
"state": "open",
"assignees": [
"whybeyoung",
"yhyang201",
"Fridge003",
"kpham-sgl"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33881",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "refactor(kernels): split zero-KV fixup by alignment",
"text": " ## Motivation This is a follow-up to #32890, which introduced the TRT-LLM ragged attention zero-KV fixup path. That path clears `out` and sets `lse` to `-inf` for sequences whose KV length is zero. The original fix already handles potentially misaligned rows by checking alignment inside the device helper before using vectorized `float4` stores. Per @nvpohanh's suggestion, this PR avoids the device-side if/else by compiling separate aligned and scalar store paths, then choosing the appropriate specialization from the host launcher. This keeps the aligned vectorized fast path while routing potentially misaligned layouts, such as TP cases where `lse` row stride is not a 16-byte multiple, to scalar stores. ## Modifications - Compile the zero-KV fixup kernel with independent store-mode template parameters for `out` and `lse`. - Dispatch four host-side variants: vector/vector, vector/scalar, scalar/vector, and scalar/scalar. - Include both base pointer alignment and row-stride alignment in the host predicates. This covers contiguous tensors with non-zero PyTorch `storage_offset`. - Preserve the existing public Python/JIT wrapper API. - Add a registered CUDA unit test for: - contiguous tensors with non-zero storage offsets; - mixed-alignment layouts where `out` can remain vectorized while `lse` uses scalar stores. ## Accuracy Tests Added: ```bash test/registered/kernels/test_fixup_zero_kv.py ``` Local checks run: ```bash git diff --check upstream/main...HEAD PYTHONPYCACHEPREFIX=/private/tmp/sglang_pycache python3 -m py_compile test/registered/kernels/test_fixup_zero_kv.py ``` Not run locally: ```bash python3 -m pytest test/registered/kernels/test_fixup_zero_kv.py -q ``` The local Python environment does not have `pytest` installed. ## Speed Tests and Profiling Added a dedicated microbenchmark for this PR: ```bash python3 test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py ``` It is also registered in the CUDA kernel benchmark suite and can be run through: ```bash python3 test/run_suite.py --hw cuda \\ --suite base-b-kernel-benchmark-test-1-gpu-large \\ --continue-on-error ``` The benchmark exercises the four host-dispatched specializations introduced here: | mode | `out` store | `lse` store | coverage | |---|---:|---:|---| | `vec_vec` | vectorized | vectorized | fully 16-byte aligned fast path | | `vec_scalar` | vectorized | scalar | TP-like layouts where `lse` row stride is not 16-byte aligned | | `scalar_vec` | scalar | vectorized | non-zero storage offset / misaligned `out` base pointer | | `scalar_scalar` | scalar | scalar | both buffers potentially misaligned | Shapes covered: `(max_seq_len, batch_size) = (32, 32), (128, 32), (512, 16)`. The benchmark is intentionally a kernel microbenchmark so it isolates the zero-KV fixup cost instead of hiding it inside a full model run. I did not post local timing numbers because this workstation cannot run the CUDA benchmark (`nvidia-smi` is unavailable and the local Python env does not have `torch`). Light local validation completed: ```bash PYTHONPYCACHEPREFIX=/tmp/sglang-pr33881-pycache \\ python3 -m py_compile \\ test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py \\ test/registered/kernels/test_fixup_zero_kv.py git diff --check ``` Expected perf behavior: the fully aligned `vec_vec` path preserves the existing vectorized fast path, while mixed-alignment cases avoid the previous device-side per-row alignment branch by selecting the scalar/vector specialization once in the host launcher. ## Microbenchmark ## Microbenchmark **GPU:** GB200 | **dtype:** fp16 out / fp32 lse | **v_head_dim:** 128 Latency in µs (p50). `before` = device-side alignment branch per row; `after` = host-dispatched specialization. | max_seq_len | batch_size | mode | before (µs) | after (µs) | Δ latency | |-------------|------------|------|-------------|------------|-----------| | 32 | 32 | `vec_vec` | 3.12 | 2.98 | ▼4.6% | | 32 | 32 | `vec_scalar` | 3.36 | 3.00 | ▼10.7% | | 32 | 32 | `scalar_vec` | 3.29 | 3.24 | ▼1.6% | | 32 | 32 | `scalar_scalar` | 3.06 | 2.76 | ▼10.0% | | 128 | 32 | `vec_vec` | 7.19 | 6.84 | ▼4.9% | | 128 | 32 | `vec_scalar` | 6.52 | 5.94 | ▼8.9% | | 128 | 32 | `scalar_vec` | 6.71 | 6.40 | ▼4.6% | | 128 | 32 | `scalar_scalar` | 5.98 | 5.21 | ▼12.8% | | 512 | 16 | `vec_vec` | 12.02 | 11.39 | ▼5.3% | | 512 | 16 | `vec_scalar` | 10.55 | 9.65 | ▼8.5% | | 512 | 16 | `scalar_vec` | 10.89 | 10.23 | ▼6.0% | | 512 | 16 | `scalar_scalar` | 9.14 | 7.80 | ▼14.6% | | max_seq_len | batch_size | mode | before (GB/s) | after (GB/s) | Δ bandwidth | |-------------|------------|------|---------------|--------------|-------------| | 32 | 32 | `vec_vec` | 317.5 | 332.8 | ▲4.8% | | 32 | 32 | `vec_scalar` | 221.5 | 247.9 | ▲11.9% | | 32 | 32 | `scalar_vec` | 301.7 | 306.5 | ▲1.6% | | 32 | 32 | `scalar_scalar` | 242.9 | 270.0 | ▲11.2% | | 128 | 32 | `vec_vec` | 551.8 | 579.9 | ▲5.1% | | 128 | 32 | `vec_scalar` | 456.4 | 500.9 | ▲9.8% | | 128 | 32 | `scalar_vec` | 591.4 | 619.8 | ▲4.8% | | 128 | 32 | `scalar_scalar` | 497.5 | 570.8 | ▲14.7% | | 512 | 16 | `vec_vec` | 660.0 | 696.8 | ▲5.6% | | 512 | 16 | `vec_scalar` | 564.1 | 616.5 | ▲9.3% | | 512 | 16 | `scalar_vec` | 728.8 | 775.6 | ▲6.4% | | 512 | 16 | `scalar_scalar` | 651.0 | 762.6 | ▲17.1% | **Reproduce (after / this PR):** ```bash cd /sgl-workspace/sglang python test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py Reproduce (before / baseline): cd /sgl-workspace/sglang_baseline python test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py ``` ## end-to-end ``` python3 -m sglang.launch_server \\ --model-path moonshotai/Kimi-K3 \\ --served-model-name k3 \\ --tp 16 \\ --nnodes 4 \\ --node-rank 0 \\ --dist-init-addr : \\ --host 127.0.0.1 \\ --port \\ --trust-remote-code \\ --moe-runner-backend marlin \\ --decode-attention-backend trtllm_mla \\ --mem-fraction-static 0.85 \\ --max-running-requests 64 \\ --reasoning-parser kimi_k3 \\ --skip-server-warmup \\ --disable-flashinfer-autotune \\ --weight-loader-prefetch-checkpoints \\ --watchdog-timeout 7200 \\ --disable-cuda-graph # eager mode only gsm8k: python3 benchmark/gsm8k/bench_sglang.py \\ --num-shots 8 \\ --num-questions 1316 \\ --parallel 1316 \\ --host 127.0.0.1 \\ --port ``` Results: | Run | Job ID | Accuracy | |--------|---------|----------| | Before | 2981071 | 95.9% | | After | 2982462 | 95.4% | ## How to trigger fix_zero_kv code path Backend requirement: --decode-attention-backend cutedsl_mla (this automatically selects trtllm_mla for prefill, which calls fixup_zero_kv_rows) Trigger condition: A prefill batch containing a mix of: - Sequences with kv_len > 0 (prefix cache hit) - Sequences with kv_len = 0 (no cached prefix) Client script (trigger_fixup_zero_kv.py): ### Step 1: warmup — send one request to populate prefix cache send(LONG_PREFIX + \" Final: what is 1+1?\") ### Step 2: concurrent mixed batch #### - 8 requests WITH cached prefix → kv_len > 0 #### - 8 requests WITHOUT shared prefix → kv_len = 0 #### Server batches them together → triggers fixup_zero_kv_rows ``` threads = [ Thread(send, LONG_PREFIX + f\" Final: what is {i}+{i}?\") # cache hit Thread(send, f\"What is {i} times {i}?\") # cache miss for i in range(8) ] ``` Server launch command: ``` python3 -m sglang.launch_server \\ --model-path /models/goldenprairie-final-weights_vv1 \\ --tp 16 --nnodes 4 \\ --decode-attention-backend cutedsl_mla \\ --trust-remote-code \\ ... ``` ### Step 3: verification ``` output [cached-3] '6' [cached-4] '8' [fresh-2] '2 times 2' [fresh-7] '7 times 7' [fresh-4] '4 times 4' [cached-0] '0+0 =' [cached-1] '2' [fresh-3] '3 times 3' [fresh-5] '5 times 5' [cached-6] '12' [fresh-0] '0 times 0' [cached-7] '14' [fresh-6] '6 times 6' [cached-5] '10' [fresh-1] '1 times 1' [cached-2] 'Answer: 4' All requests done: [('cached-3', 'ok'), ('cached-4', 'ok'), ('fresh-2', 'ok'), ('fresh-7', 'ok'), ('fresh-4', 'ok'), ('cached-0', 'ok'), ('cached-1', 'ok'), ('fresh-3', 'ok'), ('fresh-5', 'ok'), ('cached-6', 'ok'), ('fresh-0', 'ok'), ('cached-7', 'ok'), ('fresh-6', 'ok'), ('cached-5', 'ok'), ('fresh-1', 'ok'), ('cached-2', 'ok')] K3 eager smoke completed on rank 0 ``` --- ### CI States Latest PR Test (Base): :x: [Run #31646002181](https://github.com/sgl-project/sglang/actions/runs/31646002181) Latest PR Test (Extra): :white_check_mark: [Run #31672020784](https://github.com/sgl-project/sglang/actions/runs/31672020784) ",
"url": "https://github.com/sgl-project/sglang/pull/33881",
"createdAt": "2026-08-06T15:23:16Z",
"updatedAt": "2026-08-13T06:51:41Z",
"timestamp": "2026-08-13T06:51:41Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci",
"jit-kernel",
"bypass-fastfail"
],
"author": "wenscarl",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33883",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] Route --file-storage-path to the file storage backend",
"text": "`--file-storage-path` is parsed into `server_args.file_storage_path` but nothing reads it. The `file` HiCache storage backend (`HiCacheFile`) only looks at the `SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR` env var and otherwise falls back to `/tmp/hicache`, so the flag is silently ignored and L3 lands in `/tmp` no matter what you pass. On a box where `/tmp` is tmpfs or a small partition that is either a surprise (the tier is much smaller than intended) or it fills the wrong disk. Repro: launch with `--hicache-storage-backend file --file-storage-path /mnt/nvme/hicache` and no env var. The backend writes under `/tmp/hicache`, not `/mnt/nvme/hicache`. Fix: route the arg to the backend through the shared storage `extra_config`, which every pool-assembler path already threads into `HiCacheStorageConfig` (the plain `HiCacheController` path and the DSA/hybrid `build_anchor_sidecar_stack` path both carry it). `HiCacheFile` then prefers, in order: the env var, `--file-storage-path`, then the `/tmp/hicache` default. I used `extra_config` rather than a first-class field because the DSA path builds the storage config inside the pool assembler, not in the plain controller branch, and `extra_config` is the one thing all of those paths already pass down. Three files, and nothing changes when the flag is unset (still `/tmp/hicache`) or when the env var is set (still wins). Validation on GLM-5.2-FP8 (TP8, DSA pool, `file` backend, env var unset): - With `--file-storage-path `: L3 wrote to `` (26882 `.bin` files, 81 GB) and `/tmp/hicache` stayed empty. - Flag unset and env unset: still resolves to `/tmp/hicache` (no regression). - Env var set: still takes precedence over the flag. - Evicted-prefix reload still served from the tier at cache_frac 0.9996 and returns the correct output: a passphrase embedded in the cached prefix comes back byte-identical on the reload, the resident hit, and a cold recompute. `HiCacheFile` resolution was also checked in isolation: flag set -> the flag path, flag unset -> `/tmp/hicache`, env var set -> the env path. --- ### CI States Latest PR Test (Base): :x: [Run #31713019368](https://github.com/sgl-project/sglang/actions/runs/31713019368) Latest PR Test (Extra): :x: [Run #31713019344](https://github.com/sgl-project/sglang/actions/runs/31713019344) ",
"url": "https://github.com/sgl-project/sglang/pull/33883",
"createdAt": "2026-08-06T16:21:52Z",
"updatedAt": "2026-08-13T16:54:45Z",
"timestamp": "2026-08-13T16:54:45Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"hicache"
],
"author": "reger-men",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33891",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] Bulk/staging reload fast-path for contiguous host runs",
"text": "When an evicted prefix is reloaded from the host KV tier, the transfer runs the per-token kernel (`transfer_kv_per_layer_mla`) at ~4.3 GB/s no matter how the bytes are laid out, because it pays a per-token indexing cost for every token. The host pool makes that worse: it hands out free slot ids first-fit and appends freed ids on free, so after a few fill/evict cycles a re-admitted prefix lands on a scrambled set of host offsets. So the reload is a fine-grained scatter/gather. Two changes turn that reload into a large contiguous copy: 1. The host slot allocator hands out a contiguous run whenever one is free, so a prefix's KV occupies a contiguous host region. Capacity and the success/failure set are identical to the stock allocator (it only returns None when there isn't enough total free, same as before), and when the pool is too fragmented for one run it falls back to gathering the largest runs first. So the slots handed out are always valid, distinct and in range; only the packing changes. 2. A reload fast-path in `load_to_device_per_layer`. When the host run for a layer is one ascending unit-stride run, it moves the slab in one shot instead of the per-token gather: if the device target slots are also one run, a single `copy_`; if the device slots are scattered (the device pool allocator does not pack them), it copies the contiguous host slab into a contiguous device staging buffer once and places the rows at their real slots with an on-device `index_copy_`. This covers both halves of a DSA reload, the MLA KV anchor and the DSA indexer, which ride the same prefix. Any non-contiguous host case, dtype/shape mismatch, or a non-contiguous destination falls back to the exact stock kernel, so a reload is never wrong and the path with nothing to gain is unchanged. Correctness is byte-identical to the per-token kernel. A CPU check reproduces the kernel's per-index gather over the whole tensor for the KV anchor, the DSA indexer page shape, and the scattered-device staging tier, and confirms the fast-path rejects dtype/shape/non-contiguous inputs so they fall back. The reload lands the data at the exact device slots the radix cache recorded, so the model reads the right KV. Measured served on GLM-5.2-FP8 (tp8, host L2 tier, `--hicache-io-backend kernel --hicache-mem-layout layer_first`, KV in bf16 so the baseline reload is the jit indexed kernel): a 128k evicted-prefix reload is 0.694 s with the fast-path vs 1.036 s on stock (median of 3 each), a 1.49x speedup, staging tier firing on every KV and indexer layer with no fallback. Both sides return the same reloaded answer (the planted needle) with cache_frac 0.9996, so the reload is byte-correct. The clock isn't pinnable on this box so treat it as first-pass; the ~5% run-to-run spread is well inside the 1.49x gap. The isolated per-layer copy is much faster (~57 vs ~4-5 GB/s); the served win is smaller because the reload also pays the fixed resident-hit and scheduling floor, which the copy doesn't touch. --- ### CI States Latest PR Test (Base): :x: [Run #31122196837](https://github.com/sgl-project/sglang/actions/runs/31122196837) Latest PR Test (Extra): :x: [Run #31122196598](https://github.com/sgl-project/sglang/actions/runs/31122196598) ",
"url": "https://github.com/sgl-project/sglang/pull/33891",
"createdAt": "2026-08-06T17:07:52Z",
"updatedAt": "2026-08-13T16:01:40Z",
"timestamp": "2026-08-13T16:01:40Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [],
"author": "reger-men",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33894",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "refactor error responses into shared utils::response helpers",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31678306810](https://github.com/sgl-project/sglang/actions/runs/31678306810) Latest PR Test (Extra): :x: [Run #31678306585](https://github.com/sgl-project/sglang/actions/runs/31678306585) ",
"url": "https://github.com/sgl-project/sglang/pull/33894",
"createdAt": "2026-08-06T18:26:02Z",
"updatedAt": "2026-08-13T08:30:32Z",
"timestamp": "2026-08-13T08:30:32Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"run-ci"
],
"author": "rainj-me",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33895",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "move the PD bootstrap registry under api_server::disaggregation",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31682372245](https://github.com/sgl-project/sglang/actions/runs/31682372245) Latest PR Test (Extra): :x: [Run #31682371946](https://github.com/sgl-project/sglang/actions/runs/31682371946) ",
"url": "https://github.com/sgl-project/sglang/pull/33895",
"createdAt": "2026-08-06T18:26:07Z",
"updatedAt": "2026-08-13T09:13:46Z",
"timestamp": "2026-08-13T09:13:46Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"dependencies",
"run-ci"
],
"author": "rainj-me",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33907",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] Free out-of-window SWA pages without a device sync",
"text": "Follows #32701, which added `free_segment` to the paged allocator so a per-request free needs no device sync. That migration deliberately left the hybrid SWA allocator on the legacy `free()` path, so SWA configurations still pay a sync on every free. This extends the same idea to the SWA side, and composes with the free-group batching from #33475 rather than replacing it. `free_swa` runs four ops whose output shape depends on device data -- `torch.unique` on the full pages, the `> 0` mapping filter, the inner allocator's own unique, and the scalar `mapping[idx] = 0` setitem -- so each call ends in a `cudaStreamSynchronize`. The inputs are a page-aligned slice of one request's kv row, so the page count is host knowledge: `free_swa_segment` takes the page representatives by stride, gathers the mapping once at fixed shape, and clears whole pages by broadcast index. Inside a free group it defers like `free_segment` does, but what it queues is owned -- a gather result and an arithmetic result -- never the caller's `req_to_token` view, so the row may be rewritten before the group closes. The gather deliberately stays at call time: moving it to the flush would make it depend on the mapping being unchanged for the whole group, trading a documented contract for an implicit one. Migrates the out-of-window SWA free; every other call site keeps `free_swa`. Verification, 1k requests freeing one page each on a Grace/Blackwell host: | | sync debug | host | |---|---|---| | `free_swa` (legacy) | trips | 33.7ms | | `free_swa_segment`, immediate | clean | 36.3ms | | `free_swa_segment`, in a free group | clean | 15.2ms | `torch.cuda.set_sync_debug_mode(\"error\")` stays clean on both paths where `free_swa` trips; reclaimed page counts are identical. Equivalence is also checked against an independent oracle and against `free_swa` over page sizes, segment alignments and `need_sort` routing on CPU, plus one case built through the real `alloc_extend` on GPU. Also wraps `maybe_evict_swa`'s free group in `try/finally` -- the assert in `free_swa_out_of_window_slots` sits inside it, and an escape would leave the allocator permanently in deferred mode. --- ### CI States Latest PR Test (Base): :x: [Run #31682484964](https://github.com/sgl-project/sglang/actions/runs/31682484964) Latest PR Test (Extra): :x: [Run #31682484951](https://github.com/sgl-project/sglang/actions/runs/31682484951) ",
"url": "https://github.com/sgl-project/sglang/pull/33907",
"createdAt": "2026-08-06T22:07:06Z",
"updatedAt": "2026-08-13T12:08:40Z",
"timestamp": "2026-08-13T12:08:40Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "hnyls2002",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33921",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Kimi K3] Preprocess CPU-transport images on the vision owner",
"text": "## Summary - keep Kimi-K3 CPU-transport image payloads as per-image uint8 tensors - defer resize, normalization, patchification, and H2D until DP load balancing selects the vision owner rank - activate the deferred path only for the resolved `cpu` transport; CUDA IPC and CUDA VMM continue to use their existing GPU transport paths - accept only PIL images or uint8 tensors for deferred preprocessing; other tensor dtypes, precomputed embeddings, and unsupported paths keep eager preprocessing - fall back to eager preprocessing when the raw payload would be larger than the processed FP32 tensor ## Why With CPU multimodal transport, Kimi-K3 currently preprocesses on the tokenizer worker GPU, copies the expanded FP32 patch tensor to CPU, broadcasts/serializes it, and copies it back to every scheduler GPU before selecting a vision owner. The DP helper already determines the owner before invoking its local feature loader, so both preprocessing and H2D can happen only on that owner. ## Server benchmark Latest-head strict A/B on full official `moonshotai/Kimi-K3` real weights (snapshot `9f62e4e9fffbd0a83ddd60e1c209d828994b3569`), NVIDIA B300 ×8, TP8, one devbox and one local-NVMe checkpoint for both runs. Server knobs: `--mm-feature-transport cpu --mm-processor-worker-num 2 --mm-io-worker-num 16 --mem-fraction-static 0.85 --mamba-full-memory-ratio 0.9`. The seeded workload uses 32 OpenAI Chat requests at each request rate, 1–4 random images per request (86 images total), random 512–1536 px dimensions, and 4 output tokens. Every measured run completed 32/32 requests and processed 5,018 input tokens, including 3,668 vision tokens, with no request errors. | Request rate | Metric | Before | After | Change | |---:|---|---:|---:|---:| | 2 | Request throughput | 2.276 req/s | 2.279 req/s | +0.1% | | 2 | Median TTFT | 490.8 ms | 372.8 ms | -24.1% | | 2 | P90 TTFT | 803.4 ms | 964.0 ms | +20.0% | | 8 | Request throughput | 8.758 req/s | 8.775 req/s | +0.2% | | 8 | Median TTFT | 901.4 ms | 436.3 ms | -51.6% | | 8 | P90 TTFT | 1436.0 ms | 594.1 ms | -58.6% | | unlimited | Request throughput | 6.457 req/s | 22.486 req/s | +248.3% | | unlimited | Median TTFT | 4742.8 ms | 1008.6 ms | -78.7% | | unlimited | P90 TTFT | 4812.8 ms | 1282.8 ms | -73.3% | At rate 2, the median improved while the single-run P90 regressed; that low-load tail is noisy and is not claimed as a win. The rate-8 and unlimited runs show the intended owner-side preprocessing benefit under concurrency. No serving-time compilation occurred during either measured sequence. Baseline source: `52afe87a08c6aa049c52f9507b4f0ca26cecb562`. PR source/current head: `d8e645cba2e805db51ca624bc8769f5e57099bc1`. ## Microbenchmark Two NVIDIA B200 GPUs, source image on GPU 0 and vision owner on GPU 1. Medians include preprocessing plus the CPU round trip; outputs were bitwise equal after BF16 conversion. | Input | CPU payload | Before | After | Speedup | |---|---:|---:|---:| | 1 × 1536×1024 | 18.3 → 4.5 MiB | 3.95 ms | 1.51 ms | 2.61× | | 4 × 1536×1024 | 73.0 → 18.0 MiB | 51.50 ms | 4.04 ms | 12.76× | | 1 × 4096×3072 | 145.1 → 36.0 MiB | 107.75 ms | 24.23 ms | 4.45× | ## Validation - latest-head real-weight server A/B: all six measured runs completed 32/32 VLM requests with zero errors and nonzero vision/output token counts - `67 passed`: K3 processor, vision, scheduling, and GPU-preprocessing targeted tests on NVIDIA B200 (pre-guard head) - follow-up CPU validation on the B300 devbox: global-config ratchet `1 passed`; full K3 vision plus focused processor coverage `31 passed` - regression coverage for empty batches, eager float tensors, non-uint8 deferral, and preserving CUDA IPC/CUDA VMM paths - pre-commit hooks on all changed files - microbenchmark output equality: BF16 bitwise equal --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31236611773](https://github.com/sgl-project/sglang/actions/runs/31236611773) Latest PR Test (Extra): :white_check_mark: [Run #31236611637](https://github.com/sgl-project/sglang/actions/runs/31236611637) ",
"url": "https://github.com/sgl-project/sglang/pull/33921",
"createdAt": "2026-08-07T01:26:34Z",
"updatedAt": "2026-08-13T11:51:08Z",
"timestamp": "2026-08-13T11:51:08Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"Multi-modal",
"run-ci",
"run-ci-extra"
],
"author": "mickqian",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33922",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix Qwen3.5 GDN multi-item scoring",
"text": "## Motivation Qwen3.5 hybrid models use recurrent GDN layers alongside full-attention layers. Multi-item scoring already gives the full-attention layers an item-branching mask, but the GDN prefill path previously processed the packed suffix as one continuous recurrent sequence. As a result, later items could inherit convolution and SSM state from earlier items. Fixes #31969. ## Modifications - Add a read-only final-state mode to the Triton chunk gated-delta-rule path while preserving in-place updates by default. - Build GDN-specific MIS metadata once per forward for compact query and item segments. - Run GDN MIS prefill in two phases: compute each request query once, then branch every item from the query-end convolution and SSM states. - Keep item recurrence read-only against the request SSM pool and use unique cloned convolution slots per item. - Reject unsupported combinations, including non-Triton GDN prefill, page-major layout, cached prefixes, non-prefill requests, and hybrid linear backends without MIS support. - Add kernel, causal-convolution, backend, metadata, validation, and Qwen3.5 end-to-end coverage. ## Accuracy Tests Tested on one NVIDIA H20 with `Qwen/Qwen3.5-4B` in BF16, FlashInfer full attention, and Triton linear-attention prefill. - Qwen3.5 MIS end-to-end: 3 tests passed, covering empty queries, concurrent requests, varied item counts and lengths, and sibling modification/reordering. - Focused recurrent, causal-convolution, GDN backend, metadata, and validation suite: 27 tests passed with 46 subtests. - Full recurrent and causal-convolution test files: 275 tests passed. - Multi-item scores match per-item MIS scoring with `atol=2e-2, rtol=2e-2`. - Formatting, import checks, spelling, syntax compilation, CI registration validation, and whitespace checks passed. ## Speed Tests and Profiling Single H20, `Qwen/Qwen3.5-4B`, query length 120, item length 180, 10 items per request, 10 warmup requests, followed by three 60-second runs. CUDA graph, radix cache, and chunked prefill were disabled in both modes. | Mode | Items/s/GPU | P50 latency | P99 latency | | --- | ---: | ---: | ---: | | Independent pointwise scoring | 52.80 | 189.06 ms | 195.25 ms | | Multi-item scoring | 62.75 | 158.74 ms | 167.21 ms | This is an 18.84% throughput increase, with P50 latency reduced by 16.04% and P99 latency reduced by 14.36%. ## Checklist - [x] Format the code with the repository formatters and static checks. - [x] Add unit and end-to-end tests. - [x] Documentation is not required because this does not change the public API or Score API schema. - [x] Provide accuracy and speed benchmark results. - [x] Follow the SGLang code style guidance. ## Review and Merge Process 1. Ping Merge Oncalls to start the process. 2. Get approvals from CODEOWNERS and other reviewers. 3. Trigger the required CI tests. 4. After green CI and required approvals, ask Merge Oncalls to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31371271748](https://github.com/sgl-project/sglang/actions/runs/31371271748) Latest PR Test (Extra): :x: [Run #31371271514](https://github.com/sgl-project/sglang/actions/runs/31371271514) ",
"url": "https://github.com/sgl-project/sglang/pull/33922",
"createdAt": "2026-08-07T01:39:34Z",
"updatedAt": "2026-08-13T06:05:33Z",
"timestamp": "2026-08-13T06:05:33Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "daii-0818",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33991",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(diffusion): add MiniMax H3 AdaLN cache",
"text": "## Summary - add an explicit MiniMax H3 AdaLN sidecar-cache format and CUDA builder - skip original AdaLN checkpoint keys only when a matching cache is selected - reject timestep embeddings absent from the cache rather than changing the inference path - document the opt-in CLI and cookbook workflow ## Motivation The MiniMax H3 model card notes that its AdaLN branches can be precomputed for inference-only serving. The public checkpoint ships those original branches, not a cache artifact, so this keeps the existing path as the default and makes cache use explicit. ## Status Draft: remote end-to-end numerical and peak-memory validation are pending. --- ### CI States Latest PR Test (Base): :x: [Run #31182344526](https://github.com/sgl-project/sglang/actions/runs/31182344526) Latest PR Test (Extra): :x: [Run #31182344502](https://github.com/sgl-project/sglang/actions/runs/31182344502) ",
"url": "https://github.com/sgl-project/sglang/pull/33991",
"createdAt": "2026-08-07T12:45:01Z",
"updatedAt": "2026-08-13T07:53:18Z",
"timestamp": "2026-08-13T07:53:18Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"documentation",
"diffusion"
],
"author": "mickqian",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33998",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] Optimize LogicalHostPool free-list release",
"text": "
![\"image\"](\"https://github.com/user-attachments/assets/56e4bff3-019f-4378-8ac3-ebddfe142b1d\")
After: ![\"image\"](\"https://github.com/user-attachments/assets/2a5eba97-34e8-48c4-bc69-e89b0f7d665b\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #29729414329](https://github.com/sgl-project/sglang/actions/runs/29729414329) Latest PR Test (Extra): :x: [Run #29729414083](https://github.com/sgl-project/sglang/actions/runs/29729414083) ",
"url": "https://github.com/sgl-project/sglang/pull/29202",
"createdAt": "2026-06-24T19:36:59Z",
"updatedAt": "2026-08-13T07:50:20Z",
"timestamp": "2026-08-13T07:50:20Z",
"metrics": {
"reactions": 0,
"comments": 17
},
"labels": [
"amd",
"deepseek",
"run-ci",
"bypass-fastfail"
],
"author": "RolaoDenthu",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29328",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs",
"text": "## Summary Adds support in SGLang for loading and inference for ModelOpt (and Quark) NVFP4 checkpoints by dequantizing the supplied NVFP4 weights and requantizing them to MXFP4 at load time to allow efficient inference using supported AMD hardware (e.g. AMD MI355x). Enabled via `--quantization quark_mxfp4`. Weights dequantized then requantized to MXFP4 during loading, so the resulting model occupies MXFP4 memory without ever holding a full-precision copy. Also includes the config-routing and shared-expert-fusion fixes needed to load ModelOpt exports across various architectures using the existing SGLang AMD Quark integration. ## Modifications ### Online Requantization Path - **`layers/quantization/quark/quark.py`**: `_detect_nvfp4_source` detects ModelOpt and AMD Quark NVFP4 layouts; `_parse_nvfp4_excludes` extracts producer exclude lists; `from_config` routing gated on `requantization_method`. `_maybe_disable_shared_experts_fusion` delegates to the existing `can_fuse_shared_expert()` to correctly handle `shared_expert_gate` and `mtp.` exclusions. - **`layers/quantization/quark/schemes/quark_w4a4_mxfp4.py`**: NVFP4 linear scheme: `_create_weights_from_nvfp4` allocates NVFP4 staging buffers directly on device, loads all source shards, then dequant-requant to MXFP4 once all bytes are present. Registers (and discards) the NVFP4 per-tensor `input_scale` slot since MXFP4 uses dynamic activation quantization. - **`layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py`**: NVFP4 fused-MoE scheme: per-expert weight/scale registration, online expert-wise dequant-requant, no-op `input_scale` slots for w13/w2. - **`layers/quantization/dequantization.py`**, **`layers/quantization/online_quantization.py`**, **`layers/quantization/quark/utils.py`**: `Nvfp4SourceConfig` + dequant helpers and `CopyNumelCounter` to track progressive shard loading ### Fixes required to load ModelOpt exports - **Config-loading** (`model_loader/weight_utils.py`): Some ModelOpt NVFP4 checkpoints store quant metadata only in `hf_quant_config.json`; others duplicate it in `config.json`. `get_quant_config` now reads `hf_quant_config.json` preferentially when present and FP4-typed, routing through `from_config`. - **Shared-expert-fusion** (`layers/quantization/quark/quark.py`): Some NVFP4 checkpoints keep shared experts in BF16 while routed experts are FP4. Shared-expert fusion remaps `mlp.shared_experts` into the quantized `FusedMoE`, causing a shape mismatch. `QuarkConfig.__init__` now calls `_maybe_disable_shared_experts_fusion` which delegates to `can_fuse_shared_expert()` (which is the same detection path introduced in #22948 and scoped to Quark in #27057) and sets `disable_shared_experts_fusion` globally before model construction. ### Tests - **`test/registered/quant/test_quark_mxfp4.py`**: Added `TestNVFP4ToMXFP4MOETP1` testing NVFP4→MXFP4 MoE requantization accuracy on `nvidia/Qwen3-30B-A3B-NVFP4`. ### Docs - **`docs_new/docs/advanced_features/quantization.mdx`**: documents the NVFP4-MXFP4 requant path (supported checkpoint layouts, and example command. ## Accuracy Tests All numbers below are obtained using lm_eval using SGLang backend with `--quantization quark_mxfp4` on AMD MI355X GPUs. (`max_gen_toks`=32768) ### GSM8k (lm_eval, flexible-extract) | Model | Original Release (bf16/fp8) | NVFP4-to-MXFP4 | % Recovery | |---|---|---|---| | MiniMax-M2.7-NVFP4 | 0.918 | 0.920 | 100.2% | | GLM-5.1-NVFP4 | 0.952 | 0.928 | 97.5% | | Kimi-K2.6-NVFP4 | 0.939 | 0.930 | 99.0% | | Qwen3.5-397B-A17B-NVFP4 | 0.954 | 0.945 | 99.1% | | Qwen3.5-397B-A17B-NVFP4-V2 | 0.954 | 0.941 | 98.7% | | DeepSeek-R1-NVFP4 | 0.958 | 0.950 | 99.2% | --- ### CI States Latest PR Test (Base): :x: [Run #31619623512](https://github.com/sgl-project/sglang/actions/runs/31619623512) Latest PR Test (Extra): :x: [Run #31619623176](https://github.com/sgl-project/sglang/actions/runs/31619623176) ",
"url": "https://github.com/sgl-project/sglang/pull/29328",
"createdAt": "2026-06-25T20:10:55Z",
"updatedAt": "2026-08-13T17:57:19Z",
"timestamp": "2026-08-13T17:57:19Z",
"metrics": {
"reactions": 1,
"comments": 22
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"lora",
"Multi-modal",
"deepseek",
"speculative-decoding",
"hicache",
"blackwell",
"npu",
"run-ci",
"diffusion",
"mthreads",
"apple-silicon",
"jit-kernel",
"bypass-fastfail"
],
"author": "ColinZ22",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:29593",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[CPU][QUANT] add amx cpu support for auto-round",
"text": " ## Motivation Currently supports offline inference for CUDA devices with auto-round quantized models, but CPU inference is not yet supported. This is a blocker for users who want to run INT4 quantized models on CPU-only machines or in CPU-based serving pipelines using SGLang as the backend. issue description: https://github.com/sgl-project/sglang/issues/27568 ## Modifications add CPU auto-round format quantized model loading support. bugfix of FusedMoE layer loading with marlin backend, pass backend args to apply_quant_layer refactor auto_round into package layout. ## Accuracy Tests Evaluated `tmp/Qwen3-4B-int4-AutoRound` (auto-round INT4, W4G128, sym) served on CPU via SGLang using `lm-eval` (0-shot): | Tasks | Metric | Value | Stderr | |----------------|------------|-------:|-------:| | lambada_openai | acc | 0.5909 | 0.0068 | | lambada_openai | perplexity | 7.7950 | 0.3152 | | piqa | acc | 0.7410 | 0.0102 | | piqa | acc_norm | 0.7443 | 0.0102 | original BF16 reference: | Tasks | Metric | BF16 | INT4 AutoRound | |----------------|------------|-------:|---------------:| | lambada_openai | acc | 0.6024 | 0.5909 | | lambada_openai | perplexity | 7.3054 | 7.7950 | | piqa | acc | 0.7497 | 0.7410 | | piqa | acc_norm | 0.7497 | 0.7443 | moe model generate and eval : ![\"image\"](\"https://github.com/user-attachments/assets/e8384c20-7fd4-43b1-800c-36b17c317b17\")
## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31575342235](https://github.com/sgl-project/sglang/actions/runs/31575342235) Latest PR Test (Extra): :x: [Run #31575342321](https://github.com/sgl-project/sglang/actions/runs/31575342321) ",
"url": "https://github.com/sgl-project/sglang/pull/29593",
"createdAt": "2026-06-29T01:37:30Z",
"updatedAt": "2026-08-13T07:51:00Z",
"timestamp": "2026-08-13T07:51:00Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"documentation",
"quant",
"intel",
"cpu",
"run-ci",
"run-ci-extra"
],
"author": "WeiweiZhang1",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29668",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] fix: resolve Mooncake local_hostname per node for runtime attach",
"text": " ## Motivation When Mooncake HiCache storage is attached at runtime via `PUT /hicache/storage-backend`, `extra_config` is broadcast to all ranks. `load_from_extra_config()` previously fell back to the fixed default `\"localhost\"` when `local_hostname` was omitted, and a head-node `local_hostname` in the attach payload was applied to every rank. This breaks multi-node Mooncake deployments. Fixes sgl-project/sglang#23457 ## Modifications - Add `MooncakeStoreConfig._resolve_local_hostname()` so `MOONCAKE_LOCAL_HOSTNAME` / `LOCAL_HOSTNAME` from the current process take precedence over shared `extra_config`. - Reuse the helper in `from_file()`, `load_from_env()`, and `load_from_extra_config()`. - Document multi-node runtime attach guidance and JSON config precedence in the Mooncake store README. - Add unit tests covering env precedence, broadcast override behavior, and `from_file` env-over-file resolution. - Register tests for CI with `register_cpu_ci` and `CustomTestCase`. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31722158250](https://github.com/sgl-project/sglang/actions/runs/31722158250) Latest PR Test (Extra): :x: [Run #31722157962](https://github.com/sgl-project/sglang/actions/runs/31722157962) ",
"url": "https://github.com/sgl-project/sglang/pull/29668",
"createdAt": "2026-06-29T17:37:06Z",
"updatedAt": "2026-08-13T16:56:12Z",
"timestamp": "2026-08-13T16:56:12Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"run-ci"
],
"author": "stmatengss",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29723",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Add fused all-reduce RMSNorm per-token FP8/MXFP4 quant",
"text": "## Motivation The existing fused AllReduce → RMSNorm → quant path only supports per-group FP8 scales. Checkpoints that use per-token FP8 activation scaling (and MXFP4) therefore fall back to the unfused AllReduce → RMSNorm → quantize sequence — three kernel launches plus intermediate bf16 round-trips — before every tensor-parallel linear. ## Modifications Adds a per-token FP8 variant (and a 1-stage MXFP4 variant) of the fused kernel, collapsing those three launches into a single aiter kernel. The change is additive and gated: when the per-token fast path is not applicable it falls back to the existing per-group / plain AR+RMSNorm path with no behavior change. ### Key changes **Collective entry points** (`distributed/parallel_state.py`, `distributed/communication_op.py`) - `tensor_model_parallel_fused_allreduce_rmsnorm_quant_per_token` returns `(fp8_output, residual_out, per_token_scale)` with `per_token_scale` shaped `(M, 1)`, or `None` when the backend cannot fuse (fallback signal). - `tensor_model_parallel_fused_allreduce_rmsnorm_mxfp4_quant` plus the `_should_use_1stage_mxfp4_ar` heuristic for the MXFP4 1-stage path. - Device-communicator methods `fused_allreduce_rmsnorm_quant_per_token` / `fused_allreduce_rmsnorm_mxfp4_quant`, calling aiter `custom_fused_ar_rms_quant(post_per_token_quant=True)`. **LayerNorm hook** (`layers/layernorm.py`) - `forward_with_allreduce_fusion_quant_per_token` (plus the shared `_forward_with_allreduce_fusion_quant_per_token` impl), with a cached `_aiter_per_token_quant` functor and a non-aiter / `residual is None` fallback. **Communicator gating** (`layers/communicator.py`) - `_try_fused_allreduce_rmsnorm_quant` dispatches to the per-token path when `quant_format == \"fp8_per_token\"` and the LayerNorm exposes the per-token hook; otherwise it falls through to per-group / unfused. **Quant fast path** (`layers/quantization/fp8_utils.py`) - Consume pre-quantized `(fp8, per_token_scale)` inputs directly, skipping the linear's internal quantization. **Qwen3.5 wiring** (`models/qwen3_5.py`) - `_detect_fused_ar_quant_format` returns `\"fp8_per_token\"` under `SGLANG_USE_AITER_FP8_PER_TOKEN`; consumes the fused 2-tuple/3-tuple handoff in both the GDN linear-attention (`in_proj_qkvz` / `in_proj_ba`) path and the full-attention (`qkv_proj`) path. ### Enablement `SGLANG_USE_AITER=1` + `--enable-aiter-allreduce-fusion` (existing), plus `SGLANG_USE_AITER_FP8_PER_TOKEN=1` to select the per-token format. MXFP4 checkpoints auto-select the MXFP4 variant. ROCm/aiter/gfx95-gated; other backends and non-eligible configs are unchanged. ## Accuracy Tests The fused kernel is numerically equivalent to the reference AR → RMSNorm → per-token quant sequence; when the fast path is unavailable it defers to the existing per-group / plain path. Verified on Qwen3.5-397B MXFP4-AttnFP8 with GSM8K (1319q): fusion ON = 0.932, and toggling the per-token FP8 fusion off (`SGLANG_DISABLE_FUSED_AR_QUANT`) leaves accuracy unchanged at 0.932, confirming the fallback is numerically faithful. Unit tests: `test/registered/ops/test_aiter_allreduce_fusion_amd.py` adds per-token FP8 and MXFP4 cases (fused vs unfused reference) and the `None`-fallback contract. ## Speed Tests and Profiling Kernel benchmark: `benchmark/kernels/all_reduce/benchmark_fused_ar_rms_quant_amd.py` adds per-token FP8 and MXFP4 coverage against the 3-launch baseline. ### End-to-end Setup: Qwen3.5-397B MXFP4-AttnFP8, TP2 on MI355X, fp8 KV cache, `--attention-backend aiter`, random dataset, output len 1024, range ratio 0.8, num-prompts = 10 x cc. (1k,1k) uses `--enable-mixed-chunk`; (8k,1k) does not. The arms differ only by `SGLANG_USE_AITER_FP8_PER_TOKEN` (OFF = per-token fusion disabled → per-group/plain fallback; ON = per-token fused path). Accuracy is identical between arms (GSM8K 1319q = 0.932 both, see Accuracy Tests), so this is a like-for-like numerical comparison. Total token throughput (tok/s), higher is better: | cc | (1k,1k) OFF | (1k,1k) ON | Δ | (8k,1k) OFF | (8k,1k) ON | Δ | |---:|---:|---:|---:|---:|---:|---:| | 4 | 804.5 | 805.4 | +0.11% | 3265.4 | 3299.7 | +1.05% | | 8 | 1401.0 | 1411.1 | +0.72% | 5357.8 | 5413.8 | +1.05% | | 16 | 2129.0 | 2182.1 | +2.49% | 7729.3 | 7749.9 | +0.27% | | 32 | 3097.5 | 3168.6 | +2.29% | 10341.4 | 10327.2 | -0.14% | | 64 | 4578.9 | 4590.5 | +0.26% | 13255.2 | 13269.0 | +0.10% | Mean TPOT (ms/token), lower is better: | cc | (1k,1k) OFF | (1k,1k) ON | Δ | (8k,1k) OFF | (8k,1k) ON | Δ | |---:|---:|---:|---:|---:|---:|---:| | 4 | 9.57 | 9.56 | -0.10% | 10.38 | 10.27 | -1.06% | | 8 | 10.95 | 10.92 | -0.27% | 12.58 | 12.45 | -1.03% | | 16 | 14.63 | 14.28 | -2.39% | 17.58 | 17.54 | -0.23% | | 32 | 19.99 | 19.53 | -2.30% | 26.22 | 26.23 | +0.04% | | 64 | 27.05 | 26.98 | -0.26% | 41.44 | 41.36 | -0.19% | The fused path is faster or equal in 9 of 10 cells, and TPOT moves in lockstep with throughput, which is the signature of removing launches from a decode-bound step rather than of noise. The largest gains are ~2.4% at (1k,1k) cc=16/32; the rest is neutral. That is the expected magnitude: the fused region is a small share of a decode step, so collapsing three launches into one removes real work without dominating end-to-end serving throughput. Caveat on provenance: these are single samples per cell, and the ON and OFF arms were collected in separate sessions rather than interleaved, so deltas below ~1% should be read as neutral. ### Applicability The saving is largest when the consumer of the fused output accepts the quantized tensor directly. If the downstream linear is not quantized in a matching format, the kernel must still produce a bf16 side output, and the benefit reduces to the removed launches alone. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 5. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 6. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 7. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31709151314](https://github.com/sgl-project/sglang/actions/runs/31709151314) Latest PR Test (Extra): :x: [Run #31709151159](https://github.com/sgl-project/sglang/actions/runs/31709151159) ",
"url": "https://github.com/sgl-project/sglang/pull/29723",
"createdAt": "2026-06-30T07:34:19Z",
"updatedAt": "2026-08-13T15:17:28Z",
"timestamp": "2026-08-13T15:17:28Z",
"metrics": {
"reactions": 2,
"comments": 15
},
"labels": [
"high priority",
"quant",
"amd",
"run-ci"
],
"author": "mqhc2020",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29858",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Build SWA window kv buffers for the EAGLE draft-extend cuda-graph path",
"text": "## Motivation Intermittent GPU memory access fault during EAGLE multi-layer speculative decode on hybrid sliding-window models (a mix of full-attention and SWA layers, e.g. MiMo-V2.5-Pro). It only shows up with cuda graph enabled and only when the SWA kv pool is smaller than the full pool. With `--disable-cuda-graph` the model runs clean, which is the tell that this is a cuda-graph-only metadata bug and not a kernel bug. Root cause: on hybrid-SWA models the triton backend keeps the SWA layers on a smaller swa kv pool and builds a separate set of window kv buffers (`window_kv_indptr` / `window_kv_indices` and the window offsets) that index that pool. Three of the four buffer-building paths do this: decode cuda graph (`_update_decode_kv_buffers`), target_verify cuda graph (`_update_target_verify_buffers`), and every eager path (`init_forward_metadata` -> `update_sliding_window_buffer`). The EAGLE draft-extend cuda-graph path is the odd one out: `_update_draft_extend_buffers` fills only the full kv buffers, and the `is_draft_extend_v2` branch of `_build_cuda_graph_forward_metadata` hard-codes `window_kv_indices=None` (the decode/verify branches use `... if swa else None`). So on a hybrid-SWA model the draft step runs the SWA layers with no window indices. `forward_extend` takes the sliding-window branch (the layer has a window size), reads `window_kv_indices` (None), and the captured extend kernel indexes the small swa K/V buffer with full-pool locations, an out-of-bounds read. Under rocgdb the fault lands in the extend kernel (`_fwd_kernel`) on the SWA K-buffer load, and the faulting virtual addresses are foreign to every live `kv_indices` buffer, which is the signature of an OOB read off a too-small buffer rather than a stale/freed pointer. ## Modifications Build the swa window buffers for draft-extend the same way target_verify already does: 1. `_update_draft_extend_buffers`: after filling the full kv buffers, call `update_sliding_window_buffer` to fill the persistent swa window buffers (`cuda_graph_window_kv_indices` / offsets). The window covers the prefix portion (`kv_lens`), matching how draft-extend builds its full `kv_indices`. 2. `is_draft_extend_v2` branch of `_build_cuda_graph_forward_metadata`: point `window_kv_indptr` / `window_kv_indices` / `window_num_kv_splits` / `window_kv_offsets` at those buffers when `swa` is set, mirroring the verify branch. Both capture and replay route through `_apply_cuda_graph_metadata` -> `_update_draft_extend_buffers`, so the persistent swa buffers are refilled on every replay and the ForwardMetadata view (built once at capture) keeps pointing at the same address-stable buffers. The new code is gated on `sliding_window_size > 0` and the metadata fields stay None otherwise, so it is a no-op for non-SWA models. Eager is already correct (its extend path calls `update_sliding_window_buffer`), so this change is intentionally cuda-graph-only. ## Accuracy Tests MiMo-V2.5-Pro on MI300X-class hardware, TP8, default hybrid window sizing, cuda graph on. Before the fix the config faults intermittently (~1 in 3 runs, across all ranks). After the fix, full GSM8K runs clean across 6 consecutive runs, strict-match ~0.963-0.967, in line with the baseline. ## Speed Tests and Profiling Cuda-graph throughput is preserved (~195 s/pass on the validation config), so there is no need to fall back to `--disable-cuda-graph` (~4x slower on decode) or to the `--swa-full-tokens-ratio 1.0` workaround (which costs ~17% KV headroom). The change is a metadata build on the draft-extend path with no kernel changes, so no per-kernel profiling delta is expected. Notes for reviewers: - The same defect exists on the `v0.5.12.post1` release tag but in the pre-refactor shape (separate `init_forward_metadata_capture/replay_cuda_graph` + `update_sliding_window_buffer_cuda_graph`), so a backport to that release needs the older-shape version of this fix, not this diff. - Deployments hitting this have been carrying a small guard in `forward_extend` that falls back to the full kv buffers when `window_kv_indices` is None. That guard turns the NoneType failure into the OOB read described above, so it is a symptom mask, not a fix, and is intentionally not included here since this PR makes `window_kv_indices` non-None on the draft path. - Follow-up worth doing separately: a small assertion that SWA forward modes always have a non-None `window_kv_indices` under cuda graph would catch this class of missing-window-build bug early. Also worth a confirmation run on gfx950 (validated on gfx942-class so far). --- ### CI States Latest PR Test (Base): :x: [Run #31189512886](https://github.com/sgl-project/sglang/actions/runs/31189512886) Latest PR Test (Extra): :x: [Run #31189511878](https://github.com/sgl-project/sglang/actions/runs/31189511878) ",
"url": "https://github.com/sgl-project/sglang/pull/29858",
"createdAt": "2026-07-01T16:53:51Z",
"updatedAt": "2026-08-13T16:01:39Z",
"timestamp": "2026-08-13T16:01:39Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [],
"author": "reger-men",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:29935",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature][Intel XPU] Add memory saver support for Intel XPU via upstream torch_memory_saver",
"text": "## Motivation Enable release/resume_memory_occupation on Intel XPU using the upstream torch_memory_saver package (Level Zero VMM backend). Replaces in-repo implementation with pip-installable upstream dependency. Features: - Pause/resume physical GPU memory while preserving virtual addresses - Tag-based selective memory management (kv_cache, weights, cuda_graph) - Optional CPU backup for content preservation - Multi-device support (verified via the package's multi-device test) - Sysman-based memory verification (torch accounting doesn't reflect unmapped pages) ## Modifications Changes: - Add TorchMemorySaverAdapter for unified memory saver API - XPU uses hook_mode='torch' (in-process pluggable allocator) - configure_subprocess() and cuda_graph() are no-ops on XPU - Update Dockerfile to build the upstream package for XPU - Add test suite (10 tests, all passing) Installation (XPU builds from source; TMS_PLATFORM=xpu forces the XPU backend and --no-build-isolation lets the build match libsycl to the installed torch): source /opt/intel/oneapi/setvars.sh TMS_PLATFORM=xpu pip install --no-build-isolation \\ git+https://github.com/fzyzcjy/torch_memory_saver.git Usage: python -m sglang.launch_server --model-path ![\"image\"](\"https://github.com/user-attachments/assets/9a8170f6-d77a-47da-8391-54db303aeb88\")
PR: ![\"image\"](\"https://github.com/user-attachments/assets/62947b7e-b05f-472e-bc9a-ccb496853556\")
Qwen/Qwen3.5-397B-A17B-FP8 B200 pareto 8k1k no MTP: ```bash python3 -m sglang.launch_server \\ --model-path=\"${MODEL}\" \\ --host=\"${HOST}\" \\ --port=\"${PORT}\" \\ --trust-remote-code \\ --tensor-parallel-size=4 \\ --data-parallel-size=1 \\ --expert-parallel-size=1 \\ --enable-symm-mem \\ --disable-radix-cache \\ --quantization fp8 \\ --kv-cache-dtype fp8_e4m3 \\ --mamba-ssm-dtype bfloat16 \\ --attention-backend trtllm_mha \\ --moe-runner-backend flashinfer_trtllm \\ --linear-attn-prefill-backend flashinfer \\ --cuda-graph-max-bs 128 \\ --max-prefill-tokens 16384 \\ --chunked-prefill-size 16384 \\ --mem-fraction-static 0.8 \\ --stream-interval 50 \\ --cuda-graph-backend-prefill=breakable \\ --scheduler-recv-interval 30 \\ --tokenizer-worker-num 6 \\ --context-length 9236 ``` ![\"image\"](\"https://github.com/user-attachments/assets/93d930a4-ac00-46d9-b843-29b487926b87\")
```bash series conc TPS/user (sp) out TPS/GPU (sp) total TPS/GPU (sp) TTFT ms (sp) ---------------------------------------------------------------------------------------------------------------------- base 4 148.53 ( 1.00x) 135.67 ( 1.00x) 1230.91 ( 1.00x) 366.08 ( 1.00x) base 8 111.83 ( 1.00x) 202.51 ( 1.00x) 1842.56 ( 1.00x) 542.41 ( 1.00x) base 16 75.32 ( 1.00x) 286.28 ( 1.00x) 2553.16 ( 1.00x) 671.15 ( 1.00x) base 32 52.16 ( 1.00x) 395.21 ( 1.00x) 3578.48 ( 1.00x) 813.98 ( 1.00x) base 64 34.60 ( 1.00x) 539.48 ( 1.00x) 4875.69 ( 1.00x) 978.56 ( 1.00x) base 128 22.91 ( 1.00x) 693.82 ( 1.00x) 6263.06 ( 1.00x) 1345.94 ( 1.00x) opt 4 150.98 ( 1.02x) 143.51 ( 1.06x) 1301.99 ( 1.06x) 295.00 ( 1.24x) opt 8 122.60 ( 1.10x) 230.37 ( 1.14x) 2096.01 ( 1.14x) 310.17 ( 1.75x) opt 16 90.26 ( 1.20x) 343.61 ( 1.20x) 3064.46 ( 1.20x) 381.25 ( 1.76x) opt 32 62.95 ( 1.21x) 477.41 ( 1.21x) 4322.79 ( 1.21x) 568.09 ( 1.43x) opt 64 42.50 ( 1.23x) 653.23 ( 1.21x) 5903.75 ( 1.21x) 775.03 ( 1.26x) opt 128 27.96 ( 1.22x) 832.00 ( 1.20x) 7510.41 ( 1.20x) 1112.19 ( 1.21x) ``` --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31580932305](https://github.com/sgl-project/sglang/actions/runs/31580932305) Latest PR Test (Extra): :x: [Run #31580932580](https://github.com/sgl-project/sglang/actions/runs/31580932580) ",
"url": "https://github.com/sgl-project/sglang/pull/30797",
"createdAt": "2026-07-10T16:33:19Z",
"updatedAt": "2026-08-13T08:18:15Z",
"timestamp": "2026-08-13T08:18:15Z",
"metrics": {
"reactions": 0,
"comments": 13
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "mattteochen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30805",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103",
"text": " ## Motivation Integrates TRT-LLM attention kernel for DSv4 style attention (CSA, HCA). ## Modifications ## Accuracy Tests ``` SGLANG_DSV4_ATTN_DECODE_BACKEND=flashmla/trtllm_gen \\ python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4-Pro \\ --trust-remote-code --tp 8 --moe-runner-backend flashinfer_mxfp4 \\ --chunked-prefill-size 4096 --disable-flashinfer-autotune \\ --mem-fraction-static 0.88 --max-running-requests 32 ``` ### GSM8K Accuracy GSM8K V4-Pro flashmla: 0.965 GSM8K V4-Pro trtllm_gen: 0.985 ### GPQA (add --chat-template to above server command) ``` python3 -m sglang.test.run_eval --port 21000 --eval-name gpqa \\ --num-examples 198 --max-tokens 120000 --repeat 8 \\ --thinking-mode deepseek-v3 ``` FlashMLA: Repeat: 8, mean: 0.885 Scores: ['0.889', '0.879', '0.889', '0.879', '0.869', '0.899', '0.904', '0.874'] TRTLLM: Repeat: 8, mean: 0.907 Scores: ['0.919', '0.904', '0.894', '0.904', '0.909', '0.919', '0.904', '0.904'] ## Speed Tests and Profiling ### Unit Kernel Benchmarking (ran with private benchmarking repo): ``` # prefill (varlen), long contexts python benchmark.py --backends flashmla_dsv4,trtllm_gen_dsv4 --model deepseek-v4 \\ --hw b200 --dtype fp8 --mode prefill --tp 1 \\ --batch-sizes 1,2 --seq-lens 4096,16384,65536 --graph # decode, low/mid concurrency x long contexts python benchmark.py --backends flashmla_dsv4,trtllm_gen_dsv4 --model deepseek-v4 \\ --hw b200 --dtype fp8 --mode decode --tp 1 \\ --batch-sizes 1,4,16,64 --seq-lens 1024,4096,16384,65536,131072 --graph # decode, high concurrency python benchmark.py --backends flashmla_dsv4,trtllm_gen_dsv4 --model deepseek-v4 \\ --hw b200 --dtype fp8 --mode decode --tp 1 \\ --batch-sizes 128,256,512,1024 --seq-lens 256,1024,4096,16384 --graph ``` Prefill: ~1.2x faster Decode: ~1.45x faster ![\"fp8_tp1_prefill\"](\"https://github.com/user-attachments/assets/b5adb0e5-9dce-46ee-a6c8-40f892492b3b\")
![\"fp8_tp1_decode](\"https://github.com/user-attachments/assets/7a0411bc-332a-452f-aed2-0fed800b2d71\")
![\"fp8_tp1_decode\"](\"https://github.com/user-attachments/assets/fe284e92-5568-4485-8504-25999d821011\")
### E2E bench-serving ```bash python -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 21000 \\ --dataset-name random --random-input-len 131072 --random-output-len 128 \\ --random-range-ratio 1 --num-prompts 8 --max-concurrency 4 ``` flashmla: ``` Benchmark duration (s): 59.92 Total input tokens: 1048576 Input token throughput (tok/s): 17500.55 Mean E2E Latency (ms): 29950.67 Mean TTFT (ms): 19009.87 (median 19636.34) Mean TPOT (ms): 86.15 Median ITL (ms): 12.12 ``` trtllm_gen: ``` Benchmark duration (s): 60.09 Total input tokens: 1048576 Input token throughput (tok/s): 17448.78 Mean E2E Latency (ms): 30039.27 Mean TTFT (ms): 16808.71 (median 15734.66) Mean TPOT (ms): 104.18 Median ITL (ms): 13.11 ``` ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31707526521](https://github.com/sgl-project/sglang/actions/runs/31707526521) Latest PR Test (Extra): :x: [Run #31707526162](https://github.com/sgl-project/sglang/actions/runs/31707526162) ",
"url": "https://github.com/sgl-project/sglang/pull/30805",
"createdAt": "2026-07-10T19:01:43Z",
"updatedAt": "2026-08-13T14:34:42Z",
"timestamp": "2026-08-13T14:34:42Z",
"metrics": {
"reactions": 0,
"comments": 37
},
"labels": [
"high priority",
"deepseek",
"blackwell",
"run-ci",
"release-highlight"
],
"author": "akhilg-nv",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:30808",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [GLM5] Enable dense-MHA short-context prefill fallback on gfx950",
"text": "## Motivation On gfx950 (MI355X), GLM-5.2 DSA prefill always ran the triton sparse-MLA path, even at short context where the sparse indexer top-k + gather + mask overhead exceeds the KV it prunes. The dense-MHA prefill fallback — already used on NVIDIA SM90/SM100 and gated by `SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` (default = model `index_topk`) — was hard-gated to NVIDIA and never taken on ROCm, despite the dense kernel (aiter `flash_attn_varlen_func`) already being available. ## Modifications - Add gfx950 detection (`gcnArchName`) and include it in the `use_mha` prefill device gate in `dsa_backend.py` (previously `device_sm == 90 or 100 <= device_sm < 110`, NVIDIA only). - Guard `_forward_standard_mha` so ROCm routes through aiter `flash_attn_varlen_func`, never the flashinfer/trtllm (Blackwell) path. - NVIDIA behavior unchanged. On gfx950 the dense path triggers only when `max_kv_len <= SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD` (GLM-5.2: index_topk = 2048); longer context stays on sparse-MLA. Off switch: set the threshold to 0. ## Accuracy Tests GLM-5.2-MXFP4, MI355X TP4, GSM8K (200 questions): dense fallback = 0.955. ## Speed Benchmarks GLM-5.2-MXFP4, MI355X TP4, graphs-on, random 1024/1024, `sglang.bench_serving`, median. A. Baseline (dense fallback off)*: | concurrency | TTFT (ms) | ITL (ms) | E2EL (ms) | output tok/s | |---|---|---|---|---| | 4 | 277.3 | 11.33 | 11860 | 342.0 | | 8 | 467.6 | 12.98 | 13790 | 592.7 | | 16 | 872.9 | 16.48 | 17767 | 915.8 | | 32 | 1371.1 | 20.25 | 22655 | 1443.9 | | 64 | 1920.5 | 26.05 | 30112 | 2174.0 | B. Dense fallback on (Δ vs baseline): | concurrency | TTFT (ms) | Δ | ITL (ms) | Δ | E2EL (ms) | Δ | output tok/s | Δ | |---|---|---|---|---|---|---|---|---| | 4 | 196.9 | −29.0% | 11.31 | −0.2% | 11775 | −0.7% | 346.9 | +1.4% | | 8 | 366.8 | −21.6% | 13.00 | +0.2% | 13660 | −0.9% | 582.1 | −1.8% | | 16 | 526.6 | −39.7% | 16.46 | −0.1% | 17432| −1.9% | 935.0 | +2.1% | | 32 | 783.2 | −42.9% | 20.48 | +1.1% | 21895| −3.4% | 1498.13 | +3.8% | | 64 | 1125.6 | −41.4% | 26.10 | +0.2% | 29352 | −2.5% | 2227.3 | +2.5% | Aside from the consistent TTFT decreases (−22% to −43%), the ITL, E2EL, and output throughput deltas are within run-to-run noise (dense fallback is prefill-only, so decode is unaffected). *Baseline and Feature both measured with #30519, #30715 and aiter tuned MoE configs for GLM5.2; we expect those will be merged first and aiter version updated in recent images. ## Checklist - [x] Format your code according to the Format code with pre-commit. - [ ] Add unit tests according to the Run and add unit tests. - [x] Provide accuracy and speed benchmark results. - [x] Follow the SGLang code style guidance. ## Review and Merge Process 1. Ping Merge Oncalls to start the process. 2. Get approvals from CODEOWNERS and other reviewers. 3. Trigger CI tests with comments. --- ### CI States Latest PR Test (Base): :x: [Run #31635016485](https://github.com/sgl-project/sglang/actions/runs/31635016485) Latest PR Test (Extra): :x: [Run #31635016282](https://github.com/sgl-project/sglang/actions/runs/31635016282) ",
"url": "https://github.com/sgl-project/sglang/pull/30808",
"createdAt": "2026-07-10T20:30:40Z",
"updatedAt": "2026-08-12T22:22:42Z",
"timestamp": "2026-08-12T22:22:42Z",
"metrics": {
"reactions": 0,
"comments": 16
},
"labels": [
"documentation",
"amd",
"run-ci"
],
"author": "Raiden-Makoto",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30827",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat: add cache salt support to KV cache events",
"text": "## Motivation External KV-aware routers need request cache namespaces to remain distinct from SGLang's caller-defined `extra_key`. Concatenating the two values can collide, and the existing KV-event hashes and payloads do not expose a typed cache namespace. ## Modifications - Add an explicit scalar/batched `cache_salt` contract to `Engine.generate`, `Engine.async_generate`, request normalization, tokenization, scheduling, sessions, and the supported Python radix-cache variants. - Keep `cache_salt` separate from `extra_key` and index salted radix keys with the structured pair `(extra_key, cache_salt)` while preserving existing unsalted key behavior. - Namespace salted external KV-event hashes by seeding the root chain with `SHA256(b\"sglang-cache-salt-v1\\\\0\" + cache_salt.encode())`; internal cache hashes and unsalted event hashes remain unchanged. - Append typed `{ \"cache_salt\": \"...\" }` metadata to salted `BlockStored` events while preserving the legacy unsalted positional wire layout. - Add unit coverage for normalization, OpenAI forwarding, collision resistance, radix isolation, event hashing/removal, and msgspec wire compatibility. Cache-salt-plus-LoRA routing semantics are outside this PR. ## Accuracy Tests Not applicable; this does not modify model computation or generated outputs. ## Speed Tests and Profiling Not run. Unsalted radix keys and KV-event hashes retain their existing paths. Namespace hashing is limited to salted requests with KV events enabled. ## Validation - `python3 -m compileall -q python/sglang/srt ...` for all changed source and test modules - Black 26.1.0, isort 7.0.0, and Ruff 0.15.1 checks on all changed Python files - Direct msgspec checks for legacy 7-slot unsalted events, typed 8-slot salted events, and existing typed `KVEventBatch` consumers - Full SGLang unit/GPU suites were not available in this local checkout; the focused tests are included for CI ## Checklist - [x] Format the changed code with the repository-pinned tools. - [x] Add focused unit tests. - [x] No documentation update is required for this internal routing/event contract. - [x] Accuracy and speed benchmarks are not applicable to model computation. - [x] Follow the SGLang code style guidance. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31618650333](https://github.com/sgl-project/sglang/actions/runs/31618650333) Latest PR Test (Extra): :x: [Run #31618650207](https://github.com/sgl-project/sglang/actions/runs/31618650207) ",
"url": "https://github.com/sgl-project/sglang/pull/30827",
"createdAt": "2026-07-10T22:36:11Z",
"updatedAt": "2026-08-12T23:14:05Z",
"timestamp": "2026-08-12T23:14:05Z",
"metrics": {
"reactions": 0,
"comments": 30
},
"labels": [
"run-ci"
],
"author": "jthomson04",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:30929",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support decode radix cache on DeepSeek-V4 (hybrid-SWA, SWA-tail prealloc)",
"text": "## Motivation `--disaggregation-decode-enable-radix-cache` is unconditionally rejected for **every** hybrid-SWA model in `build_kv_cache`: ``` ValueError: --disaggregation-decode-enable-radix-cache is incompatible with sliding window attention (SWA) models ``` But `disaggregation/decode.py` already implements a full **SWA-tail-prealloc decode-radix path** (`DecodeTransferQueue._uses_swa_tail_prealloc` → `_swa_tail_len` / `alloc_extend_swa_tail`) for pools whose allocator exposes `alloc_extend_swa_tail` with `page_size > 1` (e.g. `SWAKVPool` / `DeepSeekV4TokenToKVPool`): decode receives the full KV for full-attention layers and only the sliding-window tail for SWA layers, which is compatible with the prefix-match-and-lock allocation path. The blanket guard is stale — even `server_args` help documents decode radix as incompatible only with `--enable-hisparse`, speculative decoding, and the fake transfer backend, **not SWA**. As a result, decode-side prefix reuse is unavailable on DeepSeek-V4 and other SWA-tail-capable models, forcing redundant KV transfers. ## Modifications 1. **`mem_cache/kv_cache_builder.build_kv_cache`** — only reject SWA models whose allocator lacks the SWA-tail-prealloc capability (`hasattr(allocator, \"alloc_extend_swa_tail\") and page_size > 1`). Mamba/SSM stays rejected. 2. **`mem_cache/unified_radix_cache.UnifiedRadixCache.cache_unfinished_req`** — fix a fatal assert on hybrid-SWA decode radix. `match_prefix()` gates device indices on **all** components, so the SWA validator collapses `device_indices` to length 0 at the first out-of-window tombstone for a long prefix whose full-attention KV is entirely device-resident (`new_prefix_len=![\"Screenshot](\"https://github.com/user-attachments/assets/12336044-9447-4e27-a25c-dedc5b4c54cd\")
![\"image\"](\"https://github.com/user-attachments/assets/8f0a3cf1-aa93-423e-986e-dc8014f0b8de\")
### Isolated prefill step Measured with `sglang.benchmark.one_batch` (Qwen3.5-397B-A17B-FP8, TP4, B200, `--linear-attn-prefill-backend flashinfer --cuda-graph-backend-prefill=breakable`, batch size 1, 4 measured reps per point). Baseline is `main` @ `b129e8a29`. | prefill ISL | main | this PR | delta | 95% CI | absolute | |---|---|---|---|---|---| | 1k | 0.03454 s | 0.03389 s | **-1.89%** | [-2.42%, -1.27%] | 0.652 ms | | 2k | 0.05120 s | 0.05050 s | **-1.36%** | [-1.67%, -1.07%] | 0.695 ms | | 4k | 0.07980 s | 0.07919 s | **-0.76%** | [-0.83%, -0.68%] | 0.605 ms | | 8k | 0.14304 s | 0.14238 s | **-0.45%** | [-0.56%, -0.35%] | 0.650 ms | Every point's 95% CI (bootstrap on the mean) excludes zero. The column to read is the **absolute** one: the saving is flat at **~0.65 ms/forward across an 8x token range**. That is the expected shape for this change — hoisting layer-invariant metadata removes a fixed amount of per-forward host work (the per-layer `clamp`/dtype conversions for `ssm_cache_indices` and `cu_seqlens`, now done once instead of once per GDN layer), independent of how many tokens flow through it. The percentage varies only because the denominator does, so the effect is largest at short ISL and shrinks as prefill becomes more GPU-bound. Methodology: the first rep at each new input shape is discarded as capture-bucket/autotune warmup (e.g. 4k: 0.107 s on the first forward vs 0.0798 s steady-state). Including it inflates the standard deviation from ~0.05 ms to ~12 ms and leaves the 4k/8k points unresolvable. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31007005071](https://github.com/sgl-project/sglang/actions/runs/31007005071) Latest PR Test (Extra): :x: [Run #31007004651](https://github.com/sgl-project/sglang/actions/runs/31007004651) ",
"url": "https://github.com/sgl-project/sglang/pull/31172",
"createdAt": "2026-07-14T08:44:57Z",
"updatedAt": "2026-08-13T07:53:11Z",
"timestamp": "2026-08-13T07:53:11Z",
"metrics": {
"reactions": 1,
"comments": 3
},
"labels": [
"run-ci"
],
"author": "mattteochen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31193",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "(fix) Prevent unknown HTTP paths from overloading Prometheus metrics generation",
"text": "## Motivation This change prevents unbounded Prometheus cardinality growth caused by arbitrary unknown HTTP paths. One real-world example is vulnerability scanners, which can probe tens of thousands of unique random endpoints. When the raw request path is used as the endpoint metric label, every unique unknown path creates a new Prometheus time series for the HTTP request/response metrics. In one production incident, a vulnerability scanner generated approximately **17,700 unique unknown endpoints**, producing metric series such as: ```text sglang:http_requests_total{endpoint=\"/.../.../.../.../.../.../.../.../.../.../../../../../../../../../../etc/passwd\",method=\"GET\"} 1.0 sglang:http_requests_total{endpoint=\"/${jndi:ldap://localhost\",method=\"GET\"} 2.0 ``` This significantly inflated the Prometheus multiprocess metric files: ``` counter_1.db 8 MB gauge_livesum_1.db 4 MB histogram_1.db 64 KB ``` The `/metrics` endpoint rebuilds the Prometheus exposition on every scrape. In multiprocess mode, this means scanning and reading the *.db files, merging samples, serializing all metrics, and optionally compressing the response. This work runs in Python on the main ASGI thread and can execute under the GIL. While reproducing the issue, I continuously sent chat completion requests while sampling the SGLang process with [pyspy](https://github.com/benfred/py-spy). Across all samples, the main thread was consistently inside the Prometheus metrics generation path: for example ``` Thread 1 (active): \"MainThread\" compress (gzip.py:616) _bake_output (prometheus_client/exposition.py:116) prometheus_app (prometheus_client/asgi.py:24) handle (starlette/routing.py:448) ``` or ``` Thread 1 (active+gil): \"MainThread\" add_sample (prometheus_client/metrics_core.py:35) _read_metrics (prometheus_client/multiprocess.py:82) merge (prometheus_client/multiprocess.py:43) collect (prometheus_client/multiprocess.py:158) collect (prometheus_client/registry.py:97) generate_latest (prometheus_client/exposition.py:289) ``` Together with the enlarged multiprocess metric files, this explained why the frontend accepted HTTP connections but stopped making progress on completion requests: the main ASGI thread was repeatedly rebuilding and serializing a high-cardinality metrics payload during `/metrics` scrapes. With this change, even under vulnerability scans or other arbitrary requests, these files remain small: ``` counter_1.db 64 KB gauge_livesum_1.db 64 KB histogram_1.db 64 KB ``` Since the metrics endpoint reconstructs the Prometheus exposition by reading and merging these multiprocess files on every scrape, the increased number of time series significantly increased metrics generation cost. In our production incident, the server eventually spent a significant amount of CPU time generating /metrics, reducing its ability to serve completion requests. ## Modifications Unhandled FastAPI paths are now reported as `endpoint=\"__unhandled__\"` instead of using the raw request path. Known routes continue to use their normalized route paths, preserving the existing metrics for handled endpoints while preventing unbounded cardinality for unknown paths. ## Accuracy Tests ## Speed Tests and Profiling Not applicable. This PR only changes HTTP metrics path labeling for unhandled FastAPI routes and does not affect model outputs, kernels, or model forward code. --- ### CI States Latest PR Test (Base): :x: [Run #31628343531](https://github.com/sgl-project/sglang/actions/runs/31628343531) Latest PR Test (Extra): :x: [Run #31628343324](https://github.com/sgl-project/sglang/actions/runs/31628343324) ",
"url": "https://github.com/sgl-project/sglang/pull/31193",
"createdAt": "2026-07-14T13:20:10Z",
"updatedAt": "2026-08-12T18:35:25Z",
"timestamp": "2026-08-12T18:35:25Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [],
"author": "ntny",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31320",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] [Diffusion] support distributed inference pipeline for GLM-Image",
"text": "## Summary This PR adds GLM-Image distributed serving: the public head batches external AR generation and dispatches each AR-complete request to an independent batch-1 denoiser that runs request preparation, DiT, and VAE decoding. ![\"GLM-Image](\"https://github.com/user-attachments/assets/b079ec92-4bb8-4a1b-bfa8-02f7251cc2eb\")
## Modifications - Add GLM-Image distributed mode for a head configured with `--srt-encoder-url` and `--denoiser-urls`, with encoder and decoder URL lists omitted. - Group compatible text-to-image requests by resolution and send one batched `/generate` call to the external AR server. Batch capacity is measured in output slots, consistent with #30683. - Run the blocking external AR request on a dedicated executor so AR generation overlaps with active denoiser work. - Assign generated prior tokens and usage to each request, then dispatch AR-complete requests across connected denoisers with one active request per worker. - Run glyph/text preparation, sequential batch-1 DiT inference, and VAE decoding on each distributed denoiser. `num_outputs_per_prompt > 1` is processed sequentially and regrouped into the original response. - Transfer only request metadata and CPU prior-token IDs over ZMQ; this mode bypasses Mooncake/RDMA and does not transfer latent or embedding tensors. - Return each completed request immediately to the head, which preserves client ordering, seeds, output paths, output saving, and image URL registration. - Track denoiser connectivity, stop dispatching to disconnected workers, and reclaim request state and worker capacity after timeouts. - Preserve the existing encoder → denoiser → decoder disaggregation path when GLM distributed mode is not enabled. - Validate per-role TP/SP/CFG/DP device requirements and disable only auto-enabled CFG parallelism when incompatible with the worker device count. - Document the topology, deployment commands, limitations, and measured NVIDIA and Ascend results. ## Deployment ### Run 14 independent Cache-DiT denoisers ~~~bash #!/bin/bash DISAGG_SERVER=\"tcp://127.0.0.1:19655\" MODEL_PATH=\"zai-org/GLM-Image\" BASE_MASTER_PORT=29005 export SGLANG_CACHE_DIT_FN=2 export SGLANG_CACHE_DIT_BN=1 export SGLANG_CACHE_DIT_WARMUP=4 export SGLANG_CACHE_DIT_RDT=0.4 export SGLANG_CACHE_DIT_MC=4 export SGLANG_CACHE_DIT_TAYLORSEER=true export SGLANG_CACHE_DIT_TS_ORDER=2 export SGLANG_CACHE_DIT_ENABLED=true for i in $(seq 2 15); do scheduler_port=$((19001 + i)) master_port=$((BASE_MASTER_PORT + i)) sglang serve \\ --model-path \"$MODEL_PATH\" \\ --disagg-role denoiser \\ --disagg-server-addr \"$DISAGG_SERVER\" \\ --srt-encoder-url http://127.0.0.1:30020 \\ --scheduler-port \"$scheduler_port\" \\ --master-port \"$master_port\" \\ --num-gpus 1 \\ --base-gpu-id \"$i\" \\ --denoiser-sp 1 \\ --cfg-parallel-size 1 \\ --batching-max-size 1 \\ --attention-backend fa & done wait ~~~ ### Run the AR server ~~~bash sglang serve \\ --model-path zai-org/GLM-Image/vision_language_encoder/ \\ --tokenizer-path zai-org/GLM-Image/processor/ \\ --enable-multimodal \\ --cuda-graph-max-bs 28 \\ --device npu \\ --attention-backend ascend \\ --disable-fast-image-processor \\ --tp-size 2 \\ --host 0.0.0.0 \\ --port 30020 \\ --mem-fraction-static 0.8 ~~~ ### Run the public head ~~~bash sglang serve \\ --model-path zai-org/GLM-Image \\ --disagg-role server \\ --srt-encoder-url http://127.0.0.1:30020 \\ --srt-encoder-timeout 300 \\ --denoiser-urls \"tcp://127.0.0.1:19003;tcp://127.0.0.1:19004;tcp://127.0.0.1:19005;tcp://127.0.0.1:19006;tcp://127.0.0.1:19007;tcp://127.0.0.1:19008;tcp://127.0.0.1:19009;tcp://127.0.0.1:19010;tcp://127.0.0.1:19011;tcp://127.0.0.1:19012;tcp://127.0.0.1:19013;tcp://127.0.0.1:19014;tcp://127.0.0.1:19015;tcp://127.0.0.1:19016\" \\ --batching-mode dynamic \\ --batching-max-size 28 \\ --batching-delay-ms 30 \\ --enable-batching-metrics \\ --host 0.0.0.0 \\ --port 30052 \\ --scheduler-port 19655 \\ --output-path ./outputs ~~~ ### Run the benchmark Tested on [longtext-bench.zip](https://github.com/user-attachments/files/29779516/longtext-bench.zip). ~~~bash python fetch_images.py \\ --base-url http://localhost:30052/v1 \\ --model GLM-Image-distributed-test \\ --output-dir generated_images \\ --max-concurrency 56 ~~~ ## Limitations - GLM distributed mode is text-to-image only; non-string prompts and image-conditioned requests are rejected. - In-flight jobs are not replayed if a denoiser disconnects. - Distributed-denoiser `RequestMetrics` are not propagated back to the head; median request E2E latency and peak memory below are reconstructed from server logs. ## Results All runs use 1280×1280 output, 30 denoising steps, and Cache-DiT. Configurations are ordered from lowest to highest throughput. The tables retain the sustained-throughput and batch-relative latency values from the original performance summary. Median request E2E latency and peak memory are reconstructed from server logs after excluding the initial BS2 warm-up where applicable. Median request E2E is measured from request arrival. Batch-relative latency is measured from batch/AR dispatch and excludes pre-dispatch waiting; for non-batched configurations, it is approximately the request latency. ### 8-GPU NVIDIA comparison | Metric | No Batch + No External AR *(baseline)* | No Batch + External AR | Monolithic Batch (8 GPUs) + External AR | Disaggregated (7 denoisers) + External AR | |---|---:|---:|---:|---:| | **Sustained throughput (img/s)** | **0.018** | **0.09** | **0.24** | **0.37** | | **Throughput improvement (baseline = 1×)** | **1.0×** | **5.0×** | **13.3×** | **20.6×** | | Median request E2E latency (s) | 55.22 | 10.83 | 134.30 | 72.07 | | Batch-relative latency (s) | ~56 | ~11 | ~80 | ~55 | | Latency improvement | 1.0× | **5.1×** | *0.7×* | ~1.0× | | Peak GPU memory (MB) | 46,698 | 27,296 | 27,982 | 28,046 per denoiser | - Using these sustained-throughput values, seven-denoiser disaggregation is 1.54× faster than monolithic batching and 20.6× faster than the local-AR baseline. ### 16-NPU Ascend comparison | Metric | No Batch + Local AR *(baseline)* | No Batch + External AR | Monolithic Batch (28) + External AR | Disaggregated (7 denoisers) + External AR | Disaggregated (14 denoisers) + External AR | |---|---:|---:|---:|---:|---:| | **Sustained throughput (img/s)** | **0.0087** | **0.0435** | **0.20** | **0.41** | **0.74** | | **Throughput improvement (baseline = 1×)** | **1.00×** | **5.0×** | **23.0×** | **47.1×** | **85.1×** | | Median request E2E latency (s) | 122.18 | 22.96 | 141.82 | 69.57 | 73.56 | | Batch-relative latency (s) | ~115 | ~23 | ~90 | ~52 | ~37 | | Latency improvement | 1.00× | **5.0×** | 1.28× | 2.2× | **3.1×** | | Peak NPU memory (MB) | 48,228 | 28,302 | 28,332 | 28,316 per denoiser | 28,566 per denoiser | - Using these sustained-throughput values, 14-denoiser disaggregation is 3.70× faster than monolithic batching, 1.80× faster than seven-denoiser disaggregation, and 85.1× faster than the local-AR baseline. - The 14-denoiser request E2E latency range is 57.25–129.25 s; median request E2E latency is 73.56 s. ### Methodology - The initial BS2 client warm-up is excluded from every affected median request E2E calculation. - Sustained throughput and batch-relative latency values are retained from the original performance summary. - For batched configurations, batch-relative latency is measured from batch/AR dispatch; median request E2E is measured from request arrival and includes pre-dispatch waiting. - Incomplete trailing BS28 groups are excluded from server-log latency calculations. - Peak memory is the runtime-reported diffusion process or denoiser value; it is not aggregate cluster memory. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://github.com/sgl-project/sglang/blob/main/CONTRIBUTING.md#code-style). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from CODEOWNERS and other reviewers. 3. Trigger CI tests with comments or contact authorized users to do so. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31715006037](https://github.com/sgl-project/sglang/actions/runs/31715006037) Latest PR Test (Extra): :x: [Run #31715005390](https://github.com/sgl-project/sglang/actions/runs/31715005390) ",
"url": "https://github.com/sgl-project/sglang/pull/31320",
"createdAt": "2026-07-15T10:41:56Z",
"updatedAt": "2026-08-13T15:22:54Z",
"timestamp": "2026-08-13T15:22:54Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"documentation",
"npu",
"diffusion"
],
"author": "OrangeRedeng",
"state": "open",
"assignees": [
"ping1jing2"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31324",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [GLM5] Skip DSA decode indexer when kv_len <= index_topk (dense k-only fast path)",
"text": "## Summary - On GLM-5.2 DSA decode, when a request's `kv_len <= index_topk` the top-k selects **all** valid positions, so the indexer's logits GEMM + `paged_mqa_logits` + top-k selection is wasted work. Add a **k-only** fast path that skips the indexer, stores the K cache, and generates the identity index directly (`[0, 1, ..., kv_len-1, -1, ...]`), feeding the same sparse-MLA decode attention kernel. - **CUDA-graph \"Design A\" dual-graph:** capture a dense (k-only) and a sparse (full indexer) decode graph per batch-size bucket and dispatch on `max_kv_len` vs `index_topk` at replay. This is correct for mixed lengths — any request with `kv_len > index_topk` in the batch falls back to the sparse graph. - **Auto-enabled for DSA models, no env toggle.** Since DSA models expose `index_topk` in their HF config, the dual-graph is turned on automatically whenever `index_topk` is present (`dsa_dual_graph = dsa_index_topk is not None`); non-DSA models and other archs are unaffected. Eager decode takes the same fast path per-step when safe. (No `SGLANG_DSA_DECODE_*` / `SGLANG_KONLY_DEBUG` environment variables — an earlier revision gated this behind opt-in/debug flags; those were removed per review.) ## Changes | File | Change | | --- | --- | | layers/attention/dsa/dsa_indexer.py | k-only fast path (`_forward_cuda_k_only`): skip logits/top-k, store K cache; for MLA generate the identity index; extended to DECODE (eager + graph) | | model_executor/runner/decode_cuda_graph_runner.py | Design A dual-graph: capture dense (k-only) + sparse decode graphs per bs bucket; host-dispatch on `max_kv_len` vs `index_topk`; auto-enabled for DSA models via `dsa_index_topk` | | model_executor/runner_utils/capture_mode.py | capture-time DSA variant flag (dense/sparse) read by the indexer skip branch | | model_executor/runner/shape_key.py | include the DSA decode variant in the cuda-graph shape key | ## Scope GLM-5.2 DSA decode on gfx950 (MI355X). Auto-enabled for DSA models; correct for mixed lengths (long context stays on the sparse indexer path). Other archs / hardware unaffected. ## Test plan **Accuracy (GSM8K, MI355X TP4, GLM-5.2-MXFP4):** | | Baseline | This PR | | --- | --- | --- | | GSM8K | 0.922 | 0.941 | Within margin of error. **Performance (MI355X TP4, GLM-5.2-MXFP4, docker `rocm/sgl-dev:v0.5.15.post1-rocm720-mi35x-20260714`, tilelang DSA backend):** token throughput per GPU (tok/s/gpu, higher better) and median TPOT (ms, lower better), dense-decode OFF vs ON (this PR). Isolated effect — same build/backend, only this PR's commits added, everything else identical. Only **i1024 / o1024** is shown: this optimization applies when `kv_len <= index_topk` (2048). At i8192 every decode step has `kv_len > 2048`, so it stays on the sparse path and is unchanged. | Concurrency | TPUT off | TPUT PR | Δ TPUT | TPOT off | TPOT PR | Δ TPOT | | --- | --- | --- | --- | --- | --- | --- | | 4 | 71.8 | 75.5 | +5.2% | 13.28 | 12.62 | −5.0% | | 8 | 126.8 | 133.2 | +5.1% | 15.18 | 14.41 | −5.1% | | 16 | 196.9 | 205.1 | +4.2% | 19.46 | 18.67 | −4.1% | | 32 | 312.4 | 322.8 | +3.3% | 24.66 | 23.90 | −3.1% | | 64 | 468.9 | 480.3 | +2.4% | 32.88 | 32.14 | −2.3% | ``` python3 -m sglang.launch_server \\ --model amd/GLM-5.2-MXFP4 \\ --tp 4 \\ --trust-remote-code \\ --tool-call-parser glm47 \\ --reasoning-parser glm45 \\ --watchdog-timeout 1200 \\ --mem-fraction-static 0.85 \\ --kv-cache-dtype fp8_e4m3 \\ --disable-radix-cache \\ --model-loader-extra-config '{\"enable_multithread_load\": true, \"num_threads\": 32}' \\ --dsa-prefill-backend tilelang \\ --dsa-decode-backend tilelang \\ --tokenizer-worker-num 8 \\ --enable-aiter-allreduce-fusion ``` Token throughput per GPU improves ~2.4–5.2% and TPOT drops ~2.3–5.1% at i1k, from skipping the wasted indexer when the sparse top-k would select all positions anyway. The gain is largest at low concurrency and converges to ~2.4% at conc 64. (Δ is the isolated dense-decode effect — same build/backend, feature off vs on — not a full-stack-vs-stock comparison.) --- ### CI States Latest PR Test (Base): :no_entry_sign: [Run #31655055850](https://github.com/sgl-project/sglang/actions/runs/31655055850) Latest PR Test (Extra): :x: [Run #31655055641](https://github.com/sgl-project/sglang/actions/runs/31655055641) ",
"url": "https://github.com/sgl-project/sglang/pull/31324",
"createdAt": "2026-07-15T11:13:56Z",
"updatedAt": "2026-08-13T05:20:31Z",
"timestamp": "2026-08-13T05:20:31Z",
"metrics": {
"reactions": 1,
"comments": 12
},
"labels": [
"amd",
"run-ci"
],
"author": "Jacob0226",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31362",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Speculative Decoding with NGRAM support for XPU",
"text": "This PR adds support for NGRAM on XPU target. JH...! --- ### CI States Latest PR Test (Base): :x: [Run #31673353486](https://github.com/sgl-project/sglang/actions/runs/31673353486) Latest PR Test (Extra): :x: [Run #31673353388](https://github.com/sgl-project/sglang/actions/runs/31673353388) ",
"url": "https://github.com/sgl-project/sglang/pull/31362",
"createdAt": "2026-07-15T18:28:59Z",
"updatedAt": "2026-08-13T06:23:10Z",
"timestamp": "2026-08-13T06:23:10Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"speculative-decoding",
"jit-kernel"
],
"author": "ANSHUMAN87",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31443",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache]: Optimize hybrid/DSA L3 prefetch result sync and usable-prefix clamping",
"text": " ## Motivation Fix: https://github.com/sgl-project/sglang/issues/30321 Thanks @cfbdsirlijun-maker 、[AlejandroParedesLT](https://github.com/AlejandroParedesLT) for reporting the issue. Thanks @stepinto for reproducing the issue: https://github.com/sgl-project/sglang/pull/31348 ## Modifications - Extracted a helper for the HiCache prefetch completion logic: _sync_and_check_hybrid_prefetch_result (Unified) / _sync_and_clamp_prefetch_result (HiRadix), pulling the cross-ATTN-group sync + usable-prefix decision out of check_prefetch_progress. - Added DSA-style clamp in Unified: for Full + KV-derived ALL_PAGES sidecars (DSA/MiniMax indexer), clamp the usable prefix to min(Full KV completion, each sidecar hit) instead of discarding; SWA/Mamba/DeepSeekV4 keep all-or-nothing. - Fixed PoolTransferResult.update_extra_pool_hit_pages: count the leading contiguous run of successes (rs.index(False)) instead of summing, so a gap in the middle of a fetch can't be over-counted as a longer safe prefix. - ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #29657971984](https://github.com/sgl-project/sglang/actions/runs/29657971984) Latest PR Test (Extra): :white_check_mark: [Run #29658092089](https://github.com/sgl-project/sglang/actions/runs/29658092089) ",
"url": "https://github.com/sgl-project/sglang/pull/31443",
"createdAt": "2026-07-16T08:45:58Z",
"updatedAt": "2026-08-13T11:55:58Z",
"timestamp": "2026-08-13T11:55:58Z",
"metrics": {
"reactions": 1,
"comments": 20
},
"labels": [
"high priority",
"hicache",
"run-ci",
"run-ci-extra"
],
"author": "hzh0425",
"state": "closed",
"assignees": [
"xiezhq-hermann",
"ispobock"
]
},
{
"id": "github:sgl-project/sglang:pull_request:31447",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "perf: fix overlap scheduling and all-reduce fusion for NVIDIA Confidential Computing(CC) on Blackwell",
"text": "## Motivation Two fixes so SGLang runs efficiently under **NVIDIA Confidential Computing (CC)**, where the GPU operates behind a bounce buffer: 1. **Per-step D2H result readback stalls the scheduler.** Under CC, `cudaMemcpyAsync` (D2H) is forced synchronous — it blocks at issue — so the per-step readback of `next_token_ids` / logprobs serializes onto the scheduler's critical path and kills decode overlap. 2. **FlashInfer AR+RMSNorm fusion disables itself.** `create_allreduce_fusion_workspace` allocates a symmetric-memory (cuMulticast) workspace whose preflight fails under CC, so the fusion silently falls back off — even though the fusion kernels themselves are multicast-free. Both paths are gated on `is_confidential_compute()` (NVML, overridable via `SGLANG_CONFIDENTIAL_COMPUTE`); **off-CC behavior is unchanged.** ## Modifications **`cc: async device->host result-readback worker`** - New `python/sglang/srt/managers/async_d2h_copy_worker.py` (`AsyncD2HCopyWorker`): runs the per-step D2H copy on a dedicated thread with its own CUDA stream, off the scheduler's critical path, so overlap is preserved. Wired into `scheduler.py` / `managers/utils.py`; only routed through the worker when CC is detected. - Adds `is_confidential_compute()` to `srt/utils/common.py`. **`cc(fix): enable FlashInfer AR+RMSNorm fusion under Confidential Computing`** (`layers/flashinfer_comm_fusion.py`) - FlashInfer now auto-selects a multicast-free IPC workspace under CC, so SGLang reuses the normal off-CC fusion path: `_resolve_backend` forces the `trtllm` backend under CC (SM100 \"auto\" defaults to `mnnvl`, which needs NVLink multicast), the symmetric-memory preflight is skipped, and the workspace stays a normal `AllReduceFusionWorkspace`. Enabled by default whenever CC is detected. ## Accuracy Tests No model-output change: the AR+RMSNorm fusion produces the same result as the off-CC path (same kernels, only the workspace allocator differs), and the D2H worker only relocates the copy, not its contents. Off-CC paths are untouched. ## Speed Tests and Profiling Intent is to *restore* CC performance to the off-CC baseline — preserve decode overlap and keep AR+RMSNorm fusion enabled under CC. CC-hardware e2e benchmarks pending. `test/registered/core/test_async_d2h_copy_worker.py` validates the worker on a single GPU: correctness off-thread (tiny + vocab-sized tensors), many sequential readbacks, error re-raise, idempotent shutdown, and the CC-detection env override. ## Checklist - [x] Format code with pre-commit (all hooks pass). - [x] Added unit tests (`test/registered/core/test_async_d2h_copy_worker.py`). - [ ] Documentation update — n/a. - [ ] Accuracy/speed benchmark — n/a (no output change; CC-hardware benchmark pending). - [x] Follow the SGLang code style guidance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31664799631](https://github.com/sgl-project/sglang/actions/runs/31664799631) Latest PR Test (Extra): :x: [Run #31664799513](https://github.com/sgl-project/sglang/actions/runs/31664799513) ",
"url": "https://github.com/sgl-project/sglang/pull/31447",
"createdAt": "2026-07-16T10:25:54Z",
"updatedAt": "2026-08-13T05:19:44Z",
"timestamp": "2026-08-13T05:19:44Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"run-ci"
],
"author": "elvischenv",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31466",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] DSpark support prefill/decode disaggregation",
"text": " ## Motivation Roadmap: https://github.com/sgl-project/sglang/issues/30344 Reference PR: #30513 This PR adds DSpark support for prefill/decode disaggregation on DeepSeek-V4. On `main`, DSpark can run in the normal non-disaggregated path, but the PD decode path does not receive the target-model hidden states required to bootstrap the draft-side DSpark state. As a result, decode may enter speculative decoding without valid DSpark `spec_info` / draft KV state. This PR fixes that gap by transferring DSpark target hidden states from Prefill to Decode through the disaggregation protocol, then injecting them into the decode-side draft KV cache before the first draft step. The implementation keeps the existing non-disaggregated DSpark behavior unchanged and scopes the new behavior to PD mode. ## What Changed Compared With main ### DSpark Hidden State PD Transfer - Add a new disaggregation state type: `DSPARK_HIDDEN`. - Let Decode describe the required DSpark hidden transfer in metadata: - target layer ids - hidden start offset - hidden length - decode radix-cache policy - PP-aware slice metadata - dynamic destination buffer information - Let Prefill capture target hidden states only on the PP rank that owns the DSpark target layers. - Transfer DSpark hidden states through Mooncake/NIXL using row-addressed dynamic destination buffers. - Trim hidden transfer windows according to the Prefill-side cached prefix while preserving absolute token offsets. - Keep fail-fast checks when required hidden rows are incomplete before transfer. ### Decode-Side Draft Bootstrap - Assemble received DSpark hidden slices on Decode before committing the transferred request. - Attach the assembled hidden rows to `prefill_tail_hidden_states`. - Build DSpark draft input from the Prefill output token as the first decode anchor. - Inject transferred target hidden states into the decode-side draft KV cache before the first draft step. ### PP-Aware Prefill Support - Add PP-aware hidden slice metadata so only the PP rank owning the target layers sends DSpark hidden states. - Propagate DSpark auxiliary hidden tensors through PP proxy outputs. - Match PP outputs by microbatch id / request identity to avoid FIFO mismatch. - Snapshot PP batch sequence lengths so delayed PP result processing does not read mutated request state. - Drain failed PP prefill bootstrap requests consistently across PP ranks. - Preserve PP admission behavior using side-effect-free resource credit probing before consensus. ### Hidden Buffer and Transfer Resource Management - Add `DSparkHiddenTransferPlan` to describe row/chunk transfer layouts. - Add decode-side `DSparkHiddenPagePool` to reuse registered GPU receive buffers. - Add optional decode hidden receive prewarm: - `SGLANG_DSPARK_PD_HIDDEN_RECV_PREWARM_ROWS` - `SGLANG_DSPARK_PD_HIDDEN_RECV_PREWARM_PAGES` - Release Prefill hidden rows as soon as the hidden transfer finishes in the Mooncake worker, instead of waiting for the full KV request success. - Prefer contiguous hidden row allocation and slice copy on Prefill. - Use `torch.empty` instead of `torch.zeros` when assembling Decode hidden tensors to avoid unnecessary clearing. ### Metadata and Speculative Decode Integration - Extend disaggregation metadata buffers to carry DSpark prefill-tail hidden state information. - Extend `DFlashDraftInputV2` / DSpark draft input to carry: - `prefill_tail_hidden_states` - `prefill_tail_valid_mask` - `prefill_tail_start_positions` - `prefill_tail_hidden_projected` - Add non-padded token count metadata needed by DSpark draft forward batches. - Ensure Decode radix-cache and Prefill radix-cache policy are aligned for DSpark hidden correctness. ### Streaming Hidden Chunk Transfer and Release Semantics - Add streaming DSpark hidden transfer for Mooncake so long prompts do not require materializing the full `prompt_len * hidden_width` tensor on Prefill or Decode. - Split DSpark hidden transfer state from KV request completion: - hidden chunk ACK only controls streaming window reuse. - hidden request done does not imply KV request success. - request success still follows the original KV / metadata success path. - Release Prefill-side hidden source rows after the hidden request is done, before full KV request completion. - Release Decode-side hidden receive rows on normal request release / abort paths. - Serialize Mooncake DSpark hidden chunks per room to preserve chunk ordering under ACK-based flow control. - Send DSpark hidden-only chunks even when the aligned KV page count is zero. - Flush pending streaming hidden chunks before writing the next chunk, preventing a later chunk from overwriting an unsent `dspark_hidden_current_*` slot. - Use offset-based streaming source row writes so multi-chunk hidden transfer preserves the absolute hidden token range. - Add fail-fast checks for source-window overwrite hazards and invalid streaming hidden ordering. - Add backend capability boundaries: - Mooncake supports streaming DSpark hidden transfer. - NIXL / MORI keep the non-streaming/default path and do not enter partially implemented streaming release semantics. ## Why These Changes Are Needed DSpark speculative decoding depends on target-model hidden states to initialize the draft-side state correctly. In PD mode, Prefill and Decode run in separate processes, so Decode cannot derive those hidden states locally. The new `DSPARK_HIDDEN` transfer path makes the hidden state an explicit part of the PD protocol, similar to KV/state transfer, but with PP-aware slicing because only specific Prefill PP ranks own the DSpark target layers. The resource probing and transactional allocation logic are needed to avoid PP rank divergence: all PP ranks must agree on which requests enter the pipeline before any rank performs side-effectful resource allocation. The dynamic registered GPU receive buffer pool avoids CPU bounce and keeps hidden transfer on the GDR path. ## Compatibility - Non-disaggregated DSpark behavior is unchanged. - Existing KV / C128 / auxiliary state transfer paths remain intact. - DSpark PD hidden transfer is activated only when Decode requests `DSPARK_HIDDEN` metadata. - Default decode hidden receive prewarm is disabled, so default memory footprint is unchanged. Current Scope and Future Work This PR supports DSpark PD hidden-state transfer for Prefill with PP. The implementation is PP-aware: each Prefill PP rank only captures and transfers the DSpark target hidden slice owned by its local layer range. Mooncake now supports streaming DSpark hidden transfer with hidden/KV/request completion semantics separated. NIXL and MORI do not yet implement streaming hidden release semantics and therefore keep the default non-streaming capability boundary. Prefill radix cache cannot be enabled alone for DSpark PD today. DSpark hidden transfer must match the KV transfer window. If Prefill radix cache is enabled but Decode radix cache is disabled, Prefill may skip cached prompt tokens and the corresponding target hidden states will be missing on Decode. Therefore, DSpark PD currently requires Prefill and Decode radix-cache policies to be consistent. Until DeepSeek-V4 Decode radix cache is supported, Prefill radix cache should stay disabled. #31097 Future work will extend this to Prefill with PP + TP + CP and streaming support for NIXL/MORI. That requires generalizing the hidden metadata from PP-only layer slicing to a full PP/TP/CP layout, including tensor-parallel hidden sharding and context-parallel token-range reconstruction on Decode. ## Accuracy Tests ``` Prefill SGLANG_PP_LAYER_PARTITION=\"6,5,5,6,5,5,6,5\" SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 NCCL_SOCKET_IFNAME=eth0 NCCL_IB_DISABLE=0 SGLANG_DSV4_FP4_EXPERTS=1 GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path /data00/models/DeepSeek-V4-Flash-DSpark --host 0.0.0.0 --port 30000 --trust-remote-code --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.8 --max-running-requests 64 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --pp-size 8 --attention-backend dsv4 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --disable-overlap-schedule --disable-piecewise-cuda-graph --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --enable-metrics --disaggregation-ib-device mlx5_1,mlx5_2,mlx5_3,mlx5_4 --moe-runner-backend flashinfer_mxfp4 --disable-radix-cache Decode SGLANG_DSPARK_PD_HIDDEN_RECV_POOL_TOKENS=65536 SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_JIT_DEEPGEMM_PRECOMPILE=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=1 SGLANG_OPT_USE_TILELANG_MHC_PRE=1 GLOO_SOCKET_IFNAME=eth0 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8 sglang serve --trust-remote-code --model-path /data00/models/DeepSeek-V4-Flash-DSpark --tp 8 --dp-size 8 --enable-dp-attention --cuda-graph-max-bs 32 --max-running-requests 256 --enable-metrics --host 0.0.0.0 --port 30000 --mem-fraction-static 0.85 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --moe-runner-backend flashinfer_mxfp4 --disaggregation-mode decode --disaggregation-ib-device \"mlx5_1,mlx5_2,mlx5_3,mlx5_4\" --speculative-algo DSPARK --tokenizer-worker-num 8 --enable-dp-lm-head --load-balance-method round_robin --swa-full-tokens-ratio 0.8 ``` Hardward | MMLU | GSM8K | QPQA | aime25 repeats 16 | status -- | -- | -- | -- | -- | -- H20 | 0.885 | 0.954 | 0.910 | 93.96 | PD ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #29816426168](https://github.com/sgl-project/sglang/actions/runs/29816426168) Latest PR Test (Extra): :x: [Run #29816426138](https://github.com/sgl-project/sglang/actions/runs/29816426138) ",
"url": "https://github.com/sgl-project/sglang/pull/31466",
"createdAt": "2026-07-16T13:27:50Z",
"updatedAt": "2026-08-13T11:09:17Z",
"timestamp": "2026-08-13T11:09:17Z",
"metrics": {
"reactions": 2,
"comments": 8
},
"labels": [
"deepseek"
],
"author": "zhangxiaolei123456",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:31470",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NVIDIA] Support flashinfer Mega Moe",
"text": "Fork from https://github.com/djns99/sglang/tree/djns99/mega_moe_flashinfer @djns99 is the main author of this PR. ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. # MoE backend benchmark: `flashinfer_trtllm_routed` (baseline) vs `flashinfer_megamoe` Benchmark of the DeepSeek-V4-Flash NVFP4 model comparing two MoE backend configurations on the serving path. `flashinfer_trtllm_routed` (trtllm routed MoE runner + flashinfer A2A) is used as the **baseline**; `flashinfer_megamoe` is the compared configuration. All deltas are reported relative to the trtllm_routed baseline. ## Environment - Model: `nvidia/DeepSeek-V4-Flash-NVFP4` - Parallelism: TP=4, DP=4, DP-attention enabled (`attn_dp_size=4`, so per-rank decode batch = `max_running_requests / 4`) - 1x node, same hardware for both runs ## Server commands **Baseline — `flashinfer_trtllm_routed`:** ```bash python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 4096 \\ --disable-flashinfer-autotune \\ --swa-full-tokens-ratio 0.1 \\ --host 0.0.0.0 --port 30000 \\ --moe-runner-backend flashinfer_trtllm_routed \\ --moe-a2a-backend flashinfer \\ --max-running-requests 1024 ``` **Compared — `flashinfer_megamoe`:** ```bash python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --chunked-prefill-size 4096 \\ --disable-flashinfer-autotune \\ --swa-full-tokens-ratio 0.1 \\ --host 0.0.0.0 --port 30000 \\ --moe-runner-backend flashinfer_megamoe \\ --moe-a2a-backend flashinfer_megamoe \\ --max-running-requests 1024 ``` ## Client command ```bash python -m sglang.bench_serving \\ --backend sglang \\ --model nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dataset-name random \\ --random-input-len 8 --random-output-len 1024 --random-range-ratio 1.0 \\ --num-prompts $((5 * C)) \\ --max-concurrency C # C in {32, 128, 1024}, num-prompts = 5 x C ``` ## Results ### Output token throughput (tok/s, higher is better) | max_conc | trtllm_routed (baseline) | megamoe | Δ (megamoe vs baseline) | |---------:|-------------------------:|--------:|------------------------:| | 32 | 876.88 | 674.21 | −23.1% | | 128 | 1215.93 | 1092.54 | −10.1% | | 1024 | 3223.92 | 3245.20 | +0.7% | ### Mean TPOT (ms, lower is better) | max_conc | trtllm_routed (baseline) | megamoe | Δ (megamoe vs baseline) | |---------:|-------------------------:|--------:|------------------------:| | 32 | 34.49 | 38.77 | +12.4% | | 128 | 109.93 | 106.83 | −2.8% | | 1024 | 422.61 | 394.92 | −6.6% | ### Full metrics | config | max_conc | out_tput (tok/s) | mean TPOT (ms) | median TPOT (ms) | p99 TPOT (ms) | achieved conc | mean TTFT (ms) | |--------|---------:|-----------------:|---------------:|-----------------:|--------------:|--------------:|---------------:| | trtllm_routed | 32 | 876.88 | 34.49 | 33.32 | 102.42 | 28.76 | 421 | | megamoe | 32 | 674.21 | 38.77 | 40.27 | 78.13 | 26.53 | 321 | | trtllm_routed | 128 | 1215.93 | 109.93 | 104.71 | 395.33 | 116.12 | 797 | | megamoe | 128 | 1092.54 | 106.83 | 101.22 | 345.54 | 103.68 | 776 | | trtllm_routed | 1024 | 3223.92 | 422.61 | 343.54 | 2429.78 | 994.81 | 3142 | | megamoe | 1024 | 3245.20 | 394.92 | 322.13 | 2356.07 | 951.20 | 3261 | ## Takeaways - **Low/mid concurrency (c32, c128): `flashinfer_trtllm_routed` wins on throughput** — +23% at c32 and +11% at c128 output token throughput, with comparable or better median TPOT. It also sustains higher achieved concurrency (28.8/116.1 vs 26.5/103.7), i.e. it drains the request queue faster. - **High concurrency (c1024): the two are effectively tied on throughput** (3224 vs 3245 tok/s, +0.7% for megamoe — within run-to-run noise). At this point both are decode-bound / saturated (achieved conc < target, p99 TPOT ~7x median), so the MoE backend is no longer the bottleneck. - **TPOT crossover:** megamoe has slightly lower mean TPOT at c1024 (−6.6%) but a worse p99 tail at low load (c32: 102 ms vs 78 ms). trtllm_routed gives more consistent per-token latency at low concurrency. - **Net:** `flashinfer_trtllm_routed` is the better default for latency-sensitive / low-to-mid concurrency serving; at saturation the choice is throughput-neutral. ## FlashInfer MegaMOE env knobs This PR exposes `SGLANG_FLASHINFER_MEGAMOE_COMBINE_DTYPE` for the FlashInfer NVFP4 MegaMOE path. It selects the cross-rank combine wire format passed to `Nvfp4CutedslMegaMoeConfig.combine_dtype`. Supported values: - `bf16`: default, exact combine path. - `mxfp8`: quantized combine traffic with a small accuracy tradeoff. - `nvfp4`: lower combine traffic with a small accuracy tradeoff. Example: ```bash SGLANG_FLASHINFER_MEGAMOE_COMBINE_DTYPE=nvfp4 \\ python -m sglang.launch_server \\ --trust-remote-code \\ --model-path nvidia/DeepSeek-V4-Flash-NVFP4 \\ --dp-size 4 --tp-size 4 --enable-dp-attention \\ --moe-runner-backend flashinfer_megamoe \\ --moe-a2a-backend flashinfer_megamoe ``` `mxfp8` and `nvfp4` are incompatible with `SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE=1`; server args validation now rejects that combination early. > Note: the `flashinfer_megamoe` in-kernel FC2-reduce variant (`SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE=1`) was excluded from this comparison due to a known issue and is not reflected here. Update 7/27/2026: ## MoE backend comparison — DeepSeek-V4-Flash-NVFP4, TP4/DP4/EP4 Baseline: `trtllm`. Positive throughput and negative latency are both wins. Tokens/rank = per-DP-rank prefill chunk (DP attention routes each request wholly to one rank; `--chunked-prefill-size` is divided by `dp_size` internally). ## Prefill ### 2048 tokens/rank (input len 2048, concurrency 128) | Backend | Input tput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | |---|---|---|---|---| | trtllm (baseline) | 38,884 | 3,084 | 3,233 | 4,104 | | **deepgemm megamoe** | **46,117** | **2,572** | **2,719** | **3,372** | | flashinfer_cutedsl_megamoe_plain | 42,324 | 2,952 | 3,018 | 3,849 | | flashinfer_cutedsl_megamoe_ikr | 41,291 | 3,036 | 3,122 | 3,955 | | flashinfer_cutedsl_megamoe_combine_nvfp4 | 42,408 | 2,953 | 3,044 | 3,838 | Change vs `trtllm`: | Backend | Input tput | Mean TTFT | Median TTFT | P99 TTFT | |---|---|---|---|---| | **deepgemm megamoe** | **+18.60%** | **-16.59%** | **-15.88%** | **-17.84%** | | flashinfer_cutedsl_megamoe_plain | +8.85% | -4.28% | -6.65% | -6.21% | | flashinfer_cutedsl_megamoe_ikr | +6.19% | -1.57% | -3.43% | -3.63% | | flashinfer_cutedsl_megamoe_combine_nvfp4 | +9.06% | -4.23% | -5.84% | -6.48% | > Note: trtllm/deepgemm rows are 640 requests (676,871 input tokens); the flashinfer rows are > 1,280 requests (1,327,019 input tokens). Same input len and concurrency, so the rates are > comparable, but the request counts differ. ### 8192 tokens/rank (input len 8192, 640 requests, 2,665,479 input tokens, concurrency 128) | Backend | Input tput (tok/s) | Mean TTFT (ms) | Median TTFT (ms) | P99 TTFT (ms) | |---|---|---|---|---| | trtllm (baseline) | 151,213 | 3,137 | 3,339 | 4,361 | | deepgemm megamoe | 154,359 | 3,047 | 3,286 | 4,163 | | flashinfer_cutedsl_megamoe_plain | 162,342 | 2,945 | 3,159 | 4,146 | | flashinfer_cutedsl_megamoe_ikr | 165,658 | 2,831 | 3,018 | 3,835 | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **169,175** | **2,745** | **2,905** | **3,753** | Change vs `trtllm`: | Backend | Input tput | Mean TTFT | Median TTFT | P99 TTFT | |---|---|---|---|---| | deepgemm megamoe | +2.08% | -2.84% | -1.58% | -4.52% | | flashinfer_cutedsl_megamoe_plain | +7.36% | -6.10% | -5.39% | -4.93% | | flashinfer_cutedsl_megamoe_ikr | +9.55% | -9.73% | -9.60% | -12.06% | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **+11.88%** | **-12.50%** | **-13.00%** | **-13.93%** | ## Decode Input len 32, output len 1024, 10,240 requests, concurrency 8192, 5,256,337 generated tokens. | Backend | Output tput (tok/s) | Mean TPOT (ms) | Median TPOT (ms) | P99 TPOT (ms) | |---|---|---|---|---| | trtllm (baseline) | 8,783 | 795.2 | 727.4 | 2,244.7 | | deepgemm megamoe | 8,620 | **762.8** | 721.3 | **1,944.3** | | flashinfer_cutedsl_megamoe_plain | 8,832 | 783.1 | 722.0 | 2,167.6 | | flashinfer_cutedsl_megamoe_ikr | 8,849 | 780.0 | 717.7 | 2,182.1 | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **8,886** | 776.6 | **715.8** | 2,151.7 | Change vs `trtllm`: | Backend | Output tput | Mean TPOT | Median TPOT | P99 TPOT | |---|---|---|---|---| | deepgemm megamoe | -1.86% | **-4.08%** | -0.84% | **-13.38%** | | flashinfer_cutedsl_megamoe_plain | +0.55% | -1.53% | -0.75% | -3.44% | | flashinfer_cutedsl_megamoe_ikr | +0.75% | -1.91% | -1.33% | -2.79% | | **flashinfer_cutedsl_megamoe_combine_nvfp4** | **+1.17%** | -2.34% | -1.60% | -4.14% | ## Takeaways - **Prefill: the best backend depends on tokens/rank.** deepgemm wins at 2048 tok/rank (+18.6% throughput, -16.6% mean TTFT) but only edges trtllm at 8192 tok/rank (+2.1%). flashinfer_cutedsl_megamoe is the reverse: +9% at 2048, +11.9% at 8192. - **Prefill: the flashinfer variant ranking does not hold across sizes.** At 8192 tok/rank it is `plain < ikr < combine_nvfp4`. At 2048 tok/rank, `plain` and `combine_nvfp4` are tied (+8.85% vs +9.06%, mean TTFT within 2 ms) and `ikr` is weakest (+6.19%). - **Decode is nearly a wash on throughput.** All backends land within ±2% of trtllm (8,620-8,886 tok/s), far tighter than the prefill spread. At concurrency 8192 this workload is memory-bound, so the MoE kernel is not the bottleneck. - **Decode tail latency is where backends separate.** deepgemm cuts P99 TPOT by 13.4% and mean TPOT by 4.1%, at the cost of 1.9% throughput — the only backend that trades throughput for tail latency. flashinfer improves both modestly. - **combine_nvfp4 is the best flashinfer variant in decode**, leading on every metric (+1.17% throughput, -2.34% mean TPOT), reproducing the 8192 tok/rank prefill ordering. - **Overall recommendation:** `flashinfer_cutedsl_megamoe_combine_nvfp4` for large-prefill and decode-heavy serving; `deepgemm megamoe` if small-prefill throughput or decode P99 dominates. ## Reproduce `![\"image\"](\"https://github.com/user-attachments/assets/91d92211-28e5-4ca8-8555-47bd81ebf65a\")
Recall is **exact by construction**. With the per-row affine $b(x) = \\lfloor (x - o) \\cdot \\delta^{-1} \\rfloor$ (code: `origin` $= o$, `inv_delta` $= \\delta^{-1}$) over negated scores $x = -\\text{score}$ (bucket 0 = best), the scan emits exactly $\\{\\, j : b(x_j) \\le \\text{th} \\,\\}$, and every refresh recomputes $\\text{th}$ from counts of genuine row elements — so it only ever *tightens* while staying above $b(x_{(k)})$, the bucket of the true k-th best score: $$\\text{th}_0 \\ge \\text{th}_1 \\ge \\cdots \\ge b\\big(x_{(k)}\\big) \\quad\\Longrightarrow\\quad \\{ j : b(x_j) \\le \\text{th}_t \\} \\supseteq \\text{top-}k \\text{at every } t$$ A stale threshold therefore admits extra candidates but never drops one, and `select` trims the superset exactly. In B200, GLM-5.2 tensors, Q=8192, topk=2048, **1.09–1.25×** over dense logits + top-k at 256K–1M contexts, with **8–32 GB of transient logits memory removed**. It is **opt-in** (`SGLANG_ENABLE_DSA_LITETOPK=1`), **off by default**, SM100 (Blackwell) only, GLM DSA shape only (H=32, D=128), and gated to configurations it can serve exactly (`num_init_tokens == num_local_tokens == 0`); everything else stays on the dense path unchanged, so it adds **zero risk** to existing deployments. ## Background: how LiteTopk replaces the dense top-k The dense prefill path scores **all** KV into a `[num_q, seq_len]` fp32 logits buffer (8–32 GB transient at 256K–1M, forcing a memory-budget chunk loop), then runs top-k over it. LiteTopk streams KV in tiles and keeps only *plausible* candidates: 1. **`seed_prep` (calibration)** — per request, score a bounded KV prefix (`min(8192, kv_len)`) with the existing dense `fp8_mqa_logits`, and derive per-row bucket-space parameters (`origin`, `inv_delta`) plus an initial gate threshold = the bucket of the k-th best sample score. A sample k-th value can only be *worse* than the true global k-th, so the initial threshold is a valid loose bound. 2. **`scan`** — the fused kernel: UMMA-scores each KV tile, folds the per-row bucket affine into the register weights (scores accumulate directly in bucket units), and emits positions whose bucket passes the gate into a compact candidate buffer via warp-local queues. Two spare warps run a threshold-refresh daemon that walks the live bucket histogram and **only tightens** the gate as real counts accumulate. 3. **`select`** — exact top-k over the compact candidates (radix on float bits in bucket space), `-1`-padded to `topk`. ## Modifications **New** - `python/sglang/jit_kernel/csrc/dsa_litetopk/dsa_indexer_kernels.cuh` — the warp-specialized scan kernel (vendored 1:1 from vLLM PR #48726). - `python/sglang/jit_kernel/csrc/dsa_litetopk/dsa_indexer.cuh` — `seed_prep` / `select` kernels, TMA helpers, shape config (vendored). - `python/sglang/jit_kernel/csrc/dsa_litetopk/entry.cuh` — the only sglang-facing layer (TVM-FFI launchers); mirrors the upstream launcher argument order so future syncs stay mechanical. - `python/sglang/jit_kernel/csrc/dsa_litetopk/vendor_deep_gemm/` — minimal vendored DeepGEMM headers (tcgen05 / TMA / UMMA PTX), with the DeepSeek license preserved. - `python/sglang/jit_kernel/dsa/litetopk.py` — Python orchestration: per-request calibration loop, scratch allocation, the three primitive calls. - `test/registered/jit/test_dsa_litetopk.py` — recall tests vs an fp32 torch reference (details below). **Modified** - `python/sglang/srt/layers/attention/dsa/dsa_indexer.py` — `use_dsa_litetopk` gate in `Indexer.__init__` + `_get_topk_ragged_litetopk` branch in `_get_topk_ragged`. - `python/sglang/srt/environ.py` — `SGLANG_ENABLE_DSA_LITETOPK` (`EnvBool`, default `False`). ## sglang deviations - **Per-request gate calibration.** Upstream calibrates every row from a single shared `kv[:sample_len]` prefix of the gathered buffer — in a multi-request ragged batch that samples positions a row can never attend, which can over-tighten the gate. Here each request samples **its own** KV prefix with per-row causal `ke`, so thresholds are causally valid by construction. - **`-1` index padding** (upstream pads `0`), matching the dense ragged transform contract (`fast_topk_transform_ragged_fused`) so downstream sparse attention sees the same invalid-slot marker. Output coordinates are identical to the dense path: gathered-KV absolute positions. - **Real SM count** for the KV-split heuristic instead of the hardcoded 148 (the kernel template keeps `NUM_SMS` for 1:1 vendoring). - **Flat-score bucket-span floor** (second commit). With all-identical K rows the sample span degenerates to ~0, the original $10^{-20}$ floor makes `inv_delta` explode, and cross-kernel fp32 accumulation-order noise then spans whole buckets — the gate dropped *every* candidate. Floor the span at $\\sim \\text{mag}/256$ (one bucket $\\geq \\text{mag} \\cdot 2^{-16}$, ~16× above the noise) plus an absolute $10^{-6}$ floor. Coarser buckets only *loosen* the gate, so the fix is recall-safe by construction. Upstream has the same latent issue (only validated on real tensors); found by the adversarial test below. - **Float4 row-alignment padding** (third commit). `seed_prep` reads sample-logits rows with 16 B `float4` loads; the per-request sample width `min(sample_len, kv_len)` is arbitrary, and any width not a multiple of 4 misaligns every odd row base — a device-side fault upstream never hits (its shared prefix has a fixed width). The wrapper pads the width to a multiple of 4 with `-inf` (already skipped by every pass's `isfinite` guard), and the entry layer fails loudly on misaligned widths instead of faulting. ## Implementation notes - **Recall safety is monotonic end to end**: the initial threshold is a loose upper bound from a causally-valid sample; every refresh recomputes from counts of genuine row elements and only tightens; the gate consumes a one-window-stale prefetched threshold (staleness admits extras, never drops); `select` is exact over the surviving candidates. - The scan keeps the **DeepGEMM 2.5 scoring loop** (register-held weights, row-pair TMEM loads, early UMMA release) with **non-persistent KV-split scheduling** (`blockIdx.x` = q-block, `blockIdx.y` = KV window), which keeps all SMs busy on the small-Q chunks long-context prefill actually produces. - Ragged final q-blocks are handled by forcing padded rows to an empty KV range; TMA out-of-bounds reads zero-fill and are dropped by the per-row range checks. - The JIT build pins `sm_100a` and vendors only the DeepGEMM headers the kernel needs; nothing is added to the AOT `sgl-kernel` wheel. ## Validation Hardware: NVIDIA B200 (SM100), CUDA 13. ### Correctness — recall vs an independent fp32 torch reference `test/registered/jit/test_dsa_litetopk.py`, registered on the `4-gpu-b200` CI runner (`base-b-kernel-unit`). The reference recomputes $$\\text{score}[r, j] \\;=\\; \\sum_{h} w[r, h] \\cdot \\mathrm{relu}\\!\\big(q[r, h, :] \\cdot k[j, :]\\big) \\cdot s[j], \\qquad j \\in [\\, ks[r],\\ ke[r] \\,)$$ in fp32 (with $s[j]$ the per-position dequant scale `kv_scale`); checks are tie-tolerant exactness: every selected index must score $\\geq v_k - \\varepsilon$, and every index scoring $> v_k + \\varepsilon$ must be selected (with $v_k$ the k-th best reference score, $\\varepsilon = 10^{-3} \\cdot \\max(|v_k|, 1)$), plus count / `-1` padding / no-duplicate / causal-validity assertions. - `[(32, 32768)] @ topk=2048` — single long request. - `[(16, 8192), (16, 24576), (8, 4096)] @ topk=2048` — multi-request ragged. - `[(32, 32768)] @ topk=512`. - `[(8, 1024)] @ topk=2048` — short rows: valid < topk → `-1` padding. - `[(9, 4099), (5, 1023)] @ topk=512` — odd sample widths (float4 alignment regression) + ragged final q-block padding. - **Flat-score adversarial** (`[(8, 16384)]`, all-identical K rows): every position ties at the k-th value — worst case for the threshold-bucket boundary and candidate-buffer pressure. Caught the bucket-span bug fixed in the second commit (0 candidates selected pre-fix; count + validity + uniqueness green post-fix). ### Performance — upstream kernel-level measurement From the vLLM PR #48726 evaluation (B200, GLM-5.2 tensors, Q=8192, topk=2048), LiteTopk vs dense `fp8_mqa_logits` + top-k: **1.09× / 1.17× / 1.25×** at 256K / 512K / 1M context, with the 8–32 GB transient logits allocation removed. sglang-side e2e numbers will accompany the PR that turns the path on by default. ## Notes - **Exact top-k SET, unspecified order**: tie-breaking at the k-th value follows atomic arrival order, and the within-row output order is unsorted (the dense radix path is also unordered). Not valid for deterministic mode. - The exact-set guarantee holds while the candidate buffer bound (`cand_cap` $= \\max(4 \\cdot \\text{topk},\\ 16384)$ per row) is not exceeded; the calibration sample must also be $\\geq$ `topk` (default $8192 \\geq 2048$). - Prefill (ragged extend) only; decode keeps the existing paged path. GLM DSA shape only (H=32, D=128); fp8 index-K cache only. - The dense path is untouched and remains the default; the env flag flips a per-`Indexer` boolean, nothing else. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31557797915](https://github.com/sgl-project/sglang/actions/runs/31557797915) Latest PR Test (Extra): :x: [Run #31557797919](https://github.com/sgl-project/sglang/actions/runs/31557797919) ",
"url": "https://github.com/sgl-project/sglang/pull/32094",
"createdAt": "2026-07-22T15:39:13Z",
"updatedAt": "2026-08-13T14:14:31Z",
"timestamp": "2026-08-13T14:14:31Z",
"metrics": {
"reactions": 2,
"comments": 6
},
"labels": [
"documentation",
"deepseek",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "yuan-luo",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:32114",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Delete cutlass_mla, non-Marlin GPTQ, AWQ AOT kernel, and Dual Chunk Flash Attention",
"text": "## Summary - Delete the `cutlass_mla` attention backend (kernel + Python integration): SM10.0-only decode kernel, already disabled on GB300 (SM10.3), falls through to `FlashInferMLABackend` for everything except plain decode, no CI coverage. - Delete the non-Marlin GPTQ CUDA kernel (`gptq_gemm`/`gptq_shuffle`) and the `\"gptq\"` GPU quantization choice: superseded by `gptq_marlin` (fully JIT), which `GPTQMarlinConfig` already auto-upgrades compatible checkpoints to. `auto-round` and `moe_wna16` lose their non-Marlin GPU fallback as a result (their Marlin paths are unaffected). - Delete the AWQ AOT dequantize kernel (`awq_dequantize`): every call site already tries a JIT kernel then a Triton kernel first; the AOT path was an unreachable-in-practice last resort. XPU is unaffected (resolves `sgl_kernel.awq_dequantize` from the separate `sgl-kernel-xpu` package). - Delete Dual Chunk Flash Attention (`dual_chunk_flash_attn`, Qwen-1M long-context backend): single-model-family opt-in, no functional kernel change in ~1 year. - Update `docs_new` (quantization/attention-backend/server-arguments pages, DeepSeek-V3 cookbook) and delete the now-orphaned examples/tests for each of the above. ## Test plan - [x] CI (not triggered on this draft — hold for review) - `python3 -m py_compile` on all modified Python files (done locally) - pre-commit hooks (done locally) Fixes #32111 Fixes #32112 --- ### CI States Latest PR Test (Base): :x: [Run #31394593765](https://github.com/sgl-project/sglang/actions/runs/31394593765) Latest PR Test (Extra): :x: [Run #31394590747](https://github.com/sgl-project/sglang/actions/runs/31394590747) ",
"url": "https://github.com/sgl-project/sglang/pull/32114",
"timestamp": "2026-08-12T16:11:27Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"documentation",
"quant",
"speculative-decoding",
"sgl-kernel",
"blackwell",
"run-ci"
],
"author": "b8zhong",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:32166",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Use SYCL kernels for DeepSeek V4 MHC on XPU",
"text": "## Motivation Enable XPU-specific optimized kernels for the DeepSeek V4 MHC (Multi-Head Compression) path on Intel XPU hardware. The existing CUDA-path kernels (deep_gemm, CUDA-first device selection) are either unavailable or suboptimal on XPU, causing incorrect device dispatch or missing performance opportunities. ## Modifications deepseek_v4.py: On XPU, dispatch hc_head to sgl_kernel.mhc.fused_hc_head (SYCL kernel) instead of the Triton implementation; dispatch mhc_fused_post_pre to the XPU-specific kernel; skip deep_gemm when running on XPU; fix device selection to avoid falling back to CUDA when XPU is available. kernels/ops/layernorm/mhc.py: Guard CUDA-only code paths so they are not triggered on XPU. test/registered/xpu/test_hc_head.py: Add XPU CI test comparing SYCL fused_hc_head against the Triton reference for both accuracy (assert_close, atol/rtol=2e-2) and performance (assertLess median latency). ## Accuracy Tests Tested fused_hc_head (SYCL) vs Triton reference on Intel XPU with hidden_size=7168, hc_mult=4, dtype=bfloat16, num_tokens=[1, 16, 128, 1024]: All configurations pass torch.testing.assert_close(atol=2e-2, rtol=2e-2). Max absolute difference is within bfloat16 precision bounds (≤ 0.03125), mean difference ≈ 0. ## Speed Tests and Profiling Benchmarked fused_hc_head SYCL kernel vs Triton on Intel XPU (BMG), hc_mult=4, dtype=bfloat16 (median latency, 50 iterations): | batch_size | seq_len | hidden_size | dtype | sycl_ms | triton_ms | speedup | |-------------:|----------:|--------------:|:---------|----------:|------------:|----------:| | 1 | 16 | 4096 | bfloat16 | 0.033488 | 0.06552 | 1.957 | | 1 | 16 | 7168 | bfloat16 | 0.054496 | 0.094744 | 1.739 | | 1 | 128 | 4096 | bfloat16 | 0.043862 | 0.075218 | 1.715 | | 1 | 128 | 7168 | bfloat16 | 0.073606 | 0.11596 | 1.575 | | 1 | 1024 | 4096 | bfloat16 | 0.222664 | 0.366938 | 1.648 | | 1 | 1024 | 7168 | bfloat16 | 0.38298 | 0.732524 | 1.913 | | 4 | 16 | 4096 | bfloat16 | 0.038324 | 0.068536 | 1.788 | | 4 | 16 | 7168 | bfloat16 | 0.06162 | 0.110292 | 1.79 | | 4 | 128 | 4096 | bfloat16 | 0.137748 | 0.194922 | 1.415 | | 4 | 128 | 7168 | bfloat16 | 0.23192 | 0.428142 | 1.846 | | 4 | 1024 | 4096 | bfloat16 | 0.746642 | 1.21654 | 1.629 | | 4 | 1024 | 7168 | bfloat16 | 1.29662 | 2.64555 | 2.04 | | 16 | 16 | 4096 | bfloat16 | 0.076388 | 0.09984 | 1.307 | | 16 | 16 | 7168 | bfloat16 | 0.12974 | 0.223678 | 1.724 | | 16 | 128 | 4096 | bfloat16 | 0.390572 | 0.627458 | 1.607 | | 16 | 128 | 7168 | bfloat16 | 0.679328 | 1.32392 | 1.949 | | 16 | 1024 | 4096 | bfloat16 | 2.88233 | 5.00391 | 1.736 | | 16 | 1024 | 7168 | bfloat16 | 5.07629 | 10.7714 | 2.122 | | 64 | 16 | 4096 | bfloat16 | 0.22256 | 0.366912 | 1.649 | | 64 | 16 | 7168 | bfloat16 | 0.383344 | 0.731328 | 1.908 | | 64 | 128 | 4096 | bfloat16 | 1.45382 | 2.46189 | 1.693 | | 64 | 128 | 7168 | bfloat16 | 2.55819 | 5.32249 | 2.081 | | 64 | 1024 | 4096 | bfloat16 | 11.4992 | 20.5583 | 1.788 | | 64 | 1024 | 7168 | bfloat16 | 20.1847 | 43.6769 | 2.164 | --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31376396565](https://github.com/sgl-project/sglang/actions/runs/31376396565) Latest PR Test (Extra): :x: [Run #31376396388](https://github.com/sgl-project/sglang/actions/runs/31376396388) ",
"url": "https://github.com/sgl-project/sglang/pull/32166",
"createdAt": "2026-07-23T07:27:38Z",
"updatedAt": "2026-08-13T06:21:16Z",
"timestamp": "2026-08-13T06:21:16Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"deepseek",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "cyxlily",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32196",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[PD] Keep EAGLE DP graph and token metadata consistent",
"text": "Fixes #32182. In PD decode, target and draft can use different MoE A2A backends. The scheduler kept target-local token counts for the draft worker, so DP ranks disagreed on draft padding and collective shapes. Missing GLM DSA seeds could also make only active ranks fall back to eager. This change synchronizes graph eligibility across DP ranks and keeps draft-specific token counts without changing the target DeepEP path. Validation: - GLM-5.2 EAGLE PD, DP32/TP32, target DeepEP + draft A2A none - 32/32 routed warmup requests returned HTTP 200; no NaN/Inf, traceback, or CUDA error --- ### CI States Latest PR Test (Base): :x: [Run #31573591543](https://github.com/sgl-project/sglang/actions/runs/31573591543) Latest PR Test (Extra): :x: [Run #31573591347](https://github.com/sgl-project/sglang/actions/runs/31573591347) ",
"url": "https://github.com/sgl-project/sglang/pull/32196",
"createdAt": "2026-07-23T11:20:26Z",
"updatedAt": "2026-08-13T10:47:44Z",
"timestamp": "2026-08-13T10:47:44Z",
"metrics": {
"reactions": 0,
"comments": 10
},
"labels": [
"bug",
"run-ci"
],
"author": "weireweire",
"state": "open",
"assignees": [
"Fridge003",
"kpham-sgl"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32214",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Strict bit-exact SWA HiCache for DeepSeek-V4 with unified_kv: SWA-window + c4/indexer state riding across L1/L2/L3",
"text": " ## Motivation DeepSeek-V4's `unified_kv` backend on ROCm packs three KV families in one pool: the content-addressed compressed KV (C4/C128), the per-request **SWA ring** (addressed by `req_pool_idx * window + pos % window`), and the compressor's **c4 / indexer overlap state**. Only the compressed KV is content-stable; the SWA ring and the c4/indexer overlap state are not, and are never written into the radix tree. Two prior PRs made reuse of such prefixes *safe* but only **approximately correct**: - **#29417** enabled unified_kv HiCache, but it offloads **only** the compressed KV (C4/C128). The SWA ring is not preserved, so on a hierarchical hit the trailing sliding window is **re-prefilled** (`swa_reprefill_tail_tokens()`) rather than restored. - **#30339** applied the same **re-prefill** idea to the radix-only path: cap the match by the trailing window and recompute it into the request's own ring, so decode never reads a stale ring. Re-prefill recomputes the trailing window (and, implicitly, the c4/indexer overlap state for that region) instead of reproducing the exact values the original prefill wrote. That is enough to prevent gross stale-ring errors and to keep GSM8K-level accuracy within noise — but it is **not bit-exact**: a partial re-prefill cannot reconstruct the original SWA window and c4/indexer overlap state byte-for-byte, so the reused output can differ from a full fresh compute at the bit level. This PR closes that gap. Instead of re-prefilling an approximation, it **captures and offloads the true values** of the SWA window and the c4/indexer overlap state across L1→L2→L3, and **restores them exactly** on a hierarchical hit — making unified_kv SWA prefix reuse **strict and bit-exact**. A hard reuse gate re-prefills only when the truth is unavailable (never a stale/approximate read). This is an **opt-in** gated by `SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE` (default **off**), and it engages only when **all** of the following hold at once: - running **DeepSeek-V4** on **ROCm/HIP**; - the **`unified_kv_triton`** attention backend is chosen (`SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton`); - **HiCache is enabled** (`--enable-hierarchical-cache`). In every other case, the feature is a strict no-op, no pool is allocated, no capture, no restore; behavior is byte-identical to `main`. ## Modifications Organized as three layers: **1. Substrate (Commit 1–5):** - `SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE` flag + `--hicache-swa-offload-page-stride` arg (`environ.py`, `server_args.py`). - Dedicated pinned-host SWA-window offload pool with capture-done handshake + upstream deferred release (`memory_pool_host.py`). - Two independent, tile-start-addressed L3 pools for c4 state and indexer state, packing (`deepseek_v4_memory_pool.py`, `hybrid_cache/hybrid_pool_assembler.py`). - Version-namespaced L3 keys (`hicache_storage.py`, `storage/nixl/hicache_nixl.py`). - Startup fail-fast guard (`hybrid_pool_assembler.py`): with the strict flag on, raise a clear `ValueError` unless `--hicache-write-policy write_through`, before any host pool is pinned. **2. Capture (Commit 6–8):** - Capture the SWA ring window at each completed prefill/decode page with EAGLE-safe geometry (`deepseek_v4_backend_hip_radix.py`, `model_runner.py`, `deepseek_v4.py`). - Snapshot c4/indexer overlap state for the same window into the independent state pools (`compress_hip.py`, `compressor_v2.py`). **3. Reuse/restore + strict gate (Commit 9–10):** - Ride the captured window + state back to device at the correct `buf_lo` offset on a hierarchical hit; **strict reuse gate** caps the match and re-prefills any tail that isn't provably bit-exact, never a stale read; commit-coupling guard keeps window+state+full-KV committed atomically (`swa_component.py`, `unified_radix_cache.py`, `base_prefix_cache.py`, `schedule_batch.py`, `schedule_policy.py`). ## Note to reviewers This PR can be split into three cumulative sub-PRs on request — substrate / capture / reuse (boundaries C1–5 / C6–8 / C9–10) — each gated by the same flag and a strict no-op by default. ## Accuracy Tests Two detectors (like #30339): a byte-exact harness is the real detector; GSM8K is only a no-regression guard. ### 1. Unit Tests (5 test files, 163 UTs total) **How:** `cd test && python3 -m pytest -q srt/mem_cache/test_swa_*.py` **Expected:** `163 passed` — byte-exact swa+c4_state/c4_indexer_state capture/restore across L1→L2→L3, strict-gate reject, commit-coupling guard, dirty-read protection, L3-only restore. ### 2. E2E (DeepSeek-V4 unified_kv on ROCm, HiCache + strict flag) **How:** launch with the gates on and `SGLANG_SWA_DBG_CHECKSUM=1` (emits a per-window checksum vs a fresh device re-read), trigger prefix reuse, grep the log. ```bash export SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton export SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=1 # dev; set 0 for the \"off\" baseline export SGLANG_SWA_DBG_CHECKSUM=1 # + standard dsv4 env as in #30339 python3 -m sglang.launch_server --model-path /data/models/DeepSeek-V4-Pro --attention-backend dsv4 \\ --page-size 256 --swa-full-tokens-ratio 0.1 --enable-cache-report --port 30001 \\ --enable-hierarchical-cache --hicache-write-policy write_through \\ --hicache-io-backend direct --hicache-mem-layout layer_first \\ --tp 4 --dp 4 --enable-dp-attention 2>&1 | tee server.log # decode + EAGLE spec path (tp8/dp8): add --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-num-draft-tokens 4 --speculative-eagle-topk 1 # L3-only variant: add --hicache-storage-backend file --hicache-storage-prefetch-policy wait_complete # + SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR=/ssd/dir, populate, kill, restart same dir, resend prefix ``` **Trigger reuse** — send the same long prompt N rounds; `cached_tokens` climbs 0 → large: ```bash python3 - <<'PY' import json, urllib.request P = \"This is a long shared prefix sentence. \" * 4000 # >> page_size (256) tokens for r in range(6): # 6 for prefill; 20 for decode dp8 (every rank must hit) b = json.dumps({\"model\": \"x\", \"prompt\": P, \"max_tokens\": 8, \"temperature\": 0}).encode() u = json.loads(urllib.request.urlopen(urllib.request.Request( \"http://127.0.0.1:30001/v1/completions\", data=b, headers={\"Content-Type\": \"application/json\"})).read())[\"usage\"] print(\"round\", r, \"cached_tokens\", (u.get(\"prompt_tokens_details\") or {}).get(\"cached_tokens\", 0)) PY ``` **Expected** (grep `server.log`): | check | grep | expect | | --- | --- | --- | | byte-exact SWA + c4/indexer | `[C4-STATE-DBG]` state ride bit-exact / `[LB-DEV]` device landing byte-exact / `[SWA-DBG]` restore verified | **L1/L2 same-process reuse**: all three **> 0**. **L3-only cross-restart**: `[LB-DEV]` + `[C4-STATE-DBG]` **> 0** (`[SWA-DBG]` is **0** on this path — the prefill checksum is not persisted across a process restart, so the window's byte-exact verification is carried by `[LB-DEV]` device-landing). | | no failure / no dirty read | `MISMATCH\\|reuse_reject\\|BIND-MISS\\|AssertionError\\|Traceback` | **0** | | L3-only source | `prefetch .*matched=[0-9]+ loaded=[0-9]+` | `matched=0 loaded>0` | **L3-only restore validated** (cold restart, empty L1/L2): `prefetch matched=0 loaded=2304`, the first request immediately reports `cached_tokens=2304`, `[LB-DEV]` + `[C4-STATE-DBG]` byte-exact all pass, zero `MISMATCH`. **gsm8k eval**: Two arms, each run once per server profile — off (SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=0) and dev (=1) — with the cache flushed (/flush_cache) before each pass and temperature=0, so the only variable is the strict flag. GSM8K is a no-regression guard (the byte-exact harness above is the real correctness detector); the acceptance criterion is that dev is not below off and invalid ≈ 0. - non-thinking ``` python3 benchmark/gsm8k/bench_sglang.py \\ --host 127.0.0.1 --port 30001 --num-shots 8 --num-questions 1319 --parallel 256 ``` For off (SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=0) and dev (=1) , all above tests we got acc ~0.94(invalid ≈ 0). - thinking ``` python3 -m sglang.test.run_eval \\ --eval-name gsm8k --host 127.0.0.1 --port 30001 \\ --num-threads 256 --temperature 0.0 --repeat 1 --num-shots 8 --max-tokens 8192 ``` For off, we got 0.933, and for dev, we got 0.942(invalid 0). ## Speed Tests and Profiling Both arms HiCache on, `DBG=0`, `--hicache-ratio 1.0` (single variable = strict flag); customer clients. Just take some of the configs as example. | scenario | metric | off | dev | note | | --- | --- | --- | --- | --- | | Prefill (dp4, 50k ISL, prefix-cache0.6) | tokens/s | ≈ 32.7k | ≈ 30.8k | dev ≈ −5.8%| | Prefill (dp4, 50k ISL, prefix-cache0.8) | tokens/s | ≈ 31.4k | ≈ 29.1k | dev ≈ −7.3%| | Decode TPOT (dp8, EAGLE, batch 32/48/64) | ms/token | ≈ 70.6 / 82.4 / 96.8 | ≈ 70.8 / 83.8 / 98.4 | dev slightly slower, no regression | The prefill cost is the deterministic price of strict correctness (positional H2D of the SWA window + c4/indexer state on reuse, not a content-addressed page copy), applied only with the flag on. **Optional in-repo reproduction.** The table above comes from the customer perf clients. For a self-contained check that needs no external scripts, the stock bench_serving client reproduces the strict overhead directly in-repo (it is a different, smaller workload, so absolute numbers differ). Launch the same server twice with DBG=0 — once SGLANG_UNIFIED_KV_BIT_EXACT_HICACHE=1 (dev), once =0 (off) — then run against each: ```bash python3 -m sglang.bench_serving --backend sglang --dataset-name random \\ --random-input-len 50000 --random-output-len 200 --random-range-ratio 1 \\ --num-prompts 64 --max-concurrency 64 --seed 42 --port 30001 ``` Compare peak output token throughput: dev trails off by a small, consistent margin (measured off ≈ 1536 tok/s vs dev ≈ 1472 tok/s, ≈ −4%), the same direction as the customer-client table. Note that at this concurrency/input length the Median/P99 TTFT is dominated by request queueing (~50–100 s) and is not a reliable reuse detector — prefix-reuse correctness is verified separately by the byte-exact harness and the climbing cached_tokens above, not by this client's TTFT. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31686468859](https://github.com/sgl-project/sglang/actions/runs/31686468859) Latest PR Test (Extra): :x: [Run #31686468531](https://github.com/sgl-project/sglang/actions/runs/31686468531) ",
"url": "https://github.com/sgl-project/sglang/pull/32214",
"createdAt": "2026-07-23T13:24:12Z",
"updatedAt": "2026-08-13T09:25:37Z",
"timestamp": "2026-08-13T09:25:37Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"deepseek",
"hicache"
],
"author": "amd-danli103",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32269",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support XQA backend for SpecDec verify",
"text": " ## Motivation Adds support for TRT-LLM XQA decode backend for SM90/120 speculative decoding, by building a causal mask it requires for the verify forward pass (more than 1 q token). Agent generated summary: Before: pure `--attention-backend trtllm_mha` + NEXTN on SM90/120 died at server startup during decode CUDA-graph capture — verify routes to the XQA decode kernel, which demands a mask for `q_len > 1` that the backend never passed: ``` File \"/SGL/flashinfer/flashinfer/xqa.py\", line 332, in xqa assert mask is not None, \"Mask is required for speculative decoding\" AssertionError: Mask is required for speculative decoding ``` Never caught before because the verify→decode-kernel routing was written for SM100 (trtllm-gen needs no mask); XQA enablement for SM90/120 came later and nobody ran spec decoding on that combo. ## Modifications Fix/After: precompute XQA's bit-packed causal mask (constant for chain/topk=1 speculation) at init and pass it on verify — flashinfer already plumbed the argument. ## Accuracy Tests Validated: boots through capture, GPQA-10 0.70, acceptance 2.20/0.40 matching the FA3-verify reference (see MTP data reported in https://github.com/sgl-project/sglang/pull/23112). ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31565406415](https://github.com/sgl-project/sglang/actions/runs/31565406415) Latest PR Test (Extra): :x: [Run #31565405956](https://github.com/sgl-project/sglang/actions/runs/31565405956) ",
"url": "https://github.com/sgl-project/sglang/pull/32269",
"createdAt": "2026-07-24T02:07:30Z",
"updatedAt": "2026-08-13T06:49:43Z",
"timestamp": "2026-08-13T06:49:43Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"performance",
"blackwell",
"run-ci",
"hopper"
],
"author": "akhilg-nv",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32280",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] Enable LLaDA2.2 block routing and graph-captured JTI on Ascend",
"text": "## Summary This PR adds self-contained LLaDA2.2-flash support for block-routing MoE and JointThresholdInDel (JTI) on Ascend NPU. Model-forward graph capture continues to use SGLang's existing NPU dLLM graph path; this PR adds graph capture only for the fixed-width JTI token-update step. It incorporates only the block-routing contract needed from #31768 and the JTI semantics needed from #31773. Those PRs are implementation references, not merge dependencies; this branch remains buildable on `main`. ## Scope and boundaries Model- and algorithm-specific code stays under LLaDA2/JTI: - `python/sglang/kernels/ops/llada2/npu.py`: Ascend block routing and edit-token reductions. - `python/sglang/srt/dllm/algorithm/joint_threshold_indel.py`: fixed-shape eager and NPU-graph JTI. - `python/sglang/srt/models/llada2.py`: selects block routing only for checkpoints that declare `expert_capacity`; the LLaDA2.1 path is unchanged. There are no changes to shared model-executor, prefill-runner, NPU attention, or NPU graph-backend code. Both dLLM prefill and iterative decode already use `ForwardMode.DLLM_EXTEND`, which is captured by the existing `NPUGraphRunner`. ## Block-routing behavior The NPU block-routing path follows this contract: 1. select `expert_capacity` experts from the maximum corrected score in each token block; 2. select each token's ordered top-k within that compact capacity; 3. use unbiased sigmoid scores for the returned weights; 4. resolve exact ties by lower expert ID without perturbing close but unequal scores; 5. normalize in FP32 when `top_k > 1`; preserve the raw sigmoid score for `top_k == 1`. The implementation keeps the CANN top-k fast path and folds `routed_scaling_factor` into `topk_weights`, matching the existing NPU MoE contract. ## JointThresholdInDel The JTI implementation preserves the required termination and accuracy semantics: - the prompt is the immutable contiguous prefix before the first MASK; - an original MASK predicted as MASK uses the best non-MASK fallback; - DELETE/SPLIT move original-mask tracking through compaction; - regular and post-edit budgets force a final scrub that excludes MASK, DELETE, and SPLIT; - a cleanup that changes token IDs gets one final forward to persist KV state; - graph and non-graph paths use fixed-shape state and are covered for parity. `enable_graph` applies only to the fixed-width JTI token-update step and is NPU-only. Model-forward graph capture remains on the existing `NPUGraphRunner`. FDFO keeps the non-graph fixed-shape path because its batch composition changes between rounds. ## Historical hardware results BF16, BS1, TP4, native top-8, block size 32, GSM8K first 200: | Model/path | Aggregate tok/s | Correct | Output tokens | |---|---:|---:|---:| | LLaDA2.1-flash reference | 190.419 | 183/200 (91.5%) | 62,931 | | LLaDA2.2 before routed-scale fix | 204.882 | 156/200 (78.0%) | 56,987 | | LLaDA2.2 with routed-scale fix | 258.354 | 189/200 (94.5%) | 52,329 | | LLaDA2.2 validated standalone tree | **252.332** | **191/200 (95.5%)** | 52,567 | The validated run used one warmup request followed by 200 serial requests on Ascend 910B3 NPU 0-3. Aggregate throughput is total output tokens divided by the sum of request durations; wall-clock throughput was 252.282 tok/s. BFCL-V4 `simple_python` was run once at temperature 1.0 and scored with BFCL's official AST checker: **371/399 (92.98%)**. This is a directional compatibility check, not a reproduction of the model card's five-run aggregate score. These hardware results predate the final scope-only cleanup. The full hardware benchmark was not rerun after that cleanup. Artifacts: - [LLaDA2.2 GSM8K-200 raw JSON](https://huggingface.co/babba82200/test1/resolve/main/llada22_pr32280_gsm8k200_raw.json?download=true) - [Measurement summary](https://huggingface.co/babba82200/test1/resolve/main/llada22_pr32280_gsm8k200_results.md?download=true) - [BFCL raw JSON](https://huggingface.co/babba82200/test1/resolve/main/llada22_bfcl_simple_python_399_raw.json?download=true) - [BFCL scored JSON](https://huggingface.co/babba82200/test1/resolve/main/llada22_bfcl_simple_python_399_scored.json?download=true) - [LLaDA2.1 GSM8K-200 JSON](https://huggingface.co/babba82200/test1/resolve/main/llada21_flash_npu_bf16_bs1_tp4_jt_gsm8k200.json?download=true) - [Warmed one-block Chrome trace](https://huggingface.co/babba82200/test1/resolve/main/llada22_flash_npu_bf16_bs1_tp4_jt_graph_206tps_oneblock.trace.json?download=true) ## Configuration ```yaml threshold: 0.5 edit_threshold: 0.0 enable_graph: true ``` ```text --dllm-algorithm JointThresholdInDel --dllm-algorithm-config /path/to/jti.yaml ``` The model config is the source of truth for `delete_token_id` and `split_token_id`; SGLang does not contain a checkpoint-specific fallback. The currently published LLaDA2.2-flash config does not declare these fields, so it requires: ```text --json-model-override-args '{\"delete_token_id\": 156930, \"split_token_id\": 156931}' ``` ## Current-head validation - JTI graph/non-graph parity, termination, configuration, and LLaDA2 routing tests: **11 passed** on CPU. - Python compilation and `git diff --check`: passed; GitHub Lint is pending on this head. - Rebased cleanly on current `upstream/main`. - NPU kernel regression tests cover tail blocks, exact ties, close unequal scores, ordered top-k, `top_k == 1`, and final scrub behavior. Device CI awaits the protected `run-ci` / `run-ci-extra` labels. ## Known limitations - DeepEP block routing is rejected until its capacity-routing semantics are implemented. - JTI can require more forwards per output token than JointThreshold because DELETE/SPLIT may introduce masks and post-edit rounds. - Current NPU graph mode does not support FDFO state packing. --- ### CI States Latest PR Test (Base): :x: [Run #31691927991](https://github.com/sgl-project/sglang/actions/runs/31691927991) Latest PR Test (Extra): :x: [Run #31691927830](https://github.com/sgl-project/sglang/actions/runs/31691927830) ",
"url": "https://github.com/sgl-project/sglang/pull/32280",
"createdAt": "2026-07-24T03:59:43Z",
"updatedAt": "2026-08-13T10:38:28Z",
"timestamp": "2026-08-13T10:38:28Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"npu",
"jit-kernel"
],
"author": "leedongkun30-arch",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32313",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] Optimize TP LMHead with All-to-All",
"text": " ## Motivation When the model is deployed with pure DP and dp-attention, **--enable-dp-lm-head will suffer from poor gemm efficiency when batchsize of each rank is small**. For example, in DeepseekV4 Pro, we use dp=8 and --enable-dp-lm-head. In such case, the lmhead gemm itself costs **320us** when bs=36 on each dp rank. ![\"image\"](\"https://github.com/user-attachments/assets/c63cd93d-b672-4834-92d4-b046decfd6da\")
In small batch, TP LMHead is a better choice. However, current implementation of TP LMHead will introduce allgather communication on vocab_size. In deepseek v4, each rank will send and receive [bs*8, vocab_size // 8 * 7] = **31MB** data, which costs around **150us**. ![\"image\"](\"https://github.com/user-attachments/assets/c15348d5-cc77-440c-b3dc-49d59b73a081\")
Therefore, we propose to use **All2All to replace the allgather + scatter communication in TP LMHead**. With All2All, the data volume that each rank sends and receives decreases to: - send: [bs * 7, vocab_size // 8]. When bs=36, it's **3.88MB**. - receive: [bs, vocab_size // 8 * 7]. When bs=36, it's **3.88MB**. The all2all costs around 46us in profile. ![\"image\"](\"https://github.com/user-attachments/assets/bc11c4be-f844-45e9-bfa0-58da5078fe53\")
Note that the **all2all optimization is tailored for dp-attention with pure DP parallel**. As for dp-tp hybrid parallel, it's hard to support with significant improvement. ## Modifications 1. Use all2all operation in _get_logits after lmhead computation is completed. 2. Only support pure dp. Dp-lm-head, cp, dp-tp hybrid parallel are not supported to be turned on together. ## Accuracy Tests GSM8k test on deepseek v4 is OK. ![\"image\"](\"https://github.com/user-attachments/assets/87181b21-7d71-44bd-8741-592f521f533e\")
## Speed Tests and Profiling Profile | Setup | gemm (us) | communication (us) | Total duration of LMHead (us) | | --- | --- | --- | --- | | dp-lm-head | 320 | \\ | 320 | | tp-lm-head-allgather | 71 | 151 | 278 | | tp-lm-head-all2all | **71** | **46** | **169** | Launch and benchmark command ``` # prefill GLOO_SOCKET_IFNAME=eth1 NCCL_SOCKET_IFNAME=eth1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT=1 SGLANG_OPT_SWA_EVICT_DROP_PAGE_MARGIN=1 SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW=1 SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 SGLANG_OPT_USE_ONLINE_COMPRESS=1 SGLANG_EXPERIMENTAL_ONLINE_C128_MTP=1 SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0 SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1 sglang serve --trust-remote-code --model-path ./models/deepseek-ai/DeepSeek-V4-Pro/ --tp 8 --ep 8 --moe-a2a-backend megamoe --enable-dsa-prefill-context-parallel --dsa-prefill-cp-mode round-robin-split --chunked-prefill-size 63488 --max-prefill-tokens 63488 --swa-full-tokens-ratio 0.1 --mem-fraction-static 0.78 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --disaggregation-mode prefill --disaggregation-transfer-backend mooncake --tokenizer-worker-num 8 --host 0.0.0.0 --port 9091 > log 2>&1 & # decode GLOO_SOCKET_IFNAME=eth1 NCCL_SOCKET_IFNAME=eth1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_OPT_USE_ONLINE_COMPRESS=1 SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=0 SGLANG_OPT_FIX_MEGA_MOE_MEMORY=1 SGLANG_OPT_USE_DEEPGEMM_MEGA_MOE=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_FP4_ACTS=1 SGLANG_OPT_DEEPGEMM_MEGA_MOE_USE_MXF4_KIND=1 sglang serve --trust-remote-code --model-path ./models/deepseek-ai/DeepSeek-V4-Pro/ --tp 8 --dp 8 --enable-dp-attention --moe-a2a-backend megamoe --mem-fraction-static 0.92 --swa-full-tokens-ratio 0.09 --disaggregation-mode decode --disaggregation-transfer-backend mooncake --enable-hisparse --hisparse-config '{\\\"top_k\\\": 1024, \\\"device_buffer_size\\\": 4096, \\\"host_to_device_ratio\\\": 5}' --tokenizer-worker-num 8 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --host 0.0.0.0 --port 9092 > log 2>&1 & # benchmark python3 -m sglang.bench_serving --model ./models/deepseek-ai/DeepSeek-V4-Pro/ --flush-cache --host 0.0.0.0 --port 9090 --backend sglang --dataset-name generated-shared-prefix --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 1024 --gsp-num-groups 1 --gsp-prompts-per-group 1024 --gsp-system-prompt-len 61440 --gsp-question-len 4096 --gsp-output-len 1024 --request-rate 10 --warmup-requests 10 ``` Benchmark result The TPOT decreases **36.97ms** (dp-lmhead) => **36.21ms** (tp-lmhead-allgather) => **35.67ms** (all2all). The E2E latency decreases **3082.24ms** compared with dp-lmhead, **1151.63ms** compared with allgather. ![\"image\"](\"https://github.com/user-attachments/assets/7f4ec3fa-e9f1-4e58-ba6a-c7a9f2d1f8ff\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 3. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 4. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 5. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31684326632](https://github.com/sgl-project/sglang/actions/runs/31684326632) Latest PR Test (Extra): :x: [Run #31684326467](https://github.com/sgl-project/sglang/actions/runs/31684326467) ",
"url": "https://github.com/sgl-project/sglang/pull/32313",
"createdAt": "2026-07-24T10:27:12Z",
"updatedAt": "2026-08-13T11:21:00Z",
"timestamp": "2026-08-13T11:21:00Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci"
],
"author": "SYChen123",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32327",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DeepSeek-V4] Add Q8KV8 sparse MLA prefill runtime backend",
"text": "Co-authored-by:@JackChuang ## Summary This PR is part of the roadmap tracked in https://github.com/sgl-project/sglang/issues/25746. This PR ports the Q8KV8 sparse MLA prefill path to the DeepSeek-V4 runtime backend and adds a runtime dispatch path via `--dsv4-prefill-backend flashmla_sparse_q8`. When the KV cache uses `fp8_e4m3`, DeepSeek-V4 can run the FP8 query × FP8 KV sparse MLA kernel during prefill instead of falling back to the existing BF16 sparse prefill path. The runtime side also adds Q-head padding, attention-sink padding, kernel input-contract validation, and regression coverage for a real serving shape that previously caused a DeepSeek-V4 hang. Validation has been completed on DeepSeek-V4-Flash with TP8, FP8 KV cache, `flashinfer_mxfp4` MoE, and CUDA graph default serving. Performance experiments show that Q8KV8 delivers consistent throughput and TTFT gains over the existing `flashmla_sparse` baseline, while GSM8K and LongBench-v2 do not show a material accuracy regression. ## Motivation DeepSeek-V4 already supports FP8 KV cache, but the existing sparse prefill backend still primarily uses the BF16 sparse MLA prefill path. As a result, even when KV is stored in FP8, the runtime cannot fully benefit from the native FP8 Q8KV8 sparse prefill kernel. This PR adds a DeepSeek-V4 Q8KV8 runtime path so prefill can directly use FP8 tensor-core computation. ## Modifications | file | role | |---|---| | `deepseek_v4_backend.py` | DeepSeek-V4 integration + `flashmla_sparse_q8` routing (`_forward_prefill_sparse_q8kv8`), Q/head-sink padding, FP8 workspace, C0/C4/C128 sparse-prefill dispatch | | `dsv4/sparse_prefill_utils.py` | Q8KV8 backend selection helper, debug env override, dtype-aware sparse-prefill workspace reuse | | `dsv4/dequant_k_cache.py` | fused paged KV gather + dequant + requant to FP8; padded Q cast for Q8KV8 prefill | | `sparse_mla_q8kv8_prefill_sm90.py` | fail-fast Python-side input contract validation before launching the SM90 Q8KV8 sparse prefill kernel | | `server_args.py` | `flashmla_sparse_q8` in DeepSeek-V4 prefill backend choices via `--dsv4-prefill-backend` | | `test_q8kv8_sparse_prefill_backend.py` | runtime wrapper tests for Q8KV8 DeepSeek-V4 sparse prefill, including C0/C4/C128 behavior | | `test_server_args.py` | CLI tests for `--dsv4-prefill-backend` choices | ## Usage | Variant | `--dsv4-prefill-backend` | Notes | |---|---|---| | Baseline / q16 | `flashmla_sparse` | Existing sparse prefill backend | | Q8KV8 / q8 | `flashmla_sparse_q8` | New Q8KV8 sparse prefill backend | ```bash python -m sglang.launch_server \\ --model-path ![\"屏幕截图](\"https://github.com/user-attachments/assets/000ef4f4-621b-4f4a-b600-435e7dad5114\")
## Speed Tests and Profiling ![\"屏幕截图](\"https://github.com/user-attachments/assets/efd4bbf2-f52e-42b1-be4d-ecf64fc195e8\")
## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 3. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 4. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 5. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31557221230](https://github.com/sgl-project/sglang/actions/runs/31557221230) Latest PR Test (Extra): :x: [Run #31557221071](https://github.com/sgl-project/sglang/actions/runs/31557221071) ",
"url": "https://github.com/sgl-project/sglang/pull/32637",
"createdAt": "2026-07-28T09:25:57Z",
"updatedAt": "2026-08-13T11:15:47Z",
"timestamp": "2026-08-13T11:15:47Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "Estrella-xx",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32673",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts",
"text": "# [Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts ## Motivation In [arXiv:2607.21535](https://arxiv.org/abs/2607.21535) we show that a built-in MTP draft head's full-attention KV read grows with the prefix until it dominates the decode step, and that restricting the *draft's* attention to a sink plus recent window cuts the per-decode-step cost by +28% to +44% at 1M context across three architecture families at near-unchanged acceptance, with the target still verifying every token at full attention. The same idea motivated `--speculative-draft-window-size` in #24664, but that flag reaches the drafter through the model file (`llama_eagle3.py` sets `sliding_window_size` per layer) and the backend's per-layer SWA path, so it applies to `LlamaForCausalLMEagle3` checkpoints. This PR applies the window one level down, in the shared draft-decode index builder, so the built-in EAGLE / MTP (NEXTN) draft-decode path gets it on the Triton and FlashInfer draft backends with no model-file change and no second attention wrapper — the draft simply gets a shorter KV list. Only the windowing mechanism is upstreamed here; the paper's ring-buffer reclaim of the unread draft KV is possible follow-up work. ## Modifications 1. `generate_draft_decode_kv_indices` gains `window_size` and `sink_size` `tl.constexpr` params, both defaulting to `0`. With `window_size > 0` a draft-decode step gathers `[first sink_size tokens] + [most recent window_size tokens]` instead of the whole prefix, then appends the draft tree tokens unwindowed as before. `window_size == 0` collapses every added op to the identity, so the default path and the kernel's other callers (AITER, MLA) are unchanged. 2. `--speculative-draft-window-size` is now also honored by the built-in EAGLE / MTP draft-decode path on the Triton and FlashInfer draft backends. `spec_utils.resolve_draft_decode_window()` is the single mapping from server args to `(window_size, sink_size)`; both `*MultiStepDraftBackend`s call it once in `__init__`. 3. New `--speculative-draft-sink-size` for the attention sink, since a pure recent window drops the initial tokens attention reliably parks on. The window keeps DFLASH's unit and anchor — KV entries, counted back from the end of the prefix — so a step reads `min(seq_len, window + sink)` prefix entries, which at the default `sink=0` is `min(seq_len, window)`, the same expression as `_compute_compact_draft_seq_lens`, with no `-1`. Tree tokens are never capped, so the existing `window_size >= num_draft_tokens` check still holds. 4. A draft model with a sliding window of its own keeps it, and this windowing stays off. That covers an MTP head with SWA in its config and a `LlamaForCausalLMEagle3` drafter, whose window comes from this same flag. One KV list is shared by every draft layer, so the index builder cannot express a per-layer window, and a checkpoint's own window should win over a server flag either way. We warn only when the resulting window differs from the requested one, so the EAGLE-3 case stays quiet. CUDA graphs are safe: the per-request kept length rides in `kv_indptr`, exactly like the variable-seqlen path the graph already replays. Same launch grid, no host sync, no dynamic shapes. ## Speed Tests and Profiling Qwen3.6-35B-A3B bf16, one B200, `trtllm_mha` target, EAGLE `steps=6 topk=1 draft_tokens=7`, `W=4032 S=64` (4096 prefix entries per draft step), batch 1, RULER `niah_multiquery_enum`, 512 output tokens. `window` vs `native` on the same draft backend; the target is identical in every arm. `T_iter` is per-iteration wall time — the mechanism, without acceptance mixed in — and TPOT is `T_iter / AL`. | context | draft backend | TPOT native → window | speedup | T_iter native → window | AL native → window | |---|---|---|---|---|---| | 32K | triton | 3.895 → 3.068 ms | 1.27× | 12.67 → 10.05 ms (−21%) | 3.25 → 3.28 | | 64K | triton | 4.414 → 3.264 ms | 1.35× | 19.06 → 13.48 ms (−29%) | 4.32 → 4.13 | | 128K | triton | 7.047 → 4.568 ms | 1.54× | 31.65 → 20.02 ms (−37%) | 4.49 → 4.38 | | 256K | triton | 10.448 → 6.093 ms | 1.71× | 57.35 → 33.45 ms (−42%) | 5.49 → 5.49 | | 32K | flashinfer | 2.199 → 2.142 ms | 1.03× | 7.20 → 7.01 ms (−3%) | 3.28 → 3.28 | | 64K | flashinfer | 1.728 → 1.738 ms | 0.99× | 7.46 → 7.18 ms (−4%) | 4.32 → 4.13 | | 128K | flashinfer | 1.799 → 1.696 ms | 1.06× | 8.08 → 7.44 ms (−8%) | 4.49 → 4.38 | | 256K | flashinfer | 1.735 → 1.499 ms | 1.16× | 9.36 → 8.05 ms (−14%) | 5.39 → 5.37 | Dense (no speculation) TPOT at those four contexts, for reference: 2.997 / 3.158 / 3.320 / 3.698 ms. - `T_iter` drops at every point on both backends and the drop grows with context, which is the mechanism doing what it should: the draft's read is capped while native's grows. - TPOT also depends on acceptance, so the 64K FlashInfer point is a wash — a 4% per-iteration saving against a 4% acceptance dip. The window is a clear win on FlashInfer from ~128K up and at every point on Triton, whose full-prefix draft read is far more expensive to begin with. - Acceptance never beats native in this sweep. It does past 512K in the paper, but that needs RoPE extension beyond this checkpoint's 256K native context, so we leave it out of the PR's numbers. ## Testing `test/registered/unit/spec/test_draft_decode_window.py` — 13 tests / 28 subtests, 16 s on one GPU. Every expectation comes from a closed-form oracle over `seq_lens`, never from the kernel's own packing, so a bug can't be mirrored on both sides. - `window_size == 0` reproduces the full-KV read plan, offsets and gathered slots, for single- and multi-request batches at `topk` 1 and 2. This is the off-by-default regression guard. - Windowed `kv_indptr` lengths are `min(seq_len, sink + window) + tree`, and the gather really is `[sink] + [recent] + [tree]`, parametrized past 256K/512K/1M. - Tree drafts (`topk > 1`): each `(request, branch)` slot gets its own windowed base list plus its own branch at the oracle offset. Covered at `page_size == 1` and on the `page_size > 1 and topk > 1` paged branch, where tree tokens are read from the uncapped prefix end but written at the capped offset. - `resolve_draft_decode_window`, including the own-window opt-out and exactly when it warns, plus a CPU-only guard that both draft backends resolve through it. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/development_guide_using_docker.html#benchmark-the-speed). - [x] Follow the SGLang code style guidance. --- ### CI States Latest PR Test (Base): :x: [Run #31621192148](https://github.com/sgl-project/sglang/actions/runs/31621192148) Latest PR Test (Extra): :x: [Run #31621191599](https://github.com/sgl-project/sglang/actions/runs/31621191599) ",
"url": "https://github.com/sgl-project/sglang/pull/32673",
"createdAt": "2026-07-28T18:51:04Z",
"updatedAt": "2026-08-12T17:45:55Z",
"timestamp": "2026-08-12T17:45:55Z",
"metrics": {
"reactions": 1,
"comments": 10
},
"labels": [
"speculative-decoding",
"run-ci",
"jit-kernel"
],
"author": "avalliappan-nvidia",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:32741",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] MXFP4 KV Cache Decode for DSV4 on Hopper",
"text": "## Summary MXFP4 KV cache for DeepSeek V4 on Hopper (SM90/H20), end to end: codec + memory pool + fused decode attention kernel + backend integration. The decode kernel is a JIT-compiled port of the FlashMLA three-stage split-KV design (scheduler metadata kernel + persistent WGMMA main kernel + combine kernel), covering all 44 layers (SWA + C4/C128 + attn_sink) in one call. The PR is self-contained: the kernel and its vendored dependencies live under `python/sglang/kernels/jit/` and compile at first use via `load_jit` — no sgl-kernel wheel changes required. ## MXFP4 Layout (368 B/token) ``` [224 B packed E2M1 | 14 B E8M0 + 2 B pad | 128 B BF16 RoPE] ``` - E2M1 (4-bit FP: 1S+2E+1M) nope data, block-size 32 (14 groups) - E8M0 8-bit exponent-only scale per group (dequantized in-register via `bits << 23`) - BF16 RoPE (unquantized) - Row-major contiguous, no page tail padding; no global scale needed (unlike NVFP4) - 16-byte-aligned row (368 B = 16×23) enables 128-bit loads; the NVFP4 reference's 380 B is only 4-byte aligned Memory capacity gain vs FP8: ~+37% full-token capacity (bytes/token 7705 → 5609; the NVFP4 reference gains +31.75%). ## Decode Kernel (`kernels/jit/csrc/mxfp4_dsv4_decode_sm90/`) FlashMLA-style fused split-KV decode for MXFP4: 1. **Scheduler metadata kernel** — 1 CTA × 1 thread computes per-SM split assignments with device-side length clamping (CUDA-graph safe: runs inside the captured graph, re-executes with replayed top-k lengths) 2. **Persistent main kernel** — 384 threads (3 warp groups: producer dequant→SMEM, QK+PV local, PV remote), WGMMA 64×64×16 / 64×256×16; E8M0 block-32 dequant via bit-shift, E2M1 via LUT+PRMT (4 instructions per 16 elements) 3. **Combine kernel** — merges split-K partials with PDL (programmatic dependent launch) Entry point: `entry.cuh` (TVM-FFI single translation unit, explicit instantiations for h_q 64/128). Python wrapper: `kernels/ops/attention/mxfp4_dsv4_decode_sm90.py` (FlashMLASchedMeta, per-geometry scratch reuse, C++-side stream resolution via `TVMFFIEnvGetStream`). ### Vendored dependencies (provenance) | Component | Source | |---|---| | `params.h`, `defines.h`, `utils.h`, `flashmla_utils.h`, `kerutils/`, `combine` | FlashMLA upstream @ 05e26647 | | `config.h`, `dequant.h`, `layout.h`, `components/helpers.h`, splitkv skeleton | SGLang reference PR #31269 (NVFP4) | | `splitkv_mla.cuh` (MXFP4 variant), `entry.cuh` | This PR | CUTLASS/CUTE headers come from the JIT build's `extra_dependencies=[\"cutlass\"]` (pinned cutlass @ 147f5673, matching the reference PR). `kerutils` includes were converted to relative paths during vendoring. ## Changes - **Codec** (`srt/layers/attention/dsv4/mxfp4_k_cache.py`): quantize/dequant for the 368 B row (Triton), paged variants - **Compressor** (`compressor_v2.py`): `_forward_mxfp4` covering all 44 layers (C0/C4/C128), decode-plan boundary semantics - **Memory pool**: parameterized enablement via `--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4` (env alias with deprecation warning); pool sizing/accounting unified - **Startup guards**: HiSparse + MXFP4 combination rejected at init (incompatible sparse layout); DeepseekV4 + fp4_e2m1 on non-SM90 fails fast with a clear error - **Backend** (`deepseek_v4_backend.py`): `_forward_mxfp4_decode_flashmla` — one fused call per layer; scheduler metadata shared per geometry across layers, re-created during CUDA-graph capture (top-k lengths are replayed device inputs) - **Tests**: `test/registered/kernels/ops/attention/test_mxfp4_dsv4_decode.py` (17 kernel cases: C0/C4/C128 × flash/profiling geometries, graph replay + length clamping, invalid contracts, scheduler-refresh regressions) + `test_mxfp4_k_cache.py` (5 codec cases) + `test_dsv4_hisparse_pool.py` (3 pool-guard cases) + `test_dsv4_compress_write_pad.py` (3 restored cases); benchmark in `test/registered/kernels/benchmark/attention/bench_mxfp4_dsv4_decode.py` - **E2E**: `test/registered/kernels/e2e/test_mxfp4_dsv4_e2e.py` — 8-GPU H200 server + gsm8k, exercising pool configurator → compressor → CUDA-graph capture (CI stage extra-b) - Removed the first-generation standalone JIT kernel (`mxfp4_decode.cuh` + `_forward_mxfp4_decode`) superseded by the FlashMLA port ## Validation - Kernel SASS was byte-identical to the AOT (cmake) build of the same sources at port time; the subsequent refactors (V32 path removal, aligned 128-bit row loads) are instruction-level changes covered by the same unit suite - Unit: 28/28 kernel + codec + pool-guard tests pass; 8-GPU e2e integration test passes locally - End-to-end on 8×H20 (TP8, `DeepSeek-V4-Flash`), FP8 (`--kv-cache-dtype fp8_e4m3`) vs MXFP4 (`--kv-cache-dtype fp4_e2m1 --fp4-kv-cache-recipe mxfp4`), identical server flags otherwise: | Metric | FP8 baseline | MXFP4 (this PR) | Δ | |---|---|---|---| | Decode throughput (server-side, 4k×256, conc 32) | 1013 tok/s | 980 tok/s | −3.3% | | GSM8K (1319, greedy, max_tokens 1024; truncated samples excluded) | 96.89% | 96.89% | 0.0pt | | AIME25 (480 = 30×16, greedy, max_tokens 32768; 73 truncated samples excluded from both) | 99.57% pass@1 | 99.78% pass@1 | +0.2pt | ## Known gaps / follow-ups - Small-batch (b≤8) kernel-level latency vs FP8: +8–12% (kernel-inherent scheduling granularity; end-to-end TPOT already within 5%) - Prefill-path throughput: −6.8% vs FP8 (out of scope for this PR; triton-based MXFP4 prefill) - Deep-context + high-concurrency decode (64 concurrent, 8k ctx, no rolling prefill): decode TPOT +17% vs FP8 (33.1 vs 28.3 ms); end-to-end 8k→8k throughput on par (1677 vs 1652 tok/s). One full AIME25 eval run showed −44% throughput vs FP8, but four controlled reruns (5-question replica, rolling short-prompt, 8k×8k fixed) could not reproduce it and measured MXFP4 at parity or faster — attributed to environment, not the kernel. - sm_90a-only (Hopper); no SM100 support (non-SM90 + fp4_e2m1 is rejected at startup) - HiSparse (sparse KV cache) not supported with MXFP4 — rejected at startup ## References - PR #31269 — NVFP4 KV cache for DSV4 in sgl-kernel (reference implementation for the FlashMLA three-stage decode) - FlashMLA upstream @ 05e26647 — scheduler/combine kernels and kerutils - This PR takes the jit_kernel approach (JIT CUDA via TVM-FFI) instead of AOT CUTLASS in sgl-kernel, keeping the PR self-contained --- **Status**: Ready for review — kernel + integration complete, end-to-end validated on 8×H20. --- ### CI States Latest PR Test (Base): :x: [Run #31670942627](https://github.com/sgl-project/sglang/actions/runs/31670942627) Latest PR Test (Extra): :x: [Run #31670942449](https://github.com/sgl-project/sglang/actions/runs/31670942449) ",
"url": "https://github.com/sgl-project/sglang/pull/32741",
"createdAt": "2026-07-29T07:39:49Z",
"updatedAt": "2026-08-13T16:39:47Z",
"timestamp": "2026-08-13T16:39:47Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"quant",
"deepseek",
"jit-kernel"
],
"author": "TobyMint",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32746",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix][AMD] MoRI EP: drop record_stream in TBO dispatch/combine (HSA out-of-resources)",
"text": "## Motivation MoRI EP prefill with `--enable-two-batch-overlap` on MI355X aborts with `HSA_STATUS_ERROR_OUT_OF_RESOURCES`. The traceback blames whichever kernel needed scratch at that moment (for us, aiter `per_1x32_mx_quant_hip`), which is a victim, not the cause. The MoRI EP TBO path calls `record_stream(comm_stream)` on every dispatch/combine tensor. That parks each block in the caching allocator's deferred-free list until a comm-stream event retires, and with `async_finish` the compute stream never blocks on comm — so the deferred frees accumulate across layers. `reserved` grows far past the live set until allocations fail, the allocator dumps its cache via `release_cached_blocks()`, and eventually ROCr cannot find scratch and aborts the queue. On one DeepSeek-V4 rank (TP8/EP8/DP8, TBO on) a `hipMalloc`/`hipFree` replay showed **283.89 GiB of 288 GiB** at the abort. The non-EP DP TBO path already fixed exactly this — see `_TBO_PERSIST_BUF` in `layers/dp_attention.py`: > per-layer `torch.empty` churn ... kept alive by the comm-stream record_stream ballooned `reserved` to ~270GB and tripped HSA_STATUS_ERROR_OUT_OF_RESOURCES ... even though the live (allocated) working set was only ~10GB. The EP path never got the equivalent treatment. ## Modifications `python/sglang/srt/layers/moe/token_dispatcher/moriep.py` only: - Remove the four `record_stream(comm_stream)` calls in `_MoriEPDispatcherImplNormal` (`_dispatch_core` inputs/outputs, `_combine_core` inputs/output). - Add `_CROSS_STREAM_SAFETY_NOTE` documenting the failure mode and the safety argument. Unlike the DP path, `record_stream` is *removable* here rather than needing a keep-alive replacement: `dispatch_b`/`combine_b` launch on the comm stream and then run `compute_stream.wait_event(done_event)` in the same call, with no TBO yield between, so existing event ordering already covers both directions — inputs stay referenced by that frame until after the wait, and outputs are gated by the `previous_event` / `wait_stream(compute_stream)` before the next comm launch. Scope: only the dual-stream branch, which exists solely under TBO (`enable_dual_stream = is_tbo_enabled()`). Non-TBO MoRI and `_MoriEPDispatcherImplLowLatency` are unaffected. ## Accuracy Tests Not run — no numerics change, only when the allocator may recycle a block. Would appreciate a reviewer with MoRI EP CI access running the TBO accuracy job; I can run gsm8k on 2×MI355X if preferred. ## Speed Tests and Profiling Not benchmarked. Expected neutral to slightly positive — `record_stream` is off the critical path, and removing it should drop the `release_cached_blocks()` stalls (two within 40 s in the trace above). ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). (black, isort, `ruff --select=F401,F821,UP037`, codespell clean; full-ruleset ruff findings unchanged from base.) - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). (Needs a multi-GPU MoRI EP TBO prefill to surface; suggestions for a cheaper check welcome.) - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No user-facing change; rationale is in-code.) - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). (See above.) - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31560569469](https://github.com/sgl-project/sglang/actions/runs/31560569469) Latest PR Test (Extra): :x: [Run #31560569402](https://github.com/sgl-project/sglang/actions/runs/31560569402) ",
"url": "https://github.com/sgl-project/sglang/pull/32746",
"createdAt": "2026-07-29T08:05:49Z",
"updatedAt": "2026-08-13T02:11:41Z",
"timestamp": "2026-08-13T02:11:41Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"amd",
"run-ci"
],
"author": "TianDi101",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32754",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Enable gfx1250 Support",
"text": "## Motivation AMD gfx1250 is a new GPU architecture that differs substantially from the existing gfx95x and gfx942 targets SGLang already supports. Several components require gfx1250-specific paths due to architectural differences: - **No MFMA/WMMA FP4 scale instructions** (`V_WMMA_SCALE_F32_32X16X128_F4`): the existing CK/ASM `fused_moe` and fp4 dense GEMM paths produce garbage or crash. - **Triton `tl.dot(fp8, fp8)` miscompiles** for contraction dim ≥ 128 on gfx1250 (verified correct at K=64, broken at K≥128), requiring bf16 upcasting in the MLA attention Triton kernels. - **`llvm.amdgcn.raw.buffer.*` buffer instructions are absent** on RDNA4 (gfx12): the `quick_all_reduce` kernel would fail to compile. - **aiter CK-backed kernels (rope, QK RMSNorm, etc.) do not build** against the gfx1250 ROCm 7.15 image's composable_kernel. - **TileLang CK-backed buffer addressing does not compile** on gfx1250, blocking the Sinkhorn/MHC path. - **`num_stages > 1` Triton pipelines miscompile** on gfx1250, causing NaN in block-FP8 matmuls. - **ROCm SDK is pip-installed** (not at `/opt/rocm`) on this new image, requiring build and runtime path fixups for AITER, TileLang, MORI, and SGLang itself. This PR brings up gfx1250 support for DSv4, DSR1, and GPT-OSS-120B models with MXFP4/FP8 quantization. ## Modifications ### Infrastructure & Docker (`docker/rocm.Dockerfile`, `scripts/ci/amd/amd_ci_install_dependency.sh`, `python/pyproject_other.toml`) - Added a new `gfx1250-rocm7_15` Docker build stage starting from `ubuntu:24.04` with pip-installed ROCm 7.15 SDK, PyTorch 2.11, and Triton 3.7. Includes path and symlink workarounds for a pip-based (non-`/opt/rocm`) ROCm SDK. - Added `GPU_ARCH_LIST_ARG` build-arg and made `PYTORCH_ROCM_ARCH` configurable; per-arch `TRITON_COMMIT_DEFAULT` env vars. - Added gfx1250-specific AITER build path (`ENABLE_CK=0`, pip `--no-build-isolation`) and MORI build path using `CMAKE_PREFIX_PATH` for the ROCm SDK. - Added gfx1250-aware TileLang `ROCM_PATH` override during CMake build. - Updated Triton rebuild logic to preserve the installed version string (workaround for ROCm 7.15 / [rocm-systems#7643](https://github.com/ROCm/rocm-systems/issues/7643)). - Renamed `SETUPTOOLS_SCM_PRETEND_VERSION` → `SGLANG_VERSION` to avoid leaking SGLang's nightly version into AITER's version. - Added `compressed-tensors==0.16.0` pin to `runtime_common` (supports torch ≥ 2.10) and dropped the old per-`srt_hip` `==0.15.0` pin. - CI install script: ROCm-version-aware AITER rebuild (`pip --no-build-isolation` for ROCm ≥ 7.15), MORI `CMAKE_PREFIX_PATH` fix, and MORI commit extraction validation. ### Core utility (`python/sglang/srt/utils/common.py`) - Added `is_gfx1250_supported()` (cached) to detect AMD RDNA4 by `gcnArchName`. - Extended `mxfp_supported()` to include `\"gfx1250\"`. ### sgl-kernel (`sgl-kernel/setup_rocm.py`, `sgl-kernel/csrc/allreduce/quick_all_reduce_base.h`) - Added `gfx1250` to the allowed arch list in `setup_rocm.py`; gfx1250 shares the `E4M3` FP8 type and the 128KB dynamic smem budget with gfx950. - Guarded `llvm.amdgcn.raw.buffer.load/store.v4i32` declarations behind `#if !defined(__gfx1250__)`, providing stub no-ops for RDNA4 where those intrinsics don't exist. ### Attention kernels (`python/sglang/kernels/ops/attention/`) - **`decode_attention.py`**: Do not downcast `q` to `fp8` before the QK dot; instead upcast K to `q`'s dtype (`bf16`) for the dot product, avoiding the gfx1250 fp8 dot bug at K≥128. Also keep softmax weights `p` in fp32 for the P·V accumulation (recovers GSM8K accuracy ~0.82 → ~0.92 on gfx1250 R1). - **`extend_attention.py`**: Same fp8 upcast fix and fp32 P·V fix for the prefill extend/unified kernels. - **`dsv4/unified_kv_kernels/paged_decode.py`**: On gfx1250, dispatch DSv4 sparse decode attention to `aiter.ops.triton.attention.pa_decode_sparse.pa_decode_sparse` (the triton path) instead of the custom Triton implementation. - **`dsv4/unified_kv_kernels/paged_prefill.py`**: Disable the OPUS (gfx950 ASM) prefill path on gfx1250. ### MoE gate JIT kernel (`python/sglang/kernels/jit/csrc/moe/moe_fused_gate.cuh`) - Widened `__shfl_down_sync` mask from `0xFFFFFFFF` to a 64-bit `0xFFFFFFFFFFFFFFFFULL` on gfx1250 (which requires a 64-bit sync mask), while staying 32-bit on all other arches. ### Layernorm (`python/sglang/kernels/ops/layernorm/mhc.py`, `python/sglang/srt/layers/layernorm.py`) - On gfx1250, use aiter's Triton RMSNorm and `rmsnorm2d_fwd_with_add` instead of the CK-backed aiter equivalents (which do not build). - Added pure-torch (`_hc_split_sinkhorn_torch`) and Triton (`_hc_split_sinkhorn_triton`) implementations of the Sinkhorn MHC kernel, dispatched on gfx1250 since TileLang's CK-backed addressing does not compile there. ### Quantization – FP8 (`python/sglang/kernels/ops/quantization/fp8_kernel.py`, `python/sglang/srt/layers/quantization/fp8.py`, `python/sglang/srt/layers/quantization/fp8_utils.py`) - **`fp8_kernel.py`**: Added `_w8a8_block_fp8_matmul_gfx1250`, a gfx1250-specific block-FP8 matmul that: (1) upcasts fp8 operands to bf16 before `tl.dot`, (2) uses only `num_stages=1` (no SW pipelining), and (3) avoids modulo-wrap index tricks that miscompile on gfx1250. Selected automatically when `_is_gfx1250` is set. - **`fp8_utils.py`**: Force `_use_aiter = False` on gfx1250 (CK/MFMA-dependent aiter quant/GEMM kernels cannot run), directing gfx1250 to the pure-Triton block-FP8 path. Extended `triton_w8a8_block_fp8_linear` to accept pre-quantized `(input, input_scale)` from gfx1250's fused quant ops (skipping re-quantization). - **`fp8.py`**: On gfx1250, use `moe_shuffle_scale` (n32k4 layout) instead of the gfx950 `shuffle_scale` (e8m0 layout) for FP8 MoE weight scales. Respect the `AITER_FORCE_A8W4` flag to select the triton a8w4 kernel shuffle layout. ### Quantization – MXFP4 Quark schemes (`python/sglang/srt/layers/quantization/quark/schemes/`) - **`quark_w4a4_mxfp4.py`**: On gfx1250 with `AITER_FORCE_A8W4=1`, dequantize FP4 weights to bf16 at load time (since no dense fp4 GEMM is available) and run a plain bf16 linear forward. Includes a pure-Python MXFP4 e2m1 → bf16 dequantization LUT. - **`quark_w4a4_mxfp4_moe.py`**: On gfx1250, use `moe_shuffle_scale` (n32k4 layout) for W4A4 MoE weight scales (instead of `e8m0_shuffle`). Shuffle weights with the (16,16) aiter layout when `AITER_FORCE_A8W4` is set. - **`quark_w4a8_mxfp4_moe.py`**: Added a dedicated gfx1250 W4A8 MoE path (`_use_gfx1250_w4a8`): bypasses the aiter CK/ASM fused_moe entirely; uses aiter's triton `moe_gemm_a8w4` kernel with a contraction-major, gate/up-interleaved, unshuffled weight layout. Weight preparation (`_process_weights_gfx1250`) and forward (`aiter_w4a8_gfx1250_forward` in new file `aiter_mxfp4_w4a8_moe.py`) are implemented separately from the gfx950 path. ### New file: `python/sglang/srt/layers/moe/fused_moe_triton/aiter_mxfp4_w4a8_moe.py` - Implements MXFP4 W4A8 GPT-OSS MoE forward for gfx1250 via `moe_gemm_a8w4`: routing, manual activation gather (in-kernel TDM gather disabled on gfx1250), static-FP8 downcast, two-GEMM (gate_up + down) execution with fused SwiGLU, and scatter back to token order. ### RoPE (`python/sglang/srt/layers/rotary_embedding/factory.py`) - Fall back from aiter's CK rope kernels to SGLang's native rope on gfx1250 (aiter rope does not build on this arch). ### Communicator / fused quant (`python/sglang/srt/layers/communicator.py`) - On gfx1250, use `fused_rms_fp8_group_quant` (triton, group_size=hidden_size = per-token) instead of aiter's CK `rmsnorm_quant` / `add_rmsnorm_quant` kernels. ### Sampler (`python/sglang/srt/layers/sampler.py`) - Disable aiter's greedy sample kernel on gfx1250 (not supported), falling back to `torch.argmax`. ### Capture mode (`python/sglang/srt/model_executor/runner_utils/capture_mode.py`) - Disable `torch.compile` during CUDA graph capture on gfx1250 (compilation + capture together triggers issues on this arch). ### MLA attention (`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`, new `triton_qk_rmsnorm.py`) - On gfx1250 (`AITER_FORCE_A8W4=1`), substitute the aiter `fused_qk_rmsnorm` (CK-backed, doesn't build) with a self-contained Triton RMSNorm or pure-torch fallback from the new `triton_qk_rmsnorm.py`. - Fixed the RoPE skip condition to check membership in `FORWARD_ABSORB_CORE_ATTENTION_BACKENDS` (not a string equality check), preventing an incorrect empty-K bug on non-aiter backends. ### DSv4 decoder layer (`python/sglang/srt/models/deepseek_v4.py`, `deepseek_common/utils.py`, `amd/deepseek_v4_fused_mhc.py`) - Extended all `_is_gfx95_supported` guards that gate MXFP4 fused quant, MQA q/k-norm, post-attention layernorm fused rmsnorm, and the fused MHC post/pre path to also include `_is_gfx1250_supported`. - On gfx1250, the fused MHC path is enabled by `SGLANG_OPT_FUSE_MHC_POST_PRE` alone (no TileLang dependency), and the aiter Triton `try_fused_hc_post_pre` is used at all token batch sizes (no M-threshold cap, since there is no TileLang fallback). - gfx1250 does not skip the post-layernorm step after the Triton MHC kernel (since `norm_fused=False` for the aiter path). ### C4 attention indexer (`python/sglang/srt/layers/attention/dsv4/indexer.py`) - Added `_use_torch_fn` guard so the `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH` env var correctly prevents the 1D c4sl unsqueeze on gfx1250. ### Server args (`python/sglang/srt/server_args.py`) - Track `_mem_fraction_static_set_by_user`: when the user explicitly sets `--mem-fraction-static`, skip the automatic 0.85 scaling for `aiter` backend with long contexts (instead, emit a warning so the user knows what is happening). ## Accuracy Tests Passed E2E accuracy tests of DSv4, DSR1, and GPT-OSS-120B. GPT-OSS-120B > AITER_FORCE_A8W4=1 ENABLE_CK=0 SGLANG_USE_AITER=1 SGLANG_USE_AITER_MOE_GU_ITLV=1 SGLANG_USE_AITER_UNIFIED_ATTN=1 python3 -m sglang.launch_server --model-path /dockerx/model/gpt-oss-120b-w-mxfp4-a-fp8/ --trust-remote-code --tp 1 --prefill-attention-backend triton --decode-attention-backend aiter --max-running-requests 1024 --mem-fraction-static 0.9 --disable-radix-cache --page-size 64 DSR1 > ENABLE_CK=0 SGLANG_USE_AITER=1 AITER_FORCE_A8W4=1 AITER_GROUPED_FORCE_SPLIT_K1=1 SGLANG_MOE_SHUFFLE_GFX1250=1 ROCM_QUICK_REDUCE_QUANTIZATION=NONE SGLANG_AITER_FP8_PREFILL_ATTN=0 SGLANG_AITER_MLA_PERSIST=0 SGLANG_INT4_WEIGHT=0 SGLANG_MOE_PADDING=1 SGLANG_SET_CPU_AFFINITY=1 SGLANG_ROCM_FUSED_DECODE_MLA=0 SGLANG_USE_ROCM700A=1 AITER_GROUPED_CONTIGUOUS_TOKEN_THRESHOLD=16 python3 -m sglang.launch_server --model-path /dockerx/model/DeepSeek-R1-0528-MXFP4 --tensor-parallel-size 1 --trust-remote-code --host 0.0.0.0 --port 8000 --mem-fraction-static 0.90 --chunked-prefill-size 16384 --attention-backend triton --max-running-requests 128 --kv-cache-dtype auto --page-size 64 DSv4-Flash > SGLANG_DEFAULT_THINKING=1 AITER_GROUPED_CONTIGUOUS_TOKEN_THRESHOLD=16 SGLANG_DSV4_REASONING_EFFORT=max SGLANG_USE_ROCM700A=0 SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton AITER_BF16_FP8_MOE_BOUND=0 AITER_FORCE_A8W4=1 SGLANG_USE_AITER_MOE_GU_ITLV=0 SGLANG_OPT_FUSE_MHC_POST_PRE=0 ENABLE_CK=0 SGLANG_USE_AITER=1 AITER_GROUPED_FORCE_SPLIT_K1=1 python3 -m sglang.launch_server --model-path /dockerx/model/DeepSeek-V4-Flash --trust-remote-code --tp 1 --attention-backend dsv4 --page-size 256 --mem-fraction-static 0.60 --swa-full-tokens-ratio 0.15 --disable-shared-experts-fusion --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --chunked-prefill-size 8192 --cuda-graph-max-bs 256 --max-running-requests 256 --disable-radix-cache --kv-cache-dtype fp8_e4m3 ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31350355796](https://github.com/sgl-project/sglang/actions/runs/31350355796) Latest PR Test (Extra): :x: [Run #31350355703](https://github.com/sgl-project/sglang/actions/runs/31350355703) ",
"url": "https://github.com/sgl-project/sglang/pull/32754",
"createdAt": "2026-07-29T08:50:52Z",
"updatedAt": "2026-08-13T16:04:54Z",
"timestamp": "2026-08-13T16:04:54Z",
"metrics": {
"reactions": 1,
"comments": 12
},
"labels": [
"high priority",
"amd",
"dependencies",
"deepseek",
"sgl-kernel",
"run-ci",
"jit-kernel",
"bypass-fastfail"
],
"author": "akao-amd",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32755",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] Occupancy tuning for DSA indexer fp8-quant Q kernel",
"text": "Optimize fused_q_indexer_rope_hadamard_quant scheduling: 8 warps/block (256 threads) + lane-0-only weights_out write. The math path (RoPE / 128-pt Hadamard / fp8 quant) is unchanged, so output is bitwise-identical for both the V4 (kRopeFirst=false, kHadamard=true) and V3.2 (kRopeFirst=true, kHadamard=false) template configs. Block size / cap default to 8 / 16 (tuned for sm_100 / B200), overridable via -DQ_BLOCK_SIZE / -DQ_MIN_BLOCKS_PER_SM. ## Motivation `fused_q_indexer_rope_hadamard_quant` (the DSA C4 indexer fp8-quant Q kernel, shared by the V4 rope-hadamard path and the V3.2/GLM rope-first path) is latency-bound with low occupancy: the baseline launches 4-warp blocks (128 threads, `__launch_bounds__(128,16)`), which run the schedulers at only ~38% achieved warp occupancy. NCU shows the top stall is `long_scoreboard` (warps waiting on global loads) with compute pipelines under 35% utilized — so the lever is scheduling (more warps in flight to hide the load latency), not the math. ## Modifications Pure launch-config changes to the quant kernel; the math path (RoPE, 128-pt Hadamard, dynamic fp8-e4m3 quant, weight scaling) is **untouched**: 1. **8 warps/block (256 threads) + `__launch_bounds__(256,16)`** — the kernel gains `kNumWarps` / `kMinBlocksPerSM` template params (defaulting to 8 / 16), doubling the warps resident per block so more work is in flight to cover the `long_scoreboard` stall. Occupancy goes ~38% → ~86%. Overridable at compile time via `-DQ_BLOCK_SIZE` / `-DQ_MIN_BLOCKS_PER_SM`. 2. **lane-0-only `weights_out` write** — the scale/weight are warp-uniform, so the other 31 same-address stores were pure waste. Each (token, head) row is a fully self-contained warp work-item (no cross-row state), so which SM / warp / order runs it does not change its 128 output bits. Output is therefore **bitwise-identical** to the previous kernel for both template configs — V4 (`kRopeFirst=false, kHadamard=true`) and V3.2/GLM (`kRopeFirst=true, kHadamard=false`). > Note: an earlier revision of this PR also added a single-wave grid cap + > persistent grid-stride loop. Ablation (below) showed the CTA change carries > essentially all of the speedup, while the persistent path was perf-neutral at > mid batch and ~3% *slower* at B≥2048 (extra loop bookkeeping once occupancy is > saturated). It has been dropped; this PR is now just the two changes above. ## Accuracy Tests `test/registered/kernels/ops/attention/test_dsv4_indexer_quant.py` — checks both template paths against a torch reference (dequantized q within fp8-e4m3 precision; `weights_out` to atol/rtol 1e-3), plus a strided-weight test. Batch sizes span small (latency-bound) through large (occupancy-saturated). ``` 19 passed ``` (V4: B in {1,8,64,256,512,2048} x {int32,int64}; V3.2: same batches; strided weight.) Additionally verified byte-exact against the pre-change kernel across B in {1,8,64,128,256,512,1024,2048,4096,8192,16384} for both configs: q_fp8 0 bytes differ, weights_out 0 elements differ, all finite. ## Speed Tests and Profiling NCU pure-kernel time on B200 (sm_100), interleaved baseline/candidate to cancel clock drift: | B | baseline (ns) | this PR (ns) | ratio | |-------:|--------------:|-------------:|:-----:| | 1 | 3104 | 3216 | 1.04 | | 8 | 3440 | 3360 | 0.98 | | 64 | 3968 | 3792 | 0.96 | | 128 | 5184 | 4400 | 0.85 | | 256 | 7392 | 6544 | 0.89 | | 512 | 11616 | 10176 | 0.88 | | 1024 | 20208 | 17312 | 0.86 | | 2048 | 37552 | 30784 | 0.82 | | 4096 | 71952 | 57344 | 0.80 | | 8192 | 141072 | 110512 | 0.78 | | 16384 | 279296 | 216352 | 0.78 | Small batch (≤64) is launch-bound and stays at parity (the grid can't fill the SMs); the benefit appears once work fills the GPU and grows to ~22% at large batch. Wall-clock cross-check (CUDA-event HOT) matches: B=256 0.95, B=2048 0.81, B=16384 0.79 — not a profiler artifact. **CTA-size sweep** (why 256 threads), ncu duration / achieved occupancy / registers-per-thread: | config | B=256 dur / occ / regs | B=2048 dur / occ / regs | |------------|------------------------|-------------------------| | 4w / 128t | 8704ns / 66% / 21 | 38896ns / 53% / 21 | | **8w / 256t** | **7728ns / 79% / 21** | **32320ns / 87% / 21** | | 12w / 384t | 7936ns / 77% / 21 | 32912ns / 81% / 21 | | 16w / 512t | 7792ns / 80% / 21 | 32880ns / 87% / 21 | 256 threads is the knee: registers stay at 21 (not register-bound; the limiter is the resident-block cap), occupancy is already saturated, and 384/512 give no further speedup. ## Checklist - [✓] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [✓] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [✓] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [✓] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31592945179](https://github.com/sgl-project/sglang/actions/runs/31592945179) Latest PR Test (Extra): :white_check_mark: [Run #31592944948](https://github.com/sgl-project/sglang/actions/runs/31592944948) ",
"url": "https://github.com/sgl-project/sglang/pull/32755",
"createdAt": "2026-07-29T08:59:02Z",
"updatedAt": "2026-08-13T08:18:06Z",
"timestamp": "2026-08-13T08:18:06Z",
"metrics": {
"reactions": 0,
"comments": 31
},
"labels": [
"quant",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "Rainchar9119",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32798",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "DFLASH support added for XPU",
"text": "This PR add support for XPU with DFLASH algo in speculative decoding. JH...! --- ### CI States Latest PR Test (Base): :x: [Run #30608529938](https://github.com/sgl-project/sglang/actions/runs/30608529938) Latest PR Test (Extra): :x: [Run #30608529765](https://github.com/sgl-project/sglang/actions/runs/30608529765) ",
"url": "https://github.com/sgl-project/sglang/pull/32798",
"createdAt": "2026-07-29T18:09:28Z",
"updatedAt": "2026-08-13T06:24:31Z",
"timestamp": "2026-08-13T06:24:31Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"speculative-decoding"
],
"author": "ANSHUMAN87",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32882",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Accept int64 top-k IDs in FlashInfer routed MoE packer",
"text": "[by Codex] ## Summary - accept both `torch.int32` and `torch.int64` router IDs in `PackTopkIds` - keep the existing in-kernel conversion to int32, avoiding a temporary cast allocation during CUDA-graph capture - add focused coverage for both input dtypes, multiple shapes, exact reference parity, CUDA-graph capture/replay, and invalid input This fixes a Qwen3 NVFP4 startup crash on Blackwell when torch compile uses the native top-k path: ```text AssertionError: topk_ids must be int32, got torch.int64 ``` ## Regression history This is a recent regression introduced by #25702. Before #25702, `_pack_topk_for_flashinfer_routed` explicitly converted `topk_ids` with `.to(torch.int32)`. The PR replaced that helper with `PackTopkIds.execute`, whose Python wrapper asserted that IDs were already int32. The Triton kernel itself has always converted loaded IDs with `ids.to(tl.int32)`, so the wrapper rejected an input the kernel could process. The strict packer originally came from #23882, where its initial caller supplied int32 IDs. The incompatibility became a regression only when #25702 reused it for the general FlashInfer routed-MoE paths. ## Before and after Reproduced with `lmsysorg/sglang:dev-cu13` on NVIDIA B200 (`torch==2.11.0+cu130`, CUDA 13.0): Before: ```text int32_eager=PASS int64_eager=FAIL: AssertionError: topk_ids must be int32, got torch.int64 ``` After: ```text int32_eager=PASS int64_eager=PASS int64_cuda_graph=PASS ``` The patched output is bit-for-bit identical to `PackTopkIds.vanilla`. ## Why testing did not catch it PR #25702 validated the routed MoE backend and reran the Blackwell model test, but that test did not enable torch compile. The normal CUDA top-k implementation returns int32 IDs, so it satisfied the new assertion. With compile enabled for a batch-size-1 graph, `TopK.enter_torch_compile` switches to `forward_native`; native `torch.topk` returns int64 indices. There was no direct unit test defining `PackTopkIds`' accepted input dtypes, so this contract mismatch escaped both the end-to-end and unit suites. - Covered: routed FlashInfer MoE with the normal CUDA top-k path (int32) - Missed: routed FlashInfer MoE + torch compile/native top-k (int64) - Remediation in this PR: direct kernel-contract tests for int32 and int64, including int64 CUDA-graph capture/replay ## Validation ```text python3 -m pytest -q test_pack_topk_ids.py ......... [100%] 9 passed in 14.55s ``` The focused reproducer also passed eager and CUDA-graph execution on B200. ## Checklist - [x] Added regression tests - [x] Verified the failure before the change on B200 - [x] Verified eager and CUDA-graph behavior after the change on B200 - [x] Ran repository pre-commit checks cc @trevor-m for visibility on the #25702 regression analysis. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31579462931](https://github.com/sgl-project/sglang/actions/runs/31579462931) Latest PR Test (Extra): :x: [Run #31579462668](https://github.com/sgl-project/sglang/actions/runs/31579462668) ",
"url": "https://github.com/sgl-project/sglang/pull/32882",
"createdAt": "2026-07-30T04:01:03Z",
"updatedAt": "2026-08-13T06:01:11Z",
"timestamp": "2026-08-13T06:01:11Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "nvpohanh",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32902",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Fix Llama 4 FA3 local attention with paged KV cache",
"text": "[by Codex] ## Motivation Llama 4 Scout with FlashAttention 3 crashes during warmup on H100 when the KV cache uses pages larger than one token. The same workload succeeds with `--page-size 1`. This is a long-standing paged-local-attention bug rather than a new 0.5.7 regression: the explicit page-size-16 reproducer fails on 0.5.6.post2, 0.5.7, and current `main`. Related to #7936. ## Reproduction Environment: - 8x NVIDIA H100 80GB HBM3 - Llama 4 Scout 17B-16E Instruct - Tensor parallel size 8 - FlashAttention 3 - Dummy weights Run: ```bash export CUDA_LAUNCH_BLOCKING=1 python -m sglang.bench_one_batch \\ --model-path ![\"image\"](\"https://github.com/user-attachments/assets/1b50ff05-a282-4744-af12-9293248ee637\")
## Speed Tests and Profiling Results for 16K input with 50 concurrent requests ![\"image\"](\"https://github.com/user-attachments/assets/0cf878fc-52e7-4887-8785-67150b1a1522\")
## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31661553050](https://github.com/sgl-project/sglang/actions/runs/31661553050) Latest PR Test (Extra): :x: [Run #31661553025](https://github.com/sgl-project/sglang/actions/runs/31661553025) ",
"url": "https://github.com/sgl-project/sglang/pull/32941",
"createdAt": "2026-07-30T14:41:08Z",
"updatedAt": "2026-08-13T03:23:14Z",
"timestamp": "2026-08-13T03:23:14Z",
"metrics": {
"reactions": 0,
"comments": 16
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"lora",
"Multi-modal",
"deepseek",
"speculative-decoding",
"sgl-kernel",
"blackwell",
"npu",
"run-ci",
"diffusion",
"apple-silicon",
"jit-kernel"
],
"author": "vstone-w",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32944",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MoE] Fuse swiglu moe up gemm epilogue",
"text": "## Motivation **What the MoE activation costs today.** The triton fused-MoE path runs three steps per layer: an up-GEMM that writes `intermediate_cache1` at the full gate+up width `N`, a standalone `silu_and_mul` that reads that buffer and writes `intermediate_cache2` at width `N/2`, and a down-GEMM. The middle step is pure data movement dressed as compute — it round-trips the entire intermediate tensor through HBM to apply an elementwise function whose inputs the up-GEMM already had sitting in registers. ![\"image\"](\"https://github.com/user-attachments/assets/9aa2f424-5571-4bd1-bde7-115f5114a5de\")
**Why that is worth removing at small batch.** At batch 1 a MoE layer's GEMMs degenerate into `M=4..8` slivers, so the step is bound by weight bandwidth and by the intrinsic latency floor of several hundred tiny kernels rather than by arithmetic. In that regime an extra launch plus a full round trip of the intermediate buffer, paid once per MoE layer per decode step, is a visible fraction of the step. It is also a link in the dependency chain — `GEMM1 -> act -> GEMM2` — that no amount of overlap can shorten while the activation stands between the two GEMMs. **Why it cannot simply be folded in.** The activation needs `gate[i]` and `up[i]` together, but the checkpoint stores W13 as two contiguous halves, so those two values come out of output columns `i` and `i + N/2` — different tiles of the up-GEMM. A tile holding `gate[i]` does not hold `up[i]`, and vice versa. **The fix.** Permute the rows of W13 once at load so gate and up of the same intermediate channel become adjacent columns. Each GEMM output column is an independent dot product of the input with one row of W13, so reordering rows can only reorder columns — the permute is value-neutral. After it, every `(gate[i], up[i])` pair lands in the *same* output tile as an even/odd column pair, the epilogue applies `silu(gate) * up` in-register, and only the half-width result is stored. `intermediate_cache1` and the standalone activation launch disappear, and the chain shortens to `GEMM1 -> GEMM2`. ## Modifications - `python/sglang/kernels/ops/moe/fused_moe_triton_kernels.py`: `FUSE_SWIGLU` constexpr on `fused_moe_kernel`, plus the epilogue that splits the accumulator into even/odd column pairs, applies the activation, and stores the half-width result. `invoke_fused_moe_kernel` gains a `fuse_swiglu` argument and asserts the output flavors it is incompatible with. - `python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py`: allocates the half-width buffer as the up-GEMM destination, skips the standalone activation, and threads the flag through `fused_experts` / `fused_experts_impl` / `_fused_moe_kernel_sequence`. - `python/sglang/srt/layers/moe/moe_runner/triton.py`: `TritonMoeQuantInfo.fuse_swiglu_interleaved` carries the layout fact from the weight owner to both `fused_experts` entry points. - `python/sglang/srt/layers/quantization/unquant.py`: `_maybe_interleave_w13_for_fused_swiglu` performs the load-time permute, per expert so the gather temporary stays at one expert's slice. - `python/sglang/srt/environ.py`: `SGLANG_OPT_FUSE_SWIGLU_INTERLEAVED`, default off. ### Bit parity is instruction-level, not approximate The kernel being replaced is compiled with `-use_fast_math`, so matching it means matching the instructions it emits. The epilogue uses `tl.inline_asm_elementwise` to do exactly that: `mul + ex2.approx.ftz` for `__expf`, `div.approx.ftz` for the division, and `mul.ftz.f32` for the product, with a single final rounding to bf16. The accumulator is already `compute_type` at that point, which matches the reference's bf16 store to `intermediate_cache1` followed by a reload. One detail is easy to get wrong and invisible from the call site: the reference instantiates its activation functor at `float`, so silu never lands in bf16 before the multiply. An implementation that rounds silu to bf16 first double-rounds and diverges on a large fraction of inputs. The first version of this change did exactly that, and only an element-wise bit comparison caught it. ### Why this is opt-in The permuted layout is understood by the fused epilogue and by nothing else, so every consumer that reads W13 — or the pre-activation buffer — in the checkpoint's halves layout has to be excluded at load rather than trapped later. The gate requires CUDA, the plain triton runner, standard a2a, no bias, bf16, and gated silu without `alpha` / `clamp_limit` / `swiglu_limit`, and additionally excludes: - **LoRA.** `_add_lora_gate_up_delta` adds its delta into the full-width pre-activation buffer this path eliminates, and indexes gate/up as contiguous halves of it — both assumptions break. Excluding it at load matters more than asserting at run time: permuting first and asserting later would turn `--enable-lora` into a crash after warmup instead of a clean fallback. The virtual-experts LoRA path is unaffected, since it consumes `lora_a` / `lora_b` and never the base W13. - **EPLB**, which rearranges experts by copying checkpoint-layout weights into the parameter. - Runtime weight updates, for the same reason; this one is documented on the env var rather than detected. Two further guards fail loudly rather than silently: `_fused_moe_kernel_sequence` asserts the call carries no hooks and no quantization, and `invoke_fused_moe_kernel` asserts that none of `fuse_add_to_output` / `mask_output` / `fuse_sum_all_reduce` / `lora_preserve_base` are set — those address `C` at the full width `N` and would write past the row end of the half-width buffer. Filtered (`-1`) expert blocks skip the store entirely for the same reason. Under the unfused path those rows are zero-filled at full width; here the row is only `N/2` wide, so an `N`-wide zero store would run into the next token's data. The rows are never read, because the down-GEMM CTA for such a block early-exits before loading its input. ## Accuracy Tests New: `test/registered/kernels/ops/moe/test_fused_swiglu_epilogue.py`, registered `base-b-kernel-unit` / `1-gpu-large`. Both cases assert **bitwise** equality; a tolerance would defeat their purpose, since the failure mode this change risks is a plausible-looking few-ULP drift that no `assert_close` would flag. - `test_fused_matches_unfused_bitwise` drives the real `fused_experts` entry point twice — halves layout with the standalone activation, then interleaved layout with the epilogue — and requires the final MoE output to be bit-identical. Shapes cover a bs=1 decode row and a ragged token count that forces the `BLOCK_SIZE_M` tail. It also covers the filtered-expert claim: if those unwritten rows were ever read, the comparison would fail immediately. - `test_epilogue_matches_silu_and_mul_bitwise` isolates the epilogue arithmetic against `silu_and_mul` over random values plus the saturating tails and signed zeros. This is the guard against a future \"simplification\" to `tl.sigmoid` or an early bf16 cast. All three pass on H20-3e (sm90). End-to-end on `Kimi-Linear-48B-A3B-Instruct` (256 experts, `moe_intermediate_size` 1024, bf16, TP4, H20-3e): six greedy prompts at 128 new tokens each, served once with the flag off and once with it on. The two output sets are **byte-identical** (equal md5). The fused run logs `Interleaved w13 gate/up: ...` on all four TP ranks and the baseline run logs it zero times, so the comparison is not vacuously passing on an inactive fast path. ## Speed Tests and Profiling Removed, per MoE layer per forward: | | before | after | |---|---|---| | kernel launches | up-GEMM, activation, down-GEMM | up-GEMM, down-GEMM | | intermediate traffic | `intermediate_cache1` written at width `N`, read back at width `N` | none | | dependency chain | `GEMM1 -> act -> GEMM2` | `GEMM1 -> GEMM2` | Measured on `Kimi-Linear-48B-A3B-Instruct`, TP4 on H20-3e (sm90), `bench_serving --dataset-name random --num-prompts 24 --random-input-len 512 --random-output-len 256 --max-concurrency 1`, four runs per configuration: | | mean TPOT (4 runs) | output throughput | |---|---|---| | baseline | 2.49 / 2.49 / 2.49 / 2.49 ms | 362.2 - 368.6 tok/s | | fused | **2.46 / 2.46 / 2.46 / 2.46 ms** | 364.5 - 372.4 tok/s | **-1.2% mean TPOT**, with the two sets fully separated — no overlap across four runs each. Throughput moves the same way but its run-to-run spread (~1.8%) is wider than the effect, so only the TPOT column supports a conclusion here. The win scales with how launch-bound the step is, so it is largest at small batch and shrinks toward zero once the GEMMs grow big enough to dominate. sm90 with a 27-layer model at 512/256 is a conservative setting for it; the shape this was originally built for is batch-1 decode on Blackwell with a deeper MoE stack, which is not measured here. That asymmetry is the other reason this ships opt-in rather than on by default. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31598793624](https://github.com/sgl-project/sglang/actions/runs/31598793624) Latest PR Test (Extra): :white_check_mark: [Run #31598793191](https://github.com/sgl-project/sglang/actions/runs/31598793191) ",
"url": "https://github.com/sgl-project/sglang/pull/32944",
"createdAt": "2026-07-30T15:08:08Z",
"updatedAt": "2026-08-13T15:24:53Z",
"timestamp": "2026-08-13T15:24:53Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"quant",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "yuan-luo",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32947",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "GLM5.2 DSA Indexer Fallback",
"text": "## Motivation `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1` is meant to be the escape hatch that runs the FP8 MQA-logits indexer without DeepGEMM. Today it only covers the **dsv4** indexer path (`dsv4/metadata.py`, `dsv4/indexer.py`). GLM 5.2 (`GlmMoeDsaForCausalLM`, derived from `DeepseekV2ForCausalLM`) runs the **dsa** indexer instead, which calls `deep_gemm.get_paged_mqa_logits_metadata` / `fp8_paged_mqa_logits` / `fp8_mqa_logits` unconditionally. The env var is therefore a no-op for GLM 5.2, and DeepGEMM stays a hard requirement. This PR makes the flag effective on the dsa path so GLM 5.2 can be functionally validated on hosts where DeepGEMM is unavailable or where the DeepGEMM MQA-logits kernels are not usable. ## Modifications **1. Route the dsa indexer's DeepGEMM entry points through resolvers** (`dsa/utils.py`, consumed by `dsa_backend.py` and `dsa/dsa_indexer.py`): | resolver | `SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=1` | default | |---|---|---| | `resolve_paged_mqa_logits_metadata_fn()` | `sglang.kernels.ops.attention.dsv4.get_paged_mqa_logits_metadata` (JIT) | `deep_gemm.get_paged_mqa_logits_metadata` | | `resolve_fp8_paged_mqa_logits_fn()` | `_fp8_paged_mqa_logits_torch` → existing `dsv4.indexer.fp8_paged_mqa_logits_torch` | `deep_gemm.fp8_paged_mqa_logits` | | `resolve_fp8_mqa_logits_fn()` | `_fp8_mqa_logits_torch` (new) | `deep_gemm.fp8_mqa_logits` | The paged and metadata variants reuse existing kernels; only the prefill path needed a new pure-torch implementation. It accumulates per-head `relu(q·k)` weighted by the gate and applies the fp8 scale, iterating over heads to keep the `[num_q, num_kv]` intermediate small. Out-of-range positions are filled with `0.0`, matching the already-validated `dsv4.indexer.fp8_paged_mqa_logits_torch` reference. All four call sites pass `clean_logits=False`, which is the only mode implemented (asserted). **2. Remove the last hard DeepGEMM dependency on this path.** `dsa_backend.py` imported `deep_gemm` at module scope solely for `get_num_sms()`, so the module could not even be loaded without DeepGEMM installed. Added `resolve_num_sms()`, which reads `torch.cuda.get_device_properties(...).multi_processor_count` on the fallback path, and dropped the import. Both consumers only use the value to shape the schedule-metadata buffer (`new_empty(num_sm + 1, 2)`), so the raw device SM count is an acceptable stand-in. `configure_deep_gemm_num_sms` (`dsa_indexer.py:488/575`) needs no change: it yields immediately when `not ENABLE_JIT_DEEPGEMM`, and that flag is already `False` when the DeepGEMM import fails. No behavior change when the env var is unset. ## Accuracy Tests Resolver wiring verified in both directions: - **Env var set, DeepGEMM blocked from the import system** (a `sys.meta_path` finder raising `ImportError` for `deep_gemm*`, simulating a host without it): all three dsa modules import successfully and every resolver resolves — `resolve_num_sms()` → 152, metadata → JIT kernel, paged/prefill → the torch implementations. - **Env var unset:** all resolvers return the `deep_gemm` callables and `resolve_num_sms()` matches `deep_gemm.get_num_sms()` (152 == 152). ## Speed Tests and Profiling Not applicable — this is a correctness/portability fallback, not a performance path. `_fp8_mqa_logits_torch` loops over heads in Python with `torch.mm`, and `_pad_heads_for_deep_gemm` pads to 32 heads when the model has fewer, so several of those matmuls are on zero-weight padding. The default DeepGEMM path is unchanged. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31630938791](https://github.com/sgl-project/sglang/actions/runs/31630938791) Latest PR Test (Extra): :x: [Run #31630938531](https://github.com/sgl-project/sglang/actions/runs/31630938531) ",
"url": "https://github.com/sgl-project/sglang/pull/32947",
"createdAt": "2026-07-30T17:15:50Z",
"updatedAt": "2026-08-13T08:26:35Z",
"timestamp": "2026-08-13T08:26:35Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"feature",
"run-ci",
"GLM"
],
"author": "wenscarl",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32963",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NVIDIA][comm] Merge EP+MoE-TP post-experts all-reduces into one _TP reduction",
"text": "## Problem statement With `--tp-size 4 --ep-size 2`, the post-experts reduction requires two all-reduces over orthogonal groups (`_MOE_EP` then `_MOE_TP`). The allreduce fusion gate was skipping *both* once `fuse_mlp_allreduce` was published, then Observed as GSM8k accuracy 0.012 on `nvidia/DeepSeek-V4-Flash-NVFP4` with silently returning under-reduced activations with no error. Observed as GSM8k accuracy 0.012 on `nvidia/DeepSeek-V4-Flash-NVFP4` with the fused residual+LN kernel reduced over only `_MOE_EP` (2 of 4 peers) — silently returning under-reduced activations with no error ## Motivation Under hybrid MoE EP+TP (`moe_ep_size > 1` **and** `moe_tp_size > 1`), allreduce fusion silently produces under-reduced activations. There is no error — the model just returns garbage. Reproduced on `main`, 4×GB200: ```bash python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B \\ --tp-size 4 --ep-size 2 --host 0.0.0.0 --port 30000 ``` ```bash curl -X POST http://localhost:30000/v1/chat/completions \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"Qwen/Qwen3-30B-A3B\", \"messages\": [ { \"role\": \"user\", \"content\": \"Where is capital city of France? Answer in one short word.\" } ], \"chat_template_kwargs\": { \"enable_thinking\": false }, \"max_tokens\": 8, \"temperature\": 0 }' ``` The completion is garbage unless `--enforce-disable-flashinfer-allreduce-fusion` is passed. `--ep-size 4` and `--ep-size 1` are both fine. ### Root cause With `--tp-size 4 --ep-size 2`, the post-experts reduction spans **two** disjoint groups (`Qwen3MoeSparseMoeBlock.forward_normal`): ``` moe_expert_parallel_all_reduce -> _MOE_EP moe_tensor_model_parallel_all_reduce -> _MOE_TP ``` `_MOE_EP × _MOE_TP` is an orthogonal decomposition of the TP world (`moe_tp_size = tp_size // moe_ep_size // moe_dp_size`), so both reductions are needed to cover all 4 ranks. When the decoder publishes `fuse_mlp_allreduce`, `should_skip_post_experts_all_reduce()` skips **both** of them — it keys off `should_skip_mlp_all_reduce()`, which does not distinguish the EP path from the TP path. The reduction is then supposed to be absorbed by the next layer's fused residual+RMSNorm, but that kernel reduces over a **single** group: the MoE fusion workspace, which `ensure_workspace_initialized()` rendezvouses on `_MOE_EP` when `moe_ep_size > 1`. Net effect: the MoE output is reduced over 2 of the 4 required ranks. `--ep-size 4` (`moe_tp_size == 1`) and `--ep-size 1` (`moe_ep_size == 1`) are unaffected, because there a single group already covers every peer. This is pre-existing on `main` and independent of any in-flight FlashInfer work. It was surfaced while reviewing #30700 (thanks @mmangkad for the hybrid EP+TP question that led here). ### Graph illustration ## Why hybrid EP+TP specifically With `--tp-size 4 --ep-size 2`, `moe_tp_size = 4 // 2 // 1 = 2`, and the MoE output needs reducing over two orthogonal subgroups that together tile the TP world: ``` rank: 0 1 2 3 ep_rank: 0 0 1 1 <- which expert shard it holds tp_rank: 0 1 0 1 <- which FFN shard it holds _MOE_EP: {0,2} {1,3} _MOE_TP: {0,1} {2,3} ``` output needs reducing over two orthogonal subgroups that together tile the TP world: ``` rank: 0 1 2 3 ep_rank: 0 0 1 1 <- which expert shard it holds tp_rank: 0 1 0 1 <- which FFN shard it holds _MOE_EP: {0,2} {1,3} _MOE_TP: {0,1} {2,3} ``` Either reduction alone covers 2 of the 4 ranks; both are required. ``` layer N (MoE output) | layer N+1 ----------------------------------------------|------------------------ main experts --> [EP skipped] [TP skipped] -----|--> fused AR + RMSNorm | workspace group = _MOE_EP | sums {0,2} only -> WRONG this PR experts --> AR(_MOE_EP) --> AR(_MOE_TP) ----|--> RMSNorm {0,2} {0,1} | all 4 ranks summed -> ok ``` Publishing `fuse_mlp_allreduce` makes `should_skip_post_experts_all_reduce()` drop **both** reductions, because it keys off that flag without distinguishing the EP path from the TP path. The next layer's fused residual+LN then reduces over a single group — the MoE fusion workspace, which `ensure_workspace_initialized()` rendezvouses on `_MOE_EP` when `moe_ep_size > 1`. Half the peers never contribute, and the result is wrong without any error. No other configuration breaks, because no other configuration has two distinct subgroups: ``` ep_size=1 -> moe_tp_size == tp_size, _MOE_TP *is* _TP (parallel_state.py:2483) ep_size=4 -> moe_ep_size == tp_size, _MOE_EP *is* _TP (parallel_state.py:2454) ep_size=2 -> neither aliases _TP -> the only broken case ## Modifications `LayerCommunicator.should_fuse_mlp_allreduce_with_next_layer()` now returns `False` when `moe_ep_size > 1 and moe_tp_size > 1`. Fusion is only valid when one group covers every peer of the reduction, which is not the case here. Both post-experts all-reduces then run inline as usual. The guard sits before the backend-specific checks, so it covers the aiter fusion branch too — that path reduces over a single group for the same reason. Attention-side fusion is untouched: it uses the attention-TP workspace (`use_attn_tp_group=True`) whose group spans the full TP world, and it is gated on `apply_flashinfer_allreduce_fusion` alone, not on `fuse_mlp_allreduce`. New unit test `test/registered/unit/layers/test_layer_communicator_fusion_gate.py` (CPU, 3 cases): hybrid EP+TP must not fuse; pure-TP and pure-EP must still fuse. The hybrid case fails on pre-fix code and passes on the fix; the other two guard against the gate degrading to always-off. After fix: ``` {\"id\":\"0dc05777d22f4efdbbba2ba896d4d196\",\"object\":\"chat.completion\",\"created\":1785440815,\"model\":\"nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4\",\"choices\":[{\"index\":0,\"message\":{\"role\":\"assistant\",\"content\":\"Paris\",\"reasoning_content\":null,\"tool_calls\":null},\"logprobs\":null,\"finish_reason\":\"stop\",\"matched_stop\":151645}],\"usage\":{\"prompt_tokens\":25,\"total_tokens\":27,\"completion_tokens\":2,\"prompt_tokens_details\":null,\"reasoning_tokens\":0},\"metadata\":{\"weight_version\":\"default\"}} ``` ## Accuracy Tests - `--tp-size 4 --ep-size 2` on 4×GB200 with Qwen3-30B-A3B: garbage before, correct completions after, **without** `--enforce-disable-flashinfer-allreduce-fusion`. - `--ep-size 4` and `--ep-size 1` re-checked for no regression. - `python -m pytest test/registered/unit/layers/test_layer_communicator_fusion_gate.py` — 3 passed. GSM8k: ``` on GB200 python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1316 --parallel 1316 reference: (--tp-size 4 --ep-size 4) Accuracy: 0.743 Invalid: 0.001 Latency: 18.692 s Output throughput: 17213.786 token/s before fix: (--tp-size 4 --ep-size 2) Accuracy: 0.012 Invalid: 0.014 Latency: 37.922 s Output throughput: 17453.860 token/s after fix: (--tp-size 4 --ep-size 2) Accuracy: 0.777 Invalid: 0.003 Latency: 17.508 s Output throughput: 16609.659 token/s ``` Also support DSV4: ``` python -m sglang.launch_server --trust-remote-code --model-path nvidia/DeepSeek-V4-Flash-NVFP4 --ep-size 2 --tp-size 4 --enable-dp-attention --chunked-prefill-size 4096 --disable-flashinfer-autotune --swa-full-tokens-ratio 0.1 --host 0.0.0.0 --port 30000 --moe-runner-backend flashinfer_trtllm_routed --max-running-requests 1024 --mem-fraction-static 0.8 ``` gsm8k: ``` python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1316 --parallel 1316 Accuracy: 0.954 Invalid: 0.000 Latency: 90.583 s ``` --- ### CI States Latest PR Test (Base): :x: [Run #31630301030](https://github.com/sgl-project/sglang/actions/runs/31630301030) Latest PR Test (Extra): :x: [Run #31630300806](https://github.com/sgl-project/sglang/actions/runs/31630300806) ",
"url": "https://github.com/sgl-project/sglang/pull/32963",
"createdAt": "2026-07-30T19:40:02Z",
"updatedAt": "2026-08-13T07:53:27Z",
"timestamp": "2026-08-13T07:53:27Z",
"metrics": {
"reactions": 0,
"comments": 22
},
"labels": [
"bug",
"deepseek",
"run-ci"
],
"author": "wenscarl",
"state": "open",
"assignees": [
"mmangkad"
]
},
{
"id": "github:sgl-project/sglang:pull_request:32982",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "docs: sync LMSYS SGLang blog cards",
"text": "Automated sync of the LMSYS SGLang blog cards in `docs_new/index.mdx`. 🤖 Generated with GitHub Actions --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #30594717790](https://github.com/sgl-project/sglang/actions/runs/30594717790) Latest PR Test (Extra): :x: [Run #30594717633](https://github.com/sgl-project/sglang/actions/runs/30594717633) ",
"url": "https://github.com/sgl-project/sglang/pull/32982",
"createdAt": "2026-07-31T00:51:05Z",
"updatedAt": "2026-08-13T12:24:12Z",
"timestamp": "2026-08-13T12:24:12Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"documentation"
],
"author": "sglang-bot",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:32991",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(attention): add architecture-owned SM12x FA4 kernels",
"text": "## Motivation SGLang currently routes SM12x FA4 through an SM80-derived forward path. That path uses the older mainloop and does not expose the SM12x TMA dataflow, paged KV, SplitKV, or relative/shared-bias support. This PR adds an architecture-owned SM12x implementation and keeps its kernel, scheduler, and runtime policy under `sglang.kernels.ops.attention.fa4_sm120`. ## Modifications - Add TMA-based SM12x forward and paged-decode kernels with warp MMA. - Add persistent scheduling and runtime-SM-count-aware tile selection instead of assuming a fixed GPU size. - Support BF16/FP16, MHA/GQA/MQA, dense and variable-length attention, paged KV, SplitKV, global/local/noncausal masks, attention sinks, packed GQA, and relative/shared bias with shearing. - Add SM12x-owned host policy, launch-plan caching, preallocated-output paths, CUDA Graph-safe decode plans, and TVM-FFI launch fast paths. - Route SM12x through a narrow architecture dispatch bridge while leaving SM80/SM9x/SM10x kernel ownership unchanged. - Replace the previous SM80-derived SM120 fallback and add comprehensive SM12x regression tests. ## Accuracy Tests The focused SM12x test file passes in full: ```text test_flash_attention_4_sm120.py: 53 passed ``` Coverage includes dense/varlen attention, BF16/FP16, MHA/GQA/MQA, asymmetric Q/V dimensions, packed and unpacked GQA, paged KV, SplitKV, attention sinks, relative/shared bias, preallocated output, CUDA Graph replay, PDL, and specialization-cache ordering. The 684 measured prefill SKU-case pairs all passed comparison against the SGLang Triton baseline: | GPU | Cases | Maximum absolute difference | | --- | ---: | ---: | | RTX PRO 6000 Blackwell | 228 | 0.0078125 | | RTX PRO 5000 Blackwell | 228 | 0.0078125 | | GB10 (SM121) | 228 | 0.015625 | ## Speed Tests and Profiling ### Prefill The common three-way matrix contains 228 matched cases per GPU: BF16/FP16, MHA/GQA/MQA, head dimensions 32/64/96/128/192->128/256, sequence lengths 128/512/2048/6144, dense and ragged varlen inputs, and causal-global, causal-local-with-sink, and noncausal-global modes. Each case uses five warmups followed by three shuffled trials with 20 CUDA event samples per trial. PR and SM80-fallback runs are each paired with the same byte-identical Triton implementation. `PR / SM80` below is the ratio of those paired speedups, which normalizes process-level clock drift. | GPU | SMs | PR / Triton | Wins | SM80 fallback / Triton | Wins | PR / SM80 fallback | Wins | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | RTX PRO 6000 Blackwell | 188 | 3.027x | 228/228 | 1.274x | 112/228 | 2.377x | 222/228 | | RTX PRO 5000 Blackwell | 110 | 3.311x | 228/228 | 1.574x | 150/228 | 2.104x | 224/228 | | GB10 (SM121) | 48 | 3.610x | 228/228 | 2.087x | 171/228 | 1.730x | 205/228 | Across all 684 equally weighted SKU-case pairs, the PR is **3.308x over Triton with 684/684 wins**, and **2.053x over the SM80-derived fallback with 651/684 wins**.  Representative absolute latency: | GPU | Case | PR | Triton | SM80 fallback | | --- | --- | ---: | ---: | ---: | | RTX PRO 6000 | MHA, BF16, HD64, L128 | 22.66 us | 32.22 us | 60.85 us | | RTX PRO 6000 | local+sink GQA, BF16, HD128, L2048 | 31.12 us | 161.34 us | 84.64 us | | RTX PRO 6000 | MHA, BF16, HD256, L6144 | 456.29 us | 2588.77 us | 571.95 us | | RTX PRO 5000 | MHA, BF16, HD64, L128 | 14.64 us | 21.10 us | 33.42 us | | RTX PRO 5000 | local+sink GQA, BF16, HD128, L2048 | 30.37 us | 198.30 us | 91.81 us | | RTX PRO 5000 | MHA, BF16, HD256, L6144 | 745.12 us | 3487.39 us | 827.04 us | | GB10 (SM121) | MHA, BF16, HD64, L128 | 18.14 us | 24.93 us | 38.37 us | | GB10 (SM121) | local+sink GQA, BF16, HD128, L2048 | 66.75 us | 437.20 us | 158.91 us | | GB10 (SM121) | MHA, BF16, HD256, L6144 | 2108.42 us | 13977.18 us | 1823.94 us | The few PR/fallback losses are concentrated in long ragged HD256 and dense noncausal GQA, most visibly on the 48-SM device. Every measured PR case still beats Triton. Theoretical useful-compute SOL and logical-memory roofline ratios: | GPU | Compute SOL median / max | Logical-memory ratio median / max | | --- | ---: | ---: | | RTX PRO 6000 Blackwell | 14.2% / 72.4% | 10.8% / 83.3% | | RTX PRO 5000 Blackwell | 35.6% / 82.0% | 14.2% / 80.0% | | GB10 (SM121) | 34.0% / 74.3% | 40.8% / 165.1% | The logical-memory ratio uses one compulsory Q/K/V read and one output write against peak DRAM bandwidth. Repeated resident tensors can be served from cache, so values above 100% are not literal measured DRAM SOL. ### Paged decode Paged decode covers 108 cases per GPU across MHA/GQA/MQA, head dimensions 64/128/256, batch sizes 1/8/32, KV lengths 128/512/2048/8192, and page size 64. CUDA Graph replay is the kernel-focused comparison; eager and synchronized wall timings include host overhead. | GPU | Graph PR / Triton | Graph wins | Eager PR / Triton | Wall PR / Triton | | --- | ---: | ---: | ---: | ---: | | RTX PRO 6000 Blackwell | 1.311x | 92/108 | 1.432x | 1.317x | | RTX PRO 5000 Blackwell | 1.317x | 96/108 | 1.424x | 1.286x | | GB10 (SM121) | 1.214x | 90/108 | 1.377x | 1.307x | The SM80-derived fallback explicitly rejects paged KV and SplitKV, so it is reported as unsupported rather than assigned a synthetic timing. The fallback also does not support the relative/shared-bias path. ## Checklist - [x] Format changed code with the repository `prek` hooks. - [x] Add focused SM12x unit and regression tests. - [x] No user-facing documentation update is required for this internal attention-kernel implementation. - [x] Provide accuracy and speed benchmark results. - [x] Follow the SGLang code-style guidance. --- ### CI States Latest PR Test (Base): :x: [Run #31651905677](https://github.com/sgl-project/sglang/actions/runs/31651905677) Latest PR Test (Extra): :white_check_mark: [Run #31651905587](https://github.com/sgl-project/sglang/actions/runs/31651905587) ",
"url": "https://github.com/sgl-project/sglang/pull/32991",
"createdAt": "2026-07-31T02:10:01Z",
"updatedAt": "2026-08-13T06:58:43Z",
"timestamp": "2026-08-13T06:58:43Z",
"metrics": {
"reactions": 0,
"comments": 9
},
"labels": [
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "lucifer1004",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33005",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[FP8][MoE] Honor UE8M0 activation scales in Triton MoE",
"text": "## Motivation DeepSeek-V4 checkpoint metadata declares `scale_fmt: ue8m0` for its FP8 path. The official DSV4 reference applies this policy when quantizing activations before every expert Linear, so both FC1 and FC2 require power-of-two activation scales. The Triton MoE path previously dropped that policy and used continuous `amax / 448` FP32 scales instead. ## Test Made Triton MoE recognize `use_scale_ue8m0` and apply power-of-two activation scales to both FC1 and FC2, then ran the following model-level A/B evaluation. - Machine: 8×NVIDIA B200 - Model: `DeepSeek-V4-Flash` - Runtime: `lmsysorg/sglang:latest` (`sha256:7b6a35df9839fd593a94a1eaee82d7777f472225d9f3ad1f8a2e0cb2bd1785d0`) - Parallelism: TP8 / EP1 - Expert path: the checkpoint's packed FP4 expert weights were converted once to FP8 at load time with `SGLANG_DSV4_FP4_DEQUANT=1`, allowing the Triton FP8 MoE path to be evaluated - MoE: `--moe-a2a-backend none`; runtime logs confirmed the Triton MoE implementation | Benchmark / protocol | Baseline (`c039e1a7`) | PR (`47f3c98c`) | Delta | |---|---:|---:|---:| | GSM8K, temperature 0, 1 run | 1266 / 1319 (95.9818%) | 1264 / 1319 (95.8302%) | -0.1516 pp | | GPQA Diamond, temperature 1, top-p 1, max thinking, 10 paired samples/question | 1676 / 1980 (84.6465%) | 1672 / 1980 (84.4444%) | -0.2020 pp | --- ### CI States Latest PR Test (Base): :x: [Run #31609723050](https://github.com/sgl-project/sglang/actions/runs/31609723050) Latest PR Test (Extra): :x: [Run #31609722764](https://github.com/sgl-project/sglang/actions/runs/31609722764) ",
"url": "https://github.com/sgl-project/sglang/pull/33005",
"timestamp": "2026-08-12T16:15:28Z",
"metrics": {
"reactions": 0,
"comments": 14
},
"labels": [
"deepseek",
"jit-kernel"
],
"author": "kaixih",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:33006",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(dsa): use FlashInfer fused top-k for packed PAGED rows",
"text": "## Motivation @humansand - [sgl-project/sglang#32490](https://github.com/sgl-project/sglang/pull/32490) added an SGL-kernel fallback because FlashInfer previously used one `row_starts` tensor for both score and page-table windows. - [flashinfer-ai/flashinfer#4169](https://github.com/flashinfer-ai/flashinfer/pull/4169) adds `page_table_row_starts`, allowing independent score and page-table origins. It is available in [FlashInfer v0.6.17rc1](https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.6.17rc1). - The rebased PR is validated against the official FlashInfer v0.6.17rc1 CUDA 13 artifacts. It is waiting for final v0.6.17 and the aligned SGLang Python, Docker, and runtime pins. ## Modifications - Remove the packed-PAGED FlashInfer-to-SGL-kernel fallback. - Pass batch-global `row_starts` as the score-window origin and request-local `page_table_row_starts` as the page-table origin. - Preserve `row_to_batch`, `dsa_graph_safe=True`, and the SGL-only top-k-v2 policy. - Extend the fused backend-equivalence test to cover shifted packed-PAGED rows and verify both FlashInfer start tensors. This PR intentionally does not change FlashInfer dependency pins. Upstream main currently pins `0.6.15.post1`. ## Accuracy Tests Static validation at rebased head `e074a275de1290cc5f284ee16a90644d5119cd74`: ```text git diff --check python3 -m compileall -q python/sglang/srt/layers/attention/dsa/dsa_topk_backend.py test/registered/kernels/ops/attention/test_dsa_indexer.py pre-commit run --all-files ``` All passed. B200 runtime validation used: - Image: `lmsysorg/sglang:nightly-dev-cu13-20260806-ae5f8c94` - Image digest: `sha256:96cbac6a4c834f8233873b19df6f0e025658d53fb81b8baa510bc9cd3ea62b95` - `flashinfer-python==0.6.17rc1` - `flashinfer-cubin==0.6.17rc1` - `flashinfer-jit-cache==0.6.17rc1+cu130` SGLang was installed from the exact rebased PR head with `--no-deps`. Focused DSA top-k coverage: ```bash python3 -m pytest -vv --tb=short \\ test/registered/kernels/ops/attention/test_dsa_indexer.py::TestDSAIndexer::test_topk_unfused_backends_valid_selection \\ test/registered/kernels/ops/attention/test_dsa_indexer.py::TestDSAIndexer::test_topk_fused_backends_equivalence \\ test/registered/kernels/ops/attention/test_dsa_indexer.py::TestDSAIndexer::test_topk_v2_respects_topk_backend ``` ```text 3 passed, 15 warnings, 30 subtests passed in 14.34s ``` A CUDA graph smoke test captured and replayed the packed-PAGED FlashInfer path, then mutated the scores and shifted every global `row_starts` origin before replaying again: ```text capture/replay passed updated-input replay passed ``` No model-level accuracy benchmark was run. ## Speed Tests and Profiling Not run. This change selects the configured FlashInfer fused transform instead of the temporary SGL-kernel fallback; no performance claim is made. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No user-facing API change.) - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31669056967](https://github.com/sgl-project/sglang/actions/runs/31669056967) Latest PR Test (Extra): :x: [Run #31669056828](https://github.com/sgl-project/sglang/actions/runs/31669056828) ",
"url": "https://github.com/sgl-project/sglang/pull/33006",
"createdAt": "2026-07-31T03:40:14Z",
"updatedAt": "2026-08-13T09:47:23Z",
"timestamp": "2026-08-13T09:47:23Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "zianglih",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33030",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] add Ascend 950 (Atlas A5) backend paths for DeepSeek-V4",
"text": "## Motivation Ascend 950 (Atlas A5) is a new-generation NPU whose numeric capabilities differ substantially from the currently supported 910B/910C (Atlas A3) parts: it natively supports FP8-e4m3 KV cache, E8M0-scaled MXFP8 GEMM, and MXFP4 grouped matmul, and the corresponding sparse-attention / KV-compression kernels take different signatures and buffer layouts. As a result DeepSeek-V4 cannot currently be brought up on A5 from community `main` at all. This PR upstreams an A5 adaptation that has been developed and validated on an internal branch (`A5_dsv4_acc`), rebuilt as a thin overlay on top of `main`. Every A5 code path is placed behind a new `is_npu_before_atlas_a5()` device gate, so **910B/910C behavior on this branch is byte-for-byte the same as `main`**, and CUDA/ROCm paths are untouched. ## Modifications ### Device gate - `utils/common.py`: new `is_npu_before_atlas_a5()` — `lru_cache`d, returns `True` on Ascend NPUs older than Atlas A5. Queries `torch_npu.npu.current_device()` rather than device 0, so a rank pinned to another die does not create an ACL context on the wrong one. ### Attention / KV cache (DSV4) - `npu/attention/ascend_dsv4_backend.py`: on A5, dispatch sparse attention to `npu_kv_quant_sparse_attn_sharedkv{,_metadata}` (FP8 KV) instead of `npu_sparse_attn_sharedkv{,_metadata}`, passing `kv_quant_mode` / `tile_size` / `rope_head_dim`. Adds the FP8 Q path and the fused `indexer_compress_epilog` write; the lightning-indexer KV dtype becomes `float8` on A5 and stays `int8` on A3. `_forward_full` passes `cmp_ratio` only on A5 — the pre-A5 op signature is untouched. - `npu/dsv4/dsv4_memory_pool.py`: A5 KV buffers are allocated as `float8_e4m3fn` with a packed per-token row (nope + 2×rope + per-64 scale, padded to 128B) via the new `a5_packed_kv_dim`; indexer buffers become `float8_e4m3fn` / `float32` instead of `int8` / `float16`. Both `set_swa_buffer` and `set_compress_buffer` write through a shared `_write_a5_packed_kv()` helper backed by `kv_compress_epilog`. - `npu/attention/ascend_backend.py`: fix hybrid-SWA `block_tables_swa` construction to slice `req_to_token` with a stride instead of materializing the full `[bs, max_len]` gather and then striding the result. Behavior-preserving, applies to all Ascend parts. ### Quantization - **New** `npu/quantization/fp4_moe_methods.py` (624 lines): `NPUW4A4Fp4MoEMethod` plus the W4A4-MXFP expert kernels — prefill/decode fused-expert entry points, MXFP4 scale reshaping for the NPU layout, SwiGLU limit application, and the grouped-matmul chain (`w4a4_mxfp_gmm_npu` / `_w4a4_mxfp_gmm` / `_w4a8_mxfp_gmm`). Three dispatch-format handlers are provided: `..._ascend_tp`, `..._deepep`, and `npu_apply_without_routing_weights_w4a4_mxfp`. The AscendTP handler owns only the grouped-matmul chain, because `AscendTPDispatcher` already performs `npu_moe_init_routing_v2` on dispatch and `npu_moe_finalize_routing` on combine. - `npu/quantization/linear_method_npu.py`: new `npu_w8a8_block_fp8_linear()` — an MXFP8 GEMM built on `npu_dynamic_mx_quant` + `npu_quant_matmul` with `float8_e8m0fnu` scales. - `layers/quantization/fp8_utils.py`: `_dispatch_auto_backend()` selects `npu_w8a8_block_fp8_linear` on A5 only (NPU imports are lazy so non-NPU builds are unaffected). - `layers/quantization/fp8.py`: on A5, `process_weights_after_loading_block_quant` reinterprets DeepSeek's `[128, 128]` block-FP8 fp32 scales as E8M0 exponent bytes — extract the exponent field, expand it over the block it covered, pair it, and transpose both weight and scale K-major. DSV4's `wo_a` is additionally batched into the `[G, D, R]` / `[G, D/64, R, 2]` layout that `npu_transpose_quant_batchmatmul` expects. Also routes FP4 experts to `NPUW4A4Fp4MoEMethod` on NPU. ### Model - `models/deepseek_v4.py`: `_use_npu_a5_mxfp8_wo_a()` decides whether `wo_a` is allocated as an FP8 parameter; `MQALayer.__init__` and `load_weights` now share that single decision, so the load-side dequant and the allocated parameter dtype cannot disagree. Adds the A5 `npu_transpose_quant_batchmatmul` forward branch, gates the batched `npu_hc_post` on `is_npu_before_atlas_a5()`, and reinterprets FP4 E8M0 scales as `uint8` in the expert weight loader. - `models/deepseek_v2.py`: DSV4's non-hash `MoEGate` layers run the router GEMM in fp32 on NPU, mirroring what the CUDA path already does via `linear_bf16_fp32`. These layers route on near-degenerate logits and the top-k boundary flips in bf16. ## Accuracy Tests ```plaintext ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: 64 Successful requests: 128 Benchmark duration (s): 215.64 Total input tokens: 1048576 Total input text tokens: 1048576 Total generated tokens: 131072 Total generated tokens (retokenized): 131072 Request throughput (req/s): 0.59 Input token throughput (tok/s): 4862.58 Output token throughput (tok/s): 607.82 Peak output token throughput (tok/s): 3622.00 Peak concurrent requests: 84 Total token throughput (tok/s): 5470.41 Concurrency: 55.60 Accept length: 2.74 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 93662.17 Median E2E Latency (ms): 94025.97 P90 E2E Latency (ms): 143925.22 P95 E2E Latency (ms): 144095.34 P99 E2E Latency (ms): 146977.56 ---------------Time to First Token---------------- Mean TTFT (ms): 13803.26 Median TTFT (ms): 9604.62 P90 TTFT (ms): 28401.46 P95 TTFT (ms): 35824.40 P99 TTFT (ms): 39820.20 -----Time per Output Token (excl. 1st token)------ Mean TPOT (ms): 78.06 Median TPOT (ms): 72.07 P90 TPOT (ms): 120.75 P95 TPOT (ms): 128.16 P99 TPOT (ms): 135.14 ---------------Inter-Token Latency---------------- Mean ITL (ms): 78.06 Median ITL (ms): 18.21 P90 ITL (ms): 27.23 P95 ITL (ms): 28.67 P99 ITL (ms): 2281.07 Max ITL (ms): 11796.57 ``` ## Benchmarking and Profiling ```plaintext 2026-08-05 03:51:55 - evalscope - INFO: Benchmark aime26 evaluation finished. 2026-08-05 03:51:55 - evalscope - INFO: Running[eval] 100%| 1/1 [Elapsed: 59:38 < Remaining: 00:00, 3578.99s/benchmark] Running[eval]: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [59:38<00:00, 3578.99s/benchmark] 2026-08-05 03:51:55 - evalscope - INFO: Overall report table: ┌─────────┬───────────┬──────────┬──────────┬───────┬─────────┬─────────┐ │ Model │ Dataset │ Metric │ Subset │ Num │ Score │ Cat.0 │ ├─────────┼───────────┼──────────┼──────────┼───────┼─────────┼─────────┤ │ dsv │ aime26 │ mean_acc │ default │ 30 │ 0.9667 │ default │ └─────────┴───────────┴──────────┴──────────┴───────┴─────────┴─────────┘ ``` ## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation / docstrings / example guides as outlined in [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#write-documentations). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as outlined in [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR. --- ### CI States Latest PR Test (Base): :x: [Run #31682486675](https://github.com/sgl-project/sglang/actions/runs/31682486675) Latest PR Test (Extra): :x: [Run #31682486234](https://github.com/sgl-project/sglang/actions/runs/31682486234) ",
"url": "https://github.com/sgl-project/sglang/pull/33030",
"createdAt": "2026-07-31T06:49:22Z",
"updatedAt": "2026-08-13T08:33:00Z",
"timestamp": "2026-08-13T08:33:00Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"documentation",
"quant",
"deepseek",
"npu",
"run-ci"
],
"author": "AndyLi429",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33040",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[minimax m3][npu]Adaptation of Minimax M3(w8a8) for NPU platforms [2/2]",
"text": " ## Motivation Adaptation of Minimax M3 for NPU platforms ## Modifications 1、Define the complete attention implementation for MiniMax-M3 on the NPU platform, with deep optimization based on NPU‑specific features. 2、Implement speculative inference adaptation for MiniMax‑M3 based on the Eagle3 draft model. 3、Fix the W8A8 weight loading method for MiniMax‑M3. 4、Fix the memory allocation issue in CUDA graph where the number of predicted tokens is not considered when CUDA graph is enabled. **5、Complete implementation of fuseep based on SwiGLU‑OAI (THIS PR).** previous PR: https://github.com/sgl-project/sglang/pull/32941 relative kernel PR: https://github.com/sgl-project/sgl-kernel-npu/pull/629 ## Accuracy Tests ![\"image\"](\"https://github.com/user-attachments/assets/742f4d4a-b24e-4a84-9129-70952ff8a784\")
## Speed Tests and Profiling Results for 16K input with 58 concurrent requests ![\"image\"](\"https://github.com/user-attachments/assets/a83da91b-5589-4104-8e44-fbbb70af4910\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31704989069](https://github.com/sgl-project/sglang/actions/runs/31704989069) Latest PR Test (Extra): :x: [Run #31704988802](https://github.com/sgl-project/sglang/actions/runs/31704988802) ",
"url": "https://github.com/sgl-project/sglang/pull/33040",
"createdAt": "2026-07-31T07:47:34Z",
"updatedAt": "2026-08-13T13:28:08Z",
"timestamp": "2026-08-13T13:28:08Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [
"run-ci"
],
"author": "vstone-w",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33048",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Hold references to fire-and-forget tasks in disaggregation",
"text": "## Motivation Five `asyncio.create_task()` calls in `srt/disaggregation` throw the returned task away. The event loop only keeps a **weak** reference to a running task, so one whose sole reference was the `create_task()` expression can be garbage collected before it finishes — [the asyncio docs call this out explicitly](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task): > Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn't referenced elsewhere may get garbage collected at any time, even before it's done. `MMEncoder` in `encode_server.py` already handles this correctly — `self.background_tasks` at line 346, used at 1060, 2022 and 2069. This PR applies that same pattern to the classes in the package that miss it. | site | task | what a collection costs | |---|---|---| | `encode_server.py:2929` `DPDispatcher.start` | `_result_listener` | dispatcher stops consuming worker results | | `encode_server.py:2930` | `_worker_watchdog` | dead ranks are never detected | | `encode_server.py:2931` | `_cleanup_stale_mappings` | stale mooncake mappings accumulate | | `common/conn.py:1721` `CommonKVBootstrapServer._run_server` | `_cleanup_expired_entries` | `room_to_dp_rank` grows without bound | | `encode_receiver.py:1668` `MMReceiverBase` | the encode dispatch | the encode never happens; the paired `_recv_mm_data` just times out | The first four are process-lifetime loops, which is the worst case for this bug — nothing else ever holds them. `encode_receiver.py` deserves a note: the code creates the encode task and then awaits `_recv_mm_data` on a socket, so the task is awaited *indirectly*, never through the task object. A local binding would only cover the happy path — if `wait_for` raises `TimeoutError` the frame unwinds while the encode may still be in flight — so this uses the same instance-level set as the rest. ## Modifications - `DPDispatcher`: add `self.background_tasks`, register the three loops from `start()`. - `CommonKVBootstrapServer`: add `self._background_tasks`, register the cleanup loop. - `MMReceiverBase`: add `self._background_tasks`, register the encode dispatch. Each uses `set.discard` in the done callback so a double-callback can't raise. No control flow, scheduling order or awaited behaviour changes — the tasks were fire-and-forget before and remain so; they simply can no longer be collected while still running. ## Accuracy Tests Not applicable — no kernel or model forward code is touched. ## Speed Tests and Profiling Not applicable. The added work is one `set.add` per task and one `discard` on completion. ## Checklist - [x] Format your code according to the pre-commit config — run with the pinned versions: ```console $ isort --check-only ![\"image\"](\"https://github.com/user-attachments/assets/a54a7907-ea87-4913-baff-4fdea79f19d6\")
After ![\"image\"](\"https://github.com/user-attachments/assets/c6e7b382-5f46-408b-8291-254381cd2fa5\")
**Throughput** (pure-TP eager, output tok/s, single-shot): | conc | baseline (einsum) | +flag (aiter) | Δ | |------|-------------------|---------------|-----| | 4 | 31.5 | 31.2 | −1% (noise) | | 32 | 257.3 | 266.2 | **+3.5%** | | 64 | 516.2 | 532.2 | **+3.1%** | The win scales with decode batch (attention on the critical path in pure-TP); under DP-attention / high concurrency it is expected to be ~neutral (comm-bound). ## How to enable ``` SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 ``` Requires `SGLANG_USE_AITER=1` on gfx95 (MI355X). Applies to the decode path only. ## Checklist - [x] Numerically validated (bf16-equivalent vs einsum) - [x] Coherent end-to-end on MI355X TP8 - [x] GSM8K accuracy preserved - [x] Opt-in (default off); CUDA / non-gfx95 / prefill unchanged - [ ] Multi-rep + DP-attention A/B (single-shot pure-TP numbers above) --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31552829437](https://github.com/sgl-project/sglang/actions/runs/31552829437) Latest PR Test (Extra): :x: [Run #31674829342](https://github.com/sgl-project/sglang/actions/runs/31674829342) ",
"url": "https://github.com/sgl-project/sglang/pull/33313",
"createdAt": "2026-08-03T02:27:25Z",
"updatedAt": "2026-08-13T06:43:01Z",
"timestamp": "2026-08-13T06:43:01Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"amd",
"deepseek",
"run-ci"
],
"author": "karverma-amd",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33316",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "docs: note OpenAI client base_url for multi-model gateways",
"text": "## Summary The OpenAI client docs already show pointing `base_url` at a local SGLang server. This PR adds a short tip that the same `base_url` pattern works with any OpenAI-compatible multi-model gateway when you are not self-hosting SGLang, using [DaoXE](https://daoxe.com) (`https://api.daoxe.com/v1`) as one concrete example. Docs only — no runtime behavior changes. Updated: - Merged with `main`, which moved this file from `docs_new/docs/` to `docs/docs/`. The branch is now up to date and conflict-free. - Removed the `utm_*` tracking parameters that were previously on the example link; it is now a plain `https://daoxe.com`. ## Test plan - [x] No conflicts with `main` - [x] Tip landed in the renamed `docs/docs/basic_usage/` path - [x] Local SGLang example unchanged - [x] Link contains no tracking parameters - [x] Tip is clearly optional --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31626133904](https://github.com/sgl-project/sglang/actions/runs/31626133904) Latest PR Test (Extra): :x: [Run #31626133648](https://github.com/sgl-project/sglang/actions/runs/31626133648) ",
"url": "https://github.com/sgl-project/sglang/pull/33316",
"createdAt": "2026-08-03T02:35:49Z",
"updatedAt": "2026-08-12T18:10:43Z",
"timestamp": "2026-08-12T18:10:43Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"documentation"
],
"author": "seven7763",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33318",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Use SYCL kernels for topk_transform_512 on XPU",
"text": "## Motivation Route `topk_transform_512` and `topk_transform_512_v2` to the native SYCL kernels on XPU. The SYCL implementations are added in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) and exposed as `torch.ops.sgl_kernel.topk_transform_512{,_v2}`. Without this dispatch, XPU runs fall through to the JIT CUDA path and fail to load, or drop to a significantly slower vectorized PyTorch fallback. ## Modifications - `python/sglang/kernels/ops/attention/dsv4/topk.py`: - `topk_transform_512(...)`: added an `elif scores.device.type == \"xpu\"` branch that calls `torch.ops.sgl_kernel.topk_transform_512(...)`, before the existing generic JIT path. - `topk_transform_512_v2(...)`: added an early-return branch for `scores.device.type == \"xpu\"` that calls `torch.ops.sgl_kernel.topk_transform_512_v2(...)`. On XPU the `metadata` argument is accepted for API parity but ignored by the kernel. ## Accuracy Tests Correctness of the XPU dispatch is verified against the vectorized PyTorch reference at the kernel level in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) $ pytest tests/test_topk_transform.py -k topk_transform_512 ============ 450 passed, 196 deselected in 164.22s (0:02:44) ============ ## Speed Tests and Profiling Speedup of the XPU dispatch is compared against the vectorized PyTorch reference at the kernel level in the `sgl-kernel-xpu` PR (https://github.com/sgl-project/sgl-kernel-xpu/pull/366) **topk_transform_512: SYCL vs Torch** | bs | topk | seq_len | page_size | sycl_ms | torch_ms | speedup | |-----:|-------:|----------:|------------:|----------:|-----------:|----------:| | 132 | 512 | 4096 | 64 | 0.150384 | 0.373984 | 2.487 | | 132 | 1024 | 4096 | 64 | 0.146692 | 0.307580 | 2.097 | | 132 | 512 | 16384 | 64 | 0.373464 | 1.723852 | 4.616 | | 132 | 1024 | 16384 | 64 | 0.383864 | 1.741480 | 4.537 | | 132 | 512 | 65536 | 256 | 1.354756 | 6.357572 | 4.693 | | 132 | 1024 | 65536 | 256 | 1.356524 | 6.405880 | 4.722 | | 256 | 512 | 16384 | 64 | 0.659724 | 3.232632 | 4.900 | | 256 | 1024 | 16384 | 64 | 0.667940 | 3.279172 | 4.909 | | 4096 | 512 | 4096 | 64 | 3.748472 | 5.637736 | 1.504 | | 4096 | 1024 | 4096 | 64 | 3.671408 | 6.628648 | 1.805 | | 4096 | 512 | 16384 | 64 | 9.758008 | 49.686104 | 5.092 | | 4096 | 1024 | 16384 | 64 | 9.875788 | 50.707020 | 5.134 | **`topk_transform_512_v2`: SYCL vs Torch** | bs | topk | seq_len | page_size | sycl_ms | torch_ms | speedup | |-----:|-------:|----------:|------------:|----------:|-----------:|----------:| | 132 | 512 | 4096 | 64 | 0.150540 | 0.308152 | 2.047 | | 132 | 1024 | 4096 | 64 | 0.147264 | 0.304356 | 2.067 | | 132 | 2048 | 4096 | 64 | 0.108524 | 0.322608 | 2.973 | | 132 | 512 | 16384 | 64 | 0.374972 | 1.723956 | 4.598 | | 132 | 1024 | 16384 | 64 | 0.384748 | 1.750684 | 4.550 | | 132 | 2048 | 16384 | 64 | 0.399412 | 1.785212 | 4.470 | | 132 | 512 | 65536 | 256 | 1.352936 | 6.369012 | 4.708 | | 132 | 1024 | 65536 | 256 | 1.356732 | 6.406712 | 4.722 | | 132 | 2048 | 65536 | 256 | 1.359332 | 6.461468 | 4.753 | | 256 | 512 | 16384 | 64 | 0.660400 | 3.227588 | 4.887 | | 256 | 1024 | 16384 | 64 | 0.668148 | 3.285464 | 4.917 | | 256 | 2048 | 16384 | 64 | 0.697112 | 3.342768 | 4.795 | | 4096 | 512 | 4096 | 64 | 3.750656 | 5.645016 | 1.505 | | 4096 | 1024 | 4096 | 64 | 3.671252 | 6.634992 | 1.807 | | 4096 | 2048 | 4096 | 64 | 2.630264 | 8.526544 | 3.242 | | 4096 | 512 | 16384 | 64 | 9.760660 | 49.661300 | 5.088 | | 4096 | 1024 | 16384 | 64 | 9.876672 | 50.683152 | 5.132 | | 4096 | 2048 | 16384 | 64 | 10.285704 | 52.551252 | 5.109 | --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31368297417](https://github.com/sgl-project/sglang/actions/runs/31368297417) Latest PR Test (Extra): :x: [Run #31368297269](https://github.com/sgl-project/sglang/actions/runs/31368297269) ",
"url": "https://github.com/sgl-project/sglang/pull/33318",
"createdAt": "2026-08-03T02:41:45Z",
"updatedAt": "2026-08-13T06:26:24Z",
"timestamp": "2026-08-13T06:26:24Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "cyxlily",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33354",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5",
"text": " ## Motivation To use a fused GDN kernel provided by sgl-kernel-xpu for better performance than what the existing triton kernels in SGLang produce. ## Modifications - Added a subclass of `GDNAttnBackend` for XPU, which does some checks and calls the fused GDN kernel. - Added a simple dispatch mechanism for the fused GDN kernel on XPU with existing paths unchanged. ## Accuracy Tests Qwen3.5-9B, GSM8K, 200 samples Fused GDN kernel from sgl-kernel-xpu - Accuracy: 0.910 - Invalid: 0.000 Triton GDN in SGLang - Accuracy: 0.900 - Invalid: 0.000 ## Speed Tests and Profiling Qwen3.5-9B on a Intel Arc BMG60 machine ``` python -I -m sglang.bench_one_batch --batch-size 1 --input 4096 --output 1024 --model Qwen/Qwen3.5-9B --tp 1 --device xpu --mem-fraction-static 0.9 --attention-backend intel_xpu --disable-radix-cache --profile --profile-stage all --profile-activities CPU XPU --chunked-prefill-size -1 --max-prefill-tokens 20480 --max-total-tokens 20480 ``` Latency | Old | New -- | -- | -- Prefill E2E | 0.937 s | 0.808 s Prefill GDN | 165 ms | 59.17 ms Decode E2E | 0.0404 s | 0.0402 ms Decode GDN | 0.522 ms | 0.452 ms ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31660770933](https://github.com/sgl-project/sglang/actions/runs/31660770933) Latest PR Test (Extra): :x: [Run #31660770766](https://github.com/sgl-project/sglang/actions/runs/31660770766) ",
"url": "https://github.com/sgl-project/sglang/pull/33354",
"createdAt": "2026-08-03T06:33:14Z",
"updatedAt": "2026-08-13T03:30:04Z",
"timestamp": "2026-08-13T03:30:04Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "Xia-Weiwen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33366",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU][Diffusion] Enable MiniMax H3 on XPU platforms",
"text": "This PR adds key changes to cover the enabling for Minimax H3 model specific on XPU platforms, including - Mapping to XPU attention backend with optimized kernel from sgl-kernel-xpu - Necessary device dispatching/autocast codes - Runtime XPU memory D2H/H2D fix Tested serving pipelines include: - TP + layer_wise offload for both encoder/dit/vae ( similar as cookbook on cuda 5090: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3#3-serve-minimax-h3, but preferring no offload (only encoder) when device memory is enough, i.e., x4 or x8) Next plans (as follow-up PRs, for further optimizations). - Apply more fusions (like QKnorm_RoPE and silu_and_mul, to be on par with cuda fusion pathes) - More serving pipelines/parallelism to be explored and evaluated. - XPU FA backend with non-contiguous support Tested example: - T2VA example from cookbook: https://docs.sglang.io/cookbook/diffusion/MiniMax/MiniMax-H3#4-generate-video-and-audio Result: https://github.com/user-attachments/assets/09e086a7-95ae-489b-bb9b-5e48553eca91 --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #30969623195](https://github.com/sgl-project/sglang/actions/runs/30969623195) Latest PR Test (Extra): :white_check_mark: [Run #31564550674](https://github.com/sgl-project/sglang/actions/runs/31564550674) ",
"url": "https://github.com/sgl-project/sglang/pull/33366",
"createdAt": "2026-08-03T08:07:44Z",
"updatedAt": "2026-08-13T06:13:31Z",
"timestamp": "2026-08-13T06:13:31Z",
"metrics": {
"reactions": 0,
"comments": 15
},
"labels": [
"intel",
"xpu",
"run-ci",
"diffusion",
"run-ci-extra"
],
"author": "jianan-gu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33370",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] Add process-local in-memory KV indexer and Router integration",
"text": "## Motivation This PR supersedes #32662 with a smaller, dependency-free milestone for the KV Indexer proposed in #31458. The Indexer is an experimental metadata-only service that tracks which SGLang worker may reuse each content-addressed KV-cache block. KV data remains owned by the workers. The end-to-end path is: ```text SGLang worker └─ ZMQ component-aware KV events └─ kv-indexer-bridge └─ gRPC └─ process-local in-memory indexer └─ MatchExternalKvPrefix └─ experimental sgl-router ``` This revision removes Redis, Dragonfly, and Redis Cluster from the architecture. Placement metadata is held in one Indexer process and is intentionally soft state. That makes the core event → index → prefix-query path smaller, easier to run, and directly testable without an external database. Related: - RFC: #31458 - Previous KV Indexer PR: #32662 - KV-event correctness fixes: #32537, #32514 ## Scope and failure model The process-local backend is protected by an `RwLock` and bounded request work. It supports concurrent reads and serialized placement mutations in one server process. It is not a distributed or persistent store. - Run exactly one Indexer process for a deployment. - Restarting the Indexer loses all placement metadata. - There is no missed-event replay, restart fencing, worker liveness TTL, or cross-process replication in this milestone. - A disconnected Bridge reconnects transport but cannot recover events emitted while it was absent. A batch the Indexer sheds, or refuses as oversized, costs those events and a reconnect rather than terminating the Bridge. - A prefix hit only improves worker choice, so the Router degrades rather than failing when the Indexer cannot answer. An empty result, a rejection from either admission bound, a query that exceeds its deadline, and an unreachable Indexer all become no signal: the Router selects by minimum active load and logs at `WARN`. An Indexer outage therefore costs cache affinity rather than availability. Only a rejected RPC — the Router and the Indexer disagreeing on the request contract — still fails the request with `503`, because degrading would hide that disagreement from every request. An endpoint the Router could never dial is rejected at startup instead of failing every later query. The Router's local radix tree is used only when no Indexer endpoint is configured. These constraints are explicit. This PR establishes a validated single-process soft-state baseline; persistence and recovery belong in follow-up work. ## What changed ### Process-local memory backend - Replace the Redis backend with an in-process component-aware placement map. - Track `(block hash, worker, tier)` placement snapshots and worker cache specs. - Support FULL, SWA, and MAMBA reusable-prefix semantics. - Bound prefix scanning and request hash counts to keep work predictable. - Remove Redis features, scripts, configuration, and integration requirements. ### Admission and resource protection - Bound concurrent prefix queries in the Indexer. - Shed queries whose caller deadline has already expired before backend work. - Bound concurrent external Indexer queries in the Router. - Add configurable Router timeout and max-inflight controls. - Rate-limit rejection logs with doubling cadence to avoid log storms. ### Experimental Router integration ```bash sgl-router \\ --model-id ![\"image\"](\"https://github.com/user-attachments/assets/c50bc331-9bac-480b-8385-cacb869444bc\")
## Modifications ## Accuracy Tests config | GSM8K 1319 accuracy | -- | -- cp | 0.944 cp+tbo | 0.936 ## Speed Tests and Profiling I/O = 128k/1k conc | cp TPUT | tbo TPUT | ΔTPUT | cp TTFT | tbo TTFT | ΔTTFT | cp TPOT | tbo TPOT | ΔTPOT -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 2 | 9536 | 9424 | −1.2% | 4813 | 4417 | −8.2% | 22.25 | 22.97 | +3.2% 4 | 15534 | 15713 | +1.2% | 7663 | 7047 | −8.0% | 25.57 | 25.86 | +1.1% 8 | 22449 | 23349 | +4.0% | 13479 | 12258 | −9.1% | 32.51 | 31.96 | −1.7% 16 | 28333 | 29651 | +4.6% | 25237 | 22501 | −10.8% | 47.65 | 47.11 | −1.1% I/O = 50k/1, prefill only conc | cp TPUT | tbo TPUT | ΔTPUT | cp TTFT | tbo TTFT | ΔTTFT -- | -- | -- | -- | -- | -- | -- 2 | 44133 | 49299 | +11.7% | 2141 | 1941 | −9.3% 4 | 45040 | 49594 | +10.1% | 4042 | 3711 | −8.2% 8 | 45410 | 50311 | +10.8% | 8003 | 7266 | −9.2% 16 | 45536 | 51859 | +13.9% | 15683 | 13823 | −11.9% ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31569928864](https://github.com/sgl-project/sglang/actions/runs/31569928864) Latest PR Test (Extra): :x: [Run #31569928786](https://github.com/sgl-project/sglang/actions/runs/31569928786) ",
"url": "https://github.com/sgl-project/sglang/pull/33480",
"createdAt": "2026-08-04T04:49:43Z",
"updatedAt": "2026-08-13T14:33:09Z",
"timestamp": "2026-08-13T14:33:09Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"deepseek",
"run-ci",
"bypass-fastfail"
],
"author": "At1a8",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33502",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "add prefill graph support on cpu",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31692644323](https://github.com/sgl-project/sglang/actions/runs/31692644323) Latest PR Test (Extra): :x: [Run #31692643880](https://github.com/sgl-project/sglang/actions/runs/31692643880) ",
"url": "https://github.com/sgl-project/sglang/pull/33502",
"createdAt": "2026-08-04T06:55:57Z",
"updatedAt": "2026-08-13T10:48:05Z",
"timestamp": "2026-08-13T10:48:05Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"dependencies",
"sgl-kernel"
],
"author": "CaoE",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33514",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[UT][NPU]Add npu unit test for dsv4",
"text": " ## Motivation Add npu unit test for dsv4 ## Modifications Add test/registered/npu/dsv4/test_dsv4_allocator.py, test/registered/npu/dsv4/test_dsv4_common_hooks.py, test/registered/npu/dsv4/test_dsv4_memory_pool.py, test/registered/npu/dsv4/test_dsv4_req_to_token_pool.py, and test/registered/npu/dsv4/test_dsv4_rope.py ## Accuracy Tests N/A ## Speed Tests and Profiling N/A ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31662610231](https://github.com/sgl-project/sglang/actions/runs/31662610231) Latest PR Test (Extra): :x: [Run #31662609897](https://github.com/sgl-project/sglang/actions/runs/31662609897) ",
"url": "https://github.com/sgl-project/sglang/pull/33514",
"createdAt": "2026-08-04T08:29:54Z",
"updatedAt": "2026-08-13T03:02:00Z",
"timestamp": "2026-08-13T03:02:00Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "luz3633",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33554",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add new spec-dec support and quant recipe for Nano v3",
"text": "## Motivation Add support for Nemotron 3.5 including DFlash and DSpark spec-dec ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/33554",
"createdAt": "2026-08-04T13:09:55Z",
"updatedAt": "2026-08-13T17:58:06Z",
"timestamp": "2026-08-13T17:58:06Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"quant",
"speculative-decoding",
"new-model"
],
"author": "rystewart-nvidia",
"state": "open",
"assignees": [
"b8zhong"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33559",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[kernel] add triton moe TMA up support",
"text": "## Motivation Following PR #10567 (\"Opt fused triton moe: add tma for down proj kernel\"), which added TMA support for the down-projection (second MoE GEMM), this PR extends TMA support to the up-projection (first MoE GEMM, a.k.a. gate_up). Runtime couples up/down TMA via c_sorted = down_moe_use_tma. Since c_sorted is a tl.constexpr, different values produce different kernel binaries. The old tuning script always used c_sorted=False for up, which doesn't match runtime when down TMA is enabled. Solution: Two-Round Coupled Tuning When --enable-tune-up-tma is set: Round 1 (Down): Measure down no-TMA vs TMA for each config. Extract down_use_tma_map = {BLOCK_SIZE_M: bool}. Round 2 (Up): Look up c_sorted = down_use_tma_map[BLOCK_SIZE_M] from Round 1. Measure up no-TMA vs TMA with the correct c_sorted matching runtime. Final: Select best BLOCK_SIZE_M by up_time + down_time, output both configs with USE_TMA flags. Code Changes (1 file: tuning_fused_moe_triton_sep.py) benchmark_config: Added tune_round (\"both\"/\"down\"/\"up\") and down_use_tma_map params. Conditionally builds only the kernels needed for each round. tune: Two-round loop when enable_up_tma=True; single-round unchanged when False. benchmark: Two-step (down then up) for single-config path. Key Properties Default behavior (no --enable-tune-up-tma) is unchanged — only down TMA tuned, up c_sorted=False. Total kernels per config: 4 (2 in round 1 + 2 in round 2) ## Modifications 1. sglang/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py 2. sglang/python/sglang/srt/layers/moe/moe_runner/triton.py 3. sglang/python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe.py ## Accuracy Tests ![\"image\"](\"https://github.com/user-attachments/assets/59baa7b8-9423-469f-83a6-a8ff207d3480\")
## Speed Tests and Profiling Add --enable-tune-up-tma to tune script：sglang/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py. It uses the original tune‑down TMA behavior by default when the flag is not provided. ![\"image\"](\"https://github.com/user-attachments/assets/45353028-3a0b-40ba-aa65-efa666f60642\")
baseline: python3 -m sglang.bench_serving --backend sglang --host localhost --port 25000 --dataset-path ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --num-prompts 500 --random-output-len 1000 --warmup-requests 10 --random-input-len 20000 --max-concurrency 500 --tokenizer /ufs/models/Qwen3.5-397B-A17B-FP8/ --request-rate 0.8 --model /ufs/models/Qwen3.5-397B-A17B-FP8/ --random-range-ratio 1 ![\"image\"](\"https://github.com/user-attachments/assets/b8c1b5ce-eb7d-4a79-9241-6d33e00cbd09\")
use down TMA ohly: ![\"image\"](\"https://github.com/user-attachments/assets/17473a79-50b5-454f-8a7b-d6bb9de1659c\")
use up TMA + down TMA: ![\"image\"](\"https://github.com/user-attachments/assets/19b823c6-cfcc-453f-9e14-1d0f946e85e7\")
results: ![\"image\"](\"https://github.com/user-attachments/assets/4a4c7352-a13f-4a79-8483-e80e6fd9d218\")
As summarized from the table, combining up TMA + down TMA achieves better performance than using down TMA alone. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31573435275](https://github.com/sgl-project/sglang/actions/runs/31573435275) Latest PR Test (Extra): :x: [Run #31573435123](https://github.com/sgl-project/sglang/actions/runs/31573435123) ",
"url": "https://github.com/sgl-project/sglang/pull/33559",
"createdAt": "2026-08-04T14:00:15Z",
"updatedAt": "2026-08-13T05:29:33Z",
"timestamp": "2026-08-13T05:29:33Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"jit-kernel",
"bypass-fastfail",
"run-ci-extra"
],
"author": "xieminghe1",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33561",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Model] Support Ling-3.0-flash (BailingMoeV3)",
"text": "## Motivation Day-0 support for [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) (`BailingMoeV3ForCausalLM`, `model_type=bailing_hybrid`): a hybrid MoE architecture interleaving KDA linear attention (with a safe-gate lower bound) and MLA, with MTP (NEXTN) and DSPARK speculative decoding support. ## Modifications - `BailingMoeV3ForCausalLM` model (`bailing_hybrid` config): KDA + MLA hybrid layers, 512-expert MoE, and MTP (NEXTN). - DSPARK support for the KDA hybrid path: graph-folded draft proposal, fused KDA chain-verify kernel, and DSA token-to-KV pool sizing. - KDA safe gate propagation through the Triton decode/target-verify kernels and their call sites, matching fused verification. - `ling3` reasoning and tool-call parsers. - BF16, FP8, compressed-tensors INT4, and native MXFP4 checkpoint support on Hopper and Blackwell. - Exactness, fusion-gate, quantization, and parser tests. ## Validation All serving runs kept decode CUDA Graph enabled and used the default graph sizing. No run used `--disable-cuda-graph` or `--cuda-graph-max-bs-decode`. ### Quantized checkpoints Full GSM8K uses 1,319 questions. Performance uses 80 exact-length random requests at ISL/OSL 8192/1024 with concurrency 1 and 16; throughput includes input and output tokens per second per GPU. | GPU | format | GSM8K | stop rate | throughput/GPU (c1 / c16) | P50 TTFT ms (c1 / c16) | P50 TPOT ms (c1 / c16) | default graph range | |---|---|---:|---:|---:|---:|---:|---:| | H200 | INT4 | 95.30% | 100% | 1,159 / 7,184 | 205.96 / 1,891.74 | 3.68 / 8.16 | through batch 218 | | H200 | MXFP4 | 96.29% | 100% | 709 / 5,215 | 305.23 / 2,725.52 | 6.05 / 11.17 | through batch 227 | | B200 | INT4 | 96.74% | 99.70% | 482 / 1,344 | 641.41 / 6,202.62 | 8.72 / 48.01 | through batch 305 | | B200 | MXFP4 | 96.29% | 99.62% | 785 / 7,933 | 127.97 / 1,117.16 | 5.60 / 7.98 | through batch 314 | INT4 selects Marlin on H200 and the graph-compatible Triton WNA16 path on B200. MXFP4 routed experts use the native FlashInfer runner on both GPUs. The B200 MXFP4 checkpoint also contains block-FP8 dense/shared projections, which use the Triton FP8 GEMM backend to avoid an unsupported FlashInfer FP8 tactic without converting the routed experts. BF16 and FP8 were rerun on H200 and B200 after the quantization changes. All four combinations served with CUDA Graph enabled, completed generation, and passed the 20-example GSM8K regression gate with 100% stop rate. ### Speculative decoding Ling-3.0-flash BF16, 4x B200 TP4, `temperature=1.0 / top_p=0.95`, thinking on, full GSM8K, `max_tokens=8192`: | spec decoding | score | stop rate | error rate | |---|---:|---:|---:| | DSPARK | 96.66% | 99.77% | 0.00% | | MTP (NEXTN) | 96.44% | 99.62% | 0.00% | Non-stop finishes are long reasoning outputs reaching `max_tokens`, not repetition loops. ## Checklist - [x] Format code with pre-commit. - [x] Add unit tests and register them in CI. - [x] Validate serving correctness, stop rate, and accuracy on H200 and B200. - [x] Publish deployment recipes and benchmark results in #34363. --- ### CI States Latest PR Test (Base): :x: [Run #31694712753](https://github.com/sgl-project/sglang/actions/runs/31694712753) Latest PR Test (Extra): :x: [Run #31694712651](https://github.com/sgl-project/sglang/actions/runs/31694712651) ",
"url": "https://github.com/sgl-project/sglang/pull/33561",
"createdAt": "2026-08-04T14:07:18Z",
"updatedAt": "2026-08-13T12:58:49Z",
"timestamp": "2026-08-13T12:58:49Z",
"metrics": {
"reactions": 9,
"comments": 10
},
"labels": [
"documentation",
"quant",
"amd",
"deepseek",
"sgl-kernel",
"blackwell",
"npu",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "JustinTong0323",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33569",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] [Diffusion] Support MiniMax H3 on Ascend NPU's",
"text": "## Motivation Extend the native MiniMax-H3 pipeline introduced in [#33275](https://github.com/sgl-project/sglang/pull/33275) to Ascend NPU. MiniMax-H3 packs real multimodal tokens together with sequence-parallel alignment padding. Its attention path is therefore variable-length even when request batch size is one: the real-token segment and the alignment-padding segment must remain isolated. ## Scope and related work - [#33275](https://github.com/sgl-project/sglang/pull/33275) provides the upstream MiniMax-H3 model and pipeline implementation that this PR extends. - [#33366](https://github.com/sgl-project/sglang/pull/33366) is the related XPU enablement PR. This PR does not include XPU backend code; shared paths use `current_platform` so the Ascend changes remain isolated and existing CUDA/ROCm behavior is preserved. - Experimental NPU RoPE, SwiGLU, and gate kernel paths were removed after benchmarking (since they did not improve performance). The final PR keeps the attention and platform-integration changes required for MiniMax-H3 on Ascend. ## Modifications ### Ascend Flash Attention - Add `forward_varlen` using `torch.ops.npu.npu_fused_infer_attention_score` with the `TND` layout and actual packed-sequence lengths. - Preserve independent packed-sequence boundaries. - Repack strided Ulysses Q/K/V views with one allocation instead of three independent contiguous copies. - Preserve the segmented fallback for causal attention. ### Laser Attention - Enable `laser_attn` for MiniMax-H3. - Add variable-length handling so real tokens cannot attend alignment padding. - Run attention only for the real-token segment, retain padding rows for sequence-parallel shape alignment, and zero-fill their output instead of computing attention for them. - Preserve BF16 activation range when the FP16 Laser kernel is used by compensating the Q/K softmax scale and restoring the value/output scale. - Limit this BF16 compensation to MiniMax-H3's packed variable-length path; the existing dense Laser path keeps its previous behavior. ### NPU dependencies and media output - Install `attentions*.whl` alongside the `deep_ep` and `sgl_kernel_npu` wheels from the pinned `sgl-kernel-npu` release archive in both the NPU Docker image and NPU CI. - Scope the missing-`attentions` layer-normalization warning to NPU. Layer normalization falls back to its native implementation; explicitly selecting Laser Attention without the package raises an actionable import error. - Install `ffmpeg` in the NPU Docker and CI environments and fail MiniMax-H3 startup early when required media tools are absent, preventing the late output loss reported in [#33927](https://github.com/sgl-project/sglang/issues/33927). - Keep these dependency checks and warnings isolated from CUDA, ROCm, XPU, MUSA, and CPU. ### Ascend platform integration - Use `current_platform` for shared memory residency, RNG, autocast, and denoising dispatch. - Add NPU support to component offload/cache-release handling while preserving CUDA/ROCm behavior. - Keep Qwen3-VL position IDs and rotary embeddings on the execution device only on NPU. - Avoid enabling Ascend-specific behavior on XPU, MUSA, or CPU. - Add the validated eight-NPU TP2 + SP4 launch command and a four-NPU TP2 + SP2 variant to the MiniMax-H3 cookbook. Both commands list the benchmarked Cache-DiT configuration explicitly; the cookbook records 55.07 s and 103.57 s end-to-end latency respectively. ### FL2VA keyframe encoding - Register `MiniMaxH3Qwen3VLEncoder.encode_ids` as an FSDP2 forward method when text-encoder CPU offload is active. - Ensure the root FSDP hooks materialize non-layer parameters, including the Qwen3-VL vision `Conv3d` patch embedding, before image-conditioned encoding. - Prevent the `aten.convolution` mixed `torch.Tensor`/`DTensor` failure observed by FL2VA requests. - Keep the change platform-independent and limited to FSDP-wrapped encoders that expose `encode_ids`; non-FSDP execution is unchanged. ## Accuracy Tests Validated end-to-end 5-second MiniMax-H3 T2VA generation with both Ascend Flash Attention and Laser Attention on 8 Ascend NPUs. Image-conditioned FL2VA was also validated with Laser Attention after the FSDP2 custom-forward fix. Server startup: ```bash SGLANG_CACHE_DIT_ENABLED=true \\ SGLANG_CACHE_DIT_FN=2 \\ SGLANG_CACHE_DIT_BN=1 \\ SGLANG_CACHE_DIT_WARMUP=4 \\ SGLANG_CACHE_DIT_RDT=0.4 \\ SGLANG_CACHE_DIT_MC=4 \\ SGLANG_CACHE_DIT_TAYLORSEER=true \\ SGLANG_CACHE_DIT_TS_ORDER=2 \\ HCCL_BUFFSIZE=256 sglang serve \\ --model-path ./weights/MiniMax-H3/ \\ --model-type diffusion \\ --model-variant fl2va \\ --num-gpus 8 \\ --tp-size 2 \\ --sp-degree 4 \\ --attention-backend laser_attn \\ --port 30088 ``` For a four-NPU host, use TP2 + SP2: ```bash SGLANG_CACHE_DIT_ENABLED=true \\ SGLANG_CACHE_DIT_FN=2 \\ SGLANG_CACHE_DIT_BN=1 \\ SGLANG_CACHE_DIT_WARMUP=4 \\ SGLANG_CACHE_DIT_RDT=0.4 \\ SGLANG_CACHE_DIT_MC=4 \\ SGLANG_CACHE_DIT_TAYLORSEER=true \\ SGLANG_CACHE_DIT_TS_ORDER=2 \\ HCCL_BUFFSIZE=256 sglang serve \\ --model-path ./weights/MiniMax-H3/ \\ --model-type diffusion \\ --model-variant fl2va \\ --num-gpus 4 \\ --tp-size 2 \\ --sp-degree 2 \\ --attention-backend laser_attn \\ --port 30088 ``` Use `--attention-backend fa` to validate the Ascend Flash Attention path. T2VA request: ```bash curl -sS -X POST http://127.0.0.1:30088/v1/videos \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"MiniMaxAI/MiniMax-H3\", \"prompt\": \"At night, while their owner sleeps in a bedroom, three cats march in loudly playing tiny brass instruments, then abruptly file out.\", \"seconds\": 5, \"task\": \"t2va\", \"conditions\": [], \"target\": { \"short_edge\": 768, \"aspect_ratio\": \"16:9\", \"duration_seconds\": 5.0 }, \"num_outputs_per_prompt\": 1, \"num_inference_steps\": 50, \"flow_shift\": 12.0, \"audio_flow_shift\": 3.0, \"seed\": 1101 }' ``` Result: https://github.com/user-attachments/assets/6fd97552-b6ce-446c-b8d2-c26f61f7c91f FL2VA request: ```bash curl -sS -X POST http://127.0.0.1:30088/v1/videos \\ -H \"Content-Type: application/json\" \\ -d '{ \"model\": \"MiniMaxAI/MiniMax-H3\", \"prompt\": \"The supplied frame continues with calm, natural motion and synchronized ambient sound.\", \"seconds\": 5, \"task\": \"fl2va\", \"conditions\": [ { \"type\": \"image\", \"uri\": \"./Forest.jpg\", \"role\": \"keyframe\", \"frame_index\": 0 } ], \"target\": { \"short_edge\": 768, \"aspect_ratio\": \"auto\", \"duration_seconds\": 5.0 }, \"num_outputs_per_prompt\": 1, \"num_inference_steps\": 50, \"flow_shift\": 12.0, \"audio_flow_shift\": 3.0, \"seed\": 2101 }' ``` Forest.jpg ![\"Forest\"](\"https://github.com/user-attachments/assets/3fa648b0-4edc-44cc-8e7d-fa378f48fdae\")
Result: https://github.com/user-attachments/assets/0cad1f72-86f4-45c2-bc70-5f8d4831ef97 ## Speed Tests and Profiling The Ascend FA path performs one packed `TND` attention operation using actual sequence lengths. For strided Ulysses tensors, Q/K/V are repacked through one allocation. Laser Attention computes only the real-token segment and zero-fills alignment-padding outputs, avoiding the previous short-segment fallback calculation. FA vs Laser Attention on T2VA: | Attention backend | Topology | End-to-end generation time | Speedup vs. 8-NPU FA | | --- | --- | ---: | ---: | | Ascend Flash Attention | 8 NPUs, TP2 + SP4 | 69.51 s | 1.00× | | Laser Attention | 8 NPUs, TP2 + SP4 | 55.07 s | 1.26× | | Laser Attention | 4 NPUs, TP2 + SP2 | 103.57 s | — | On the same eight-NPU topology, Laser Attention reduced end-to-end generation time by 20.8% compared with Ascend Flash Attention. The attention measurements use the same 5-second T2VA request at 1344×768, 124 frames, 24 FPS, 50 inference steps, and seed 1101. The launch commands list the measured Cache-DiT configuration explicitly: `FN=2`, `BN=1`, `WARMUP=4`, `RDT=0.4`, `MC=4`, TaylorSeer enabled, and TaylorSeer order 2. Experimental NPU RoPE, SwiGLU, and gate kernels were disabled; they are not present in the final PR. Laser Attention performance: T2VA: ![\"image\"](\"https://github.com/user-attachments/assets/69915664-c35a-49f0-8a5d-f635902cc3f9\")
FL2VA: ![\"image\"](\"https://github.com/user-attachments/assets/797bcc84-98fc-4238-ac51-1a0d12947bf1\")
## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci`. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31495020724](https://github.com/sgl-project/sglang/actions/runs/31495020724) Latest PR Test (Extra): :x: [Run #31495020587](https://github.com/sgl-project/sglang/actions/runs/31495020587) ",
"url": "https://github.com/sgl-project/sglang/pull/33569",
"createdAt": "2026-08-04T15:43:02Z",
"updatedAt": "2026-08-13T07:51:02Z",
"timestamp": "2026-08-13T07:51:02Z",
"metrics": {
"reactions": 0,
"comments": 14
},
"labels": [
"documentation",
"npu",
"run-ci",
"diffusion"
],
"author": "OrangeRedeng",
"state": "open",
"assignees": [
"ping1jing2"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33572",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Cosmos3] Add cosmos3 Reasoner to llm only inference",
"text": " ## Motivation Cosmos3 Omni model has 2 transformer towers stacked on top of each other for video generation. First tower is understanding tower that is also trained for text/visual reasoning. Metrics are included in model card under Reasoning Benchmarks: https://huggingface.co/nvidia/Cosmos3-Nano#reasoning-benchmarks . This PR adds support for Reasoner only inference. ## Modifications Should be merged after: https://github.com/sgl-project/sglang/pull/31590 * Added new model type cosmos3_omni * Modified DefaultModelLoader with allow_patterns_overrides argument to allow reading Diffusers format checkpoints * Added Cosmos3 specific checkpoint key mappings * Extended Qwen3VL execution path checks with new model type (Cosmos3 Reason follows Qwen3VL architecture) ## Accuracy Tests Simple request to make sure model generates reasonable output, since rest of the implementation relies on Qwen3VL integration. ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31727450906](https://github.com/sgl-project/sglang/actions/runs/31727450906) Latest PR Test (Extra): :x: [Run #31727450532](https://github.com/sgl-project/sglang/actions/runs/31727450532) ",
"url": "https://github.com/sgl-project/sglang/pull/33572",
"createdAt": "2026-08-04T15:58:33Z",
"updatedAt": "2026-08-13T17:48:06Z",
"timestamp": "2026-08-13T17:48:06Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"diffusion"
],
"author": "Zhylkaaa",
"state": "open",
"assignees": [],
"change": "new"
},
{
"id": "github:sgl-project/sglang:pull_request:33602",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [GLM5] Add opt-in PTPC FP8 projections on gfx950",
"text": "## Motivation The GLM-5.2 MXFP4 checkpoint keeps four attention projections in BF16: - fused `q_a + kv_a` - `q_b_proj` - `o_proj` This PR adds an opt-in FP8 path for those projections on gfx950. PTPC means per-token activation scaling and per-channel weight scaling. BF16 remains the default and fallback path. ## Modifications - Cache bpreshuffled FP8 projection weights while retaining the original BF16 parameters. - Use one flag, `SGLANG_DSA_FP8_PROJ_GEMM`, to switch all four projections between BF16 and PTPC FP8. - Reuse fused quantized inputs for `q_b_proj` and `o_proj` where available. - Keep fused `q_a + kv_a` on tuned Aiter BF16 for M≤512; use PTPC FP8 for M>512. - Limit the path to GLM-5.2 on gfx950 with Aiter enabled. ## Accuracy Tests GLM-5.2-MXFP4, TP4 MI355X: | Validation | Result | | --- | ---: | | PTPC GSM8K, 1319 examples | 0.933 | ## Speed Benchmarks GLM-5.2-MXFP4, TP4 MI355X, 8k input / 1k output. A. BF16 — `SGLANG_DSA_FP8_PROJ_GEMM=0`: | concurrency | TTFT (ms) | ITL (ms) | E2EL (ms) | output tok/s | | ---: | ---: | ---: | ---: | ---: | | 4 | 1027.49 | 11.773 | 13386.33 | 305.72 | | 8 | 1637.90 | 13.517 | 16432.02 | 498.29 | | 16 | 2915.32 | 16.130 | 21704.23 | 754.64 | | 32 | 5769.21 | 20.130 | 30926.24 | 1059.44 | | 64 | 10870.96 | 27.013 | 48250.20 | 1357.82 | B. PTPC FP8 — `SGLANG_DSA_FP8_PROJ_GEMM=1` (Δ vs BF16): | concurrency | TTFT (ms) | Δ | ITL (ms) | Δ | E2EL (ms) | Δ | output tok/s | Δ | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 4 | 972.22 | −5.38% | 11.820 | +0.40% (within noise) | 13387.67 | +0.01% | 305.76 | +0.01% | | 8 | 1615.05 | −1.40% | 13.560 | +0.32% (within noise) | 16423.83 | −0.05% | 498.43 | +0.03% | | 16 | 2877.66 | −1.29% | 15.917 | −1.32% | 21314.90 | −1.79% | 768.28 | +1.81% | | 32 | 5566.14 | −3.52% | 19.793 | −1.67% | 30263.03 | −2.14% | 1082.37 | +2.16% | | 64 | 10420.25 | −4.15% | 26.633 | −1.41% | 47426.84 | −1.71% | 1382.11 | +1.79% | **Baseline** = Image `rocm/sgl-dev:v0.5.16-rocm720-mi35x-20260804` plus these 6 PRs PR #30575: Triton sparse MLA prefill/decode PR #30715: fused indexer query Hadamard + FP8 quant PR #30808: gfx950 dense-MHA short-context fallback PR #31323: fused shared-expert append PR #31324: skip unnecessary dense-regime decode indexer work PR #33620: packed-MXFP4 absorbed K/V BMM enabled **This PR** = Baseline + the flag flip ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31623100648](https://github.com/sgl-project/sglang/actions/runs/31623100648) Latest PR Test (Extra): :x: [Run #31623100214](https://github.com/sgl-project/sglang/actions/runs/31623100214) ",
"url": "https://github.com/sgl-project/sglang/pull/33602",
"createdAt": "2026-08-04T21:06:40Z",
"updatedAt": "2026-08-12T17:34:22Z",
"timestamp": "2026-08-12T17:34:22Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"quant",
"deepseek"
],
"author": "Raiden-Makoto",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33604",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix Whisper transcription for audio over 30 seconds",
"text": "## Summary Whisper's feature extractor pads or truncates input to its fixed 3000-mel-frame (30-second) encoder window, so `/v1/audio/transcriptions` silently drops everything after 30 seconds when the serving layer sends a long file as one generation. This change: - splits long Whisper audio into contiguous chunks of at most 30 seconds; - places each seam at the quietest 100 ms RMS window within the final second of the chunk, matching the former vLLM speech-to-text behavior; - runs non-streaming chunks concurrently and stitches results in audio order; - streams chunks sequentially with one final finish frame; - offsets `verbose_json` segment timestamps by each chunk's start time; - preserves fused language auto-detection and reports the first chunk's detected language; - gates chunking through the transcription adapter, so Qwen3-ASR and other adapters remain unchanged. Each chunk receives an independent sampling-parameter dictionary because the multimodal processor consumes transcription-only keys while preparing decoder prompts. ## Validation - Added CPU unit coverage for energy-aware split points, contiguity, maximum duration, waveform preservation, short-audio behavior, ordered stitching, chunk failures and aborts, streaming, fused auto-detection, spaceless scripts, and verbose timestamp offsets. - Added GPU endpoint coverage using a 40-second clip whose speech begins after Whisper's 30-second encoder window: JSON transcription, `verbose_json` timestamps past 30 seconds, and streaming. - The audio splitter tests pass locally (5 tests). - Changed Python files pass AST, isort, Ruff, Black, codespell, CI registry, merge-conflict, and private-key checks; `compileall` and `git diff --check` pass. - Live validation with `openai/whisper-large-v3` on a 57.06-second file produced the complete transcript, including segments after 30 seconds, with one streaming stop frame. Split points and explicit-language output matched vLLM 0.11.0. --- ### CI States Latest PR Test (Base): :x: [Run #31727027933](https://github.com/sgl-project/sglang/actions/runs/31727027933) Latest PR Test (Extra): :hourglass_flowing_sand: [Run #31727027759](https://github.com/sgl-project/sglang/actions/runs/31727027759) ",
"url": "https://github.com/sgl-project/sglang/pull/33604",
"createdAt": "2026-08-04T21:15:55Z",
"updatedAt": "2026-08-13T17:54:10Z",
"timestamp": "2026-08-13T17:54:10Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"run-ci-extra"
],
"author": "shenxiul",
"state": "open",
"assignees": [
"JustinTong0323"
],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:33614",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] Fix Dspark state divergence across TP rank",
"text": " ## Motivation Related to: https://github.com/sgl-project/sglang/issues/33289 To fix https://github.com/sgl-project/sglang/issues/33289 bug, upgrade NCCL to the newest version (2.30.7). When TP > 1, Dspark makes serveral sampling decisions on reach rank: 1. The draft Markov chain samples proposal tokens step by step (with in-graph philox noise introduced since https://github.com/sgl-project/sglang/pull/33298) 2. Target verify derives `correct_len` / `bonus` / `cap_trim_lens` 3. Prefill samples `next_token_ids` In order to maintain the speedup, SGLang skips cross-rank sync of sampled tokens by default (I found the comment here: https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/sampler.py#L496). So nothing forces these values to agree. Once one rank commits a different token or accept length, sequence lengths and KV state drift apart and a later collective deadlocks (demonstrated by fault injection in the Accuracy Tests section below). ## Modifications Broadcast rank 0's sampling decisions before they affect state. The idea follows the existing `verify_lens` handling in `dspark_planner.py` and the EAGLE proposals in https://github.com/sgl-project/sglang/issues/29003 and https://github.com/sgl-project/sglang/pull/31478: 1. Broadcast every Markov proposal token in **both** eager and graph-folded sampling. Sync only at the end of a draft block is too late because each step depends on the previous token. 2. Broadcast `correct_len`, `bonus`, and `cap_trim_lens` in the eager verify path and the captured verify epilogue, before finalization, token output, or KV commit. Separate `all_reduce(MIN)` calls could combine fields from different ranks. 3. Broadcast prefill `next_token_ids` before entering the draft path. `DsparkTpSync` in `dspark_tp.py` (newly added file) uses the TP group's PyNCCL communicator. Process-group collectives cannot be captured in CUDA graphs, and using the model's existing TP communicator preserves collective ordering across eager and captured execution. With DP attention, it uses `attn_tp_group`, matching `verify_lens_broadcast_group` in original `dspark_planner.py`. TP=1 is a no-op. The reason why I don't match RNG seeds is because ranks can still diverge if they consume different numbers of random values, and it does not cover non-RNG differences. In addition, for TP > 1, this add `gamma + 3` broadcasts of a `[bs]` tensor per decode step, all within the captured graphs. ## Accuracy Tests All tests below on following configs: - 2 DGX Sparks - TP=2 - DeepSeek-V4-Flash-0731 with 131k context - Decode CUDA graphs and folded sampling enabled - Add https://github.com/sgl-project/sglang/pull/33407 code. It fixes an unrelated bug for SM12x sparse-MLA topk dispatch crash at boot. Dspark + DeepSeek V4 Flash cannot start on DGX Sparks without it. - **NCCL 2.30.7** for SGLang's PyNCCL communicator via `SGLANG_NCCL_SO_PATH` (torch's bundled 2.28.9 wedges this workload's graph/eager mix with or without this patch; a separate problem from this fix) For accuracy test, we run`python3 -m sglang.test.few_shot_gsm8k --num-questions 200` (temp=0): accuracy 0.965, invalid 0.000. This is expected since there's no mathematical changes. To demonstrate the failure mode this patch addresses, we fault-injected a rank-local accept divergence into both builds: rank 1 lowers every accept length by one (`correct_len = torch.clamp(correct_len - 1, min=0)`, NCCL 2.30.7 in both _runs). **_Note: the fault reproduction code has been removed before submitting PR._** Without the fix, the server deadlocks before finishing startup warmup. With the fix, the same injection is harmless: rank 0's broadcast overwrites the divergent values, the server boots, and a temp=0 request returns the expected output. As a stability check on the fix itself (its broadcasts run inside the captured decode graphs), I sent 42 sequential chat requests with `ignore_eos=true` and `max_tokens=512` (30 at temp=1, then 12 at temp=0) and probed `/health_generate` after every request with a 180s client timeout, so a stuck collective surfaces as a failed request instead of a silent hang. As the result, all 42 returned HTTP 200 with `finish_reason=length`. 21504 tokens generated in total and mean acc_len during the temp=1 run was 2.86 (block_size=4). Same for temp=0, neither of them occur any error. ## Speed Tests and Profiling Test on 24 natural prompts, 256 output tokens, concurrency=1, temp=1, same setup as above: | Requests | Output throughput | TTFT mean/median/P90 (ms) | TPOT mean/median/P90 (ms) | |---|---|---|---| | 24 | 31.9 tok/s | 157.6 / 153.7 / 178.6 | 30.8 / 31.7 / 35.5 | Comparing with pre-fix TP baseline is impossible because it cannot survive under such load. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31302168114](https://github.com/sgl-project/sglang/actions/runs/31302168114) Latest PR Test (Extra): :x: [Run #31302167999](https://github.com/sgl-project/sglang/actions/runs/31302167999) ",
"url": "https://github.com/sgl-project/sglang/pull/33614",
"createdAt": "2026-08-04T23:11:47Z",
"updatedAt": "2026-08-13T08:16:31Z",
"timestamp": "2026-08-13T08:16:31Z",
"metrics": {
"reactions": 0,
"comments": 12
},
"labels": [],
"author": "JackZeng0208",
"state": "open",
"assignees": [
"hnyls2002"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33623",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Kimi K3] Fuse MLA gate projection into QKV-A GEMM",
"text": "Replaces #33521, which was automatically closed when its base branch was deleted. Rebasing to main as requested in the review. ## Motivation Kimi-K3 MLA computes QKV-A and the TP-local output gate from the same hidden states. They currently run as separate GEMMs. Fuse them to reduce projection cost while keeping the gate output TP-local. ## Modifications - Merge unquantized BF16/FP16 QKV-A and g_proj weights after loading. - Run one merged projection and stash the gate for the pre-o_proj gate multiply, preserving quantized and LoRA fallback paths. - Select tile_m=32 for the K3 merged [3648, 7168] fused-A shape so its 114 CTAs fit in one H200 wave. - Add CPU equivalence and SM90 CUDA Graph tests for M=1/8/16 at K3 TP8 dimensions. ## Accuracy Tests - H200 model unit test: 2/2 passed. - Merged QKV-A and gate outputs matched separate F.linear references. - Attention-layer A/B correctness passed for M=1/4/8/16 and context 512/4096. - Full pre-commit passed on the three changed files. ## Speed Tests and Profiling Single H200, BF16, FlashInfer MLA, CUDA Graph, K3 TP8 single-rank dimensions: | Decode M | Context 512 | Context 4096 | | ---: | ---: | ---: | | 1 | +4.47% | +2.90% | | 4 | +2.99% | +2.25% | | 8 | +1.28% | +1.01% | | 16 | +3.32% | +1.57% | The merged fused-A kernel dropped from 16.8-18.0 us to 14.9-15.1 us after the tile selection fix. ## Checklist - [x] Format the code with the project pre-commit hooks. - [x] Add unit tests for the new functionality. - [x] Provide accuracy and performance results. - [x] Keep quantized and LoRA paths on their existing fallback. --- ### CI States Latest PR Test (Base): :x: [Run #31550507014](https://github.com/sgl-project/sglang/actions/runs/31550507014) Latest PR Test (Extra): :x: [Run #31550506888](https://github.com/sgl-project/sglang/actions/runs/31550506888) ",
"url": "https://github.com/sgl-project/sglang/pull/33623",
"createdAt": "2026-08-05T00:23:24Z",
"updatedAt": "2026-08-13T05:40:28Z",
"timestamp": "2026-08-13T05:40:28Z",
"metrics": {
"reactions": 1,
"comments": 2
},
"labels": [
"run-ci",
"jit-kernel",
"bypass-fastfail",
"run-ci-extra"
],
"author": "Hayden727",
"state": "closed",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33634",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] Add test for --dllm-fdfo",
"text": " ## Motivation This PR adds a test case to verify that the `--dllm-fdfo` (First-Done-First-Out) scheduling flag takes effect and improves performance for Diffusion LLM models on Ascend NPU backend. When enabled, FDFO prioritizes requests that complete their diffusion steps earlier, reducing waiting time and improving hardware utilization. ## Modifications - Add `TestLLaDA2MiniFDFO` test class in `test_ascend_llada2.py` - Benchmark LLaDA2.0-mini model with 128 prompts (3584 input tokens, 1024 output tokens) - Compare performance between `--dllm-fdfo` enabled and disabled (`--no-dllm-fdfo`) - Assert that enabling FDFO yields: - Lower mean TTFT (Time-To-First-Token) - Higher total throughput - Lower P99 TTFT - Register test with NPU CI pipelines (stage-b-test-2-npu-a3 and nightly-2-npu-a3) ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31663841896](https://github.com/sgl-project/sglang/actions/runs/31663841896) Latest PR Test (Extra): :x: [Run #31663841776](https://github.com/sgl-project/sglang/actions/runs/31663841776) ",
"url": "https://github.com/sgl-project/sglang/pull/33634",
"createdAt": "2026-08-05T03:00:21Z",
"updatedAt": "2026-08-13T03:25:23Z",
"timestamp": "2026-08-13T03:25:23Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"npu"
],
"author": "longxin9715",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33647",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[KDA] Add FlashInfer CAKE prefill and decode backends",
"text": "## Motivation This PR integrates exported FlashInfer CAKE KDA kernels into SGLang and adds an explicit `cake` backend for recurrent KDA prefill and decode: ```bash --mamba-ssm-dtype bfloat16 --linear-attn-backend cake ``` The shared flag selects CAKE for both phases. Existing per-phase flags remain authoritative when a mixed configuration is requested, and the verify backend remains independently selected. The serving-native packed decode API is being upstreamed in [flashinfer-ai/flashinfer#4378](https://github.com/flashinfer-ai/flashinfer/pull/4378). This PR depends on that PR landing and on SGLang consuming a FlashInfer revision containing it. H=12 prefill support is already available through [flashinfer-ai/flashinfer#4351](https://github.com/flashinfer-ai/flashinfer/pull/4351), and the overall work is tracked in [flashinfer-ai/flashinfer#4254](https://github.com/flashinfer-ai/flashinfer/issues/4254). ## Modifications - Add `cake` to the linear-attention backend choices and dispatcher. - Make `--linear-attn-backend cake` select CAKE for both prefill and decode, while preserving explicit per-phase overrides. - Integrate CAKE prefill through `flashinfer.kda.recurrent_kda`. - Integrate the Kimi-K3 T=1 serving hot path through the independent `flashinfer.packed_kda_decode` API. - Forward SGLang's post-convolution packed QKV, raw gate/beta logits, caller-owned state pool, and original int32 state indices directly. - Avoid Q/K/V splitting and copying, gate/beta preprocessing, and state gather/scatter on the covered packed-decode path. - Preserve `-1` CUDA-graph padding rows and update active state slots in place. - Log once when the serving-native packed CAKE route is actually exercised. - Bypass Kimi-K3's separate fused decode path when `cake` is explicitly selected, ensuring that the requested backend is exercised. ## Packed decode contract and fallback The CAKE packed path is selected only for the frozen FlashInfer contract: - `T=1`, `H=HV=12`, `K=V=128`; - BF16 packed QKV, raw gate/beta, output, and recurrent state; - FP32 `A_log[12]` and `dt_bias[1536]`; - packed QKV `[B, 4608]`; - state pool `[N, 12, 128, 128]` with compact inner dimensions and a positive valid outer slot stride; - contiguous CUDA int32 state indices `[B]`, with unique in-bounds active indices and `-1` for inactive graph rows; - `scale=1/sqrt(128)`, `lower_bound=-5`, and `1 <= B <= 65535`; - exact SM100 with CUDA >= 12.8 or SM103 with CUDA >= 12.9. Unsupported shapes, dtypes, layouts, architectures, GQA, a missing packed API, and ReplaySSM route through the existing Triton packed implementation. Triton errors are propagated rather than hidden. CAKE prefill similarly retains Triton for contracts requiring speculative or state tracking, intermediate-state returns, T=1-only batches, unsupported dimensions, or prefill CUDA-graph capture. Decode CUDA graphs are supported by the packed CAKE path. ## Validation Candidate revisions: - SGLang PR head: `a6e3452c1cf42ada75fbc301ec99439651d1dfd2`; - validated SGLang snapshot: `2c86a82dab900f4fb1589951e13a6cd8eb74bd68`; - identical tracked tree for both SGLang revisions: `a50bae03fbfb97711d5189fa778155ab82f7e095`; - FlashInfer performance commit: `89a5d3fcee8f8ddcff9da5547d095ffd73c1ef79`; - FlashInfer #4378 head after non-kernel review cleanup: `826ed5f2fdc301440aa1ec799066f2fa3dfb63d5`. GB300 / SM103 validation: ```text FlashInfer JIT specification tests: 19 passed FlashInfer packed KDA GPU tests: 26 passed SGLang CPU adapter/selector tests: 9 passed SGLang packed KDA GPU tests: 7 passed, 11 deselected Pre-commit on changed files: passed ``` The SGLang GPU tests cover B=1/8/64/128, the exact H=12 serving contract, non-contiguous packed-QKV outer strides, non-identity state slots, `-1` padding, padded state-pool outer strides, bitwise preservation of untouched slots, forwarding of original state/index pointers, GQA fallback, and CUDA Graph replay with changed inputs and indices through stable pointers. FlashInfer #4378 additionally validates B=1/8/16/31/32/64/128/256/512 on B200 and GB300, current-stream execution, production strides, a 512-step FP64 diagnostic, and synccheck/memcheck with zero errors on both architectures. ### Exported public API vs exact SGLang packed Triton Strict CUPTI, cold L2, 30 rounds per backend, same inputs, and alternating AB/BA order: | GPU | Direct B=1–512 geomean | CUDA Graph B=1–512 geomean | |---|---:|---:| | B200 / SM100 | `1.1058x` | `1.0863x` | | GB300 / SM103 | `1.0708x` | `1.0526x` | The complete per-batch tables, including retained low points, are in FlashInfer #4378. ## Same-configuration Kimi-K3 E2E Performance and accuracy use the same server launch configuration: - Kimi-K3 TP8/DCP8 on 8-GPU GB300 allocations; - BF16 recurrent state, HiCache, and full decode CUDA graphs; - identical server and workload arguments apart from the selected linear-attention backend and unavoidable port values; - random seed 42. This is a same-configuration cross-run comparison, not a current-source paired A/B in one allocation. The recorded Triton baseline used SGLang `0a3efffc2f7fe8e0898473d86320d006973fdd71` and FlashInfer `6b4ecd9d459caf389ce8c4a92bbb491214f2eac8` (Slurm `371396`). The CAKE run used the validated SGLang snapshot `2c86a82dab900f4fb1589951e13a6cd8eb74bd68` and FlashInfer `826ed5f2fdc301440aa1ec799066f2fa3dfb63d5` (Slurm `378402`). Both used the same GB300 SKU and the configuration above. ### Serving performance Workload: 128 requests, 128 input tokens, 512 requested output tokens, concurrency 64, greedy decoding, one warmup, cache flush before each measured run, and three repetitions per backend. | Metric | Triton runs | CAKE runs | Triton median | CAKE median | Speedup | |---|---|---|---:|---:|---:| | Decode output tok/s | `2008.276 / 1979.075 / 1959.839` | `1992.671 / 1985.439 / 1967.150` | `1979.075` | `1985.439` | `1.0032x` | | Total tok/s | `2510.345 / 2473.844 / 2449.798` | `2490.838 / 2481.799 / 2458.938` | `2473.844` | `2481.799` | `1.0032x` | | Median E2E latency (ms) | `16301.127 / 16542.684 / 16705.231` | `16432.781 / 16490.611 / 16645.532` | `16542.684` | `16490.611` | `1.0032x` | | Median TTFT (ms) | `650.728 / 805.163 / 741.787` | `666.114 / 748.696 / 808.828` | `741.787` | `748.696` | `0.9908x` | | Median TPOT (ms) | `30.656 / 30.806 / 30.934` | `30.794 / 30.814 / 30.987` | `30.806` | `30.814` | `0.9998x` | | Median ITL (ms) | `30.459 / 30.661 / 30.799` | `30.402 / 30.642 / 30.821` | `30.661` | `30.642` | `1.0006x` | All repetitions completed with zero request errors. The CAKE route log was checked only after real generation requests had exercised the server. All six runs completed 128/128 requests with the expected 16,384 input and 65,536 output tokens. The output-throughput median is nominally `1.0032x`, but the decode TPOT median is effectively flat (`0.9998x`). Given the cross-run method and this noise-scale delta, these data do not establish a material E2E speedup. ### GSM8K accuracy Protocol: 200 examples, five-shot completion API, greedy sampling, maximum 512 tokens, 128 client threads, and three repetitions against the same server process used for the performance measurements. | Triton runs | CAKE runs | Triton median | CAKE median | Delta | Gate | |---|---|---:|---:|---:|---| | `0.970 / 0.980 / 0.985` | `0.975 / 0.985 / 0.975` | `0.980` | `0.975` | `-0.005` | PASS | The quality gate requires a CAKE median of at least `0.97` and a median regression of no more than `0.01` relative to Triton. ## Checklist - [x] Add the serving-native packed CAKE decode route. - [x] Preserve explicit Triton fallbacks for unsupported contracts. - [x] Add targeted CPU and GPU correctness tests. - [x] Run pre-commit on all changed files. - [ ] Merge FlashInfer #4378. - [ ] Update SGLang's FlashInfer dependency to a revision containing #4378. - [x] Report same-configuration Kimi-K3 performance and GSM8K results. --- ### CI States Latest PR Test (Base): :x: [Run #31091488033](https://github.com/sgl-project/sglang/actions/runs/31091488033) Latest PR Test (Extra): :no_entry_sign: [Run #31224311301](https://github.com/sgl-project/sglang/actions/runs/31224311301) ",
"url": "https://github.com/sgl-project/sglang/pull/33647",
"createdAt": "2026-08-05T04:53:30Z",
"updatedAt": "2026-08-13T17:09:43Z",
"timestamp": "2026-08-13T17:09:43Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"high priority",
"kernel",
"release-highlight"
],
"author": "yyihuang",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33672",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DSV4] Support raw-index output in TopK v2",
"text": "## Summary Allow TopK v2 to populate a raw-index output buffer. DSV4 sparse prefill internally allocates `c4_sparse_raw_indices` even when `--enable-return-indexer-topk` is disabled. The previous `raw_indices is None` gate therefore forced the InfX sparse-prefill path to fall back to TopK v1. When `--enable-return-indexer-topk` is enabled, the capture-only temporary buffer could additionally take precedence over the functional sparse-prefill buffer. This change makes the sparse-prefill buffer canonical, passes it through TopK v2, and lets the capturer copy the same result afterward. ## Changes - Pass the raw-output buffer to `topk_transform_512_v2`. - Prioritize `core_metadata.c4_sparse_raw_indices`; the capturer reads the same result afterward. - Add a regression test covering TopK v2 with sparse prefill and capture enabled. No CUDA kernel, ABI, metadata format, default, or HiSparse behavior changes. Related: #32771 contains the same buffer-ordering change; #33288 owns chunked-path metadata refresh. ## Testing - DSV4 indexer unit tests: **8 passed, 12 subtests passed** - TopK v2 GPU subset on GB300: **36 passed, 208 deselected** - `pre-commit run --all-files` A prior GB300 microbenchmark (8,192 queries, max C4 length 2,048, `topk=1024`, metadata planning excluded) measured **0.1023 ms** for v1 and **0.0646 ms** for v2 (**36.9% lower kernel latency**). --- ### CI States Latest PR Test (Base): :x: [Run #31662394592](https://github.com/sgl-project/sglang/actions/runs/31662394592) Latest PR Test (Extra): :x: [Run #31662394475](https://github.com/sgl-project/sglang/actions/runs/31662394475) ",
"url": "https://github.com/sgl-project/sglang/pull/33672",
"createdAt": "2026-08-05T07:48:14Z",
"updatedAt": "2026-08-13T04:44:04Z",
"timestamp": "2026-08-13T04:44:04Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "weireweire",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33673",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MiniMax-M3] Add MiniMax-M3 DSpark support",
"text": "## Summary This PR adds MiniMax-M3 DSpark speculative decoding support with MiniMax-M3 as the target model and MiniMax-M3-DSpark as the draft model. MiniMax-M3-DSpark is a draft model and cannot be served standalone. When it is paired with MiniMax-M3, several integration points are required across auxiliary hidden-state capture, VL wrapper forwarding, target-verify metadata, MiniMax sparse attention geometry, DP padding handling, and DFLASH/DSPARK config loading. This PR wires those pieces together and validates two-node TP16 serving for MiniMax-M3 + MiniMax-M3-DSpark. It also keeps the implementation compatible with the latest MiniMax-M3 NPU and EAGLE3 changes on `main`. ## Modifications ### MiniMax-M3 target model - Add `set_dspark_layers_to_capture` to `MiniMaxM3SparseForCausalLM`. - Reuse the existing layer-capture setup shared with EAGLE3 so DSpark and EAGLE3 mark `_is_layer_to_capture` consistently. - Require explicit DSpark layer IDs, since DSpark needs deterministic auxiliary hidden-state capture points. - Preserve compatibility with the latest MiniMax-M3 NPU and EAGLE3 changes from `main`. ### MiniMax-M3 VL wrapper - Add `set_dspark_layers_to_capture` to the VL wrapper. - Reuse the same capture setup as EAGLE3 for layer marking. - Forward `aux_hidden_states` to the logits processor when auxiliary capture is enabled. - Keep the tuple guard for warmup/idle forwards, where the inner language model may return a plain hidden-state tensor instead of `(hidden_states, aux_hidden_states)`. ### ForwardBatch TARGET_VERIFY metadata - Initialize TARGET_VERIFY `extend_*` metadata in `ForwardBatch` instead of synthesizing it inside the MiniMax sparse attention backend. - Assign zero-length verify rows for DP-padded requests so `sum(extend_seq_lens_cpu)` reflects the real non-padded verify token count. - Add `target_verify_q_cap()` to provide a capture-safe fixed query bound for CUDA Graph replay. - Add `build_uniform_target_verify_layout()` to build uniform target-verify layouts with both device tensors and host-side metadata. - Avoid CUDA Graph capture-time device-to-host sync and CPU-to-CUDA copies by carrying `verify_lens_cpu` and `total_verify_tokens`. ### MiniMax sparse attention - Consume TARGET_VERIFY geometry from `ForwardBatch`. - Support both ragged target-verify layouts and uniform CUDA Graph capture layouts through `build_ragged_target_verify_geometry`. - Resolve extend metadata through a shared path that returns `cu_seqlens`, cache sequence lengths, prefix lengths, and host-side per-request verify lengths. - Trim DP-padded `q` and `idx_q` rows before entering the sparse kernel. - Reject DP-padded TARGET_VERIFY requests when per-request verify lengths are unavailable, instead of falling back to `batch_size * verify_len`. - Keep the latest NPU sparse attention paths, metadata cache, lazy GPU sparse-kernel import, and EAGLE3-compatible behavior from `main`. ### DFLASH / DSPARK config handling - Add a fallback for reading `sliding_window` from the raw `config.json` when the loaded Transformers config exposes it as `None`. - Preserve MiniMax-M3-DSpark compatibility with DFLASH/DSPARK draft-model loading. ## Error cases fixed This PR fixes the following MiniMax-M3 + MiniMax-M3-DSpark serving failures: ```text ValueError: DFLASH sliding_attention layers require config.sliding_window. ``` ```text ValueError: Model MiniMaxM3SparseForConditionalGeneration implements neither set_dspark_layers_to_capture nor set_dflash_layers_to_capture, one of which is required for DFLASH/DSPARK. ``` ```text AttributeError: 'NoneType' object has no attribute 'device' ``` ```text ValueError: DFLASH target_hidden feature dim mismatch. Expected shape [N, 36864] (num_context_features=6, hidden_size=6144), but got shape=(..., 6144). ``` ```text RuntimeError: CUDA error: operation failed due to a previous error during capture. ``` It also fixes two target-verify correctness issues: - DP-padded TARGET_VERIFY batches no longer use `batch_size * verify_len` as the real sparse-kernel token count. - Ragged and uniform target-verify paths now use shared per-request geometry, so padded rows are trimmed before the MiniMax sparse kernel. ## Validation ### Static checks ```bash python3 -m py_compile \\ python/sglang/srt/layers/attention/minimax_sparse_backend.py python/sglang/srt/model_executor/forward_batch_info.py python/sglang/srt/speculative/dflash_utils.py python/sglang/srt/models/minimax_m3_vl.py python/sglang/srt/models/minimax_m3.py ``` ### Two-node TP16 serving Validated with MiniMax-M3 as the target model and MiniMax-M3-DSpark as the DSPARK draft model: ```bash python3 -m sglang.launch_server \\ --model-path /models/MiniMax-M3 \\ --served-model-name MiniMax-M3 \\ --reasoning-parser auto \\ --tool-call-parser auto \\ --trust-remote-code \\ --tp-size 16 \\ --nnodes 2 \\ --node-rank <0-or-1> \\ --dist-init-addr 192.168.1.81:20000 \\ --speculative-algorithm DSPARK \\ --speculative-draft-model-path /models/MiniMax-M3-DSpark \\ --speculative-dspark-block-size 8 \\ --chunked-prefill-size 8192 \\ --mem-fraction-static 0.80 \\ --dtype bfloat16 \\ --host 0.0.0.0 \\ --port 31000 ``` ### End-to-end serving results | Metric | Input 8K / Output 512 | Input 32K / Output 512 | |---|---:|---:| | Successful requests | 256 | 256 | | Request throughput | 0.14 req/s | 0.05 req/s | | Input token throughput | 1140.76 tok/s | 1518.18 tok/s | | Output token throughput | 73.01 tok/s | 24.29 tok/s | | Peak output throughput | 80.00 tok/s | 75.00 tok/s | | Total token throughput | 1213.76 tok/s | 1542.47 tok/s | | Effective concurrency | 15.76 | 15.91 | | DSpark accept length | 3.39 | 3.40 | | Mean E2E latency | 110538.41 ms | 335299.96 ms | | Median E2E latency | 97803.34 ms | 286638.45 ms | | P99 E2E latency | 266417.01 ms | 876994.82 ms | | Mean TTFT | 7546.60 ms | 54129.68 ms | | Median TTFT | 5068.49 ms | 42576.83 ms | | P99 TTFT | 64566.62 ms | 239261.61 ms | | Mean TPOT | 201.55 ms | 550.24 ms | | Median TPOT | 177.17 ms | 444.06 ms | | P99 TPOT | 511.15 ms | 1607.82 ms | ### Accuracy | Benchmark | Examples | Sampling | Repeats | Score | Latency | |---|---:|---|---:|---:|---:| | GSM8K | 1,314 | Default evaluation settings | 8 | 0.970 mean | 796.733 s mean | | GPQA | 32 | `temperature=1.0`, `top_p=0.95`, `max_tokens=128000` | 8 | 0.992 mean | 11,345.289 s mean | --- ### CI States Latest PR Test (Base): :x: [Run #31677831954](https://github.com/sgl-project/sglang/actions/runs/31677831954) Latest PR Test (Extra): :x: [Run #31677831868](https://github.com/sgl-project/sglang/actions/runs/31677831868) ",
"url": "https://github.com/sgl-project/sglang/pull/33673",
"createdAt": "2026-08-05T08:02:19Z",
"updatedAt": "2026-08-13T07:28:21Z",
"timestamp": "2026-08-13T07:28:21Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "shiyang814-cpu",
"state": "open",
"assignees": [
"hnyls2002"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33684",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Weight Cache] Support static DP/EP layouts",
"text": "## Motivation Weight-cache daemons and engines must construct identical model-parallel layouts before CUDA IPC can safely map a cached MoE shard. The existing daemon launcher rejected `dp_size > 1`, preventing static DP/EP deployments from using the daemon-backed weight cache. This change adds static DP/EP layout support to the CUDA IPC weight-cache path. It is the implementation foundation for the DP/EP roadmap in #33522. ## Modifications - Propagate static DP/EP, MoE DP, attention DP, MoE A2A, and DeepEP-mode configuration from the engine into weight-cache daemons. - Initialize daemon-side model-parallel and MoE state with the same static layout as the engine. - Extend `CacheConfig` with MoE DP/EP shard identity and layout fields, so daemon/client compatibility validation rejects incompatible shards. - Fix static Mooncake EP dispatch to use an all-active rank mask when Elastic EP state is absent. - Route Qwen3 MoE through the existing DeepEP-class backend predicate so Mooncake uses the EP execution path. - Add CPU coverage for MoE shard-rank calculation, extended cache fingerprints, and daemon DeepEP-mode forwarding. The implementation supports static layouts. Dynamic expert placement, EPLB, redundant experts, and recovery lifecycle ownership remain roadmap work. ## Accuracy Tests - `PYTHONPATH=python python -m pytest -q test/registered/unit/model_loader/test_weight_cache_protocol.py` - `29 passed` - Eight-GPU Qwen3-30B-A3B-FP8 daemon IPC serving with `TP=DP=EP=8`, Mooncake low-latency A2A, DP attention, and DP LM head. - 8 daemons exported IPC handles. - 8 schedulers logged `Loaded model via IPC`. - A deterministic completion generated `Paris` for `The capital of France is`. ## Speed Tests and Profiling N/A. This change establishes static DP/EP correctness and does not claim a new performance result. ## Checklist - [x] Format modified Python files with Black; CI will run the repository pre-commit hooks. - [x] Add unit tests according to [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31676640818](https://github.com/sgl-project/sglang/actions/runs/31676640818) Latest PR Test (Extra): :x: [Run #31676640548](https://github.com/sgl-project/sglang/actions/runs/31676640548) ",
"url": "https://github.com/sgl-project/sglang/pull/33684",
"createdAt": "2026-08-05T09:10:25Z",
"updatedAt": "2026-08-13T09:36:11Z",
"timestamp": "2026-08-13T09:36:11Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "UNIDY2002",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33685",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU CI] Reorganize test output/log directory structure with workflow context",
"text": " ## Motivation Restructure NPU CI output/log persistence so runs from different workflows are distinguishable by directory and re-runs never overwrite prior results, and migrate nightly single-node tests from a per-case matrix to per-suite jobs. ## Modifications - Reorganize outputs/logs under `{branch_label}-{create_time}-{run_id}-{run_attempt}/{workflow_name}/{test_type}/{tc_name}`; create_time is UTC+8 date + minute precision, recorded once at run start and shared across jobs, so re-runs (via run_attempt) never overwrite prior results. - Plumb branch_label/workflow_name/create_time through nightly-test-npu via a single `run_start_metadata` JSON, consumed by the single-node stage and multi-node e2e templates; whitelist branch_label characters for safe path use. - Replace the per-case `nightly-test-npu-e2e-single-node.yml` matrix with six per-suite jobs (`nightly-{perf,acc}-{2,4,16}-npu-a3`) that run `run_suite.py --nightly --continue-on-error --timeout-from-est-time`. - Extend `_npu-single-node-test-stage.yml` with a nightly mode: install test dependencies, upload metrics.json artifacts and back up plogs so nightly logs/results keep the structured layout. - Register cases to the new `nightly-acc-*` / `nightly-perf-*` suites and add them to `run_suite.py` NIGHTLY_SUITES (keeping existing base-c registrations for PR tests). - Add RUN_LABEL/TIMESTAMP and missing env vars to k8s templates; derive run_label from the persistence directory and persist metrics/plog under the same run prefix in the accuracy/performance utilities. ## Accuracy Tests N/A. CI configuration and output/log path changes only; model outputs are not affected. ## Speed Tests and Profiling N/A. CI configuration and output/log path changes only; inference speed is not affected. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31698149322](https://github.com/sgl-project/sglang/actions/runs/31698149322) Latest PR Test (Extra): :x: [Run #31698149326](https://github.com/sgl-project/sglang/actions/runs/31698149326) ",
"url": "https://github.com/sgl-project/sglang/pull/33685",
"createdAt": "2026-08-05T09:13:18Z",
"updatedAt": "2026-08-13T17:39:58Z",
"timestamp": "2026-08-13T17:39:58Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"deepseek",
"npu",
"run-ci"
],
"author": "pllimax",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33726",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix(bcg): preserve Qwen3-VL DeepStack inputs during replay",
"text": "## Motivation Prefill breakable CUDA graph (BCG) replay should preserve the same model semantics as eager execution. Qwen3-VL passes intermediate visual features through `input_deepstack_embeds` and adds them at selected language-model layers. However, BCG did not copy this request-specific tensor into a stable graph input slot. As a result, normal BCG execution behaved like an explicit zero-DeepStack run rather than eager execution. Requests still completed successfully, making this a silent correctness issue. ## Reproduction Environment: - Model: `Qwen/Qwen3-VL-8B-Instruct` - Revision: `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b` - GPU: NVIDIA H200 - Dtype: BF16 - Decoding: greedy, `max_new_tokens=20` - Input: deterministic RGB vertical-stripe image The GPU test harness temporarily enabled Qwen3-VL for BCG testing. The experiment compares four arms: 1. `eager_normal` 2. `eager_zero_deepstack` 3. `bcg_normal` 4. `bcg_zero_deepstack` The zero-DeepStack arms replace `input_deepstack_embeds` with zeros immediately before the language-model forward call. Each result is formatted as: ```text (common_prefix_length, all_token_ids_equal, maximum_absolute_logprob_difference) ``` | Comparison | Before fix | After fix | |---|---:|---:| | `bcg_normal` vs `eager_normal` | `(7, False, 1.154)` | `(15, True, 0.071)` | | `bcg_normal` vs `bcg_zero_deepstack` | `(20, True, 0.000)` | `(7, False, 1.148)` | | `bcg_zero_deepstack` vs `eager_zero_deepstack` | `(20, True, 0.066)` | `(20, True, 0.066)` | | `eager_zero_deepstack` vs `eager_normal` | `(7, False, 1.138)` | `unchanged` | Before the fix, `bcg_normal` was identical to `bcg_zero_deepstack` and diverged from `eager_normal`. After the fix, `bcg_normal` matches `eager_normal` at the token level and correctly diverges from `bcg_zero_deepstack`. The zero-DeepStack control remains unchanged. Instrumentation also confirmed that `input_deepstack_embeds` was present and non-zero at the language-model entry before replay. The RGB image is used only as a deterministic semantic-equivalence test, not as a visual-quality benchmark. ## Modifications - Add an explicit `supports_bcg_deepstack_replay` capability to Qwen3-VL. - Add an optional stable `input_deepstack_embeds` buffer and graph slot. - Allocate the slot only for multimodal models that opt in and have a non-zero DeepStack width. - Pass the stable slot during BCG capture so the DeepStack additions are included in the graph. - Copy the request-specific DeepStack tensor into the slot before replay. - Clear missing and padded regions to prevent stale data from previous requests. - Add unit tests for capability gating, buffer allocation, slot registration, and inactive model paths. `input_embeds` cannot be reused because DeepStack has a different shape and is consumed at selected intermediate language-model layers. Current Qwen3.5 checkpoints have no active DeepStack embeddings, so the new replay path remains inactive for them. The BCG allowlist is unchanged. ## Accuracy Tests ```bash pytest \\ test/registered/unit/model_executor/test_deepstack_replay_slot.py \\ test/registered/unit/model_executor/test_cuda_graph_buffer_registry.py \\ test/registered/unit/model_executor/test_prefill_cuda_graph_runner.py \\ test/registered/unit/model_executor/test_prefill_cuda_graph_runner_helpers.py ``` Result: ```text 57 passed, 21 warnings, 2 subtests passed in 10.55s ``` This includes 11 new tests and 46 existing adjacent tests. The GPU four-arm validation results are shown in the Reproduction section. ## Speed Tests and Profiling This is a correctness fix, not a speed optimization. Models that do not activate DeepStack replay do not allocate the new buffer or perform the replay copy. For Qwen3-VL-8B with `max_num_tokens=8192`, `hidden_size=4096`, three DeepStack embeddings, and BF16, the maximum additional buffer is: ```text 8192 × 4096 × 3 × 2 bytes = 192 MiB ``` No replay-latency benchmark was performed because Qwen3-VL is not currently enabled in the production BCG allowlist. ## Checklist - [x] Format your code according to the [[Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit)](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [[Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests)](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [[Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations)](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). No user-facing API or configuration change. - [ ] Provide accuracy and speed benchmark results according to [[Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy)](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [[Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed)](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). Accuracy results and memory overhead are reported above; latency was not benchmarked. - [x] Follow the SGLang code style [[guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance)](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31674823658](https://github.com/sgl-project/sglang/actions/runs/31674823658) Latest PR Test (Extra): :x: [Run #31674823338](https://github.com/sgl-project/sglang/actions/runs/31674823338) ",
"url": "https://github.com/sgl-project/sglang/pull/33726",
"createdAt": "2026-08-05T15:24:02Z",
"updatedAt": "2026-08-13T09:28:57Z",
"timestamp": "2026-08-13T09:28:57Z",
"metrics": {
"reactions": 0,
"comments": 11
},
"labels": [
"run-ci"
],
"author": "bowenwan6",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33743",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MoE] Fix flashinfer TRT-LLM BF16 expert weight reload on refit",
"text": "## Motivation `process_weights_after_loading` rewrites BF16 MoE expert weights into the flashinfer TRT-LLM BlockMajorK layout whenever `use_flashinfer_trtllm_moe` is set, and that flag covers **both** `flashinfer_trtllm` and `flashinfer_trtllm_routed`. The inverse hook, however, was gated on `is_flashinfer_trtllm_routed()` alone. So with `--moe-runner-backend flashinfer_trtllm` the destination parameter stayed in block layout and the hot copy raised: ``` The size of tensor a (64) must match the size of tensor b (2048) at non-singleton dimension 2 ``` torch names the `copy_` destination first, so the 64 is ours: `block_k` is 128 bytes and the conversion runs on a `uint8` view, giving 128 / 2 bytes-per-bf16. Nobody opts into this — `flashinfer_trtllm` is auto-selected on sm100 for BF16 MoE models when `moe_runner_backend` is left at `\"auto\"`, so an RL weight refit on Blackwell hits it by default. Fixes #27787 ## Modifications 1. **Gate the inverse on the same flag that gates the transform** (`use_flashinfer_trtllm_moe`), so the two cannot drift apart again. 2. **The restore inverts the data, not just the shape.** RL callers batch weights, so one refit is many update RPCs; with a shape-only restore an earlier bucket's re-derive would block expert slots that a later bucket has not written yet a second time. 3. **New `repack_weights_after_hot_update` hook** to re-derive the layout once the copies are done. `update_weights_from_disk` and the checkpoint-engine IPC path already re-run `process_weights_after_loading`; `update_weights_from_tensor`, `update_weights_from_distributed` and the bucketed variants call `model.load_weights()` directly and do not. Widening the gate on its own would therefore only trade the loud copy failure for a parameter left in canonical layout while the kernel reads BlockMajorK. The re-derive is called from a `finally` on all four update paths, so a mid-update exception cannot leave parameters canonical while the kernel expects block layout. Files touched: `layers/quantization/unquant.py`, `layers/quantization/base_config.py` (no-op base hook), `model_executor/model_runner_components/weight_updater.py`, plus a new CPU unit test. Note on the rebase onto current main: the only conflict was add/add with #33905 (XPU MoE row-stride padding), which inserts helpers at the same module-level position. Both blocks are kept and there is no interaction — the XPU padding applies under an XPU default device, the BlockMajorK rewrite under flashinfer on sm100, and #33905 only touches `create_weights`, which this PR does not. ## Accuracy Tests Reproduced against the real code path on CPU with Qwen3-30B-A3B geometry (hidden 2048, moe_intermediate 768, tp=2 -> 384 per partition): ``` before: postprocess (4,32,768,64) -> restore no-op -> copy raises after: restore -> (4,768,2048) -> copy ok -> repack -> (4,32,768,64) ``` `test/registered/unit/layers/quantization/test_flashinfer_trtllm_bf16_moe_reload.py` covers this end to end and is registered as a CPU test (11 cases): the block layout is applied on cold load and changes the shape; the restore runs for the non-routed backend and *inverts* the layout rather than reshaping it; a reload reproduces the cold-load layout exactly, including through the bucketed update path; the repack is a no-op when nothing was reverted and when the backend is inactive; non-bijective permutations and unexpected element counts are rejected; and the `finally` re-derive still runs when `load_weights` raises. ## Speed Tests and Profiling Not applicable to the forward path — this PR only changes the weight-update path. The re-derive runs once per update RPC, after the copies land; no kernel or forward-pass code is touched. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31643837888](https://github.com/sgl-project/sglang/actions/runs/31643837888) Latest PR Test (Extra): :x: [Run #31643837810](https://github.com/sgl-project/sglang/actions/runs/31643837810) ",
"url": "https://github.com/sgl-project/sglang/pull/33743",
"createdAt": "2026-08-05T18:52:04Z",
"updatedAt": "2026-08-13T07:17:09Z",
"timestamp": "2026-08-13T07:17:09Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"bug",
"quant",
"run-ci"
],
"author": "Kh4L",
"state": "open",
"assignees": [
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33778",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Avoid materializing GDN QKV tensors during target verification",
"text": "## Summary This PR removes redundant Q/K/V materialization from compatible GDN speculative target-verification paths. `causal_conv1d_update` already produces packed QKV. Previously, every GDN layer launched `fused_qkv_split_gdn_prefill_kernel` to copy that output into three contiguous tensors before Triton target verification. Triton accepts explicit token strides, so it can consume zero-copy `torch.split`/`view` tensors instead. The optimization is route-aware: - Triton and ReplaySSM target verification use strided Q/K/V views. - Prefill, FlashInfer, CuTeDSL, and other routes requiring contiguous inputs retain materialization. - Model arithmetic, recurrent-state updates, speculative acceptance, logits, and sampling are unchanged. This eliminates one memory-copy kernel launch per GDN layer on every compatible target-verification pass. ## Profile evidence Matched H200 CPU/GPU traces using Qwen3.5-4B, ReplaySSM, and NEXTN T=3: | Target-verification metric | Baseline | Patch | Change | |---|---:|---:|---:| | QKV-split launches/pass | 24 | 0 | **-100%** | | QKV-split GPU time/pass | 56.946 us | 0 | **-56.946 us** | | CUDA-graph kernels | 452 | 428 | **-24** | | CUDA-graph span | 8,132.684 us | 8,040.898 us | **-91.786 us / -1.129%** | The graph-span reduction's approximate 95% interval was **79.732-103.841 us**. ## Standardized GSM8K benchmark 3 alternating baseline/QKV pairs ran on the same H200 with fresh servers, seed 0, empty prefix cache, and a 30-second cooldown. The benchmark used its defaults: 200 questions, five shots, 512 maximum output tokens, temperature 0, and parallelism 64. ### Pair 1 | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,433.642 tok/s | 6,762.467 tok/s | **+5.111%** | | Latency | 5.379 s | 5.145 s | **-4.350%** | | Engine TPOT | 9.7374 ms | 9.4735 ms | **-2.710%** | ### Pair 2 | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,202.258 tok/s | 6,366.221 tok/s | **+2.644%** | | Latency | 5.470 s | 5.401 s | **-1.261%** | | Engine TPOT | 10.2534 ms | 9.7761 ms | **-4.655%** | ### Pair 3 | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,496.073 tok/s | 6,669.102 tok/s | **+2.664%** | | Latency | 5.270 s | 5.134 s | **-2.581%** | | Engine TPOT | 9.3372 ms | 9.4393 ms | **+1.093%** | ### Three-pair aggregate | Metric | Baseline | Patch | Change | |---|---:|---:|---:| | Output throughput | 6,377.324 tok/s | 6,599.263 tok/s | **+3.480%** | | Latency | 5.373 s | 5.227 s | **-2.723%** | | Engine TPOT | 9.7760 ms | 9.5630 ms | **-2.179%** | | Accuracy | 87.667% | 87.500% | -0.167 pp | ## Reproduction Server: ```bash CUDA_VISIBLE_DEVICES=0 \\ PYTHONPATH=![\"image\"](\"https://github.com/user-attachments/assets/cb953da4-5651-4eb2-accd-678c85ea34fc\")
### Per-GPU throughput from log file We record the per-rank throughput (median) for every per-rank batchsize from the log file, and make comparison between TBO and non-TBO, which is shown below. ![\"image\"](\"https://github.com/user-attachments/assets/7d4d0923-6de1-469e-85c4-dc97e1e7fb26\")
**At what batchsize, will TBO win non-TBO？** The answer is 320. Applying TBO will lead to huge performance degradation when bs is small, e.g. bs=60 on each dp rank. **Only if bs>320 on each dp rank, TBO has around 5~10% performance improvement compared with non-TBO baseline**. ![\"image\"](\"https://github.com/user-attachments/assets/db293e08-261d-4c76-a81c-305d87636946\")
### Profile We profile the case where bs=424 on each dp rank. **Hisparse only. Each decode step costs 135ms.** ![\"image\"](\"https://github.com/user-attachments/assets/a7131ae6-e07e-4b04-937c-b66b3b786f32\")
**Hisparse + tbo. Each decode step costs 124ms. 8% performance gains on large batchsize.** ![\"image\"](\"https://github.com/user-attachments/assets/99f92b30-e976-4fa3-aad1-acdf28e44ebf\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31479832051](https://github.com/sgl-project/sglang/actions/runs/31479832051) Latest PR Test (Extra): :x: [Run #31479831902](https://github.com/sgl-project/sglang/actions/runs/31479831902) ",
"url": "https://github.com/sgl-project/sglang/pull/33834",
"createdAt": "2026-08-06T07:38:10Z",
"updatedAt": "2026-08-13T06:26:51Z",
"timestamp": "2026-08-13T06:26:51Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"deepseek"
],
"author": "SYChen123",
"state": "open",
"assignees": [
"Fridge003",
"b8zhong"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33857",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Perf] Skip trivial DSV4 nonpaged indexer logits",
"text": "## Motivation SGL Top-K v1/v2 directly emits sequential indices when a row has no more candidates than `index_topk`, so those logits are never read. The DSV4 eager nonpaged path still computed them with DeepGEMM. ## Modifications - Encode trivial DeepGEMM rows as empty `[ks, ks)` ranges. - Preserve the original lengths used by KV gathering and Top-K. - Apply the optimization only to SGL Top-K; other backends keep full ranges. - Add one focused `<=/> index_topk` boundary assertion to the existing plan test. This targets the current eager `NonPagedIndexerPlan`; unlike #25400, it does not compact rows or add a host-side decision. ## Validation - Nonpaged indexer unit test: 7 passed, 12 subtests passed. - SGL Top-K v1/v2 GPU checks: mixed and all-trivial cases preserve selected indices. - Full pre-commit passed. ## Performance GB300, 8K prefill, `index_topk=1024`, 80 samples: - MQA + Top-K: `0.14361 -> 0.13467 ms` (`-6.23%`) --- ### CI States Latest PR Test (Base): :x: [Run #31718371894](https://github.com/sgl-project/sglang/actions/runs/31718371894) Latest PR Test (Extra): :x: [Run #31718371557](https://github.com/sgl-project/sglang/actions/runs/31718371557) ",
"url": "https://github.com/sgl-project/sglang/pull/33857",
"createdAt": "2026-08-06T09:06:23Z",
"updatedAt": "2026-08-13T17:06:50Z",
"timestamp": "2026-08-13T17:06:50Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"run-ci"
],
"author": "weireweire",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:33863",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Feature] PP Support PD + DSpark",
"text": " ## Motivation Fixes #32750. The existing PD + DSpark path works without Pipeline Parallelism because the prefill worker captures all required target hidden states, injects draft KV locally, and transfers target KV + draft KV to the decode worker. With PP enabled, the captured target hidden states are distributed across PP stages. Each stage only owns a subset of the features required by the DSpark context projection. Injecting draft KV independently on each stage is not equivalent to the non-PP computation because RMSNorm must be applied after the full context projection is accumulated. ## Design This PR keeps the existing PD data plane unchanged: ```text Prefill -> target KV + draft KV -> Decode ``` Hidden states are not transferred across the PD boundary. Each prefill PP stage computes its local pre-norm context contribution: ```text ctx_i = h_i @ W_i.T ``` The accumulated context is carried through PP proxy tensors: ```text ctx_acc = recv_ctx_acc + ctx_i ``` The last PP stage performs: ```text ctx_norm = RMSNorm(ctx_acc) K, V = KVProj(ctx_norm) ``` and writes draft KV into the prefill-side draft KV pool. This preserves the original non-PP computation: ```text concat(h1, ..., hk) @ W_fc.T == sum_i(h_i @ W_i.T) ``` while keeping the additional PP payload fixed at `[num_tokens, hidden_size]`. ## Changes ### PP-aware target capture - Capture only target layers owned by the current PP rank. - Carry local captures through `dspark_aux_hidden_states`. - See [`deepseek_v4.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4.py#L2455-L2467) and [`kimi_k3.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/kimi_k3.py#L2483-L2525). ### Fixed-size PP context accumulation - Each rank computes its local projection contribution. - `dspark_ctx_acc` remains `[num_tokens, hidden_size]` across PP hops. - Raw `dspark_aux_hidden_states` is removed after local projection. - RMSNorm is applied once on the final PP rank. - See [`dspark_worker_v2.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/speculative/dspark_components/dspark_worker_v2.py#L693-L726). ### DSV4 block-FP8 column slicing - Slice matching FP8 weight columns and scale blocks. - Support canonical FP8 scales and DeepGEMM packed UE8M0 scales. - Avoid repeated zero-padded full-K GEMMs for cross-rank target features. - See [`deepseek_v4_dspark.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4_dspark.py#L70-L147). Unsupported quantization backends retain the full-K fallback in [`project_target_hidden_partial()`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4_dspark.py#L803-L846). ### Single-owner fast path When all target layers belong to the final PP rank, that rank uses the existing full projection directly. Non-owner DSV4 ranks use lifecycle-only skeleton models: - No draft stages or heads. - No checkpoint weight loading. - No draft KV pool. - No planner, proposer, or injector. - Distributed lifecycle remains aligned across ranks. See [`dspark_config.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/speculative/dspark_components/dspark_config.py#L41-L73) and [`deepseek_v4_dspark.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/models/deepseek_v4_dspark.py#L678-L708). ### Draft KV injection and transfer - Add projected-context KV injection on the final PP rank. - Apply normalization, WKV projection, KV normalization, and RoPE before writing draft SWA KV. - Assign target KV global layer IDs. - Use the `1_000_000+` namespace for draft KV entries. - Only the final PP rank transfers draft KV; every PP rank transfers its local target KV. See [`dspark_kv_inject.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/speculative/dspark_components/dspark_kv_inject.py#L77-L159), [`utils.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/disaggregation/utils.py#L965-L1001), and [`prefill.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/disaggregation/prefill.py#L197-L257). ### PP bootstrap and transfer consensus - Ordered intersection for successful bootstrap/transfer requests. - Ordered union for failures. - Any PP-rank failure becomes a global failure. - KV is not released until the local transfer reaches a terminal state. See [`scheduler_pp_mixin.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/managers/scheduler_pp_mixin.py#L861-L914) and [`prefill.py`](https://github.com/bytedance-iaas/sglang/blob/fix/dspark-lifecycle-draft-kv-pool/python/sglang/srt/disaggregation/prefill.py#L857-L938). ## Supported Scope - Supported: `PP + DSPARK + PD prefill` - Unsupported: `PP + DSPARK decode` - Unsupported: `PP + DSPARK unified serving` - Unsupported: `DSPARK + context parallelism` ## Why `scheduler_pp_mixin.py` Is Modified The last PP stage performs additional work because it injects and transfers draft KV, while the other PP stages only transfer their local target KV. This creates asymmetric transfer completion times. The previous PP scheduler logic treated successful and failed transfers as one unordered set. It could also admit requests based only on `KVPoll.WaitingForInput`, before confirming that every PP stage had sufficient local metadata credits. Under sustained load, these behaviors could cause: - different PP stages to admit requests in different orders; - early KV release on stages that completed before the last PP stage; - inflight queue divergence after a transfer failure; - PP send/receive or proxy-tensor scheduling desynchronization. The scheduler changes make the control plane deterministic: - bootstrap readiness is probed without reserving resources; - request order is preserved during PP intersection and union; - transfer success requires intersection across all PP stages; - transfer failure is propagated as a union across all PP stages; - release occurs only after the last PP stage has completed both target KV and draft KV transfer; - DSpark draft input is reconstructed after PP output processing; - PP output tokens are moved to the batch device before creating the next draft state. The scheduler does not transfer hidden states or draft KV itself. Its role is to keep admission, completion, and release decisions consistent across PP stages despite the extra work performed by the last stage. ## fixes state-component matching This patch fixes state-component matching for PD transfer when PP ranks expose different state component layouts. Previously, state components were matched positionally: ```text prefill state[i] -> decode state[i] ``` That is not always valid for DSV4 under PP. Some state components are optional per PP stage. For example, `C128_STATE` only exists on PP ranks that own C128 layers. When the final PP rank sends draft SWA but does not own any C128 layer, the source and destination layouts can diverge: ```text Prefill final PP rank: state[0] = target SWA state[1] = draft SWA Decode rank: state[0] = target SWA state[1] = target C128_STATE state[2] = draft SWA ``` With positional matching, `prefill state[1]` would incorrectly map to `decode state[1]`, so draft SWA could be written into the target C128 state buffer. The fix is to include `state_types` in the decode registration metadata and match state components by `(StateType, occurrence)` instead of list index: ```text prefill SWA #1 -> decode SWA #1 prefill SWA #2 -> decode SWA #2 ``` This keeps the existing behavior when layouts are already aligned, while correctly handling missing optional components such as `C128_STATE`. The change is applied to both Mooncake and NIXL transfer paths. For mixed-version compatibility, if the peer does not provide `state_types`, the code falls back to the old positional mapping. This does not change the actual KV/state data layout or copy logic. It only changes how the destination state component is selected. ## Accuracy Tests ``` SGLANG_PP_LAYER_PARTITION=\"6,5,6,5,6,5,5,5\" SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 NCCL_SOCKET_IFNAME=eth0 NCCL_IB_DISABLE=0 SGLANG_DSV4_FP4_EXPERTS=1 GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path /data02/models/DeepSeek-V4-Flash-0731 --host 0.0.0.0 --port 30000 --trust-remote-code --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.9 --max-running-requests 64 --chunked-prefill-size 8192 --max-prefill-tokens 16384 --pp-size 8 --attention-backend dsv4 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --disable-overlap-schedule --disable-piecewise-cuda-graph --disaggregation-transfer-backend mooncake --enable-metrics --moe-runner-backend flashinfer_mxfp4 --speculative-algo DSPARK --disaggregation-mode prefill --disaggregation-ib-device \"mlx5_1,mlx5_2,mlx5_3,mlx5_4\" --disable-flashinfer-autotune --disable-shared-experts-fusion SGLANG_DSV4_FP4_EXPERTS=1 SGLANG_JIT_DEEPGEMM_PRECOMPILE=1 SGLANG_OPT_DEEPGEMM_HC_PRENORM=1 SGLANG_OPT_USE_TILELANG_MHC_PRE=1 GLOO_SOCKET_IFNAME=eth0 NCCL_MIN_NCHANNELS=24 NCCL_IB_QPS_PER_CONNECTION=8 sglang serve --trust-remote-code --model-path /data02/models/DeepSeek-V4-Flash-0731 --tp 8 --dp-size 8 --enable-dp-attention --cuda-graph-max-bs 32 --max-running-requests 256 --enable-metrics --host 0.0.0.0 --port 30000 --mem-fraction-static 0.85 --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --moe-runner-backend flashinfer_mxfp4 --disaggregation-mode decode --disaggregation-ib-device \"mlx5_1,mlx5_2,mlx5_3,mlx5_4\" --speculative-algo DSPARK --tokenizer-worker-num 8 --enable-dp-lm-head --load-balance-method round_robin --swa-full-tokens-ratio 0.8 --disable-shared-experts-fusion ``` Hardward | MMLU | GSM8K | QPQA | aime25 repeats 16 -- | -- | -- | -- | -- H20 | 0.878 | 0.945 | 0.961 | 98.33% +/- 2.11% (SEM 0.53%) ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31662550711](https://github.com/sgl-project/sglang/actions/runs/31662550711) Latest PR Test (Extra): :x: [Run #31662550618](https://github.com/sgl-project/sglang/actions/runs/31662550618) ",
"url": "https://github.com/sgl-project/sglang/pull/33863",
"createdAt": "2026-08-06T09:42:11Z",
"updatedAt": "2026-08-13T06:49:11Z",
"timestamp": "2026-08-13T06:49:11Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"documentation",
"quant",
"amd",
"dependencies",
"Multi-modal",
"deepseek",
"speculative-decoding",
"hicache",
"sgl-kernel",
"blackwell",
"diffusion",
"model-gateway",
"mthreads",
"jit-kernel"
],
"author": "zhangxiaolei123456",
"state": "open",
"assignees": [
"whybeyoung",
"yhyang201",
"Fridge003",
"kpham-sgl"
]
},
{
"id": "github:sgl-project/sglang:pull_request:33881",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "refactor(kernels): split zero-KV fixup by alignment",
"text": " ## Motivation This is a follow-up to #32890, which introduced the TRT-LLM ragged attention zero-KV fixup path. That path clears `out` and sets `lse` to `-inf` for sequences whose KV length is zero. The original fix already handles potentially misaligned rows by checking alignment inside the device helper before using vectorized `float4` stores. Per @nvpohanh's suggestion, this PR avoids the device-side if/else by compiling separate aligned and scalar store paths, then choosing the appropriate specialization from the host launcher. This keeps the aligned vectorized fast path while routing potentially misaligned layouts, such as TP cases where `lse` row stride is not a 16-byte multiple, to scalar stores. ## Modifications - Compile the zero-KV fixup kernel with independent store-mode template parameters for `out` and `lse`. - Dispatch four host-side variants: vector/vector, vector/scalar, scalar/vector, and scalar/scalar. - Include both base pointer alignment and row-stride alignment in the host predicates. This covers contiguous tensors with non-zero PyTorch `storage_offset`. - Preserve the existing public Python/JIT wrapper API. - Add a registered CUDA unit test for: - contiguous tensors with non-zero storage offsets; - mixed-alignment layouts where `out` can remain vectorized while `lse` uses scalar stores. ## Accuracy Tests Added: ```bash test/registered/kernels/test_fixup_zero_kv.py ``` Local checks run: ```bash git diff --check upstream/main...HEAD PYTHONPYCACHEPREFIX=/private/tmp/sglang_pycache python3 -m py_compile test/registered/kernels/test_fixup_zero_kv.py ``` Not run locally: ```bash python3 -m pytest test/registered/kernels/test_fixup_zero_kv.py -q ``` The local Python environment does not have `pytest` installed. ## Speed Tests and Profiling Added a dedicated microbenchmark for this PR: ```bash python3 test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py ``` It is also registered in the CUDA kernel benchmark suite and can be run through: ```bash python3 test/run_suite.py --hw cuda \\ --suite base-b-kernel-benchmark-test-1-gpu-large \\ --continue-on-error ``` The benchmark exercises the four host-dispatched specializations introduced here: | mode | `out` store | `lse` store | coverage | |---|---:|---:|---| | `vec_vec` | vectorized | vectorized | fully 16-byte aligned fast path | | `vec_scalar` | vectorized | scalar | TP-like layouts where `lse` row stride is not 16-byte aligned | | `scalar_vec` | scalar | vectorized | non-zero storage offset / misaligned `out` base pointer | | `scalar_scalar` | scalar | scalar | both buffers potentially misaligned | Shapes covered: `(max_seq_len, batch_size) = (32, 32), (128, 32), (512, 16)`. The benchmark is intentionally a kernel microbenchmark so it isolates the zero-KV fixup cost instead of hiding it inside a full model run. I did not post local timing numbers because this workstation cannot run the CUDA benchmark (`nvidia-smi` is unavailable and the local Python env does not have `torch`). Light local validation completed: ```bash PYTHONPYCACHEPREFIX=/tmp/sglang-pr33881-pycache \\ python3 -m py_compile \\ test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py \\ test/registered/kernels/test_fixup_zero_kv.py git diff --check ``` Expected perf behavior: the fully aligned `vec_vec` path preserves the existing vectorized fast path, while mixed-alignment cases avoid the previous device-side per-row alignment branch by selecting the scalar/vector specialization once in the host launcher. ## Microbenchmark ## Microbenchmark **GPU:** GB200 | **dtype:** fp16 out / fp32 lse | **v_head_dim:** 128 Latency in µs (p50). `before` = device-side alignment branch per row; `after` = host-dispatched specialization. | max_seq_len | batch_size | mode | before (µs) | after (µs) | Δ latency | |-------------|------------|------|-------------|------------|-----------| | 32 | 32 | `vec_vec` | 3.12 | 2.98 | ▼4.6% | | 32 | 32 | `vec_scalar` | 3.36 | 3.00 | ▼10.7% | | 32 | 32 | `scalar_vec` | 3.29 | 3.24 | ▼1.6% | | 32 | 32 | `scalar_scalar` | 3.06 | 2.76 | ▼10.0% | | 128 | 32 | `vec_vec` | 7.19 | 6.84 | ▼4.9% | | 128 | 32 | `vec_scalar` | 6.52 | 5.94 | ▼8.9% | | 128 | 32 | `scalar_vec` | 6.71 | 6.40 | ▼4.6% | | 128 | 32 | `scalar_scalar` | 5.98 | 5.21 | ▼12.8% | | 512 | 16 | `vec_vec` | 12.02 | 11.39 | ▼5.3% | | 512 | 16 | `vec_scalar` | 10.55 | 9.65 | ▼8.5% | | 512 | 16 | `scalar_vec` | 10.89 | 10.23 | ▼6.0% | | 512 | 16 | `scalar_scalar` | 9.14 | 7.80 | ▼14.6% | | max_seq_len | batch_size | mode | before (GB/s) | after (GB/s) | Δ bandwidth | |-------------|------------|------|---------------|--------------|-------------| | 32 | 32 | `vec_vec` | 317.5 | 332.8 | ▲4.8% | | 32 | 32 | `vec_scalar` | 221.5 | 247.9 | ▲11.9% | | 32 | 32 | `scalar_vec` | 301.7 | 306.5 | ▲1.6% | | 32 | 32 | `scalar_scalar` | 242.9 | 270.0 | ▲11.2% | | 128 | 32 | `vec_vec` | 551.8 | 579.9 | ▲5.1% | | 128 | 32 | `vec_scalar` | 456.4 | 500.9 | ▲9.8% | | 128 | 32 | `scalar_vec` | 591.4 | 619.8 | ▲4.8% | | 128 | 32 | `scalar_scalar` | 497.5 | 570.8 | ▲14.7% | | 512 | 16 | `vec_vec` | 660.0 | 696.8 | ▲5.6% | | 512 | 16 | `vec_scalar` | 564.1 | 616.5 | ▲9.3% | | 512 | 16 | `scalar_vec` | 728.8 | 775.6 | ▲6.4% | | 512 | 16 | `scalar_scalar` | 651.0 | 762.6 | ▲17.1% | **Reproduce (after / this PR):** ```bash cd /sgl-workspace/sglang python test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py Reproduce (before / baseline): cd /sgl-workspace/sglang_baseline python test/registered/kernels/benchmark/attention/bench_fixup_zero_kv.py ``` ## end-to-end ``` python3 -m sglang.launch_server \\ --model-path moonshotai/Kimi-K3 \\ --served-model-name k3 \\ --tp 16 \\ --nnodes 4 \\ --node-rank 0 \\ --dist-init-addr ![\"tlru-mechanism\"](\"https://github.com/user-attachments/assets/256103ac-07fb-4ade-912e-353df10a9ac4\")
Concretely: a conversation or agentic turn with history length `L` whose next prompt is expected to add `Q_hat` tokens only needs `L + Q_hat - xi` tokens resident to keep its next prefill inside a TTFT budget `xi`. Everything past that point is \"TEL-safe\" — evicting it cannot worsen tail latency, because the recompute it causes still fits the budget. T-LRU frees those tokens first (phase 1) and continues in plain recency order once they run out (phase 2). Implemented as a priority function rather than a second eviction pass: a TEL-safe node reports as infinitely old, so the existing device and host eviction drivers produce both of the paper's phases unchanged. ```python budget = max(node.convo_length + self.next_prompt_estimate - self.threshold, 0) cached_without_this_node = node.depth - len(node.key) tel_safe = cached_without_this_node >= budget return (-1 if tel_safe else 0, node.last_access_time) ``` ![\"result_32\"](\"https://github.com/user-attachments/assets/0df14d77-4ec5-4219-bade-f081a0c968b1\")
The conversation's history length (`convo_length`) is tracked as a monotone high-water mark separate from its resident depth, maintained on insert and split. Deriving it from what is still cached would leave a trimmed conversation over budget on every later pass and walk it down to nothing instead of stopping after `xi - Q_hat` tokens. Thresholds are expressed in tokens, unlike the paper's blocks (multiply the paper's values by `page_size` to convert), and startup rejects the two configurations that would silently reduce the policy to LRU (`xi <= Q_hat`, and use without the unified radix tree). **Choosing `xi` and `Q_hat`.** Only the difference `delta = xi - Q_hat` reaches the policy (the protected budget is `max(L - delta, 0)`), so the two are stated separately only because each is independently interpretable. `xi` is the tolerated recompute cost in tokens: pick a TTFT budget in seconds and divide by `alpha`, the measured per-uncached-token prefill cost of the system under test (~64 us/token here), giving `xi = 14592` — p5 of the trace's context-length distribution. `Q_hat` is the expected next-prompt size, taken from the trace's per-turn prompt-growth distribution rather than guessed: `2304`, its p60 (median growth is 1,731). Both are workload-relative token counts, so porting them to a new workload means re-deriving: re-measure `alpha` and re-apply the SLO, or match the freed share `F(delta) = sum(min(delta, ctx_i)) / sum(ctx_i)` (7.4% of served context at our setting) and solve for the new `delta`. Files touched (325 insertions, 7 deletions): | file | change | | --- | --- | | `mem_cache/evict_policy.py` | `TLRUStrategy` | | `mem_cache/unified_radix_cache.py` | `convo_length` bookkeeping on insert/split | | `mem_cache/cache_init_params.py`, `kv_cache_builder.py`, `utils.py` | plumb the two params | | `srt/server_args.py` | flags, choices, validation | | `test/registered/unit/mem_cache/test_tlru_eviction_policy.py` | 9 unit tests | ## Accuracy Tests No change to model forward computation or kernels: the policy only reorders which cached prefixes are evicted, and an evicted prefix is recomputed exactly on its next request, so model outputs are unchanged by construction. The default `lru` path is untouched when the flag is not set. Policy behaviour is covered by 9 CPU-only unit tests (`test_tlru_eviction_policy.py`, no GPU required): - frees exactly the tail budget on a fresh conversation, and stops there - conversations under the threshold are entirely free - phase 1 spreads across conversations, then falls back to LRU - degenerates to exact LRU when `Q_hat` reaches `xi` - an oversized tail node is protected rather than partially freed - a compacted branch keeps its shared prefix protected - budget clamps at zero; priorities stay finite and orderable ## Speed Tests and Profiling Measured on DeepSeek-V4-Pro FP4, 4x B300, TP4/EP4/DP-attention, HiCache DRAM ratio 4, on the SemiAnalysis agentic trace via aiperf, 3,600 s measured after 600 s cache-pressure warmup, `xi=14592` / `Q_hat=2304`. C=32: 5 LRU / 12 T-LRU runs; C=40: 4 LRU / 5 T-LRU. **Methodology.** The client seed is pinned, so both arms replay identical trajectories; requests are matched across arms on `(conversation_id, turn_index)` and compared as a paired sample (~6,000+ matched requests per concurrency), which removes workload variance. Comparisons use the middle 80% of each run's measured window (opening/closing 10% trimmed from both arms alike) to exclude the residual warm-up transient and client-drain. And the trace is using [semianalysisai/cc-traces-weka-062126](https://huggingface.co/datasets/semianalysisai/cc-traces-weka-062126) on HuggingFace: 9,843 real Claude Code sessions (98,827 turns) captured by WEKA's kv-cache-tester. The actual generator code lives in the aiperf fork inside [InferenceX](https://github.com/SemiAnalysisAI/InferenceX), checked out at dd237099. **Paired quantile deltas, T-LRU vs LRU on identical requests (lower is better):** C=32: | metric | p50 | p75 | p90 | p95 | p99 | | --- | --- | --- | --- | --- | --- | | TTFT | -3.2%| -5.3% | -4.7% | -6.9% | -8.6% | | ITL | +2.2%* | -0.7% | -8.7% | -17.2% | -43.9% | C=40: | metric | p50 | p75 | p90 | p95 | p99 | | --- | --- | --- | --- | --- | --- | | TTFT | +5.1% | +2.4% | +2.1% | -4.3% | -13.6% | | ITL | +0.2% | +0.8% | -0.6% | -2.7% | -11.0% | The shape matches the mechanism: the decode tail is where retention pays off, and the effect grows monotonically toward the tail at C=32 (ITL -8.7% at p90 to -43.9% at p99), alongside TTFT body improvements. The cost is small and in the body: ITL p50 +2.2% at C=32 and TTFT p50 +5.1% at C=40 (both corroborated by run-level exact rank tests). **Throughput** (arm-level medians, higher is better) is unchanged — every delta is inside its replicate spread. The policy reshapes the latency distribution without moving aggregate throughput or the prefix hit rate. | C | metric | LRU | T-LRU | delta | replicate spread | | --- | --- | --- | --- | --- | --- | | 32 | tok/s/GPU | 28,580 | 28,228 | -1.2% | 12.2% | | 32 | tok/s/user | 23.45 | 22.89 | -2.4% | 4.9% | | 32 | GPU prefix hit | 96.04% | 96.07% | +0.04 pp | 0.5 pp | | 40 | tok/s/GPU | 30,117 | 30,558 | +1.5% | 8.0% | | 40 | tok/s/user | 20.06 | 19.72 | -1.7% | 9.8% | | 40 | GPU prefix hit | 95.70% | 95.79% | +0.09 pp | 0.3 pp | Note: this configuration is well-provisioned (working set ~0.28x of KV capacity, 96% prefix hit rate vs a 97.7% ceiling), so these numbers are a lower bound on what a cache-pressured deployment shows; and `xi`/`Q_hat` are workload-relative token counts — copying our values onto a different context distribution will not reproduce this behaviour (only `xi - Q_hat` reaches the policy). **Write-back study.** There is a nuance in the analogy to a CPU cache: in a CPU, SRAM caches are *write-back* to DRAM; here, DRAM itself is the second-level cache (backing GPU HBM), and it runs write-back in an \"allocate on eviction\" sense. That is: a KV page gets its DRAM copy only at the moment it is demoted out of HBM, never proactively. Thus, the *write-back rate* is exactly the *eviction rate*. **Measured write-back and load-back rates** (median per run, 3,600 s window, from run counters): | arm | C | written back to DRAM | rate | load-back from DRAM | | --- | --- | --- | --- | --- | | LRU | 32 | 30.0M tokens | ~8.3k tok/s 10 MB/s | 0.7M tok (~0.2 MB/s) | | LRU | 40 | 38.9M tokens | ~10.8k tok/s 13 MB/s | 3.3M tok (~1.1 MB/s) | | T-LRU | 32 | 48.1M tokens | ~13.4k tok/s 16 MB/s | 17.4M tok (~5.7 MB/s) | | T-LRU | 40 | 58.3M tokens | ~16.2k tok/s 19 MB/s | 22.8M tok (~7.5 MB/s) | Implied size is ~1.2 KB per token, consistent with MLA's compressed KV — this model is unusually cheap to demote. **Two main readings** 1. *Average bandwidth is trivial*. 10–20 MB/s sustained against an HBM↔DRAM link measured in hundreds of GB/s is negligible — write-back traffic is not a bandwidth concern here, even with T-LRU's 1.6× amplification (visible directly: e.g., T-LRU writes 48M vs LRU's 30M at C=32, and the ~17M difference is its load-back volume). Any latency cost from the demotion path arises from the moment copies happen — bursty, under memory pressure, on the eviction path while a prefill waits — not from the total data moved. 2. *The tier is write-heavy and read-light under LRU* (e.g., 30M written vs 0.7M read back at C=32, a 43:1 ratio). This is exactly when write-back is the right policy: write-through would have pre-copied everything for almost no read benefit. T-LRU narrows the ratio to ~3:1 by design — it deliberately evicts things that come back. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31549613933](https://github.com/sgl-project/sglang/actions/runs/31549613933) Latest PR Test (Extra): :white_check_mark: [Run #31687279205](https://github.com/sgl-project/sglang/actions/runs/31687279205) ",
"url": "https://github.com/sgl-project/sglang/pull/34012",
"createdAt": "2026-08-07T17:56:37Z",
"updatedAt": "2026-08-13T09:39:55Z",
"timestamp": "2026-08-13T09:39:55Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "yl3469",
"state": "open",
"assignees": [
"ispobock",
"hzh0425"
]
},
{
"id": "github:sgl-project/sglang:pull_request:34014",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Improve M3 performance on MI350",
"text": " ## Motivation Improve M3 performance on MI350. ## Modifications - Adds AITER FlyDSL MXFP8 MoE support, including pre-shuffled gate/up weights and E8M0 scales, OAI-SwiGLU activation handling, and fused shared-expert execution. - Adds selectable dense MXFP8 linear backends through `SGLANG_MXFP8_LINEAR_BACKEND`: - `auto`/`aiter`: native blockwise MXFP8 via `torch._scaled_mm` - `triton`: existing `dot_scaled` implementation - `bf16`: cached BF16 weight dequantization with hipBLASLt - Reduces decode overhead by: - Skipping redundant sparse KV-cache writes when fusion already populated the cache. - Avoiding unnecessary top-k ID sanitization when expert parallelism is disabled. - Fixes AITER top-k handling for fused shared experts and respects the configured gate/up layout. - Enables MiniMax-M3 shared-expert fusion on supported ROCm devices (gfx942 or newer), while preserving existing EP and DeepEP compatibility guards. - Allows the AITER MoE runner for MXFP8 on gfx950 and validates that `SGLANG_USE_AITER=1` is enabled when explicitly selected. - Applies the same ROCm shared-expert support to the MiniMax-M3-VL model. ``` SGLANG_USE_AITER=1 \\ sglang serve \\ --trust-remote-code \\ --model-path /persistent/models/MiniMax-M3-MXFP8 \\ --reasoning-parser auto \\ --tool-call-parser auto \\ --tp 8 \\ --quantization mxfp8 \\ --moe-runner-backend aiter \\ --fp8-gemm-backend bf16 \\ --enable-aiter-allreduce-fusion \\ --dtype bfloat16 \\ --chunked-prefill-size 8192 \\ --mem-fraction-static 0.80 \\ --host 0.0.0.0 \\ --port 30000 ``` ## Accuracy Tests baseline ``` == gsm8k == 1319 examples (single-shot) | 227.1s | 1485 tok/s | 337K tokens * score = 88.55% stop_rate = 89.61% truncated_rate = 10.39% [warn: hitting max_tokens] error_rate = 0.00% ``` this pr ``` == gsm8k == 1319 examples (single-shot) | 151.2s | 2231 tok/s | 337K tokens * score = 88.63% stop_rate = 89.99% truncated_rate = 10.01% [warn: hitting max_tokens] error_rate = 0.00% ``` ## Speed Tests and Profiling ![\"Screenshot](\"https://github.com/user-attachments/assets/513b39bc-72e1-4510-a3cd-9b09b4c4b32c\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/34014",
"createdAt": "2026-08-07T18:43:58Z",
"updatedAt": "2026-08-13T17:49:34Z",
"timestamp": "2026-08-13T17:49:34Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"quant",
"amd",
"jit-kernel"
],
"author": "zcnrex",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:34018",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[SM12x] Re-enable the FP8 wo_a GEMM on sm120/sm121",
"text": "## Motivation On SM120/SM121 `server_args.py` disables a block of features with the rationale *\"SM120 lacks tcgen05/TMEM: disable features that depend on DeepGEMM\"*. For `wo_a` (the first stage of DeepSeek-V4's o-projection) the consequence is that the FP8 weights already present in the checkpoint are **dequantised to bf16 at load** and the projection then runs as a generic bf16 WMMA GEMM. The hardware half of that rationale is correct — sm120/sm121 really have no tcgen05/TMEM (`grep tcgen05` hits only `impls/sm100_*.cuh`). The *implication* is what went stale: **DeepGEMM no longer requires tcgen05**. The installed package ships nine `impls/sm120_*.cuh` — `sm120_fp8_fp4_gemm_1d1d`, `sm120_bmk_bnk_mn`, `sm120_tf32_hc_prenorm_gemm`, `sm120_{fp8,fp4}_{paged_,}mqa_logits`, … — all built on the SM120 warp-level MMA path instead. Measured on 2× DGX Spark (GB10 / sm_121, TP=2), `deepseek-ai/DeepSeek-V4-Flash-0731` + DSPARK, the bf16 fallback costs **48 calls/step at ~174 µs = 12.7 ms/step**, about 15% of the decode step. ## Modifications Opt-in only; default behaviour on every arch is unchanged. 1. The SM120 feature-disable block honours an explicit `SGLANG_OPT_FP8_WO_A_GEMM=1` (`is_set()` guard). 2. The existing opt-in gate accepts sm12x via `is_sm120_supported()`. 3. `wo_a` takes the **whole** sm100 branch on sm12x: the dedicated ue8m0 activation quant, `transform_sf_into_required_layout` for the weight scales, and `recipe=(1, 1, 128)`. Point 3 is not cosmetic. Two constraints were found by offline probing at the real shapes (T=6, G=4, D=4096, R=1024): - **sm12x accepts only ue8m0 scales.** Every fp32-scale variant I tried (`fp8_einsum`, `fp8_gemm_nt` ×G, `m_grouped_fp8_gemm_nt_contiguous`) aborts with `cudaErrorLaunchFailure`; every ue8m0 variant works. - **`recipe` must match the weight-scale layout.** The transformed `[G, R, D/128] int32` layout pairs with `(1, 1, 128)`; the untransformed `[G, R/128, D/128] fp32` layout pairs with `(1, 128, 128)`. Crossing them trips `Assertion (utils/layout.hpp:97): sf.size(-2) == ceil_div(mn, gran_mn)`. Splitting the branch (ue8m0 weights + `(1,128,128)`) reproduces that assert, which is why sm12x reuses the sm100 branch intact rather than a hybrid. Offline microbenchmark, exclusive GPU, same shapes: | | µs/call | | --- | ---: | | bf16 einsum (current fallback) | 152–154 | | `m_grouped_fp8_gemm_nt_contiguous` (ue8m0, M padded 6→128) | 73.4 | | **dedicated quant + `fp8_einsum` + recipe (1,1,128)** | **26.0** | The resulting kernel is `deep_gemm::sm120_fp8_fp4_gemm_1d1d_impl<0u, 6u, 4096u, 128u, 128u, 4u, …>` — the same instantiation vLLM launches for this projection on the same hardware. ## Accuracy Tests GSM8K 5-shot, `--num-questions 200 --parallel 16`, 2× DGX Spark TP=2: | | Accuracy | Invalid | | --- | ---: | ---: | | baseline | 0.970 | 0.000 | | **this PR + #34019** | **0.980** | 0.000 | An earlier n=1000 run on an older base gave 0.943 (baseline) vs 0.946 (this change). Note accuracy is **not** comparable across different `--num-questions` — the first 200 questions are easier than the first 1000 (same build: 0.970 @ n=200 vs 0.943 @ n=1000). ## Benchmarking and Profiling `sglang.bench_one_batch_server`, bs=1, `SGLANG_SIMULATE_ACC_LEN=5` so the accept length is pinned and the comparison is pure step cost (`acc_length: 5.00` confirmed in both arms). Same container, same clocks (unlocked), RoCE/RDMA available to both. | in / out | baseline | this PR + MHC default | | | --- | ---: | ---: | ---: | | 1024 / 512 | 58.48 tok/s | **72.67 tok/s** | +24.3% | | 8192 / 1024 | 61.88 tok/s | **66.76 tok/s** | +7.9% | Kernel-level attribution from a 20-step profile (same container, same conditions), which separates the two changes: | kernel family | baseline | patched | attributable to | | --- | ---: | ---: | --- | | bf16 SM80-WMMA | 20.18 ms / 136 calls | **12.74 ms / 93 calls** | **this PR** (wo_a's 43 calls move out) | | `deep_gemm sm120_fp8_fp4_gemm_1d1d` | 0 | **3.29 ms / 43 calls** | **this PR** | | fp32 cuBLAS (`hc_pre`) | 9.54 ms / 93 calls | 0.86 ms / 8 calls | MHC default (separate PR) | | decode step | 70.31 ms | **63.19 ms** | −10.1% | Net for this PR alone: −7.45 ms of bf16 WMMA for +3.29 ms of FP8 GEMM ≈ **−4.2 ms/step**. ### Test environment caveat DSPARK on sm120/sm121 **cannot boot on stock flashinfer today**: the draft's index width is `ceil_align(swa_window + gamma, 64)`, which is 192 for every gamma in [1, 64], while flashinfer 0.6.15.post1 only instantiates decode top-k widths `{128, 512, 1024}`. Both known fixes are still open — flashinfer-ai/flashinfer#4309 (adds the 192 instantiation) and sgl-project/sglang#33407 (dispatch fallback in SGLang). The numbers above were collected with flashinfer#4309 carried as a local patch; `SGLANG_SM120_FLASHMLA_BACKEND=triton` is the only stock alternative and it routes *all* sparse-MLA calls through Triton, which costs throughput on the hot path (measured 4.8x slower end to end here), so it is not a usable baseline. ### On tests No unit test is included. The change is arch-gated to sm120/sm121, and CI has no such runner, so a test would not execute where the behaviour differs. Validation was done on hardware instead: GSM8K for accuracy and a kernel-level profile (reported above) to confirm the intended kernels actually run rather than inferring it from throughput. Happy to add a test if there is a preferred pattern for arch-gated paths. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#running-unit-tests-adding-tests). - [x] Update documentation / docstrings / example tutorials as needed, see [Writing Documentation](https://docs.sglang.ai/developer_guide/contribution_guide.html#writing-documentation-running-docs-ci). - [x] Provide throughput / latency benchmark results and accuracy evaluation results as needed, see [Benchmark and Profiling](https://docs.sglang.ai/developer_guide/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/developer_guide/accuracy_evaluation.html). - [x] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [x] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR. --- ### Open question for maintainers The same SM120 block also force-disables `SGLANG_OPT_DEEPGEMM_HC_PRENORM` and pushes the DSA indexer onto the torch / TileLang paths (`SGLANG_FP8_PAGED_MQA_LOGITS_TORCH=True`, `SGLANG_OPT_USE_TILELANG_INDEXER=True`). DeepGEMM ships `sm120_tf32_hc_prenorm_gemm.cuh` and `sm120_{fp8,fp4}_{paged_,}mqa_logits.cuh` for exactly those. If the tcgen05 premise is stale for `wo_a`, those three deserve a re-audit too — happy to test them on this hardware. Tracking under #19637 rather than widening this PR. Also note `deep_gemm_wrapper/configurer.py` gates on `sm_version == 120`, which misses sm_121 (GB10 reports 121), while `server_args.py` uses `is_sm120_supported()` which returns True there. The two disagree; this PR uses the helper consistently but does not change `configurer.py`. --- ### CI States Latest PR Test (Base): :x: [Run #31215115097](https://github.com/sgl-project/sglang/actions/runs/31215115097) Latest PR Test (Extra): :x: [Run #31215114872](https://github.com/sgl-project/sglang/actions/runs/31215114872) ",
"url": "https://github.com/sgl-project/sglang/pull/34018",
"createdAt": "2026-08-07T19:52:00Z",
"updatedAt": "2026-08-13T03:18:16Z",
"timestamp": "2026-08-13T03:18:16Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"deepseek"
],
"author": "zhendonghua",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34019",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[SM12x] Default the fused MHC post+pre path on",
"text": "## Motivation On sm120/sm121 the SM120 block in `server_args.py` sets `SGLANG_OPT_USE_TILELANG_MHC_PRE=False`, so DeepSeek-V4's `hc_pre` falls through to `hc_pre_torch_impl` — an fp32 `F.linear` of shape `[M, 16384] × [16384, 24]`. cuBLAS serves that with `cutlass_80_simt_sgemm`, i.e. **plain CUDA cores, no tensor cores at all**. On 2× DGX Spark (GB10 / sm_121, TP=2) running `deepseek-ai/DeepSeek-V4-Flash-0731` + DSPARK that fallback costs **93 calls/step, 9.54 ms/step** — about 14% of the decode step, for a projection whose weights are only 1.5 MB. `mhc_fused_post_pre` covers the same work, and `_is_fused_mhc_post_pre_enabled()` already special-cases this arch: ```python # SM120 disables the standalone TileLang pre path. mhc_fused_post_pre does # not read that flag and dispatches independently for both small and large # token batches, so the standalone pre flag must not veto the fused opt-in. ``` But `SGLANG_OPT_FUSE_MHC_POST_PRE` defaults to `False`, so on sm12x the combination the comment anticipates is never reached out of the box — the SIMT fallback is what everyone gets. ## Modifications Default `SGLANG_OPT_FUSE_MHC_POST_PRE=True` inside the existing `is_sm120_supported()` block, guarded by `is_set()` so an explicit setting still wins. No other arch is affected; no change to the flag's global default. ## Accuracy Tests **This change is not bit-identical.** The fused kernel is fp32 throughout — same precision as the fallback, verified from the TileLang kernel's dtype declarations (`pre_fn`, `mixes_partial_out`, `sqrsum_partial_out` and every accumulator are `T.float32`) — but it uses a **split-K reduction**, so the accumulation order differs. MHC feeds a discrete cluster assignment, so a last-bit difference can flip an assignment and diverge the whole generation at temperature 0. Greedy output is therefore **not** reproducible against the fallback. Task accuracy is unaffected. GSM8K 5-shot, `--parallel 16`, 2× DGX Spark TP=2: | build | n | Accuracy | Invalid | | --- | ---: | ---: | ---: | | baseline | 200 | 0.970 | 0.000 | | **this change** | 200 | **0.980** | 0.000 | | baseline (older base, n=1000) | 1000 | 0.943 | 0.000 | | **this change (older base, n=1000)** | 1000 | **0.945** | 0.000 | Both arms are within noise of each other at both sample sizes. Note accuracy is **not** comparable across different `--num-questions`: the same build scores 0.970 at n=200 and 0.943 at n=1000, because the first 200 questions are easier — so an apparent \"drop\" from 0.970 to 0.945 across a resize is an artifact, not a regression. Anyone who relies on bit-exact reproduction against the previous default should set `SGLANG_OPT_FUSE_MHC_POST_PRE=0`. ## Benchmarking and Profiling `sglang.bench_one_batch_server`, bs=1, `SGLANG_SIMULATE_ACC_LEN=5` so the accept length is pinned and the comparison is pure step cost (`acc_length: 5.00` confirmed in both arms). Same container, unlocked clocks, RoCE/RDMA available to both arms. | in / out | baseline | this change + FP8 wo_a (#34018) | | | --- | ---: | ---: | ---: | | 1024 / 512 | 58.48 tok/s | **72.67 tok/s** | +24.3% | | 8192 / 1024 | 61.88 tok/s | **66.76 tok/s** | +7.9% | The two changes were measured together, but a 20-step profile separates them cleanly by kernel family: | kernel family | baseline | patched | attributable to | | --- | ---: | ---: | --- | | fp32 cuBLAS / SIMT (`hc_pre`) | 9.54 ms / 93 calls | **0.86 ms / 8 calls** | **this PR** | | MHC TileLang (fused kernel) | 0.36 ms / 92 calls | **1.75 ms / 180 calls** | **this PR** | | bf16 SM80-WMMA | 20.18 ms / 136 calls | 12.74 ms / 93 calls | FP8 wo_a (#34018) | | `deep_gemm sm120_fp8_fp4_gemm_1d1d` | 0 | 3.29 ms / 43 calls | FP8 wo_a (#34018) | | decode step | 70.31 ms | **63.19 ms** | −10.1% | Net for this PR alone: −8.68 ms of SIMT fp32 GEMM for +1.39 ms of fused TileLang ≈ **−7.3 ms/step**. The 8 remaining fp32 calls are the first layer of each forward, where `prev_residual is None` and the fused post+pre has nothing to fuse against. ### Test environment caveat DSPARK on sm120/sm121 **cannot boot on stock flashinfer today**: the draft's index width is `ceil_align(swa_window + gamma, 64)`, which is 192 for every gamma in [1, 64], while flashinfer 0.6.15.post1 only instantiates decode top-k widths `{128, 512, 1024}`. Both known fixes are still open — flashinfer-ai/flashinfer#4309 (adds the 192 instantiation) and sgl-project/sglang#33407 (dispatch fallback in SGLang). The numbers above were collected with flashinfer#4309 carried as a local patch; `SGLANG_SM120_FLASHMLA_BACKEND=triton` is the only stock alternative and it routes *all* sparse-MLA calls through Triton, which costs throughput on the hot path (measured 4.8x slower end to end here), so it is not a usable baseline. ### On tests No unit test is included. The change is arch-gated to sm120/sm121, and CI has no such runner, so a test would not execute where the behaviour differs. Validation was done on hardware instead: GSM8K for accuracy and a kernel-level profile (reported above) to confirm the intended kernels actually run rather than inferring it from throughput. Happy to add a test if there is a preferred pattern for arch-gated paths. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#running-unit-tests-adding-tests). - [x] Update documentation / docstrings / example tutorials as needed, see [Writing Documentation](https://docs.sglang.ai/developer_guide/contribution_guide.html#writing-documentation-running-docs-ci). - [x] Provide throughput / latency benchmark results and accuracy evaluation results as needed, see [Benchmark and Profiling](https://docs.sglang.ai/developer_guide/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/developer_guide/accuracy_evaluation.html). - [x] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [x] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR. --- ### CI States Latest PR Test (Base): :x: [Run #31660094177](https://github.com/sgl-project/sglang/actions/runs/31660094177) Latest PR Test (Extra): :x: [Run #31660094118](https://github.com/sgl-project/sglang/actions/runs/31660094118) ",
"url": "https://github.com/sgl-project/sglang/pull/34019",
"createdAt": "2026-08-07T19:53:21Z",
"updatedAt": "2026-08-13T08:12:13Z",
"timestamp": "2026-08-13T08:12:13Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci"
],
"author": "zhendonghua",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34037",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Sampling] Bound sampling-mask returns with opt-in truncation",
"text": "## Framing PR #27408 introduced SGLang's native sampling-mask response and its serving/disaggregation plumbing. This PR is standalone, general serving hardening for that shared sampling primitive. It is not RL-specific, and it does not duplicate endpoint exposure or top-p-only admission work. When those request paths are present, they use the same bounded producer implemented here. ## Contract For the weights `w` used to sample token `y`, let `S = {i | w[i] > 0}` and let `C = --sampling-mask-max-tokens`: - sampling itself is unchanged; - if `|S| <= C`, the returned mask is the complete support `S`; - `return_sampling_mask=true` is exact by default, so `|S| > C` returns HTTP 400 rather than an incomplete mask; - `sampling_mask_mode=\"bounded\"` explicitly opts into a mask of at most `C` token IDs; - a bounded mask keeps the highest-weight entries and retains `y` by replacing the final entry only when necessary; - `output_token_sampling_mask_truncated` reports incompleteness for each generated token; - the selected-token logprob always remains `log(w[y] / sum(w))` under the complete post-filter sampling distribution, including when metadata is truncated; - a sampled token outside the captured support remains an internal consistency error. The cap bounds returned metadata only. It never changes the sampling distribution. ## Implementation - Capture support from the sampling producer's representation: - PyTorch reuses the filtered sorted weights and token-ID permutation used by `torch.multinomial`, gathers the selected weight directly, and slices the already-ranked bounded prefix; - FlashInfer joint top-k/top-p uses SGLang's existing merged renormalization primitives and intersects their positive supports, matching joint-filter semantics; - FlashInfer min-p reuses the sequentially renormalized weights passed to min-p sampling and applies the same min-p predicate; - greedy sampling emits the sampled-token singleton. - Separate fact from policy: the sampler emits `OK`, `TRUNCATED`, or `INVALID`; the scheduler accepts `TRUNCATED` only for a bounded request. - Cache opted-in GPU row indices in `SamplingBatchInfo` and maintain them through batch filter/merge operations instead of rebuilding and transferring them every token. - Reuse producer tensors when every row opts in; mixed batches copy only opted-in rows. - Add `--sampling-mask-max-tokens` with a default of 4096. - Keep the selected-token logprob based on the complete support when the returned token list is bounded. - Keep capture tensor-backed through the existing result-holder async D2H primitive and PP logits-metadata transport. Python integers are created only for the realized support length. - Omit redundant row indices from D2H/PP metadata; result processing derives the same stable row order from request flags. - Reach truncation/invalid consensus across the existing attention TP/CP groups before scheduler control flow can diverge. - Carry the per-token truncation bit through streaming, multi-tokenizer output, sessions, PP, and PD disaggregation. PD reuses an unused slot in the existing padded scalar metadata buffer instead of allocating another RDMA buffer. - Abort an exact overflow before committing its sampled token. The shared radix-cache discard path releases the full owned `kv_len_to_handle`, including overlap-scheduled KV slots that intentionally have no committed token ID. - Skip sampling-mask inspection for non-token-producing middle prefill chunks while keeping missing output strict for token-producing chunks. - Reject sampling-mask requests on producer backends that do not emit valid metadata, and keep speculative decoding unsupported until it produces one support per accepted token. ## Performance model - Requests that do not ask for a sampling mask execute no additional CUDA kernels, D2H copies, PP tensors, or TP/CP collectives. - Requested rows require dense `O(R * V)` support reconstruction because the current sampling primitives do not expose a sparse post-filter support. This work is isolated to opted-in rows. - Packing remains fixed-width and asynchronous: token IDs move `4 * R * C` bytes plus 12 scalar bytes per requested row/token. A ragged GPU `nonzero` result would require a device-to-host synchronization, and top-k ties mean `top_k` is not a safe allocation bound. - On H200 with current `main`'s dependency pins (Torch 2.13/CUDA 13, FlashInfer 0.6.17, sglang-kernel 0.4.6.post1), operator probes at `V=128256`, `C=4096` measured about 0.25-0.72 ms of opt-in overhead across the FlashInfer and PyTorch paths. Same-node TP2/4/8 status `all_reduce` measured about 22-44 us. - Measured peak allocation follows the expected row scaling: about 1.7 MB at `R=1`, 54 MB at `R=32`, and 107 MB at `R=64` for `V=128256`, `C=4096`. The existing pinned D2H path moved 16 KB at `R=1` through 1.05 MB at `R=64` in about 57 us p50 on this node. ## Behavior Existing sampling-mask consumers retain an exact-support contract. Consumers that can use bounded metadata opt in per request and receive an explicit per-token truncation signal; they never have to infer truncation from `len(mask) == C`. Requests that do not ask for a sampling mask stay off this path. In a mixed batch, unrelated rows do not pay capture or host-transfer cost. Endpoint exposure and sampling-mask admission policy remain out of scope. ## Tests Coverage includes: - complete support below the cap; - factual truncation above the cap and sampled-token retention; - exact-default HTTP 400 and bounded-mode acceptance; - explicit per-token truncation metadata; - exact selected-token logprob under the complete distribution, distinct from renormalization over the bounded subset; - invalid-support detection; - sampler token-ID mappings and opted-in row isolation; - greedy singleton capture; - PyTorch filtered-weight reuse and top-p capture with unlimited top-k; - cached sampling-mask row indices across batch filter/merge; - all-row tensor reuse and mixed-batch row selection; - PP, streaming/multi-tokenizer, session, and PD metadata transport; - non-final PP prefill chunks with skipped output communication; - current-main hidden-state and Mamba boundary behavior. Validation: - focused unit suites on rebased current `main`: 157 passed, 2 skipped, including 8 subtests; - a 4,927-assertion H200 operator matrix against independent support/logprob oracles using FlashInfer 0.6.17 and sglang-kernel 0.4.6.post1; - H200 end-to-end server checks for exact overflow abort, bounded truncation, explicit truncation metadata, selected-token logprobs, and deterministic sampling preservation; - one-node 8x H200 TP2/4/8 collective probe; - repository-native all-files `pre-commit` pipeline passed; - `git diff --check` passed; - Python bytecode compilation passed for every changed Python file. Registered CUDA/AMD server tests remain assigned to CI. --- ### CI States Latest PR Test (Base): :x: [Run #31621272165](https://github.com/sgl-project/sglang/actions/runs/31621272165) Latest PR Test (Extra): :x: [Run #31621271908](https://github.com/sgl-project/sglang/actions/runs/31621271908) ",
"url": "https://github.com/sgl-project/sglang/pull/34037",
"createdAt": "2026-08-07T21:46:24Z",
"updatedAt": "2026-08-12T17:56:52Z",
"timestamp": "2026-08-12T17:56:52Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "nanjiangwill",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34042",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "add flashinfer cute-dsl backend for mxfp8 gemm",
"text": "WIP for #32950... :D ## Motivation the cute-dsl swap-AB/split-K kernels are a lot faster than the persistent cutlass one for dense mxfp8 gemms, so it should be selectable and the default on sm10x ## Modifications adds an alias in the cli (/srt/server_args.py) for `--fp8-gemm-backend=flashinfer_cutedsl`, an enum for backend == \"flashinfer_cutedsl\" in both Fp8GemmRunnerBackend and Mxfp8DenseGemmBackend, and calls the FlashInfer API with backend=\"cute-dsl\" in (/sglang/srt/layers/quantization/fp8_utils.py) auto now resolves mxfp8 dense to flashinfer_cutedsl on sm10x, sm120 stays on cutlass since there is no cute-dsl kernel for it. flashinfer_cutlass gets its own branch in the resolver so an explicit request doesn't fall through to auto, and the old M <= 64 cutlass -> cute-dsl override is dropped now that the backend is resolved once. also runs flashinfer autotune for the cute-dsl kernel and for mxfp8 checkpoints on sm10x ## Accuracy Tests test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py passes on b300 (sm103), 8 passed 2 skipped ## Speed Tests and Profiling full speedup benchmark across CuTe DSL untuned, CuTe DSL Tuned, CUTLASS and SplitK from valid (divisible by 128, 256) GEMM shapes for M=[1, 2, 4, 8, 16, 32, 64, 128, 256, 512]+ Kimi-k3/GLM-5.2. All kernel latencies use cold-L2 rotating buffers and CUDA graph replay. #### GLM-5.2 744B-A40B shapes | M | N | K | CUTLASS (us) | CuTe untuned (us) | Untuned vs CUTLASS | CuTe tuned (us) | Tuned vs CUTLASS | Selected Split-K | Tune (s) | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1 | 16384 | 2048 | 31.771 | 7.971 | 3.986x | 8.266 | 3.844x | 1 | 1.42 | | 2 | 16384 | 2048 | 30.094 | 8.357 | 3.601x | 8.366 | 3.597x | 1 | 1.42 | | 4 | 16384 | 2048 | 29.056 | 8.382 | 3.466x | 8.586 | 3.384x | 1 | 1.42 | | 8 | 16384 | 2048 | 27.010 | 8.381 | 3.223x | 8.382 | 3.222x | 1 | 1.42 | | 16 | 16384 | 2048 | 27.226 | 8.381 | 3.249x | 8.387 | 3.246x | 1 | 1.42 | | 32 | 16384 | 2048 | 27.166 | 8.586 | 3.164x | 8.381 | 3.242x | 1 | 1.42 | | 64 | 16384 | 2048 | 16.776 | 9.390 | 1.787x | 8.587 | 1.954x | 1 | 1.42 | | 128 | 16384 | 2048 | 17.192 | 9.406 | 1.828x | 9.608 | 1.789x | 1 | 1.42 | | 256 | 16384 | 2048 | 17.800 | 12.480 | 1.426x | 11.451 | 1.554x | 1 | 1.42 | | 512 | 16384 | 2048 | 18.216 | 19.437 | 0.937x | 15.957 | 1.142x | 1 | 1.42 | | 1 | 28672 | 512 | 14.938 | 5.093 | 2.933x | 5.102 | 2.928x | 1 | 1.10 | | 2 | 28672 | 512 | 14.934 | 5.102 | 2.927x | 5.104 | 2.926x | 1 | 1.10 | | 4 | 28672 | 512 | 14.323 | 5.106 | 2.805x | 5.104 | 2.806x | 1 | 1.10 | | 8 | 28672 | 512 | 13.910 | 5.101 | 2.727x | 5.098 | 2.729x | 1 | 1.10 | | 16 | 28672 | 512 | 13.910 | 5.106 | 2.725x | 5.309 | 2.620x | 1 | 1.10 | | 32 | 28672 | 512 | 13.698 | 5.314 | 2.578x | 5.483 | 2.498x | 1 | 1.10 | | 64 | 28672 | 512 | 13.707 | 6.746 | 2.032x | 5.907 | 2.320x | 1 | 1.10 | | 128 | 28672 | 512 | 13.496 | 6.338 | 2.130x | 6.541 | 2.063x | 1 | 1.10 | | 256 | 28672 | 512 | 10.429 | 8.267 | 1.261x | 7.765 | 1.343x | 1 | 1.10 | | 512 | 28672 | 512 | 12.067 | 12.195 | 0.990x | 11.043 | 1.093x | 1 | 1.10 | #### Kimi-K3 shapes | M | N | K | CUTLASS (us) | CuTe untuned (us) | Untuned vs CUTLASS | CuTe tuned (us) | Tuned vs CUTLASS | Selected Split-K | Tune (s) | |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | 1 | 36864 | 7168 | 213.062 | 43.990 | 4.843x | 42.979 | 4.957x | 1 | 2.54 | | 2 | 36864 | 7168 | 184.304 | 44.000 | 4.189x | 42.962 | 4.290x | 1 | 2.54 | | 4 | 36864 | 7168 | 188.090 | 44.246 | 4.251x | 43.190 | 4.355x | 1 | 2.54 | | 8 | 36864 | 7168 | 180.723 | 44.202 | 4.089x | 43.395 | 4.165x | 1 | 2.54 | | 16 | 36864 | 7168 | 186.171 | 44.907 | 4.146x | 43.502 | 4.280x | 1 | 2.54 | | 32 | 36864 | 7168 | 90.102 | 44.006 | 2.047x | 43.802 | 2.057x | 1 | 2.54 | | 64 | 36864 | 7168 | 91.413 | 46.875 | 1.950x | 44.219 | 2.067x | 1 | 2.54 | | 128 | 36864 | 7168 | 98.784 | 47.592 | 2.076x | 46.056 | 2.145x | 1 | 2.54 | | 256 | 36864 | 7168 | 107.546 | 65.933 | 1.631x | 54.094 | 1.988x | 1 | 2.54 | | 512 | 36864 | 7168 | 115.429 | 115.432 | 1.000x | 90.738 | 1.272x | 1 | 2.54 | | 1 | 9216 | 7168 | 54.664 | 13.142 | 4.159x | 13.296 | 4.111x | 1 | 1.82 | | 2 | 9216 | 7168 | 50.478 | 13.189 | 3.827x | 13.293 | 3.797x | 1 | 1.82 | | 4 | 9216 | 7168 | 49.954 | 13.298 | 3.757x | 13.296 | 3.757x | 1 | 1.82 | | 8 | 9216 | 7168 | 51.534 | 13.296 | 3.876x | 13.501 | 3.817x | 1 | 1.82 | | 16 | 9216 | 7168 | 49.896 | 13.702 | 3.641x | 13.699 | 3.642x | 1 | 1.82 | | 32 | 9216 | 7168 | 25.789 | 14.109 | 1.828x | 13.914 | 1.853x | 1 | 1.82 | | 64 | 9216 | 7168 | 26.194 | 17.390 | 1.506x | 14.934 | 1.754x | 1 | 1.82 | | 128 | 9216 | 7168 | 26.816 | 17.637 | 1.520x | 14.926 | 1.797x | 1 | 1.82 | | 256 | 9216 | 7168 | 27.222 | 18.213 | 1.495x | 16.573 | 1.643x | 1 | 1.82 | | 512 | 9216 | 7168 | 28.302 | 29.718 | 0.952x | 24.150 | 1.172x | 1 | 1.82 | Fixes #32950 ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31606673227](https://github.com/sgl-project/sglang/actions/runs/31606673227) Latest PR Test (Extra): :x: [Run #31606672826](https://github.com/sgl-project/sglang/actions/runs/31606672826) ",
"url": "https://github.com/sgl-project/sglang/pull/34042",
"createdAt": "2026-08-07T22:53:49Z",
"updatedAt": "2026-08-13T00:50:02Z",
"timestamp": "2026-08-13T00:50:02Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"documentation",
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "xikronz",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34057",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[CI] /rerun-failed-ci: rerun cancelled runs and target the newest run per workflow",
"text": "`/rerun-failed-ci` only reruns runs whose conclusion is `failure` or `skipped`, so a workflow left `cancelled` (manual cancel, lost runner, cancel race) can never be restarted from a PR comment — you have to click \"Re-run all jobs\" in the UI. It also iterated over every run at the head SHA, and GitHub can have several runs of the same workflow at one commit, so when a newer run superseded an older one the command reran the stale one. Fix: collapse to the newest run per workflow, and accept `cancelled` / `timed_out` alongside `failure` / `skipped`. Those two use a full `run.rerun()` — `rerun_failed_jobs()` rejects a run with no failed jobs, and in a run holding both a failed job and cancelled ones it would leave the cancelled stages unrun. Dry-ran the old vs new selection against real run data: identical on a healthy all-failure PR; on a PR that was stuck with 6 cancelled test workflows, all 6 are now picked up (and the two workflows that had a superseded older run now target the newer one). --- ### CI States Latest PR Test (Base): :no_entry_sign: [Run #31657174782](https://github.com/sgl-project/sglang/actions/runs/31657174782) Latest PR Test (Extra): :x: [Run #31657174674](https://github.com/sgl-project/sglang/actions/runs/31657174674) ",
"url": "https://github.com/sgl-project/sglang/pull/34057",
"createdAt": "2026-08-08T02:23:52Z",
"updatedAt": "2026-08-13T06:44:49Z",
"timestamp": "2026-08-13T06:44:49Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"documentation",
"run-ci"
],
"author": "alisonshao",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34075",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] Keep DECODE token-ids logprob entries tensor-typed under no_copy_to_cpu",
"text": "## Motivation A decode batch is processed for token-ids logprobs whenever **any** request in it asks for them. Requests that did not ask contribute a `None` entry to `token_ids_logprobs_list`, and `get_token_ids_logprobs_raw` turns that entry into a bare `[]`: ```python # python/sglang/srt/layers/logprob_processor.py if stage == LogprobStage.DECODE: for i, token_ids in enumerate(token_ids_logprobs_list): if token_ids is None: vals.append([]) # <-- bare list idxs.append([]) else: ... vals.append(row if no_copy_to_cpu else row.tolist()) # <-- tensor ``` Both DECODE producers call this with `no_copy_to_cpu=True` unconditionally (`OutputLogprobProcessor.compute_logprobs` for the normal decode path, `compute_spec_v2_logprobs` for spec v2), so the val list is heterogeneous — a bare `[]` among GPU tensors — on every **mixed** decode batch. The non-overlap result path then calls `.tolist()` on every entry unconditionally: ```python # python/sglang/srt/managers/scheduler_components/batch_result_processor.py if logits_output.next_token_token_ids_logprobs_val: logits_output.next_token_token_ids_logprobs_val = [ v.tolist() for v in logits_output.next_token_token_ids_logprobs_val ] ``` so the first decode batch that mixes one request with `token_ids_logprob` and one without kills the scheduler: ``` AttributeError: 'list' object has no attribute 'tolist' ``` One logprob client takes the server down for all tenants. Found while running speculative-decoding workloads on a downstream deployment; the repro is plain upstream code (no speculative decoding required — the normal decode producer hits it too). ## Modifications `python/sglang/srt/layers/logprob_processor.py` — in the `stage == LogprobStage.DECODE` / `token_ids is None` arm, return a tensor-typed empty when `no_copy_to_cpu` is set — both branches now flow through the same append, so the entry type rule is structural: ```python if token_ids is None: row = logprobs.new_empty((0,)) # empty entry, same type rule as below else: row = logprobs[i, token_ids_tensor] vals.append(row if no_copy_to_cpu else row.tolist()) ``` `tensor.new_empty((0,)).tolist()` is `[]`, so downstream results are byte-identical to today's non-mixed behavior; the `no_copy_to_cpu=False` path is unchanged. New unit test `test/registered/unit/layers/test_logprob_processor.py` (CPU CI, `base-a-test-cpu`): - mixed batch keeps every val entry tensor-typed (and the empty entry empty); - the **real consumer** (`SchedulerBatchResultProcessor.move_logprobs_to_cpu`) survives a mixed batch and produces `[]` for the request that didn't ask; - the `no_copy_to_cpu=False` path still returns plain lists. ## Accuracy Tests Not a numerics change: the fix only changes the Python type of an empty placeholder (`[]` -> empty tensor whose `tolist()` is `[]`). Logprob values for requests that asked for them are untouched. Red/green on the new test: Without the fix (test alone on current main): ``` $ python3 -m pytest test/registered/unit/layers/test_logprob_processor.py -q ... > v.tolist() for v in logits_output.next_token_token_ids_logprobs_val E AttributeError: 'list' object has no attribute 'tolist' python/sglang/srt/managers/scheduler_components/batch_result_processor.py:421: AttributeError ... FAILED test/registered/unit/layers/test_logprob_processor.py::TestDecodeTokenIdsLogprobsMixedBatch::test_move_logprobs_to_cpu_survives_mixed_batch FAILED test/registered/unit/layers/test_logprob_processor.py::TestDecodeTokenIdsLogprobsMixedBatch::test_none_entry_stays_tensor_typed 2 failed, 1 passed, 19 warnings in 11.21s ``` (the crash is raised from the real consumer, same signature as the server death) With the fix: ``` $ python3 -m pytest test/registered/unit/layers/test_logprob_processor.py -q 3 passed, 19 warnings in 14.28s ``` ## Speed Tests and Profiling No perf-relevant change: one `new_empty((0,))` per non-asking request per decode iteration, only on batches that already compute token-ids logprobs. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31644255091](https://github.com/sgl-project/sglang/actions/runs/31644255091) Latest PR Test (Extra): :x: [Run #31644254839](https://github.com/sgl-project/sglang/actions/runs/31644254839) ",
"url": "https://github.com/sgl-project/sglang/pull/34075",
"createdAt": "2026-08-08T06:10:14Z",
"updatedAt": "2026-08-13T06:16:05Z",
"timestamp": "2026-08-13T06:16:05Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci"
],
"author": "shenxiul",
"state": "open",
"assignees": [
"Qiaolin-Yu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:34076",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] Pin overlap batch tensors freed while forward-stream kernels still read them",
"text": "## Motivation Two use-after-free seams in the overlap scheduler. Both are silent, in-range corruption: the caching allocator hands a freed block to a same-stream reallocation while the **forward stream's** already-enqueued kernels still read it, so the server keeps running and emits wrong tokens. Found while running speculative-decoding workloads on a downstream deployment. **Seam 1 — `_forward_isolation` exit restore drops mid-forward rebinds.** `Scheduler._forward_isolation` snapshots the ScheduleBatch fields before the forward and restores them in `finally:`. Spec-V2 workers rebind SB fields to **fresh tensors during the forward** (e.g. `eagle_prepare_for_verify` replaces `input_ids` / `out_cache_loc` after the snapshot was taken). The restore then drops the rebound tensors' only Python reference while forward-stream kernels may still read them. The existing pins don't cover this window: - `record_batch_in_overlap` pins the **pre-forward** attr values only; - `GenerationBatchResult.extra_keep_alive_refs` pins the verify forward batch (tensors dropped by *mid-forward* rebinds), not the values dropped by the *exit restore* itself. **Seam 2 — `mix_running_indices` freed before the forward-stream gather.** `resolve_forward_inputs` (managers/overlap_utils.py) runs on the forward stream and gathers ```python decode_gpu = future_map.output_tokens_buf[batch.mix_running_indices] ``` `mix_running_indices` is the mixed-in running batch's `req_pool_indices` (`ScheduleBatch.mix_with_running`), a schedule-stream allocation whose host-side reference drops right after the batch is staged — `resolve_forward_inputs` itself clears both staging fields at the end. There is no `record_stream` on it, so the allocator can reuse the block for the scheduler's next allocations while the gather is still queued, and the mixed batch's decode `input_ids` become whatever rows the recycled memory names. ## Modifications - `python/sglang/srt/managers/scheduler.py` — in `_forward_isolation`'s `finally:`, before the restore and only for `overlap=True`, append the **current** (post-forward) attr values into the same 2-iteration `batch_record_buf` ring slot: ```python self.batch_record_buf[self.batch_record_ct].append(list(vars(batch).values())) ``` `vars()` rather than `dataclasses.fields`: the mid-forward rebinds include ad-hoc instance attributes. This is a lifetime pin (like the existing ring slot), not a fence. Docstring updated. - `python/sglang/srt/managers/overlap_utils.py` — `record_stream` the indices on the forward stream before the gather: ```python if batch.mix_running_indices.device.type != \"cpu\": batch.mix_running_indices.record_stream( torch.get_device_module(batch.device).current_stream() ) ``` guarded by device type: CPU-device overlap has no cross-stream reuse hazard, and `Tensor.record_stream` rejects CPU tensors. New unit test `test/registered/unit/managers/test_spec_v2_overlap_lifetime.py` (CPU CI `base-a-test-cpu`; CUDA CI `base-b` / `1-gpu-small`): - exit pin captures mid-forward rebinds (dataclass field AND ad-hoc attr) into the ring slot while the restore still reverts the batch (transactional SB); the non-overlap path pins nothing; - a deterministic GPU repro of seam 2: drives the real `resolve_forward_inputs` on a real side stream behind a `torch.cuda._sleep`, drops the host reference, reallocates the block with decoy pool rows, and asserts the gathered decode `input_ids` are the true relay rows; - a CPU-device mixed gather still works (pins the device-type guard). ## Accuracy Tests Correctness fix for silent corruption; no numerics change on the healthy path (the pins only extend tensor lifetimes). Red/green on the new test (1 GPU): Without the fix (test alone on current main): ``` $ python3 -m pytest test/registered/unit/managers/test_spec_v2_overlap_lifetime.py -q ... E AssertionError: Lists differ: [500, 500, 500, 500, 500, 500, 500, 500, 5[593 chars] 500] != [200, 300, 200, 300, 200, 300, 200, 300, 2[593 chars] 300] ... FAILED test/registered/unit/managers/test_spec_v2_overlap_lifetime.py::TestForwardIsolationExitPin::test_exit_pin_covers_mid_forward_rebinds FAILED test/registered/unit/managers/test_spec_v2_overlap_lifetime.py::TestMixRunningIndicesKeepAlive::test_indices_survive_forward_stream_gather 2 failed, 2 passed, 17 warnings in 16.35s ``` (the `[500, ...]` values are the decoy rows: the queued gather read recycled memory — exactly the production corruption, reproduced deterministically) With the fix: ``` $ python3 -m pytest test/registered/unit/managers/test_spec_v2_overlap_lifetime.py -q 4 passed, 17 warnings in 16.44s ``` Stable across repeated runs, with and without `SGLANG_IS_IN_CI=true` (the debug-assert relay mode). On a CUDA-less machine the GPU repro skips (`3 passed, 1 skipped`). ## Speed Tests and Profiling No kernel or scheduling change. Cost per overlap iteration: one `list(vars(batch).values())` (a few dozen references) appended to the existing ring slot, and one `record_stream` call on mixed prefill+decode batches. Both are O(1) host-side; the ring slot already frees on the same 2-iteration cadence. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31644255351](https://github.com/sgl-project/sglang/actions/runs/31644255351) Latest PR Test (Extra): :x: [Run #31644255366](https://github.com/sgl-project/sglang/actions/runs/31644255366) ",
"url": "https://github.com/sgl-project/sglang/pull/34076",
"createdAt": "2026-08-08T06:10:16Z",
"updatedAt": "2026-08-12T21:51:40Z",
"timestamp": "2026-08-12T21:51:40Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "shenxiul",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34077",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] Fence relay staging writes behind the in-flight forward's tail relay writes",
"text": "## Motivation Under overlap scheduling, the FutureMap relay is written from two streams into the same pool-indexed rows: - the in-flight forward's **tail** relay writes on the forward stream — `on_publish` / the non-spec publish and `_relay_forward_payload` -> `stash` all execute inside `forward_stream_ctx`; - **batch-prep staging** on the schedule stream, which publishes/stashes into req_pool rows that were just freed by the result the scheduler processed and immediately reallocated to a new request: 1. PD-disagg decode bootstrap, non-spec (`ScheduleBatchDisaggregationDecodeMixin.process_prebuilt`); 2. EAGLE disagg spec bootstrap (`build_eagle_disagg_draft_input`, under `enable_overlap`); 3. the hisparse staging->decode rebuild (`Scheduler._build_hisparse_decode_batch`). Nothing orders the two: `publish`/`stash` are plain scatter stores on the current stream. If the rows' previous owner's final forward is still executing when the bootstrap stages the new owner, the stale tail write can land **after** the staging write, and the bootstrapped request's first resolve silently reads the previous owner's bonus token / topk / seq_lens. The values are in-range — no crash, just wrong output. Found while running speculative-decoding workloads on a downstream deployment. The chained `publish_ready` event records (#30435) fence **readers** of the publish event; they do not order these staging scatter **writes** behind the previous owner's forward. The scheduler already uses exactly the needed primitive for the adjacent WAR case (`_apply_war_barrier`: `schedule_stream.wait_stream(forward_stream)`), but batch-prep staging has no equivalent. ## Modifications - `python/sglang/srt/managers/overlap_utils.py` — `FutureMap` gains `set_staging_fence` / `run_staging_fence`. Unwired (direct construction in tests, MLX's stream-free loop) the fence is a no-op. - `python/sglang/srt/managers/scheduler.py` - new `Scheduler._fence_relay_staging`: overlap-only `schedule_stream.wait_stream(forward_stream)`, None-guarded (the stream exists only after `run_event_loop`'s setup); - `run_event_loop` wires it into the FutureMap once the schedule stream exists; `__init__` now pre-sets `self.schedule_stream = None`; - `_build_hisparse_decode_batch` fences before its stash. - `python/sglang/srt/disaggregation/decode_schedule_batch_mixin.py` — the non-spec bootstrap fences before its stash. - `python/sglang/srt/speculative/eagle_disaggregation.py` — the spec bootstrap fences before its publish/stash pair. Placement contract (documented on `run_staging_fence`): the seams invoke the fence **after** their H2D payload materialization and **immediately before** the publish/stash pair. That way only the enqueue-only scatter writes sit behind the fence; a pageable H2D copy on a fenced schedule stream would host-block every bootstrap iteration until the in-flight forward drains. New unit test `test/registered/unit/disaggregation/test_relay_staging_fence.py` (CPU CI `base-a-test-cpu`; CUDA CI `base-b` / `1-gpu-small`): - fence semantics (orders schedule behind forward; inert without overlap or before stream setup) and the FutureMap wired/unwired contract; - `run_event_loop` wires the bound fence; - each of the three seams calls the fence immediately before its relay writes (`[\"run_staging_fence\", \"publish\", \"stash\"]` ordering on the real code paths), and `get_new_prebuilt_batch` does NOT fence at the caller level (the stall hazard above); - an empirical race repro on real CUDA streams: the previous owner's forward tail (`sleep, publish, sleep, stash`) races the real bootstrap seam on a second stream. Unfenced, the bootstrapped request deterministically resolves the stale bonus token and seq_len; with the shipped wiring it resolves its own staged values. ## Accuracy Tests Correctness fix for silent wrong-token output in PD-disagg / hisparse bootstrap under overlap; no numerics change on the healthy path. Red/green on the new test (1 GPU): Without the fix (test alone on current main): ``` $ python3 -m pytest test/registered/unit/disaggregation/test_relay_staging_fence.py -q ... E AssertionError: Lists differ: ['publish', 'stash'] != ['run_staging_fence', 'publish', 'stash'] E AttributeError: 'FutureMap' object has no attribute 'run_staging_fence' E AttributeError: 'Scheduler' object has no attribute '_fence_relay_staging' ... 8 failed, 2 passed, 19 warnings in 13.28s ``` (in the GPU race test the unfenced arm reproduces the race first — the bootstrapped request resolves the stale `(bonus, seq_len) = (111, 40)` instead of its own `(222, 9)` — before the fenced arm fails on the missing fence API) With the fix: ``` $ python3 -m pytest test/registered/unit/disaggregation/test_relay_staging_fence.py -q 10 passed, 19 warnings in 17.11s ``` Stable across repeated runs, with and without `SGLANG_IS_IN_CI=true` (debug-assert relay mode). On a CUDA-less machine the race repro skips (`9 passed, 1 skipped`). ## Speed Tests and Profiling The fence fires only on the three staging seams (request bootstrap / rebuild transitions, not steady-state decode iterations) and only under overlap. It waits on the forward stream *from the schedule stream* — the same wait `_apply_war_barrier` already takes every launch — and the placement contract keeps H2D copies in front of it, so no host block is added. Steady-state TPOT is untouched. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31644255456](https://github.com/sgl-project/sglang/actions/runs/31644255456) Latest PR Test (Extra): :x: [Run #31644255286](https://github.com/sgl-project/sglang/actions/runs/31644255286) ",
"url": "https://github.com/sgl-project/sglang/pull/34077",
"createdAt": "2026-08-08T06:10:18Z",
"updatedAt": "2026-08-12T21:51:42Z",
"timestamp": "2026-08-12T21:51:42Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "shenxiul",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34078",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Spec] Guard multi-layer EAGLE against diverted dp-attention non-extend batches",
"text": "## Motivation `MultiLayerEagleWorkerV2.forward_batch_generation` routes on ```python if batch.forward_mode.is_extend() or batch.is_extend_in_batch: ``` `is_extend_in_batch` is the **global** dp-attention stamp (`layers/dp_attention.py`), so a rank whose **local** batch is not an extend can be diverted into the target-prefill branch while the scheduler nulled `input_ids` for the spec iteration. Today that branch forwards the batch straight into `target_worker.forward_batch_generation`: - the staggered-IDLE rank (a peer prefills while this rank has nothing to run — the only non-extend shape the default spec+dp-attention no-mix sync produces here) crashes the target forward on `input_ids=None`; - a NON-empty locally-DECODE batch (only possible when the no-mix sync is skipped via `--speculative-skip-dp-mlp-sync`) would run much further and fail later in `rotate_input_ids` — an anonymous crash with peer ranks already hung in the dp gather. Found while running multi-rank speculative-decoding workloads on a downstream deployment (dp-attention with staggered prefill arrival). ## Modifications `python/sglang/srt/speculative/multi_layer_eagle_worker_v2.py` — in the extend branch, before the target forward: 1. **Fail-loud tripwire** for a non-empty locally-non-extend divert: a `RuntimeError` naming the batch shape (`bs`, `forward_mode`), raised before the target forward instead of the later anonymous crash. It sits above the bonus-shape condition so a divert with a `None`/mismatched bonus is caught too. 2. **Staggered-IDLE fill**: for the empty batch, populate `input_ids`/`out_cache_loc` from `spec_info.bonus_tokens` and the `req_to_token` rows (both no-op empty tensors), so the IDLE batch flows through the target forward like any other idle forward instead of carrying `None`s. New unit test `test/registered/unit/spec/test_multi_layer_eagle_divert_guard.py` (CPU CI, `base-a-test-cpu`; white-box `__new__` + SimpleNamespace style like the existing spec unit tests): - non-empty DECODE divert raises the RuntimeError before the fill and before the target forward (with and without a bonus-shaped spec_info); - the staggered-IDLE batch reaches the target forward with empty int64 `input_ids`/`out_cache_loc` (no `None` left for downstream padding helpers); - a genuine local extend reaches the target forward untouched. ## Accuracy Tests No numerics change: the fill only affects empty (IDLE) batches, and the tripwire only turns an existing downstream crash into an immediate, named error. Red/green on the new test: Without the fix (test alone on current main): ``` $ python3 -m pytest test/registered/unit/spec/test_multi_layer_eagle_divert_guard.py -q ... E AttributeError: 'NoneType' object has no attribute 'numel' ... FAILED test/registered/unit/spec/test_multi_layer_eagle_divert_guard.py::TestMultiLayerEagleDivertGuard::test_non_empty_decode_divert_fails_loud FAILED test/registered/unit/spec/test_multi_layer_eagle_divert_guard.py::TestMultiLayerEagleDivertGuard::test_non_empty_decode_divert_fails_loud_without_bonus_shape FAILED test/registered/unit/spec/test_multi_layer_eagle_divert_guard.py::TestMultiLayerEagleDivertGuard::test_staggered_idle_fill_noops_into_target_forward 3 failed, 1 passed, 19 warnings in 12.29s ``` (diverted batches reach the target forward with `input_ids=None`; the IDLE batch leaves `None`s behind) With the fix: ``` $ python3 -m pytest test/registered/unit/spec/test_multi_layer_eagle_divert_guard.py -q 4 passed, 19 warnings in 16.64s ``` ## Speed Tests and Profiling Not perf-relevant: two host-side checks on the extend branch; the fill runs only for empty (IDLE) diverted batches. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31644255789](https://github.com/sgl-project/sglang/actions/runs/31644255789) Latest PR Test (Extra): :x: [Run #31644255509](https://github.com/sgl-project/sglang/actions/runs/31644255509) ",
"url": "https://github.com/sgl-project/sglang/pull/34078",
"createdAt": "2026-08-08T06:10:19Z",
"updatedAt": "2026-08-13T06:16:54Z",
"timestamp": "2026-08-13T06:16:54Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "shenxiul",
"state": "open",
"assignees": [
"Qiaolin-Yu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:34090",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix GLM streaming unknown-tool filtering",
"text": "## Motivation GLM-4 and GLM-4.7 streaming parsers currently emit undeclared tool names even when `SGLANG_FORWARD_UNKNOWN_TOOLS` is disabled, unlike their non-streaming paths. This can expose a call that the request did not advertise. Fixes #33902. ## Modifications - centralize the existing unknown-tool policy in `BaseFormatDetector` - validate each GLM streaming function name once it is complete - consume a disabled unknown call without emitting name or argument deltas - continue parsing a following declared call without consuming a tool index - add coverage for both detectors with forwarding disabled and enabled ## Accuracy Tests Targeted CPU parser results: - unknown-tool policy tests: 6 passed - existing GLM-4 and GLM-4.7 detector tests: 23 passed, 182 deselected No model weights or numerical kernels are changed. ## Speed Tests and Profiling Not applicable. The change adds one dictionary membership and environment-policy check when a streaming function name becomes complete. ## Checklist - [x] Format checked with Black 26.1.0 - [x] Added CPU unit tests - [x] Ruff checks passed - [x] No documentation change is required for this bug fix --- ### CI States Latest PR Test (Base): :x: [Run #31245894297](https://github.com/sgl-project/sglang/actions/runs/31245894297) Latest PR Test (Extra): :x: [Run #31245894209](https://github.com/sgl-project/sglang/actions/runs/31245894209) ",
"url": "https://github.com/sgl-project/sglang/pull/34090",
"createdAt": "2026-08-08T07:19:10Z",
"updatedAt": "2026-08-12T16:32:09Z",
"timestamp": "2026-08-12T16:32:09Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "Ricardo-M-L",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:34140",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [Spec] Enable stochastic tree verification on ROCm",
"text": "## Motivation ROCm EAGLE verification currently routes every HIP request through greedy `argmax`, even when `temperature`, `top_p`, or `top_k` request stochastic sampling. ROCm also lacks the CUDA target-only tree-sampling kernel and its top-k/top-p renormalization operators. While validating a portable tree verifier with GLM-5.2-FP8, EAGLE top-k > 1 exposed a second blocker: DSA’s page-64 index cache received physical token-slot IDs but indexed them as page rows: ```python index_k[tgt_loc_flat] = index_k[src_loc_flat] ``` On MI355X this failed during draft cache preparation, before target verification: ```text HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception (0x1016) torch.AcceleratorError: HIP error: unspecified launch failure ``` Both the stochastic verifier and page-aware DSA relocation are required for a complete ROCm top-k > 1 tree path. ## Modifications ### Target-only tree verification - `python/sglang/kernels/ops/speculative/tree_sampling.py` - Add the Triton target-only stochastic tree verifier and residual bonus-token sampling. - `python/sglang/kernels/ops/speculative/__init__.py` - Register the Triton tree-sampling operator for CUDA and HIP. - `python/sglang/srt/speculative/eagle_utils.py` - Route non-greedy HIP EAGLE requests to the Triton tree verifier; preserve chain rejection sampling and greedy dispatch. - `python/sglang/srt/speculative/dflash_utils.py` - Enable the same target-only Triton verifier for DFLASH on HIP. - `python/sglang/srt/speculative/spec_utils.py` - Advertise tree-kernel availability on HIP. ### Portable probability renormalization - `python/sglang/kernels/ops/sampling/renorm.py` - Add exact threshold-based top-k/top-p pivots, tie handling, and full-sort overflow fallback. - `python/sglang/kernels/ops/sampling/renorm_triton.py` - Fuse pivot application, masked reduction, and normalization for HIP. - `python/sglang/kernels/ops/sampling/__init__.py` - Select CUDA AOT, HIP Triton, or portable Torch implementations through the kernel registry. - `python/sglang/srt/layers/sampler.py` - Reuse the portable top-p semantics in the Torch sampler path. ### Page-aware DSA tree relocation - `python/sglang/kernels/ops/attention/dsa/index_buf_accessor.py` - Add overlap-safe gather/scatter kernels that move each token’s 128-byte FP8 index key and 4-byte FP32 scale by `(page, offset)`. - Preserve AITer’s 16x16 preshuffled K layout and CUDA’s row-major layout. - `python/sglang/srt/mem_cache/index_key_cache.py` - Replace token-slot-as-page-row indexing with logical K/scale relocation. - `python/sglang/srt/mem_cache/dsa_cache_layer_split.py` - Remove the duplicated broken override so layer-sharded caches inherit the corrected implementation. ### Tests - `test/registered/kernels/ops/speculative/test_tree_speculative_sampling_triton.py` - `test/registered/kernels/ops/speculative/test_spec_verify_distribution.py` - `test/registered/kernels/ops/attention/test_dsa_index_cache_move.py` ## Accuracy Tests ### MI355X unit tests on final PR commit ```text TestTreeSpeculativeSamplingTriton.test_matches_torch_oracle TestTreeSpeculativeSamplingTriton.test_seeded_inputs_are_repeatable TestTreeSpeculativeSamplingTriton.test_dflash_sampling_integration_matches_oracle TestTreeSpeculativeSamplingTriton.test_branched_tree_preserves_target_distribution TestTreeSpeculativeSamplingTriton.test_matches_cuda_aot_oracle [skipped on ROCm] TestSpecSamplingDispatch.test_hip_target_only_uses_triton_tree TestSpecSamplingDispatch.test_rejection_sampling_uses_triton_chain TestSpecVerifyDistribution.test_chain_sampler_preserves_target_distribution TestSpecVerifyDistribution.test_greedy_verify_collapses_to_argmax TestPortableSpecRenorm.test_top_k_scalar_and_per_row TestPortableSpecRenorm.test_top_p_per_row_and_zero_mass TestDSAIndexCacheMove.test_cross_page_overlap_and_identity TestDSAIndexCacheMove.test_empty_move TestDSAIndexCacheMove.test_layer_split_skips_non_owned_buffers TestDSAIndexCacheMove.test_page64_tree_prefix_tail_duplication ``` | Test file | Result | |---|---:| | `test_tree_speculative_sampling_triton.py` | 4 passed, 1 CUDA-only skip | | `test_spec_verify_distribution.py` | 5 passed, 1 sgl-kernel-only skip | | `test_dsa_index_cache_move.py` | 4 passed | | Pre-commit, all 15 changed files | Passed | CUDA-only registered renormalization oracles: ```text TestSpecRenormFallbacks.test_top_k_fallback_matches_kernel TestSpecRenormFallbacks.test_top_p_fallback_matches_kernel ``` ### GLM-5.2-FP8 customer workload Apex Shortlist problem 4, gold answer `2618`, `n=8`, `temperature=1.0`, `top_p=0.95`, `max_tokens=131072`, EAGLE top-k 2 / 5 steps / 6 draft tokens. | Platform | Correct | Stop | Length | Completion tokens | Server exceptions | |---|---:|---:|---:|---:|---:| | MI355X, TP8 | 7/8 | 8 | 0 | 317,649 | 0 | | GB300, TP4 | 7/8 | 8 | 0 | 357,376 | 0 | MI355X top-k 3 stress: 27,681 completion tokens, 0 server exceptions. ## Speed Tests and Profiling No valid end-to-end before/after comparison exists: the previous HIP path silently executed greedy verification instead of the requested stochastic algorithm, and DSA top-k > 1 page-64 serving crashed before completion. Kernel optimization will follow separately. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No user-facing API change.) - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). (No semantically valid baseline exists.) - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31638149782](https://github.com/sgl-project/sglang/actions/runs/31638149782) Latest PR Test (Extra): :x: [Run #31638149526](https://github.com/sgl-project/sglang/actions/runs/31638149526) ",
"url": "https://github.com/sgl-project/sglang/pull/34140",
"createdAt": "2026-08-09T00:52:35Z",
"updatedAt": "2026-08-13T04:35:42Z",
"timestamp": "2026-08-13T04:35:42Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"speculative-decoding",
"run-ci",
"jit-kernel"
],
"author": "Raiden-Makoto",
"state": "open",
"assignees": [
"1am9trash",
"RolaoDenthu"
]
},
{
"id": "github:sgl-project/sglang:pull_request:34141",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Reserve multimodal runtime allocations and keep padded inputs aligned",
"text": "## Motivation The multimodal embedding cache (`SGLANG_VLM_CACHE_SIZE_MB`) and the GPU feature-transport pools (`SGLANG_MM_FEATURE_CACHE_MB`, used by `--mm-feature-transport cuda_ipc`/`cuda_vmm`) allocate GPU memory only **after** the KV pool is sized: the embedding cache fills lazily per rank at runtime, and the feature pools are created on `base_gpu_id` when the HTTP tokenizer workers start. Memory profiling never sees either allocation, so the KV pool absorbs their bytes and the late allocations must fit into the runtime slack shared with CUDA graphs / NCCL / activations — risking rank-0 startup OOM or squeezed headroom. This bites hardest when the budgets are raised, e.g. scaling `SGLANG_MM_FEATURE_CACHE_MB` together with `--tokenizer-worker-num` (the budget is divided across workers, so keeping per-worker capacity requires raising the total). Padding token IDs without padding optional token-aligned embeddings can also leave batch inputs with mismatched row counts. ## Modifications - New `mm_runtime_reservation_gb()` in `kv_cache_configurator.py`: for multimodal models, reserves `SGLANG_VLM_CACHE_SIZE_MB`, plus `SGLANG_MM_FEATURE_CACHE_MB` when the feature transport is `cuda_ipc`/`cuda_vmm`; logs the reservation at boot. - The reservation is subtracted in **both** KV sizing passes: initial profiling (`_profile_available_bytes`) and the post-capture resize (`compute_post_capture_kv_resize`) — the resize re-derives the budget from measured free memory and would otherwise silently undo the initial reservation. - Pad `input_embeds` together with other token-aligned inputs. The feature pools physically land on one GPU only, but KV pools must be sized uniformly across ranks, so the reservation applies to every rank (conservative on the others). ## Accuracy Tests - Focused CPU regression check for padded input embeddings: passed. - No model-output changes from the memory-planning update. ## Speed Tests and Profiling No kernel/runtime changes; the KV pool shrinks by exactly the configured reservation. Verified on an internal multimodal deployment (tp=4, `cuda_vmm` transport, embedding cache 2048 MB, feature budget 1024 MB): the 3.00 GB reservation is logged in both sizing passes, the final KV pool is smaller by exactly the reserved bytes, and serving works. ## Original commits - `825ab512f` - `023e82c2c` ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31567036115](https://github.com/sgl-project/sglang/actions/runs/31567036115) Latest PR Test (Extra): :x: [Run #31567036010](https://github.com/sgl-project/sglang/actions/runs/31567036010) ",
"url": "https://github.com/sgl-project/sglang/pull/34141",
"createdAt": "2026-08-09T01:38:41Z",
"updatedAt": "2026-08-12T18:04:08Z",
"timestamp": "2026-08-12T18:04:08Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"documentation",
"Multi-modal",
"run-ci",
"bypass-fastfail"
],
"author": "oulgen",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34153",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Scheduler] Fix final chunked-prefill abort commit race",
"text": "## Motivation Fixes #34149. Related prior report: #34112. While testing chunked-prefill cancellation on latest `main`, we independently reproduced the same user-visible symptom reported in #34112: a cancelled request can expose one token and then finish with `abort`. The #34112 attachment used SGLang v0.5.9. Its negative batch-level output IDs were old FutureMap relay handles and are not the failure criterion or target of this PR. Current `main` no longer uses that `ScheduleBatch.output_ids` representation. Our deterministic current-main reproduction isolates a final-prefill overlap race. An abort can arrive after the final chunked-prefill `EXTEND` is launched but before its delayed result is committed. At that point `scheduler.chunked_req` is already clear, `Req.output_ids` is empty, and the result is pending in `result_queue`. ## Root cause There were two commit boundaries to protect: 1. `abort_request()` records `req.to_finish = FINISH_ABORT()`, but the prefill result processor previously appended the sampled token and token-derived metadata before `update_finish_state()` consumed the pending finish. 2. In the CUDA/HIP overlap loop, scheduling the next batch happens before processing the delayed final-prefill result. Without an admission guard, the pending-finish prefill request can enter optimistic decode, increment `decode_batch_idx` / the KV committed watermark, and perform an optimistic decode step and its associated allocation before the result is dropped (which may reserve multiple slots or pages under speculative decoding). Filtering only in the output streamer is too late: request output, reasoning state, logprobs, hidden states, sampling metadata, grammar state, and KV accounting may already be mutated. ## Modifications - When a delayed result is actually pending, exclude pending-finish prefill requests from the prefill-to-decode merge before optimistic decode preparation. - Keep decode requests in mixed batches on their existing partial-output path. - In the final-prefill result processor, promote the pending finish with `new_accepted_len=0` and reuse normal finished-request cleanup. - Do not commit the dropped token's output ID, reasoning state, hidden state, customized info, sampling mask, returned logprob values, or grammar state. - Still advance packed hidden-state and logprob cursors so later live requests in the same batch read their own slices. - Preserve synchronous, MLX, and PP scheduling semantics by applying the merge guard only while `result_queue` contains a delayed result. - Leave PD-prefill transfer lifecycle unchanged; it uses a separate scheduler/result path. ## Regression coverage The scripted test deterministically waits for this window: - the request has passed through chunked prefill; - the final `EXTEND` is launched; - `scheduler.chunked_req is None`; - no middle chunks remain in flight; - `Req.output_ids` is empty; - the final result is pending. It then aborts and verifies: - finish reason is `FINISH_ABORT`; - no token is appended; - no optimistic decode step occurs; - the request-pool row and KV object are released; - reclaimable KV capacity returns to its baseline after the scheduler drains. Additional CPU tests cover both sides of the scheduler gate (pending queue vs. already-processed result), mixed prefill/decode membership, and a two-request packed-logprob case where the aborted segment advances the cursor without attaching metadata to the aborted request. ## Accuracy Tests Cancellation control flow only; normal token generation is unchanged. Validated on top of `main` commit `ec9babe36cc172cb5d7f3882547718e99ddb2e0c`: ```text PYTHONPATH=python .venv/bin/python -m pytest -q \\ test/registered/unit/managers/test_scheduler_chunked_req_gate.py \\ test/registered/unit/managers/test_batch_result_processor_hidden_states.py \\ test/registered/unit/managers/test_batch_result_processor_spec_grammar.py 11 passed, 2 subtests passed ``` ```text SGLANG_USE_MLX=1 PYTHONPATH=python .venv/bin/python \\ test/registered/scripted_runtime/test_scripted_runtime_core.py \\ TestScriptedRuntimeCore.test_abort_at_last_chunk_does_not_append_output Ran 1 test in 125.143s OK ``` Also passed: - direct CI-style execution of the two modified CPU test files; - Python compile checks for all touched files; - registered-test and package-registration repository checks; - Black 26.1.0, Ruff 0.15.1, and `git diff --check`. The packed-logprob test was mutation-checked: it fails both if the aborted request attaches returned logprobs and if its packed cursor contribution is skipped. ## Speed Tests and Profiling Not applicable. The new checks are scheduler control-flow predicates; the result-drop branch runs only for a request that is already finishing. ## Topology notes The race is not caused by TP, DP, or PP. TP/DP use the same per-request result state. PP and MLX process their pending result before the relevant `last_batch` is reused, so the queue-gated admission change is a no-op there while the result-drop protection remains valid. PD-prefill is intentionally out of scope because it has a separate transfer and result lifecycle. The registered end-to-end regression is TP=1 / DP=1 / PP=1. Multi-rank topology behavior was source-reviewed but not claimed as a multi-GPU runtime validation. ## Checklist - [x] Rebased onto current `main`. - [x] Added deterministic registered regression coverage. - [x] Added focused scheduler and packed-metadata unit coverage. - [x] Ran relevant unit and scripted-runtime tests. - [x] Ran formatting, compile, and test-registration checks. - [x] Documentation and performance benchmarks are not applicable to this abort-only correctness fix. --- ### CI States Latest PR Test (Base): :x: [Run #31717674760](https://github.com/sgl-project/sglang/actions/runs/31717674760) Latest PR Test (Extra): :x: [Run #31717674527](https://github.com/sgl-project/sglang/actions/runs/31717674527) ",
"url": "https://github.com/sgl-project/sglang/pull/34153",
"createdAt": "2026-08-09T03:59:16Z",
"updatedAt": "2026-08-13T15:52:34Z",
"timestamp": "2026-08-13T15:52:34Z",
"metrics": {
"reactions": 0,
"comments": 7
},
"labels": [],
"author": "jeremyzhang866",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34157",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Tests] Add regression tests for reasoning parser edge cases",
"text": "## Motivation SGLang issue #20865 asks contributors to improve unit-test coverage for core runtime components that don't require a GPU, specifically calling out `test_reasoning_parser.py` as a starting point. The reasoning parser's streaming path (`parse_streaming_increment`) has several edge cases that were not previously covered: malformed/nested tags, Unicode boundaries that span multiple streaming chunks, and behavior consistency between streamed and full-input parsing. Gaps here can lead to silent state corruption, dropped output, or divergent behavior between streaming and non-streaming code paths in production. ## Modifications Added 4 new regression tests to `TestBaseReasoningFormatDetector` in `test/registered/unit/parser/test_reasoning_parser.py`: - **`test_parse_malformed_nested_tags`** — verifies nested `![\"image\"](\"https://github.com/user-attachments/assets/8987eed1-ff49-4800-8ab2-f02472f35892\")
## Modifications 1. Add a environment variable `SGLANG_ENABLE_RANK_CONSENSUS_CHECKER`, which enables the consensus checker. **Default to false**. 2. Add `rank_consensus_checker.py`. It simply creates a background thread. When `assert_same` is called, it pushes a message to a queue. The background thread consumes the queue of events, and compares events with other ranks. Unless divergence, all ranks must have the same sequence of events. 3. Add `@rank_consenus` to a few functions in hicache code. This work is in progress. Feel free to talk to me if you have better idea, e.g. more convenient decorator, better naming, which code to add the decorator, etc. ## Test ### Effectiveness With rank consensus checker enabled, we could detect PP divergence with L3 in a few minutes. The problem is fixed after patching #27010. Use the following command to reproduce this. ```sh nohup env HF_HUB_OFFLINE=1 SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1 \\ sglang serve --model-path Qwen/Qwen3.5-27B \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen3_coder \\ --mamba-radix-cache-strategy extra_buffer \\ --mem-fraction-static 0.8 \\ --tp-size=1 \\ --pp-size=2 \\ --host 0.0.0.0 \\ --port 30000 \\ --max-total-tokens=$((64*1024)) \\ --hicache-ratio=2 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"128gb\", \"protocol\": \"tcp\"}' \\ > sglang.log 2>&1 & python benchmark/hicache/bench_multiturn.py \\ --num-clients=32 \\ --num-rounds=10 \\ --disable-auto-run \\ --model=Qwen/Qwen3.5-27B \\ --enable-round-barrier \\ --request-rate 1 ``` ### Performance Tested 3 groups: - A: Baseline (4a5d7d3c). - B: This PR, with consensus checker **disabled** - C: This PR, with consensus checker **enabled** Conconlusion: There is no overhead when then checker is disabled compared the baseline. ![\"image\"](\"https://github.com/user-attachments/assets/d1c262ce-e60f-4bcf-8130-91272cbd8f73\")
Testing commands: ```sh nohup env HF_HUB_OFFLINE=1 SGLANG_ENABLE_RANK_CONSENSUS_CHECKER=1 \\ sglang serve --model-path Qwen/Qwen3.5-27B \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen3_coder \\ --mamba-radix-cache-strategy extra_buffer \\ --mem-fraction-static 0.8 \\ --tp-size=2 \\ --pp-size=1 \\ --host 0.0.0.0 \\ --port 30000 \\ --max-total-tokens=$((64*1024)) \\ --hicache-ratio=2 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"128gb\", \"protocol\": \"tcp\"}' \\ > sglang.log 2>&1 & python benchmark/hicache/bench_multiturn.py \\ --num-clients=32 \\ --num-rounds=10 \\ --disable-auto-run \\ --model=Qwen/Qwen3.5-27B \\ --enable-round-barrier \\ --request-rate 1 ``` ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 4. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 5. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 6. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31576776545](https://github.com/sgl-project/sglang/actions/runs/31576776545) Latest PR Test (Extra): :no_entry_sign: [Run #31661211450](https://github.com/sgl-project/sglang/actions/runs/31661211450) ",
"url": "https://github.com/sgl-project/sglang/pull/34406",
"createdAt": "2026-08-11T10:29:16Z",
"updatedAt": "2026-08-13T02:34:34Z",
"timestamp": "2026-08-13T02:34:34Z",
"metrics": {
"reactions": 4,
"comments": 1
},
"labels": [
"documentation",
"run-ci",
"run-ci-extra"
],
"author": "stepinto",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34407",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[sgl-router] Worker Snapshot Support for Recoverable KV Placement State in sgl-router",
"text": "## Motivation KV-aware routing depends on an accurate view of the KV blocks available on each worker replica. Previously, sgl-router built this view only from live `BlockStored` and `BlockRemoved` events. If the Router started after the Worker cache had already been populated, or if the Router restarted and lost its in-memory index, the existing KV placement state could not be reconstructed. Missing or stale placement metadata does not affect inference correctness, but it reduces cache-aware routing quality and may lower the KV cache hit rate. This PR implements the snapshot bootstrap and full-recovery portion of #33394. Short-gap recovery through the replay endpoint remains separate follow-up work related to #32729. ## Modifications ### Worker-side snapshot provider - Add an optional `snapshot_endpoint` to the built-in ZMQ KV-event publisher. The endpoint is exposed independently for each routable DP or attention-DP replica using `base_port + dp_rank`. - Maintain an in-memory KV placement mirror inside the publisher thread. Sequence assignment, event application, and snapshot capture are serialized by the same thread, so the snapshot and its live-event barrier represent one consistent placement cut. - Include the replica lifecycle `epoch`, `replica_rank`, `barrier_seq`, `resume_seq`, `barrier_id`, and the active block records required to reconstruct parent relationships. - Stream large snapshots in chunks of up to 4096 records. The chunk size limits each ZMQ message, not the total snapshot size. - Scope the publisher epoch to one DP replica lifecycle, so restarting one DP rank does not invalidate the other ranks. - Keep the replay buffer complete after a sequence number is assigned even if live PUB delivery fails, and index the snapshot mirror by `block_hash` for direct removal. ### Discovery and compatibility - Advertise the optional snapshot host, base port, and protocol version through `/server_info`. An invalid or absent snapshot configuration does not hide an otherwise valid live KV-event publisher. - Append NUL-delimited epoch and snapshot-barrier metadata to the ZMQ topic frame only for snapshot-capable publishers. The sequence frame and event payload remain unchanged. - Preserve compatibility with existing ZMQ consumers because SUB filters are prefix-based. - Keep workers without a snapshot endpoint in the existing `LegacyBestEffort` live-only mode. ### Router-side recovery For each `(worker_url, dp_rank)` replica, sgl-router now: 1. subscribes to the live event stream; 2. fetches the replica snapshot while buffering live events; 3. waits for and validates the matching snapshot barrier; 4. verifies contiguous sequence numbers and applies post-snapshot updates; 5. replaces the replica placement state in the HashTree; and 6. marks the replica `READY`. One snapshot is fetched per independently routable DP or attention-DP replica, rather than per TP rank. Snapshot-capable replicas remain absent from the cache-aware placement tree until synchronization completes. Normal routing fallback remains available while a replica is synchronizing. A sequence gap, epoch change, publisher reset, decode failure, or invalid snapshot clears only the affected replica placement state and triggers a fresh synchronization. Snapshot retries use bounded buffering, cancellation-aware tasks, and exponential backoff. ### Code organization - Add `snapshot.rs` for the chunked snapshot wire protocol and validation. - Add `sync.rs` for bootstrap, barrier catch-up, replica state transitions, gap detection, and resynchronization. - Keep `index.rs` as the lifecycle facade for worker discovery, subscriber management, and snapshot task ownership. ## Accuracy Tests This PR does not change model execution, kernels, or generated outputs. The following tests passed: ```bash PYTHONPATH=python python3 -m pytest -q \\ test/registered/unit/disaggregation/test_kv_events.py # 10 passed PYTHONPATH=python python3 -m pytest -q \\ test/registered/unit/entrypoints/test_server_info.py # 17 passed cd experimental/sgl-router cargo test --lib policies::kv_events # 102 passed cargo test --test component # 41 passed ``` The snapshot recovery E2E test also passed with two Workers and one Router: - KV blocks populated before Router startup were recovered from snapshots. - Live KV events produced after the Router became ready were applied incrementally. - After restarting the Router, the placement view was rebuilt and the same cached prefixes continued to route to the expected Workers. The legacy compatibility E2E test also passed: Workers without a snapshot endpoint continued to use live-only indexing, and two Router instances converged to the expected prefix routing results. ## Speed Tests and Profiling No model throughput benchmark was run because this PR does not modify the inference or request-forwarding path. Snapshot capture uses the publisher in-memory placement mirror, and snapshot network transfer runs on a separate thread using bounded chunks. Normal live-event processing remains incremental. ## Checklist - [x] Format the code according to the repository pre-commit configuration. - [x] Add unit tests for snapshot capture, chunking, epoch isolation, protocol validation, recovery, failure handling, and legacy compatibility. - [x] Document the snapshot protocol and synchronization lifecycle in the implementation. - [x] Provide accuracy and speed-test applicability and E2E validation results. - [x] Follow the existing SGLang Python and Rust code organization and style. Related RFC: #33394 Related replay work: #32729 --- ### CI States Latest PR Test (Base): :x: [Run #31676895343](https://github.com/sgl-project/sglang/actions/runs/31676895343) Latest PR Test (Extra): :x: [Run #31676895141](https://github.com/sgl-project/sglang/actions/runs/31676895141) ",
"url": "https://github.com/sgl-project/sglang/pull/34407",
"createdAt": "2026-08-11T11:17:10Z",
"updatedAt": "2026-08-13T07:14:33Z",
"timestamp": "2026-08-13T07:14:33Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "skajre",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34411",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[VLM] Reuse cached Kimi-K3 embeddings before preprocessing",
"text": "## Summary Stacked on #34404; review `fd40b11ac0cb0..63cf4f894c57c` for the PR-3-only diff. - acquires per-image embedding-cache leases on the request's final DP route before processor dispatch, and omits features only when every required TP rank has pinned the same entries - transfers lease ownership to the admitted request; consume, cancel, abort, flush, timeout, and internal retry paths release pins safely - moves EPD metadata and local/global embedding lookup before decode/processor/ViT through a model-declared, per-modality capability; K3 is the first adapter, with multi-image partial hits and deduplication - preserves CPU, CUDA IPC, and CUDA VMM compatibility; featureless references do not allocate IPC/VMM proxies or a new fixed HBM pool - adds low-cardinality identity, preprocess phase, embedding acquire, skipped-stage, occupancy, eviction, and single-flight metrics - adds typed Protocol/resolver interfaces instead of model-name branches or `getattr` probing, plus the canonical `VLM preprocessing and caching` guide and concise API, EPD, model-support, new-model, server-argument, and K3 Cookbook references ## Correctness and failure behavior - false hits are prevented in three stages: a type-preserving artifact identity covers exact media plus processor configuration; a full feature identity binds it to the processor-output hash; embedding lookup uses a compact 64-bit key but requires exact feature-identity equality - identity matching is symmetric: an identity-less request cannot match a strong entry, and inserting either identity kind replaces an unpinned entry of the other kind instead of inheriting it - K3 namespaces Radix prefix KV by the ordered full feature identities, so two media items whose legacy pad sentinels collide after the 30-bit reduction still cannot share language-model KV; a partial identity list is rejected rather than creating an incomplete namespace - same path/URL with changed bytes, processor/backend/config/per-image option changes, and tensor/PIL metadata changes cannot reuse an artifact; unsupported identity values are rejected - leases atomically pin cache entries against eviction and become request-owned after scheduler admission - partial TP hits and expired leases retry internally through the feature-carrying path - feature-hash mismatch clears the stale processor artifact and rejects reuse - K3 EPD carries feature identities alongside image sizes and grids to the language server, so reconstructed items use the same embedding and Radix identity checks as normal serving - K3 Mooncake/global embedding-cache entries are keyed by the full feature identity; a caller-provided global-cache key is combined with, rather than substituted for, that identity - caller-provided legacy `mm_hashes` changes only the router/pad sentinel; K3 stores its processor-derived embedding key separately, so a caller hash cannot redirect lease admission, embedding lookup, or cache write-back - untrusted identity still reads and verifies actual media bytes; only `--trust-mm-content-hashes` lets a metadata hit skip the source read - cache flush, model update, shutdown, cancellation, and timeout clear their preprocess state and outstanding leases `--trust-mm-content-hashes` is the explicit correctness boundary: a trusted hot hit does not read the source, so a stale or incorrect caller hash can return a stale artifact. Without that flag, SGLang verifies caller hashes against the actual media snapshot and rejects mismatches. In other words, false hits are fail-closed under the default configuration. A 64-bit embedding-key collision only selects a candidate and still requires the full SHA-256 feature identity to match; a 30-bit pad collision is isolated by the full-identity Radix namespace; mutable paths and URLs are re-read and re-hashed. The only intentional exception is trusted-hash mode, where correctness is explicitly delegated to the caller. ## Normal-serving performance Full real-weight Kimi-K3 (`1.56 TB`), 2 nodes x 4 NVIDIA GB300, TP8, commit `25eae3c909d7`. Each cell is the median of 3 runs over 32 conversations x 8 turns. The same image set is resent on every turn; random images are seeded from 512x512 through 2048x2048. Both sides use the same 4096 MiB lazy embedding-cache cap and auto CUDA VMM/FABRIC feature transport. The only A/B switch is K3 auto versus `--mm-preprocess-cache-size-mb 0`. Four random images, concurrency 32, 16 output tokens: | Request rate | Output tok/s off -> on | Median TTFT off -> on | Median E2E off -> on | |---:|---:|---:|---:| | 1 | 53.29 -> 100.14 (+87.9%) | 5846 -> 1122 ms (-80.8%) | 6364 -> 1479 ms (-76.8%) | | 4 | 53.83 -> 109.46 (+103.3%) | 6128 -> 1148 ms (-81.3%) | 6721 -> 1482 ms (-78.0%) | | 8 | 55.53 -> 107.57 (+93.7%) | 5830 -> 1174 ms (-79.9%) | 6294 -> 1502 ms (-76.1%) | | inf | 54.33 -> 106.05 (+95.2%) | 5973 -> 1133 ms (-81.0%) | 6366 -> 1479 ms (-76.8%) | The `inf` A/B was repeated on the same physical nodes. Cache-on duration fell from 75.39 s to 38.62 s. Median hot-turn TTFT fell from 5956 to 1147 ms (-80.8%). The first cold turn did not regress (19307 -> 15830 ms at saturated concurrency). Representative matrix: | Case | Output tok/s off -> on | Median TTFT off -> on | Median TPOT off -> on | |---|---:|---:|---:| | 4 random images, c1, o16 | 21.20 -> 29.90 (+41.0%) | 501 -> 297 ms | 12.12 -> 10.89 ms | | 4 random images, c8, o16 | 44.58 -> 80.69 (+81.0%) | 1874 -> 639 ms | 15.07 -> 13.67 ms | | 1 fixed image, c32, o16 | 197.22 -> 283.14 (+43.6%) | 1702 -> 976 ms | 21.10 -> 19.60 ms | | 4 fixed images, rate 8, c8, o16 | 59.47 -> 98.01 (+64.8%) | 1471 -> 623 ms | 14.75 -> 13.47 ms | | 4 random images, rate 8, c32, o128 | 305.14 -> 488.26 (+60.0%) | 6287 -> 1404 ms | 33.86 -> 30.49 ms | The measured 128 cold images produced 896 metadata and embedding hits. All 896 hot images skipped processor work, feature transport, and ViT. Aggregate preprocess-stage time fell from 194.28 s to 24.56 s (-87.4%). The CPU cache held 128 entries / 62,692 bytes with no eviction. Peak HBM was 257,304 MiB cache-on versus 265,202 MiB cache-off (~7.7 GiB lower); there is no new fixed HBM reservation. The environment's auto FlashInfer MXFP4 TRT-LLM Gen SiTU MoE path hung after one run, so both sides pin Marlin. These numbers establish the cache A/B, not the best possible production absolute throughput. `--enable-symm-mem` was accepted but K3 disabled it for decode-graph correctness. No EPD runtime benchmark is claimed in this PR; EPD is covered by the code and unit tests only in this round. ## Validation - changed-file pre-commit, Python compile, registered-test validation, Cookbook validation, and Mintlify link/build checks: pass - final stacked Linux suite on both GB300 nodes: 170 passed + 5 subtests per node; the latest focused false-hit/K3 suite is 143 passed + 5 subtests and covers forced 64-bit embedding-key collision, forced 30-bit pad collision, identity downgrade, caller router-hash override, partial-identity rejection, mutable path, processor-output change, EPD identity propagation, and strong global-cache keys - strict full-real-weight normal-serving matrix compares prompt/output token IDs, messages, finish reasons, and usage across cold/hot paths; it covers one/two/duplicate/reordered images, partial hits, cross input forms, mutable paths, caller-hash rejection, concurrent single-flight, and `n=2` - the latest full-real-weight 2-node GB300 rerun passes all 11 normal-serving identity cases; a Native `/generate` adversarial check also proves that changing `mm_hashes` for the same image still skips processor/transport/ViT, while two different images forced to share the same `mm_hashes` value perform a cold miss before only the true repeat becomes hot - 0 request errors across the full normal-serving matrix; input/output usage lengths stay aligned ## Stack 1. #34398 — content identity, API, bounded LRU, and single-flight infrastructure 2. #34404 — typed K3 per-image artifacts and ordinary/EPD identity plumbing 3. this PR — scheduler lease, early EPD reuse, metrics, documentation, and benchmark --- ### CI States Latest PR Test (Base): :x: [Run #31676763339](https://github.com/sgl-project/sglang/actions/runs/31676763339) Latest PR Test (Extra): :x: [Run #31676763157](https://github.com/sgl-project/sglang/actions/runs/31676763157) ",
"url": "https://github.com/sgl-project/sglang/pull/34411",
"createdAt": "2026-08-11T11:40:23Z",
"updatedAt": "2026-08-13T08:59:10Z",
"timestamp": "2026-08-13T08:59:10Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"documentation",
"Multi-modal",
"run-ci",
"run-ci-extra"
],
"author": "mickqian",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34413",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "cookbook(qwen3.5): add MI355X disaggregated inference and MTP commands, bump image to v0.5.16",
"text": "## Motivation Add disaggregated inference (DI) and MTP (EAGLE speculative decoding) deployment commands for Qwen3.5-FP8 on AMD MI355X to the cookbook page, and bump the MI355X docker image to v0.5.16. This aligns with InferenceX PR [SemiAnalysisAI/InferenceX#2557](https://github.com/SemiAnalysisAI/InferenceX/pull/2557) which adds the corresponding benchmark configurations. ## Changes - `docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx`: - MI355X docker image: `lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260715` → `lmsysorg/sglang:v0.5.16-rocm720-mi35x` - **New: Disaggregated Inference (MoRI 1P1D) commands** — TP4 prefill + TP8 decode (12 GPUs), delivering 24-31% better throughput/GPU vs TP8P+TP8D (16 GPUs) with identical decode interactivity - **New: DI + MTP (EAGLE) commands** — EAGLE speculative decoding (depth 3) on the TP4P+TP8D topology, with guidance to switch decode to TP4 at conc≥128 to avoid MTP throughput collapse (+83% throughput/GPU improvement) ## Notes - The previous cookbook PR ([#31454](https://github.com/sgl-project/sglang/pull/31454)) bumped images; this PR adds the DI/MTP deployment recipes that were missing. - Disaggregated inference commands use `--disaggregation-mode prefill/decode` and `--disaggregation-transfer-backend mori`. cc @zijiexia @yichiche Made with [Cursor](https://cursor.com) --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31708425253](https://github.com/sgl-project/sglang/actions/runs/31708425253) Latest PR Test (Extra): :x: [Run #31708424884](https://github.com/sgl-project/sglang/actions/runs/31708424884) ",
"url": "https://github.com/sgl-project/sglang/pull/34413",
"createdAt": "2026-08-11T11:43:07Z",
"updatedAt": "2026-08-13T14:07:29Z",
"timestamp": "2026-08-13T14:07:29Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"documentation"
],
"author": "ChangLiu0709",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34415",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "K3 pp",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/34415",
"createdAt": "2026-08-11T11:54:40Z",
"updatedAt": "2026-08-13T11:45:22Z",
"timestamp": "2026-08-13T11:45:22Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"deepseek",
"speculative-decoding",
"npu",
"jit-kernel"
],
"author": "JinyanYi",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34417",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Diffusion] Fix AttributeError in grouped forward_batch by installing the residency manager",
"text": "## Motivation Reported in #34000 (Bug 2). `PipelineExecutor.__init__` seeds `component_residency_manager = None`, and every `_execute_stages` run enters `_component_residency_request` → `begin_component_residency_request`, which dereferences it unguarded. Of the three forward entry points in `ComposedPipelineBase`, two install the manager and one does not: | entry point | installs manager? | |---|---| | `forward` | yes | | `forward_batch` | **no** | | `forward_batch_sequentially` | yes | `forward_batch` is the grouped path taken when a request expands to `num_outputs_per_prompt > 1`, so it reaches `execute_group_with_profiling` with the manager still `None`: ``` AttributeError: 'NoneType' object has no attribute 'begin_request' ``` That the grouped path is meant to be supported is visible in `begin_component_residency_request` itself, which explicitly handles a `list` payload (`if isinstance(batch, list): batch = batch[0]`). Only the install was missed. `forward_batch_sequentially` — added later, in #30683 — got it right, which suggests the two-way duplication is what let `forward_batch` drift. ## Modifications Extract the install into `_install_component_residency_manager` and call it from all three entry points. Consolidating removes the failure mode rather than just this instance of it — a fourth entry point can no longer forget the setup. No double-install: `forward_batch` returns early for `len(batches) == 1` (delegating to `forward`) before reaching the new call. ## Accuracy Test Manager visible to the executor at execute time, per entry point: ``` before after forward installed installed forward_batch None installed <- would raise AttributeError forward_batch_sequentially installed installed ``` Also confirmed the executor genuinely has no `None` fallback: `begin_component_residency_request` with `component_residency_manager = None` raises `AttributeError: 'NoneType' object has no attribute 'begin_request'`, i.e. a skipped install is a hard crash rather than a silent degradation. **Verification caveat, please read:** I could not run the pytest suite against `main` — the sglang install I have is `0.5.12.post1`, whose tree predates the current layout, and my only box is a CPU node where importing `sglang.multimodal_gen` off network storage takes >10 minutes. The table above comes from extracting the real methods from both `main` and this branch via `git show` and executing them side by side against a recording executor stub. **The added unit test has not been executed under a real sglang install** — please treat it as unvalidated and let CI be the first real run. ## Checklist - [x] Format the code with pre-commit hooks (isort / black / ruff clean on the changed files) - [x] Add unit tests (`test_pipeline_residency_manager_install.py`) — but see the verification caveat above - [x] Linked the tracking issue (#34000) --- ### CI States Latest PR Test (Base): :x: [Run #31503646958](https://github.com/sgl-project/sglang/actions/runs/31503646958) Latest PR Test (Extra): :x: [Run #31503646814](https://github.com/sgl-project/sglang/actions/runs/31503646814) ",
"url": "https://github.com/sgl-project/sglang/pull/34417",
"createdAt": "2026-08-11T11:56:21Z",
"updatedAt": "2026-08-13T01:54:14Z",
"timestamp": "2026-08-13T01:54:14Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"diffusion"
],
"author": "CjhHa1",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34421",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD][Perf] Fuse GatedDeltaNet QKVZBA split/reshape/cat into a single Triton kernel for Qwen3.5-architecture MoE on HIP",
"text": "## Motivation `Qwen3_5GatedDeltaNet.forward` only takes the fused Triton split/reshape/cat path when the head-group ratio `num_v_heads // num_k_heads` is 1, 2 or 4. Layouts with a ratio of 8 fall back to the eager sequence (`fix_query_key_value_ordering` -> `.contiguous()` -> `torch.cat`), which costs four separate data-movement kernels per GDN layer. Measured on a decode CUDA graph on MI355X (TP=8, aiter backend): | # | Kernel | Source expression | us | |---|---|---|---:| | 1 | `direct_copy_kernel_cuda` (elementwise) | `b.contiguous()` | 3.8 | | 2 | `direct_copy_kernel_cuda` (elementwise) | `a.contiguous()` | 4.0 | | 3 | `CatArrayBatchedCopy` | `torch.cat((q, k, v))` -> `mixed_qkv` | 4.8 | | 4 | `direct_copy_kernel_cuda` (elementwise) | materialising the non-contiguous `z` for `z.reshape(...)` | 4.4 | | | **Total** | | **17.0** | That is 17.0 us and 4 launches of pure data movement per GatedDeltaNet layer. Summed over the GatedDeltaNet layers of the model measured here it accounts for ~1.17 ms of every decode iteration, and the whole-trace `cat`/`contiguous`/`reshape` bucket was the 6th-largest kernel category at 4.9% of GPU time. `fused_qkvzba_split_reshape_cat_contiguous` already performs exactly this work in a single pass, and it also writes `z` out contiguous so the later `z.reshape(...)` becomes free — i.e. it subsumes all four kernels, not just the three obvious ones. It was simply never enabled for ratio 8, and its hard-coded `num_warps=1` is too narrow once each program has to move 8 v-heads' worth of elements. This PR enables the fused path for ratio 8 on aiter and scales `num_warps` to the per-program work, replacing 4 kernels with 1 (3 launches eliminated per layer, 207 per decode iteration) and cutting the replaced region by **-73.5%**. ## Modifications - **python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py**: in `fused_qkvzba_split_reshape_cat_contiguous`, replace the hard-coded `num_warps=1` with a heuristic based on how many v/z elements a single program moves (`v_elems_per_program = (num_heads_v // num_heads_qk) * head_v`): one warp up to 512 elements, four warps above that, so a wide head group does not serialise its vector loads/stores on a single warp. For every ratio the path already enabled at today's head dims the heuristic resolves back to `num_warps=1`, so existing callers keep the identical launch configuration. The kernel body itself is unchanged. - **python/sglang/srt/models/qwen3_5.py**: hoist the fused-path ratio whitelist into a module-level `_GDN_FUSED_QKVZBA_RATIOS`, which is `(1, 2, 4, 8)` when `_use_aiter` is set and `(1, 2, 4)` otherwise, and use it in the `Qwen3_5GatedDeltaNet.forward` dispatch instead of the inline `in [1, 2, 4]` test. Because the non-aiter tuple is exactly the previous list, the CUDA / CPU / NPU control flow is unchanged — same branch taken, same kernels, byte-identical behaviour. The eager fallback is retained untouched for every ratio not in the tuple. ## Accuracy Tests Model: a large MoE model with GatedDeltaNet linear-attention layers (MXFP4 weights, TP=8, MI355X), aiter backend, ratio-8 head groups (`num_k_heads_tp=2`, `num_v_heads_tp=16`). | Benchmark | Score | Threshold | Status | |-----------|:-----:|:---------:|:------:| | GSM8K (1319 questions, parallel=512, max-tokens 2048) | 0.9795 | 0.900 | PASS | Run via `sglang.test.run_eval` against a live server. The baseline (fused path disabled, eager fallback) scored 0.9787 on the same checkpoint and the same harness, so the fused kernel is numerically equivalent to within run-to-run sampling noise. ## Speed Tests and Profiling Workload: canonical-8k, IL=8192 / OL=1024, `num_prompts = concurrency x 10`. Baseline is the same build with the change reverted. Traces captured on TP rank 0 at both anchors (conc4, conc64). ### Kernel-level (per GatedDeltaNet layer, decode) | Kernel | Before (us) | After (us) | Notes | |--------|:-:|:-:|-------| | `direct_copy_kernel_cuda` (`b.contiguous()`) | 3.8 | -- | eliminated | | `direct_copy_kernel_cuda` (`a.contiguous()`) | 4.0 | -- | eliminated | | `CatArrayBatchedCopy` (`torch.cat((q,k,v))`) | 4.8 | -- | eliminated | | `direct_copy_kernel_cuda` (`z` materialisation for `z.reshape`) | 4.4 | -- | eliminated; `z` now written contiguous | | `fused_qkvzba_split_reshape_cat_contiguous_kernel` | -- | 4.5 | new fused kernel, 1 launch | | **Replaced region total** | **17.0** | **4.5** | **-12.5 us (-73.5%), 4 kernels -> 1** | Summed over every GatedDeltaNet layer of the model, that 12.5 us/layer is **862.5 us saved per decode iteration** (region total 1173.0 us -> 310.5 us). The full-attention layers are untouched (0 us). At conc64 the same region is 18.0 us -> 4.5 us per layer, i.e. 1242.0 us -> 310.5 us (**-75.0%**), 931.5 us saved per iteration. Kernel count in the decode GDN layer drops 13 -> 10 and its kernel-time sum 86.5 us -> 75.3 us (conc4); 117.2 us -> 102.4 us (conc64). Prefill benefits too, since the same dispatch is used there: the GDN region goes 60.4 us -> 41.9 us per layer at conc4 prefill (7112 tokens) and 63.0 us -> 41.7 us at conc64 chunked prefill (7316 tokens). Profiling confirmation (trace analysis, both anchors): the `Qwen3_5GatedDeltaNet` module has **zero** `direct_copy_kernel_cuda` and **zero** `CatArrayBatchedCopy` launches remaining in the after traces (the only surviving `direct_copy` launches belong to `LogitsProcessor`), and `fused_qkvzba_split_reshape_cat_contiguous_kernel` appears with exactly one launch per GDN invocation. Normalised per GDN invocation over the whole trace, the replaced kernels go 17.36 us -> 4.92 us at conc4 (-71.7%) and 69.82 us -> 13.07 us at conc64 (-81.3%). The intended `num_warps` branch is the one taken: `num_v_heads_tp / num_k_heads_tp = 8`, `head_v = 128` -> 1024 elements per program -> `num_warps=4`. ### E2E benchmark **Concurrency 4** (baseline n=2, after n=2; run-to-run spread < 0.2% on both sides) | Metric | Before | After | Delta | |--------|:-:|:-:|:-:| | Total throughput (tok/s) | 1910.86 | 1983.34 | +3.79% | | Output throughput (tok/s) | 212.57 | 220.63 | +3.79% | | Median ITL (ms) | 16.605 | 16.005 | -3.61% | | Median TPOT (ms) | 17.855 | 17.240 | -3.44% | | Median E2E latency (ms) | 16678.65 | 16106.44 | -3.43% | | Median TTFT (ms) | 449.72 | 446.93 | -0.62% | **Concurrency 64** (baseline n=2, after n=3) | Metric | Before | After | Delta | |--------|:-:|:-:|:-:| | Total throughput (tok/s) | 10008.69 | 10131.36 | +1.23% | | Output throughput (tok/s) | 1110.38 | 1123.99 | +1.23% | | Median ITL (ms) | 30.335 | 29.590 | -2.46% | | Median TPOT (ms) | 56.47 | 55.77 | -1.24% | | Median E2E latency (ms) | 52773.70 | 52141.40 | -1.20% | | Median TTFT (ms) | 544.98 | 499.14 | -8.41% | ### Kernel-to-E2E consistency | Anchor | Saving / iteration | Baseline iteration | Predicted ITL | Observed ITL | |---|:-:|:-:|:-:|:-:| | conc4 | 862.5 us | 16.605 ms | -5.19% | -3.61% | | conc64 | 931.5 us | 30.335 ms | -3.07% | -2.46% | Observed lands at ~70-80% of the GPU-side prediction at both anchors — same direction, same order of magnitude — with the residual being the host/scheduler overhead floor that a GPU-side fusion does not remove. Both concurrencies were repeated on both sides and the run spread (<0.2% at conc4, 0.13% for the conc64 baseline) is far smaller than the reported deltas. TTFT is not where this change pays off: the ~1.28 ms saved across a whole prefill is ~0.3% of the baseline TTFT, so the TTFT deltas above are dominated by queueing rather than by the fused kernel. ### Notes - The `num_warps` threshold (512 v/z elements per program) is a static heuristic tuned on MI355X for `head_v=128` rather than an autotuned config; ratio-8 enablement stays behind `_use_aiter` so other backends are unaffected until they are measured. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31654038211](https://github.com/sgl-project/sglang/actions/runs/31654038211) Latest PR Test (Extra): :x: [Run #31654038134](https://github.com/sgl-project/sglang/actions/runs/31654038134) ",
"url": "https://github.com/sgl-project/sglang/pull/34421",
"createdAt": "2026-08-11T12:38:20Z",
"updatedAt": "2026-08-13T07:18:36Z",
"timestamp": "2026-08-13T07:18:36Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "yichiche",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34424",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Fix ROCm VAE Conv2D fast path breaking spatial-parallel decode",
"text": "## Motivation On ROCm, running the Wan VAE decode with spatial-shard parallel decode (`--num-gpus 2 --enable-cfg-parallel`) crashes as soon as the decoding stage starts: ``` RuntimeError: The size of tensor a (39) must match the size of tensor b (43) at non-singleton dimension 3 # under torch.compile the same failure surfaces during tracing: # Attempting to broadcast a dimension of length 43 at -2! ... [1, 384, 1, 43, 160] vs [1, 384, 1, 39, 160] ``` The trigger is `SGLANG_USE_ROCM_VAE_CONV2D_BF16=1` (equivalently `SGLANG_USE_ROCM_VAE_CONV2D=1`), which enables the ROCm Conv3D→Conv2D VAE fast path. `RocmPlatform._replace_conv3d_with_conv2d` monkey-patches `forward` on every `nn.Conv3d` subclass that carries a `_padding` attribute. That predicate also matches `SpatialParallelCausalConv3d` and `SpatialParallelConv3d`, whose `_padding` intentionally carries **no height padding** — the missing rows come from the halo exchange with the neighbouring rank, and the output is trimmed afterwards. Replacing their `forward` outright discards both the halo exchange and the output trim, so every 3×3×3 conv silently loses two rows of height while the 1×1×1 `conv_shortcut` (kernel is not 3-D, so it is never patched) keeps its own. In `residual_block_forward` the two paths then disagree at `return x + h`: ``` local shard height 45 -> conv_in 43 -> conv1 41 -> conv2 39 (halo dropped) local shard height 45 -> conv_shortcut 43 (1x1x1, unpatched) => x + h ==> 39 vs 43 ``` Single-GPU decode is unaffected because plain `WanCausalConv3d._padding` does carry real height padding. `torch.compile` is not involved — the failure reproduces identically with `--enable-torch-compile false`. ## Modifications - `runtime/layers/parallel_conv.py`: add `_halo_conv_forward` to `SpatialParallelCausalConv3d` and `SpatialParallelConv3d`. It is the inner conv applied to the already halo-exchanged input, and it is the documented override point for platform fast paths. Both classes now pass `self._halo_conv_forward` to `_spatial_parallel_conv_forward` instead of `super().forward`. - `runtime/platforms/rocm.py`: for spatial-parallel convs, install the Conv2D decomposition into `_halo_conv_forward` instead of clobbering `forward`, so the halo exchange and output trim stay in place and the optimization is preserved rather than disabled. A spatial-parallel conv with no such hook, or with a non-`zeros` `padding_mode`, is now skipped rather than silently corrupted. Non-parallel `CausalConv3d` modules keep the existing `forward` patch unchanged. - `runtime/layers/parallel_conv.py`: generalize `_maybe_contiguous_for_sp_gather`. It previously only handled channels-last inputs, so the Conv2D decomposition's permuted-view output reached NCCL non-contiguous and failed with `Tensors must be contiguous` in `gather_and_trim_height`. It now materializes any non-contiguous input. - Logging now reports the two patch kinds separately: `patched N CausalConv3d + M spatial-parallel halo kernels`. ## Benchmarking and Profiling Wan2.2-T2V-A14B, 720p, MI355, `SGLANG_USE_ROCM_VAE_CONV2D_BF16=1`. Before this PR, 2-GPU spatial-parallel decode does not run at all, so there is no before/after latency to compare — the table records that the failing configurations now complete. | Config | Before | After | |---|---|---| | 2 GPU CFG-parallel, eager | crash: `39` vs `43` | 7.00 s | | 2 GPU CFG-parallel, `--enable-torch-compile true` | crash during Dynamo tracing | 179.63 s (includes cold compile) | | 2 GPU CFG-parallel, 193 frames, 8 steps, torch.compile (full workload) | crash | 114.23 s end-to-end, decoding stage 4.49 s | ## Accuracy Tests Wan2.2-T2V-A14B, 720p, 25 frames, 8 steps, seed 42, MI355. Frames decoded from the saved mp4 and compared pixel-wise. | Comparison | MAE | Max abs diff | PSNR | |---|---|---|---| | 1-GPU reference vs 2-GPU spatial-parallel (this PR, Conv2D fast path) | 0.9277 | 12 / 255 | 45.57 dB | | 1-GPU reference vs 2-GPU spatial-parallel (Conv3D reference path) | 0.9366 | 12 / 255 | 45.52 dB | `python -m pytest python/sglang/multimodal_gen/test/unit/test_vae_spatial_parallel_decode.py` — 18 passed, 12 subtests passed. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review Process 1. Ping Merge Oncalls to start the PR flow. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - `/tag-run-ci-label`, `/rerun-failed-ci`, `/tag-and-rerun-ci` 4. After green CI and required approvals, ask Merge Oncalls to merge. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31668209693](https://github.com/sgl-project/sglang/actions/runs/31668209693) Latest PR Test (Extra): :x: [Run #31668209562](https://github.com/sgl-project/sglang/actions/runs/31668209562) ",
"url": "https://github.com/sgl-project/sglang/pull/34424",
"createdAt": "2026-08-11T12:55:23Z",
"updatedAt": "2026-08-13T05:30:28Z",
"timestamp": "2026-08-13T05:30:28Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"amd",
"run-ci",
"diffusion"
],
"author": "yichiche",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34425",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Intel][XPU] Accept XPU in the shared test CLI --device choices",
"text": "add_common_sglang_args_and_parse hardcoded choices=[\"auto\", \"cuda\", \"rocm\", \"cpu\"], so --device xpu was rejected even though ServerArgs accepts it. It also offered \"rocm\", which DeviceConfig always refuses: PyTorch drives AMD GPUs through the \"cuda\" device type. This change as part of #24922 ## Motivation Any test or benchmark using the shared CLI parser fails on Intel GPUs: argparse.ArgumentError: argument --device: invalid choice: 'xpu' (choose from auto, cuda, rocm, cpu) `add_common_sglang_args_and_parse` hardcoded `[\"auto\", \"cuda\", \"rocm\", \"cpu\"]`, while the server's `SUPPORTED_DEVICES` is `[\"cuda\", \"xpu\", \"hpu\", \"cpu\", \"npu\", \"musa\", \"mps\"]` — so the harness rejected five devices the server accepts. It also offered `\"rocm\"`, which is invalid everywhere: `DeviceConfig(\"rocm\")` raises, since PyTorch has no `rocm` device type and `RocmSRTPlatform.device_type` is `\"cuda\"`. The old list offered an unusable value while rejecting every valid non-CUDA accelerator. Auto-detection itself was never broken (`ServerArgs.device=None` → `get_device()`, harness `\"auto\"` → `auto_config_device()`); the hardcoded `choices` was the only blocker. ## Modifications `python/sglang/test/test_utils.py` - `--device` uses `choices=[\"auto\"] + SUPPORTED_DEVICES`, help text derived from the same list, so the harness can't drift from the server. - Dropped `\"rocm\"`; AMD GPUs use `--device cuda`, as they already did everywhere else. - Fixed two stale docstrings, including a `run_bench_one_batch` `Args: device:` block for a parameter the function never had. `test/registered/unit/test_common_sglang_args.py` (new) - Per-vendor classes (NVIDIA / AMD ROCm / Intel XPU / CPU) plus parser-wide contracts, following `unit/platforms/test_platform_interface.py`. - `register_cpu_ci(est_time=3, suite=\"base-a-test-cpu\")` — parsing argv, reading class-level `device_type`, and building `DeviceConfig` need no accelerator, so all vendor paths are covered on CPU-only CI. No caller impact: all 26 callers checked, none reads `args.device`; `auto`/`cuda`/`cpu` still accepted; nothing in the repo ever passed `--device rocm`. Out of scope: `ServerArgs.device` and `server_arguments.mdx` still omit `mps` from their help strings — a separate doc inconsistency. ## Accuracy Tests Not applicable — CLI argument parsing only. No kernel, model forward, or sampling code is touched, and no signatures changed (docstrings only), so model outputs cannot change. ## Speed Tests and Profiling No inference-path impact — the change only affects CLI parsing at startup. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31493874055](https://github.com/sgl-project/sglang/actions/runs/31493874055) Latest PR Test (Extra): :x: [Run #31493874020](https://github.com/sgl-project/sglang/actions/runs/31493874020) ",
"url": "https://github.com/sgl-project/sglang/pull/34425",
"createdAt": "2026-08-11T12:59:03Z",
"updatedAt": "2026-08-13T05:20:18Z",
"timestamp": "2026-08-13T05:20:18Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "dayanandav",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34427",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix Kimi-K3 media token resolution",
"text": "## Motivation Kimi-K3 currently takes the `<|media_pad|>` token ID from the checkpoint configuration. When that value is stale or differs from the tokenizer vocabulary, SGLang looks for and expands the wrong token ID, breaking image-placeholder resolution for multimodal requests. The tokenizer is the source of the input IDs used at runtime, so the processor should prefer its vocabulary mapping while retaining the checkpoint value as a compatibility fallback. ## Modifications - Resolve the Kimi-K3 `<|media_pad|>` ID from the processor tokenizer vocabulary during processor initialization. - Fall back to `hf_config.media_placeholder_token_id` when the tokenizer vocabulary does not contain the placeholder. - Emit a startup warning when the tokenizer ID overrides the checkpoint-configured ID. - Propagate the resolved ID through `MultimodalSpecialTokens`, the GPU processor wrapper, and cloned processor workers. - Extend the existing Kimi processor worker test to cover tokenizer/config mismatches and the checkpoint fallback path while preserving Kimi-K2.5 coverage. ## Accuracy Tests No model-level accuracy test was run. The focused CPU regression coverage verifies that the resolved media token ID is shared by the multimodal token contract, the active Kimi-K3 wrapper, and its cloned worker. ```text python -m pytest -q test/registered/unit/models/test_kimi_k25.py 36 passed ``` ## Speed Tests and Profiling Not applicable. The change performs one tokenizer vocabulary lookup during Kimi-K3 processor initialization and does not add work to inference or preprocessing hot paths. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No documentation changes are required for this internal bug fix.) - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). (No model-level accuracy or speed benchmark was run; see above.) - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31670793085](https://github.com/sgl-project/sglang/actions/runs/31670793085) Latest PR Test (Extra): :x: [Run #31670792941](https://github.com/sgl-project/sglang/actions/runs/31670792941) ",
"url": "https://github.com/sgl-project/sglang/pull/34427",
"createdAt": "2026-08-11T13:21:00Z",
"updatedAt": "2026-08-13T06:55:26Z",
"timestamp": "2026-08-13T06:55:26Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [],
"author": "mmangkad",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34428",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Honor should_apply_lora when wrapping LoRA target modules",
"text": "## Motivation `LoRAManager.init_lora_modules()` decides what to wrap by matching the last one or two components of a module name against `--lora-target-modules`. For multimodal models that is ambiguous: encoder towers name their projections the same way the language model does. `VisionAttention` exposes `qkv_proj` and `proj`, so a target as ordinary as `qkv_proj` selects tower modules the adapter carries no weights for, and `get_layer_id()` files them into `self.lora_modules[![\"image\"](\"https://github.com/user-attachments/assets/7178f122-9b6a-4e13-b517-232b6703f6ad\")
Flashinfer benchmark on a single B300 with Muse Glimmer shape ### N=6656, K=19968 | M | cutlass (TFLOPs/s) | cute-dsl (TFLOPs/s) | |---|-------------------:|--------------------:| | 1 | 6.312 | **8.555** | | 32 | 208.153 | **256.573** | | 64 | 415.487 | **494.538** | | 128 | 817.889 | **982.648** | | 256 | 1446.592 | **1818.284** | | 512 | 2224.931 | **2624.488** | | 1024 | 2708.906 | **3201.373** | | 2048 | 2838.883 | **3499.666** | | 4096 | **3519.589** | 3449.986 | | 8192 | **3403.057** | 3193.448 | ### N=19968, K=6656 | M | cutlass (TFLOPs/s) | cute-dsl (TFLOPs/s) | |---|-------------------:|--------------------:| | 1 | 6.775 | **7.504** | | 32 | 224.885 | **244.536** | | 64 | 448.619 | **498.479** | | 128 | 895.739 | **1002.112** | | 256 | 1463.501 | **1680.373** | | 512 | 2261.627 | **2641.604** | | 1024 | 3029.742 | **3146.874** | | 2048 | **3539.739** | 3382.431 | | 4096 | **3710.748** | 3601.933 | | 8192 | **3635.707** | 3376.387 | ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31618178019](https://github.com/sgl-project/sglang/actions/runs/31618178019) Latest PR Test (Extra): :x: [Run #31618177806](https://github.com/sgl-project/sglang/actions/runs/31618177806) ",
"url": "https://github.com/sgl-project/sglang/pull/34435",
"createdAt": "2026-08-11T16:28:39Z",
"updatedAt": "2026-08-12T23:07:00Z",
"timestamp": "2026-08-12T23:07:00Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"documentation",
"quant",
"apple-silicon"
],
"author": "IwakuraRein",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34437",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[attention] Default Kimi-K3 DSPARK decode attention backend to cutedsl_mla",
"text": "## Summary Default Kimi-K3 + DSPARK (pure-TP, SM100/SM103) decode attention backend from `trtllm_mla` to `cutedsl_mla`. Applies only when the user has not explicitly set an attention backend; explicit `--attention-backend` keeps priority. Non-DSPARK default is unchanged (`trtllm_mla`), DCP path already defaults to `cutedsl_mla`. ## Motivation For MTP/spec verify (q>1), `cute-dsl` is faster than `trtllm-gen` at every context length in CUDA-graph mode, because it folds the verify tokens into the MMA tile (`fold_sq`) instead of re-scanning KV per query row. See flashinfer-ai/flashinfer#4390 (thread + eager/graph comparison table). ## Benchmarks 8x B300 (SM103), 2-node TP8, Kimi-K3 DSPARK (block 7, q=8), bs=1, isl=900k, osl=1024, `bench_one_batch_server`: | backend | latency | decode throughput | ITL | acc length | |---|---|---|---|---| | trtllm_mla (previous default) | 101.31 s | 118.35 tok/s | 8.45 ms | 5.12 | | cutedsl_mla | 95.11 s | 183.55 tok/s | 5.45 ms | 3.87 | Note: acc length differs between runs (5.12 vs 3.87) even though the attention backend does not affect verification acceptance — this is run-to-run variance of the benchmark workload, not a backend effect. Kernel-level (torch profiler, 10 verify steps x 24 MLA layers): the verify attention kernel drops from 1247.6 us/call to 311.3 us/call (~4x), consistent with the 900k-scaled values from flashinfer-ai/flashinfer#4390 (1413/359 us at 1000k). Verified end-to-end on the devbox: with no `--attention-backend` flag, the server reports `decode/verify attention backend cutedsl_mla (speculative_attention_mode=decode)` and serves normally. --- ### CI States Latest PR Test (Base): :x: [Run #31517614146](https://github.com/sgl-project/sglang/actions/runs/31517614146) Latest PR Test (Extra): :x: [Run #31517613806](https://github.com/sgl-project/sglang/actions/runs/31517613806) ",
"url": "https://github.com/sgl-project/sglang/pull/34437",
"createdAt": "2026-08-11T17:26:59Z",
"updatedAt": "2026-08-13T08:32:20Z",
"timestamp": "2026-08-13T08:32:20Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "yhyang201",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34454",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] [Perf] Accelerate ROCm top-p selection and tree verification",
"text": "## Motivation Depends on https://github.com/sgl-project/sglang/pull/34140. Until that PR merges, this draft includes its prerequisite commit; GitHub will drop the duplicated diff after merge. ROCm stochastic tree verification calls top-p renormalization over 154,880 probabilities per row. The existing path uses general-purpose `torch.topk(..., 32)`, including unused int64 indices. ## Modifications - Add a two-stage Triton value-only selector: - Select exact local top-32 values from 2,048-value vocabulary chunks. - Merge the local lists into the exact global top 32. - Preserve the established PyTorch row-sum, prefix, pivot, and normalizer reduction order. - Keep the exact 4,096-prefix/full-sort fallback for broad or boundary-tied distributions. - Use the value-only hierarchical selector for every HIP row count; dead-work removal moves it below `torch.topk` even for one active sequence. - Stop the tree verifier bonus-token CDF scan after the selected vocabulary block, avoiding residual loads for later blocks. ## Accuracy Tests ### Unit and captured-row tests | Test | Result | |---|---:| | MI355X `test_spec_verify_distribution.py` | 13 passed, 1 platform-only skip | | MI355X `test_tree_speculative_sampling_triton.py` | 4 passed, 1 CUDA-only skip | | Captured GLM-5.2-FP8 rows, bitwise output parity | 192 / 192 | | Pre-commit, 4 changed files | Passed | ### GLM-5.2-FP8 long-horizon math workload Prompt: ```text Determine the sum of the smallest 50 possible values of gcd(a^2+b^2+c^2, abc) across all triples of positive integers (a,b,c). Let's think step by step and output the final answer within \\boxed{}. ``` | Configuration | Value | |---|---:| | Model | `zai-org/GLM-5.2-FP8@ba978f7d347eaf65d22f1a86833408afdb953541` | | Hardware | MI355X, TP8 | | Expected boxed answer | `2618` | | Choices | 8 | | Temperature | 1.0 | | Top-p | 0.95 | | Maximum tokens per choice | 131,072 | | EAGLE top-k / steps / draft tokens | 2 / 5 / 6 | | Result | Value | |---|---:| | Correct boxed answers | 7 / 8 | | Normal stops | 8 / 8 | | Completion tokens | 335,661 | | Mean acceptance length | 4.406 | | Server exceptions | 0 | ## Speed Benchmarks MI355X, GLM-5.2-FP8 captured probability rows, vocabulary 154,880, p50 milliseconds. | Decode batch | Probability rows | `torch.topk32` | Triton select + merge | Selector latency reduction | Topk32 full top-p | PR full top-p | Full top-p latency reduction | |---:|---:|---:|---:|---:|---:|---:|---:| | 8 | 48 | 0.131 | 0.101 | 22.9% | 0.258 | 0.229 | 11.2% | | 256 | 1,536 | 1.815 | 1.352 | 25.5% | 2.431 | 1.964 | 19.2% | | Active sequences | Probability rows | Topk32 full top-p | PR full top-p | Latency reduction | |---:|---:|---:|---:|---:| | 1 | 6 | 0.206 | 0.188 | 8.8% | | 2 | 12 | 0.211 | 0.183 | 13.2% | | 4 | 24 | 0.243 | 0.196 | 19.4% | | 8 | 48 | 0.258 | 0.229 | 11.2% | | 128 | 768 | 1.371 | 1.055 | 23.1% | | 256 | 1,536 | 2.431 | 1.964 | 19.2% | Tree verifier, captured GLM-5.2-FP8 requests, p50 milliseconds: | Decode batch | Baseline | PR | Latency reduction | |---:|---:|---:|---:| | 1 | 0.0846 | 0.0547 | 35.3% | | 8 | 0.0951 | 0.0656 | 31.0% | | 32 | 0.1253 | 0.1097 | 12.5% | | 128 | 0.1609 | 0.1453 | 9.7% | | 256 | 0.1965 | 0.1737 | 11.6% | Matched serving-level throughput is pending after prerequisite PR #34140 merges. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). (No user-facing API change.) - [ ] Provide end-to-end speed benchmark results according to [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31642668964](https://github.com/sgl-project/sglang/actions/runs/31642668964) Latest PR Test (Extra): :x: [Run #31642668668](https://github.com/sgl-project/sglang/actions/runs/31642668668) ",
"url": "https://github.com/sgl-project/sglang/pull/34454",
"createdAt": "2026-08-11T21:27:07Z",
"updatedAt": "2026-08-12T21:29:08Z",
"timestamp": "2026-08-12T21:29:08Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"speculative-decoding",
"jit-kernel"
],
"author": "Raiden-Makoto",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34456",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support deferred MoE finalize for MXFP4 and FP8 block-scale",
"text": "## Motivation The FlashInfer TRT-LLM MoE kernels can return the unweighted expert outputs instead of performing the top-k weighted combine inside the kernel. When the combine is deferred, it is merged with the shared-expert addition into a single kernel, which removes one kernel launch per MoE layer per decode step. This mode was previously enabled only for the NVFP4 runner. This PR enables it for the MXFP4 runner and for the FP8 block-scale runner. The FP8 per-tensor runner is not changed. ## Modifications 1. The deferred combine is enabled for MXFP4 and FP8 block-scale in addition to NVFP4. Each supported quantization and runner combination accepts the deferred form for exactly one routing format, so the required format is now derived per configuration rather than assumed. 2. The routed scaling factor is handled explicitly. On the MXFP4 path the routing decision is computed in advance, and the MoE kernel therefore does not execute the stage that would apply this factor to the expert weights. The factor is instead passed to the fused combine kernel, which multiplies it into the expert weight already held in a register. No additional kernel is launched. All other callers pass a value of 1.0 and their results are unchanged. ## Accuracy AIME 2026, 30 problems, 8 repeats, 240 samples per configuration. | Configuration | pass@1 | | --- | --- | | Baseline | 96.67% | | This PR | 97.08% | Servers: ``` # baseline SGLANG_ENABLE_MOE_DEFERRED_FINALIZE=0 sglang serve --trust-remote-code \\ --model-path deepseek-ai/DeepSeek-V4-Flash-0731 --tp 4 \\ --moe-runner-backend flashinfer_mxfp4 --swa-full-tokens-ratio 0.1 --port 30001 # this PR sglang serve --trust-remote-code \\ --model-path deepseek-ai/DeepSeek-V4-Flash-0731 --tp 4 \\ --moe-runner-backend flashinfer_mxfp4 --swa-full-tokens-ratio 0.1 --port 30000 ``` Evaluation: ``` sgl-eval run aime26 --base-url http://127.0.0.1:30001/v1 \\ --n-repeats 8 --temperature 1.0 --top-p 0.95 --thinking \\ --num-threads 240 --max-tokens 131072 sgl-eval run aime26 --base-url http://127.0.0.1:30000/v1 \\ --n-repeats 8 --temperature 1.0 --top-p 0.95 --thinking \\ --num-threads 240 --max-tokens 131072 ``` ## Speed DeepSeek-V4-Flash-0731, MXFP4 MoE runner, tensor parallel size 4, B300, input length 1024, output length 1024. One measurement per point. | Batch size | Output tok/s, baseline | Output tok/s, this PR | Change | | --- | --- | --- | --- | | 1 | 193.09 | 197.46 | +2.3% | | 4 | 622.83 | 632.04 | +1.5% | | 8 | 1163.31 | 1177.10 | +1.2% | | 16 | 2107.59 | 2105.02 | -0.1% | | 64 | 6265.74 | 6272.06 | +0.1% | | 256 | 15597.53 | 15246.44 | -2.3% | Kernel counts from a decode profile of 20 steps at batch size 1, one tensor parallel rank, 43 MoE layers: | Kernel | Baseline | This PR | | --- | --- | --- | | In-kernel combine | 3.241 ms, 860 launches | not launched | | Elementwise add for the shared expert | 1.324 ms, 860 launches | not launched | | Fused combine and shared-expert add | not launched | 3.318 ms, 860 launches | At batch size 256 the same comparison gives 9.717 ms for the two baseline kernels and 6.604 ms for the single fused kernel, a reduction of 3.11 ms per 20 decode steps. --- ### CI States Latest PR Test (Base): :x: [Run #31550766019](https://github.com/sgl-project/sglang/actions/runs/31550766019) Latest PR Test (Extra): :x: [Run #31550765937](https://github.com/sgl-project/sglang/actions/runs/31550765937) ",
"url": "https://github.com/sgl-project/sglang/pull/34456",
"createdAt": "2026-08-11T21:52:01Z",
"updatedAt": "2026-08-12T20:53:28Z",
"timestamp": "2026-08-12T20:53:28Z",
"metrics": {
"reactions": 0,
"comments": 5
},
"labels": [
"deepseek",
"jit-kernel"
],
"author": "b8zhong",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34459",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Fix DeepSeek-V4 routing: sqrtsoftplus underflow and unfloored renorm",
"text": "## Motivation DeepSeek-V4's `sqrtsoftplus` gate has four routing implementations, and each was missing a different piece of the numerics its reference gate provides (`inference/model.py` `Gate.forward`, `F.softplus`). Same class as flashinfer-ai/flashinfer#3803. 1. **Triton router** (default, layers 3-42) computed `log(1.0 + exp(x))`, which in fp32 rounds to `log(1.0) == 0` for `x <= -16.64`. If every top-K expert falls below that, the row renormalizes to all-zero weights and the token loses its entire routed-expert contribution. 2. **`hash_topk`** (layers 0-2) divided by `routed_sum` with no floor, in both the CUDA kernel (default) and its torch fallback. Hash routing ignores scores, so an all-underflowed row gives `0/0` = NaN. Every sibling already floors this. 3. **CUDA `moe_fused_gate`** (test-only) used `log1pf(expf(x))`, which overflows above `88.7`. On B300: worst relative error vs `F.softplus` over `[-104, 200]` goes `1.0` -> `3.5e-07`; `hash_topk` at `-200` goes `nan` -> `0.0`; `test_moe_fused_gate.py` 39 passed. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#code-formatting-with-pre-commit). - [x] Update documentation as needed, including docstrings or example tutorials. - [x] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#running-unit-tests-adding-to-ci). --- ### CI States Latest PR Test (Base): :x: [Run #31637754619](https://github.com/sgl-project/sglang/actions/runs/31637754619) Latest PR Test (Extra): :x: [Run #31637754257](https://github.com/sgl-project/sglang/actions/runs/31637754257) ",
"url": "https://github.com/sgl-project/sglang/pull/34459",
"createdAt": "2026-08-11T22:46:25Z",
"updatedAt": "2026-08-12T22:29:17Z",
"timestamp": "2026-08-12T22:29:17Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "b8zhong",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34471",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] Support LTX-2.5",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31632222955](https://github.com/sgl-project/sglang/actions/runs/31632222955) Latest PR Test (Extra): :x: [Run #31632222552](https://github.com/sgl-project/sglang/actions/runs/31632222552) ",
"url": "https://github.com/sgl-project/sglang/pull/34471",
"createdAt": "2026-08-11T23:21:25Z",
"updatedAt": "2026-08-13T17:53:56Z",
"timestamp": "2026-08-13T17:53:56Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"documentation",
"run-ci",
"diffusion"
],
"author": "AgainstEntropy",
"state": "open",
"assignees": [],
"change": "updated"
},
{
"id": "github:sgl-project/sglang:pull_request:34474",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] Qwen3.5: guard attn layers against empty DP-attention batch",
"text": "## Problem Under wide expert-parallel disaggregated decode with DP attention + MTP speculative decoding, a decode rank can receive **zero requests** in a round while still being a non-idle (`target_verify`) rank. The existing `is_idle()` guards on the attention layers do not cover this \"non-idle but empty\" case: the GDN linear-attention and full-attention layers then run on a `seq_len == 0` batch, and the `.view(seq_len, -1)` / `.reshape(..., -1)` calls crash with `cannot reshape tensor of 0 elements into shape [0, -1]`. ## Fix Extend the attention-layer guards from `not is_idle()` to `not is_idle() and hidden_states.shape[0] > 0`, so an empty batch skips the attention compute while still participating in the surrounding collective communication. The GDN output reshape is also made zero-safe by giving the last dim explicitly instead of `-1` (which cannot be inferred from 0 elements). Non-empty batches are unaffected. ## Test Qwen3.5-397B-A17B-MXFP4 wide-EP16 2P1D + MTP on 4x MI355X (gfx950): previously crashed at warmup on the empty DP-attention rank; with this change the decode servers come up and GSM8K passes (0.973). Base / non-MTP legs unchanged (0.975). --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31556424704](https://github.com/sgl-project/sglang/actions/runs/31556424704) Latest PR Test (Extra): :x: [Run #31556424611](https://github.com/sgl-project/sglang/actions/runs/31556424611) ",
"url": "https://github.com/sgl-project/sglang/pull/34474",
"createdAt": "2026-08-11T23:36:27Z",
"updatedAt": "2026-08-13T05:33:06Z",
"timestamp": "2026-08-13T05:33:06Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"run-ci"
],
"author": "Lzy17",
"state": "open",
"assignees": [
"1am9trash",
"yichiche"
]
},
{
"id": "github:sgl-project/sglang:pull_request:34477",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[CI] Route mmlu and GB300 MMMU-Pro evals through sgl-eval",
"text": "Moves the GB300 MMMU-Pro evals and all mmlu evals onto sgl-eval, and deletes the NeMo-Skills glue that the former needed. **MMMU-Pro.** Replaces the per-process NeMo-Skills venv bootstrap and its stdout-regex score parsing with the existing sgl-eval shim. Sampling is pinned in the two GB300 tests to what `ns eval --benchmarks=mmmu-pro:1` sent implicitly (temperature 0.7, seed 0, no thinking), so those baselines carry over unchanged. Verified equivalent: same prepared dataset, byte-identical rendered messages, and with concurrency pinned to 1 on both sides, byte-identical generations and the same score question by question. **mmlu.** Now scored by sgl-eval instead of the local zero-shot prompt plus the single `Answer: [A-D]` regex in `simple_eval_common`. This changes scores for the ~95 mmlu call sites, so their thresholds need re-baselining. Two reasons it is worth it: grading comes from NeMo-Skills' `eval_mcq` rather than one regex that takes the *first* `Answer:` in the response, and 86 of those callers pass `num_examples=64`, which the local sampler drew with `random.Random(0)` — the matching seeded sampling is what sgl-eval#31 adds. `simple_eval_mmlu` stays: the ascend eval imports its `subject2category` table. Note for reviewers: `SGL_EVAL_REF` advances to the sha carrying both `mmmu_pro_vision` and that sampling fix, which also moves the nightly GSM8K eval onto this sgl-eval version. --- ### CI States Latest PR Test (Base): :no_entry_sign: [Run #31660219933](https://github.com/sgl-project/sglang/actions/runs/31660219933) Latest PR Test (Extra): :no_entry_sign: [Run #31660219787](https://github.com/sgl-project/sglang/actions/runs/31660219787) ",
"url": "https://github.com/sgl-project/sglang/pull/34477",
"createdAt": "2026-08-12T00:04:15Z",
"updatedAt": "2026-08-13T07:44:00Z",
"timestamp": "2026-08-13T07:44:00Z",
"metrics": {
"reactions": 0,
"comments": 8
},
"labels": [
"quant",
"amd",
"hicache",
"blackwell",
"npu",
"run-ci",
"run-ci-extra"
],
"author": "hnyls2002",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34479",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Fix] Decouple `--language-only` from encoder disaggregation",
"text": "Makes `--language-only` mean one thing: serve the language half of a VLM, with no local vision tower. Where image features come from is decided by encoder registration (`--encoder-urls` or the `EncoderBootstrapServer`), not by this flag. - The vision tower is now actually skipped, not built-and-left-empty, across the VLMs that support it. - A multimodal request with no encoder available is rejected instead of silently falling back to local processing against an unpopulated tower. - `--language-model-only` (#22867, #34262) is folded in as a deprecated alias; the checkpoint-declared `language_model_only` config key is unchanged. --- ### CI States Latest PR Test (Base): :x: [Run #31670647711](https://github.com/sgl-project/sglang/actions/runs/31670647711) Latest PR Test (Extra): :x: [Run #31670647625](https://github.com/sgl-project/sglang/actions/runs/31670647625) ",
"url": "https://github.com/sgl-project/sglang/pull/34479",
"createdAt": "2026-08-12T00:12:44Z",
"updatedAt": "2026-08-13T05:33:12Z",
"timestamp": "2026-08-13T05:33:12Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"documentation",
"Multi-modal"
],
"author": "hnyls2002",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34483",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] CI: cut two setup cycles from the AMD multimodal-gen lanes",
"text": "## Motivation AMD multimodal-gen jobs spend most of their runner time not testing. Measured on mi300 in [run 31443692177](https://github.com/sgl-project/sglang/actions/runs/31443692177), every job pays a container pull of 27–97 min plus a dependency install of 23–98 min, so 100–155 min of fixed cost before a single test executes. Across the eight multimodal jobs in that run that is ~15.8 runner-hours of setup against ~5.6 hours of tests — roughly 74% overhead, and because both matrices are `max-parallel: 1` it is serialized into wall-clock. CUDA for comparison: `Install dependencies` takes 27 seconds and total fixed overhead is under a minute. The two clearest wins are removing whole setup cycles. ## Modifications **Fold the `unit` suite into a warm shard.** `multimodal-gen-unit-test-amd` paid ~110 min of setup to run ~5 min of portable CPU-style tests. It now runs as a step of `multimodal-gen-test-1-gpu-amd` part 0 (and the ROCm 7.2 counterpart), on the container that job already warmed. The step runs after the diffusion tests and on failure as well, so a diffusion regression cannot mask a unit regression, and it carries its own `timeout-minutes` so it cannot eat the diffusion step's budget. Trigger conditions are unchanged — on both lanes the 1-GPU job already gates on exactly the inputs the unit job did (`multimodal_gen` on `pr-test-amd`, `main_package || sgl_kernel` on ROCm 7.2). The standalone job, its `target_stage_select` entry, and its `needs` entry are removed; the `ci_coverage_report.py` comment naming it is updated (the `\"unit\": (\"CUDA\", \"AMD\")` mapping itself still holds). **2-GPU: 3 shards → 2.** Per-case timings from the same run were 14, 26 and 21 min of healthy work across the three shards; the third shard's other 96 min was `flux_image_t2i_2_gpus` (77.7 min) and `flux2_modelopt_fp8_tp2_t2i` (18.1 min) retrying, which is the known ROCm flux bug (#34351 / #34352), not real work. Two shards therefore land near 30 min of case time each — and even if one shard inherited both flux cases while they stay broken, ~126 min is still inside the 180 min step timeout. Net effect: 8 AMD multimodal jobs → 5 per lane, i.e. ~4 fewer setup cycles across the two AMD workflows, on the order of 8 runner-hours per full PR run. ## What this deliberately does not do **1-GPU stays at 4 shards.** It has no headroom today: shard 0 already ends with `The action 'Run diffusion server tests (1-GPU)' has timed out after 90 minutes`, spent retrying the broken `flux_image_t2i`. Merging 1-GPU shards should wait until that case is fixed or skipped, and would likely want the step timeout raised to 150 min to match the 2-GPU lane. **The retry amplification and the image pull are untouched.** Retrying a deterministically broken case seven times is what turns a 20-minute shard into a 127-minute one, and the ROCm image pull is the single largest line item in the whole lane — caching or pre-pulling it on the mi300 runners would beat any amount of matrix tuning. Both are worth separate work. ## Validation Dispatched on this branch against the ROCm 7.2 lane (the PR gate since #34204): [run 31638833841](https://github.com/sgl-project/sglang/actions/runs/31638833841). **The unit fold works, including the case it had to get right.** On 1-GPU part 0 the diffusion step failed (7 passed, 1 setup error) and the folded `Run diffusion unit tests` step still ran and passed — 1360 passed, 10 skipped in 102 s. `if: matrix.part == 0 && (success() || failure())` fires after a failed predecessor, and the step is correctly skipped on parts 1-3. **The 2 two-GPU shards cover the suite and fit the timeout, but with less margin than I estimated.** They split into 13 cases + 3 standalone and 13 cases + 4 standalone. Shard 1 spent 45 min on cases plus 9 min on standalone. Shard 0 reached **149 min** against the 180 min step cap — not because two shards is too coarse, but because `flux_image_t2i_2_gpus` fails and the harness retries it: 64 min for the case block, then repeated ~13 min retry passes. Both shards' remaining failures are the pre-existing ROCm set (flux #34351 / #34352, and `test_ar_models` failing warmup with `Inference tensors do not track version counter`). So the split is safe today and drops to ~45-65 min per shard once the flux retries stop, but if a reviewer wants insurance while flux is broken, the options are to hold this until flux lands or to raise the 2-GPU step timeout. I have deliberately not raised it here: a larger cap means a wedged job holds a scarce mi300 runner longer. ## Accuracy Tests N/A — CI orchestration only. ## Speed Tests and Profiling Covered above: the change is motivated entirely by measured CI wall-clock, and the numbers come from the per-step and per-case timings of run 31443692177. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31660972929](https://github.com/sgl-project/sglang/actions/runs/31660972929) Latest PR Test (Extra): :x: [Run #31660972765](https://github.com/sgl-project/sglang/actions/runs/31660972765) ",
"url": "https://github.com/sgl-project/sglang/pull/34483",
"createdAt": "2026-08-12T00:45:24Z",
"updatedAt": "2026-08-13T02:30:07Z",
"timestamp": "2026-08-13T02:30:07Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"amd"
],
"author": "michaelzhang-ai",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34487",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD] ci: cache the CI image instead of re-pulling it in every job",
"text": " ## Motivation Every AMD CI job re-acquires the `rocm/sgl-dev` image from scratch, and the setup that precedes it re-fetches the whole git remote. On the [nightly CI dashboard](https://michaelzhang-ai.github.io/sglang-ci/nightly-ci/) this is the `Setup docker` step, and it dominates the wall clock of the AMD nightlies. Measured on the Aug 10 scheduled runs ([Nightly Test (AMD)](https://github.com/sgl-project/sglang/actions/runs/31415730698), [ROCm 7.2](https://github.com/sgl-project/sglang/actions/runs/31415528605)), summing `Setup docker` across the 90 container jobs: | Phase | Nightly (AMD) | Nightly ROCm 7.2 | |---|---|---| | `Setup docker` total | 23.1 GPU-machine-hours | 25.6 GPU-machine-hours | | ⤷ `git fetch --tags origin` | 13.7 h (p50 4.6 min, p90 54 min, max 64 min) | 12.9 h (p50 0.6 min, p90 57 min, max 62 min) | | ⤷ `docker pull` from Docker Hub | 9.2 h (p50 3.7 min, p90 26 min, max 60 min) | 12.5 h (p50 4.2 min, p90 59 min, max 70 min) | That is ~49 GPU-machine-hours of setup per night before a single test runs, and the p50/p90 spread shows most of it is contention: ~90 jobs hitting github.com and Docker Hub for the same objects at the same time. Two causes, both fixable in-tree: 1. **Every job downloads the image over the internet.** `amd_ci_start_container.sh` prefers the in-network mirror at `10.44.14.109:5000` over Docker Hub, but [#31436](https://github.com/sgl-project/sglang/pull/31436) disabled the job that populates it. 0 of 89 jobs found their tag there; all 89 fell through to Docker Hub with `manifest unknown`. 2. **The version lookup fetches the entire remote.** The scripts need the latest release tag only to build the image tag string, but they obtained it with `git fetch --tags origin`, which drags every branch and tag object into the shallow CI checkout. What is *not* fixable in-tree: the runners are docker-in-docker with ephemeral storage, and each AMD job starts the container exactly once, so a runner's image store is always cold. No local-cache logic in these scripts can help — the only lever here is making the *source* of the image closer. That is also why `find_latest_image()`'s existing \"check local cache on the runner\" probe never hits; this PR leaves the probe in place and records the constraint next to it, since it is what makes the store look reusable. ## Modifications Scope: **AMD-only.** Nothing outside `scripts/ci/amd/` and the two AMD nightly release workflows changes, plus one CPU-registered unit test. In particular `python/tools/get_version_tag.py` is untouched — setuptools-scm runs it for every pyproject (CUDA, CPU, NPU, XPU) and `release-pypi-pr.yml` / `release-pypi-nightly.yml` shell out to it, so no non-AMD wheel version can move. **`ls-remote` instead of a full tag fetch.** New `scripts/ci/amd/amd_ci_latest_release_tag.py` lists tags with `git ls-remote --tags --refs` — refs only, no objects — and falls back to the local ref store when the remote is unreachable. It imports `parse_version_tuple` from the shared helper rather than reimplementing the ordering, so the tag it picks is the one the nightly release workflow published the image under, with stable/post above rc ([#23644](https://github.com/sgl-project/sglang/pull/23644)). Locally it resolves the same `v0.5.17` in 0.55 s. **Configurable mirror address.** The in-network mirror now comes from `AMD_CI_DOCKER_REGISTRY_MIRROR` (default unchanged; empty means \"always Docker Hub\") so it can be retargeted without a code change — the address has already moved twice in-tree ([#23607](https://github.com/sgl-project/sglang/pull/23607), [#29765](https://github.com/sgl-project/sglang/pull/29765)). **Restore the in-network mirror, copying with skopeo.** [#31436](https://github.com/sgl-project/sglang/pull/31436) disabled `push_local_registry` because it was red, and the [logs](https://github.com/sgl-project/sglang/actions/runs/29415796071) give the reason: ``` The push refers to repository [10.44.14.109:5000/rocm/sgl-dev] Get \"https://10.44.14.109:5000/v2/\": http: server gave HTTP response to HTTPS client ``` `docker push` needs the host daemon to list the mirror in `insecure-registries`. That holds on the pools that run the tests — their pulls fail with a plain `manifest unknown`, so they are speaking HTTP fine — but not on the pool that ran this job. `skopeo copy --dest-tls-verify=false` takes the destination's plain-HTTP listener per invocation, so it does not depend on daemon config, and it streams blobs registry-to-registry so the runner no longer stores the image just to re-push it. The job is now also `continue-on-error`: CI already falls back to Docker Hub for any tag the mirror lacks, so a mirror outage should cost speed, not fail image publication. **One log line per job for the dashboard.** `Setup docker` currently reports a single duration that mixes both costs, which is why it took log scraping to find the split above. The scripts now emit: ``` [amd-ci-setup] image=rocm/sgl-dev:v0.5.17-rocm700-mi30x-20260812 source=registry-mirror version_resolve=1s image_acquire=0s ``` `source` distinguishes a mirror hit from a Docker Hub pull, so mirror hit rate becomes directly measurable rather than inferred. **Unit test.** `test/registered/unit/tools/test_amd_ci_latest_release_tag.py` (`base-a-test-cpu`, no GPU) pins what the image tag depends on: `ls-remote` keeps the lookup from transferring objects, `--refs` keeps annotated tags from also yielding peeled `![\"Open](\"https://cursor.com/assets/images/open-in-web-dark.png\")
![\"Open](\"https://cursor.com/assets/images/open-in-cursor-dark.png\")
![\"image\"](\"https://github.com/user-attachments/assets/e6d1a44d-d02b-4487-9ec6-6739bdec7521\")
direct_write profiling: ![\"image\"](\"https://github.com/user-attachments/assets/f6cf55bd-b0bf-4280-90fa-fb387f88fab5\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31557647154](https://github.com/sgl-project/sglang/actions/runs/31557647154) Latest PR Test (Extra): :x: [Run #31557647020](https://github.com/sgl-project/sglang/actions/runs/31557647020) ",
"url": "https://github.com/sgl-project/sglang/pull/34498",
"createdAt": "2026-08-12T02:40:51Z",
"updatedAt": "2026-08-13T09:00:10Z",
"timestamp": "2026-08-13T09:00:10Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"amd"
],
"author": "Emmanuel0612",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34502",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[ROCm] Fuse per-token fp8 activation quant into RMSNorm for per-chann…",
"text": "This PR fuses the per-token FP8 activation quantization into the preceding RMSNorm for per-channel FP8 attention projections on AMD gfx95 (validated on MI355X with Kimi-K2.7-Code-MXFP4). By emitting a pre-quantized (fp8, scale) tuple straight from the norm, it removes the standalone per-token quant kernel before each attention projection, improving throughput by up to ~3.5% with no accuracy change. ## Motivation For per-channel fp8 attention projections on gfx95 (e.g. Kimi-K2.7-Code-MXFP4), the per-token activation quantization runs as a **standalone `_per_token_group_quant_8bit` launch** immediately before each projection (`fused_qkv_a_proj_with_mqa`, `q_b_proj`, `kv_b_proj`). Profiling shows this as a separate kernel that can be folded into the preceding RMSNorm. ## Change Fold the per-token fp8 quant **into the preceding RMSNorm**, so the projection receives a pre-quantized `(fp8, x_scale[m, 1])` tuple consumed directly by `gemm_a8w8_bpreshuffle` in `apply_fp8_linear`. The standalone quant kernel is eliminated. Three fusion sites, each emitting a per-token whole-row scale `[m, 1]`: - **Entry proj** (`fused_qkv_a_proj_with_mqa`): fold into `input_layernorm`, including the fused all-reduce + RMSNorm path (`custom_fused_ar_rms_quant`). - **Decode `q_b_proj`**: fold into the fused q/kv RMSNorm (`fused_qk_rmsnorm`, `quant_type=per_Token`). - **Prefill `q_b_proj` / `kv_b_proj`**: fold into the single-tensor RMSNorm (`rmsnorm2d_fwd_with_dynamicquant`, `group_size=0`). A new `\"fp8_per_token\"` quant_format selects this path for per-channel fp8; block-scale fp8 keeps the existing group-128 path. `apply_fp8_linear` gains a tuple fast path, and the fused-a-gemm shortcuts fall back to the standard proj when handed a tuple. **All new fused-quant kernels degrade gracefully** to the prior separate-quant path when the aiter kernel is unavailable. ## Accuracy test **Accuracy (GSM8K)** — neutral, within run-to-run noise: Results (MI355X / gfx950, Kimi-K2.7-Code-MXFP4) | parallel | 100 | 300 | 600 | 1200 | | --- | --- | --- | --- | --- | | original | 0.951 | 0.953 | 0.942 | 0.950 | | fold_quant | 0.947 | 0.946 | 0.947 | 0.951 | ## Speed Tests and Profiling **Throughput / GPU (tok/s)** — consistent improvement: Results (MI355X / gfx950, Kimi-K2.7-Code-MXFP4) | Conc | 4 | 8 | 16 | 32 | 64 | 128 | | --- | --- | --- | --- | --- | --- | --- | | original | 906.82 | 1491.25 | 2128.38 | 2955.05 | 3925.4 | 5048.96 | | fold_quant | 932.15 | 1511.13 | 2203.48 | 2992.39 | 3983.7 | 5084.88 | | Δ | +2.79% | +1.33% | +3.53% | +1.26% | +1.49% | +0.71% | original profiling: ![\"image\"](\"https://github.com/user-attachments/assets/ebc5e349-68a9-4fcb-a57c-94a96f38b51d\")
fold_quant profiling: ![\"image\"](\"https://github.com/user-attachments/assets/a428c1f0-535d-4d85-8b96-75128358bb07\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31683848785](https://github.com/sgl-project/sglang/actions/runs/31683848785) Latest PR Test (Extra): :x: [Run #31683848250](https://github.com/sgl-project/sglang/actions/runs/31683848250) ",
"url": "https://github.com/sgl-project/sglang/pull/34502",
"createdAt": "2026-08-12T03:05:54Z",
"updatedAt": "2026-08-13T08:51:02Z",
"timestamp": "2026-08-13T08:51:02Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"amd",
"deepseek"
],
"author": "Emmanuel0612",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34509",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[JIT Kernel] Migrate moe_topk_softmax from AOT to JIT",
"text": "# [JIT Kernel] Migrate moe_topk_softmax from AOT to JIT Test and benchmark on an **H100 80GB (sm_90), CUDA 13.0, torch 2.12.0+cu130**. **Size comparision** of the compiled files(.so) : Kernel | Variants | JIT| AOT 1-arch | AOT ÷ JIT -- | -- | -- | -- | -- moe_topk_softmax | 3 dtypes | 379.69 KiB | 777.65 KiB | 2.048× - JIT: one instantiated runtime variant. - AOT: all variants combined into one .so for single arch. ## Motivation Part of #17865 (sgl-kernel wheel slimming). `topk_softmax` is the fused softmax + top-k MoE router. It is currently AOT-only, compiled into the wheel once per shipped arch. This PR adds the JIT implementation so the kernel is built on demand for the running arch. ## Modifications | File | Change | |---|---| | `python/sglang/kernels/jit/csrc/moe/moe_topk_softmax.cuh` | New. Header-only JIT port of `aot/csrc/moe/moe_topk_softmax_kernels.cu`. | | `python/sglang/kernels/ops/moe/moe_topk_softmax.py` | New. `load_jit` wrapper + `register_custom_op`. | | `python/sglang/kernels/ops/moe/__init__.py` | Register the JIT implementation. | | `test/registered/kernels/ops/moe/test_moe_topk_softmax.py` | New. Correctness vs torch and vs the AOT kernel. | | `test/registered/kernels/benchmark/moe/bench_moe_topk_softmax.py` | New. JIT vs AOT vs Torch on the `marker` framework. | ## Accuracy Tests test/registered/kernels/ops/moe/test_moe_topk_softmax.py ``` $ PYTHONPATH=python python -m pytest test/registered/kernels/ops/moe/test_moe_topk_softmax.py ============================= test session starts ============================== platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 rootdir: /workspace/sglang/test configfile: pytest.ini collected 845 items test/registered/kernels/ops/moe/test_moe_topk_softmax.py ............... [ 1%] ............................................................sss......... [ 10%] ........................................................................ [ 18%] sss..................................................................... [ 27%] ............sss......................................................... [ 35%] ........................sss............................................. [ 44%] ....................................sss................................. [ 52%] ... 815 passed, 30 skipped, 2 warnings in 8.23s ``` The 30 skips are `topk > num_experts` combinations. Coverage: fp32 / fp16 / bf16 x `num_experts` in {8, 128, 256, 512, 6, 160, 1024} x `topk` in {1, 2, 4, 8} x softcapping x correction bias x renormalize, plus single-token, `topk == num_experts`, and zero-token edge cases. That 845-case matrix is the local sweep; under CI `get_ci_test_range` trims it to 17 cases on one dtype. The wall time above is with a warm tvm-ffi cache -- on a cold runner the JIT compile dominates (~29s of the ~31s CI total, ~44s for the full local matrix), which is what `est_time=40` reflects. The case count itself is nearly free. ## Benchmark test/registered/kernels/benchmark/moe/bench_moe_topk_softmax.py. Representative slice at `topk=2` (full sweep is 144 configs: 6 token counts x 6 expert counts x 4 topk values): ``` ====================================================================================== num_tokens num_experts topk | jit(us) aot(us) torch(us) -------------------------------------------------------------------------------------- 1 128 32 2 | 1.8918 2.2157 12.5082 9 128 256 2 | 2.5770 2.5773 14.5789 13 128 512 2 | 2.9194 2.9190 16.2022 21 128 160 2 | 4.8010 4.7917 13.7594 49 1024 32 2 | 2.3626 2.3546 13.8614 57 1024 256 2 | 2.9763 2.9702 23.2186 61 1024 512 2 | 3.5510 3.5501 35.2432 69 1024 160 2 | 7.1798 7.1677 19.3114 97 8192 32 2 | 2.7485 2.7546 26.8995 105 8192 256 2 | 6.4257 6.3954 98.9348 109 8192 512 2 | 9.4780 9.4682 206.6805 117 8192 160 2 | 32.1358 32.0856 68.4762 121 32768 32 2 | 4.4435 4.4435 74.6025 129 32768 256 2 | 16.8421 16.7787 366.3514 133 32768 512 2 | 26.2749 26.2259 846.6365 141 32768 160 2 | 119.5377 119.5891 243.2006 ====================================================================================== ``` Across all 144 configs the JIT/AOT latency ratio is **median 1.0005, max 1.008** -- no configuration is more than 0.8% slower than AOT. That is the expected outcome, since the device code is identical and only host-side dispatch differs. Against the unfused PyTorch baseline (`softmax` + `topk`) the kernel is **~4.9x faster** at the median. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31697482215](https://github.com/sgl-project/sglang/actions/runs/31697482215) Latest PR Test (Extra): :white_check_mark: [Run #31697481874](https://github.com/sgl-project/sglang/actions/runs/31697481874) ",
"url": "https://github.com/sgl-project/sglang/pull/34509",
"createdAt": "2026-08-12T03:06:50Z",
"updatedAt": "2026-08-13T14:19:53Z",
"timestamp": "2026-08-13T14:19:53Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"sgl-kernel",
"run-ci",
"jit-kernel",
"run-ci-extra"
],
"author": "VoidC-minor",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34517",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[AMD][Spec] Accelerate Qwen3.5 verification with grouped-head shared KV",
"text": "## Motivation Qwen3.5 uses grouped-query attention, where multiple query heads share a KV head. For Qwen3.5-397B under TP2, each rank has 16 query heads sharing one TP-local KV head. During EAGLE target verification, the existing split-KV path processes query heads independently and repeatedly scans the same prefix KV cache. This becomes increasingly expensive at high concurrency, when verification is memory-bandwidth-bound. PR [#33981](https://github.com/sgl-project/sglang/pull/33981) introduced a grouped-head verification kernel for Kimi-K3's absorbed MLA layout. This PR extends that kernel to Qwen3.5's ordinary GQA layout so each program loads a KV tile once and reuses it across a block of query heads. ## Modifications - Extend grouped-head verification to ordinary GQA with 256-dimensional, already-rotated Q/K tensors (`PE_DIM=0`). - Pad short verify widths so Triton's dot operand always contains at least 16 rows. - Recognize Qwen3.5 causal and conditional-generation architectures. - Automatically select grouped-head verification only when: - running on supported AMD `gfx95` hardware with Triton attention; - performing causal EAGLE target verification with `topk=1`; - the model is Qwen3.5 with exactly one TP-local KV head; and - the existing `SGLANG_ENABLE_SPLITKV_VERIFY` performance path is enabled. - Preserve split-KV and extend attention as guarded fallbacks. - Keep the existing Kimi-K3 MLA dispatch behavior. ## Correctness Tests `test/registered/attention/test_verify_shared_kv.py` compares grouped-head output against `extend_attention_fwd` and covers: - Qwen3.5 TP8, TP4, and TP2 local head shapes. - Short verify widths of 1, 2, and 3 tokens. - BF16 and FP8 E4M3 KV caches, including K/V descale factors. - The existing Kimi-K3 absorbed-MLA shape. - Rejection of multiple TP-local KV heads. - Backend routing for model architecture, `topk`, hardware support, the split-KV switch, and local KV-head count. Environment: - AMD Instinct MI355X (`gfx950`) - ROCm 7.2 - Triton attention ```text Ran 6 tests in 4.125s OK ``` Speculative acceptance remained stable in the end-to-end measurements: - C4 three-run mean: approximately 3.36 - C8 three-run mean: approximately 3.39 - C16-C128: 3.388-3.418 ## Performance ### Kernel benchmark Environment: MI355X, Qwen3.5 TP4-local shape, eight query heads sharing one KV head, head dimension 256, four verification tokens, BF16. | Prefix tokens | Extend attention | Split-KV | Grouped-head | Speedup vs. split-KV | |---:|---:|---:|---:|---:| | 1,024 | 0.077 ms | 0.028 ms | **0.022 ms** | **1.25x** | | 2,048 | 0.138 ms | 0.042 ms | **0.028 ms** | **1.50x** | | 4,096 | 0.262 ms | 0.044 ms | **0.031 ms** | **1.42x** | | 8,192 | 0.509 ms | 0.044 ms | **0.037 ms** | **1.18x** | | 16,384 | 1.005 ms | 0.073 ms | **0.049 ms** | **1.49x** | The benchmark script was used for development measurements and is intentionally not included in this focused PR. ### End-to-end serving benchmark Environment: 2x MI355X, TP2, Qwen3.5-397B-A17B-MXFP4, Triton full attention, BF16 Mamba state, EAGLE with three speculative steps, top-k 1, four draft tokens, 8192/1024 random input/output lengths, page size 16, radix cache disabled, and seed 1. | Concurrency | Baseline tok/s | Grouped-head tok/s | Throughput change | |---:|---:|---:|---:| | 4 | 635.76 | **649.81** | **+2.21%** | | 8 | 874.53 | **910.61** | **+4.13%** | | 16 | 1,130.94 | **1,205.07** | **+6.56%** | | 32 | 1,416.33 | **1,514.97** | **+6.96%** | | 64 | 1,632.23 | **1,811.29** | **+10.97%** | | 128 | 1,922.06 | **2,105.80** | **+9.56%** | C4 and C8 are three-run means; C16-C128 are single-run pinned-seed measurements. These A/B measurements used a temporary development-only gate before the final automatic dispatch was adopted. That gate is not part of this PR. Setting `SGLANG_ENABLE_SPLITKV_VERIFY=0` is not an equivalent control because it disables both grouped-head and per-head split-KV verification. A paired same-GPU comparison before and after the PR scope cleanup measured -0.87% at C4 and +0.13% at C64, both within expected run-to-run variance. ## Checklist - [x] Format code according to the contribution guide. - [x] Add registered correctness tests. - [x] Provide accuracy and performance measurements. - [x] Keep the change scoped to the model/hardware-specific optimization. - [x] No user-facing documentation is needed for this transparent, hardware-gated kernel selection. ## Review and Merge Process 1. Obtain CODEOWNER and merge-oncall review. 2. Run all required Base A, Base B, Base C, and AMD CI jobs. 3. Investigate the root failure rather than cascade fast-fail jobs, and involve CI/CD maintainers for suspected AMD runner inconsistencies. 4. Merge only after required approvals and green CI. --- ### CI States Latest PR Test (Base): :x: [Run #31713838326](https://github.com/sgl-project/sglang/actions/runs/31713838326) Latest PR Test (Extra): :x: [Run #31713837795](https://github.com/sgl-project/sglang/actions/runs/31713837795) ",
"url": "https://github.com/sgl-project/sglang/pull/34517",
"createdAt": "2026-08-12T04:02:13Z",
"updatedAt": "2026-08-13T16:15:56Z",
"timestamp": "2026-08-13T16:15:56Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci",
"jit-kernel"
],
"author": "chuyeh",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34523",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[CI] Fix nightly test failures",
"text": "## Wait longer for the driver to release GPU memory - Reproduced in two nightly runs: the class after `test_ring_2_5_1t` (1T params, ~122 GiB per GPU) fails `setUpClass` with `GPU 0 uses 122.22 GiB (no other compute processes)` — the server process had exited well inside the 30s window but the driver had not returned the memory yet - Raise `_GPU_IDLE_TIMEOUT_SECS` to 180s. Free when the GPUs are already idle: the first poll returns immediately ## Move `test_llama4` out of the nightly suites Nobody serves Llama 4 any more, and the CI HF account has no access to `meta-llama/Llama-4-Scout-17B-16E-Instruct`, so the class has been `@unittest.skip`-ing for a while while its `est_time` still counted toward shard sizing on two suites. Moved to `test/manual/8-gpu-models/` with the skip dropped, so it is runnable by hand for anyone who does have access. ## Delete four superseded copies under `test/manual/nightly/` `test_text_models_gsm8k_eval.py`, `test_text_models_perf.py`, `test_vlms_mmmu_eval.py`, `test_vlms_perf.py` each have a same-named counterpart under `test/registered/` that is the maintained one. The manual copies are stale forks, not an alternative entry point: - Both perf copies unpack a 2-tuple from `NightlyBenchmarkRunner.run_benchmark_for_model`, which now returns 3 values — they raise `ValueError` on the first benchmark, so running them by hand does not work either - The two eval copies have drifted ~90 lines from their registered namesakes The five files under `test/manual/nightly/` that have no registered counterpart are untouched. ## Verification The GPU-idle fix is order-dependent: it only reproduces when another test leaves memory behind, so a single-file rerun cannot show it. `/rerun-test registered/8-gpu-models/test_ring_2_5_1t.py registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_nightly.py` puts both in one job in that order. --- ### CI States Latest PR Test (Base): :x: [Run #31647823366](https://github.com/sgl-project/sglang/actions/runs/31647823366) Latest PR Test (Extra): :x: [Run #31647823203](https://github.com/sgl-project/sglang/actions/runs/31647823203) ",
"url": "https://github.com/sgl-project/sglang/pull/34523",
"createdAt": "2026-08-12T04:39:14Z",
"updatedAt": "2026-08-12T22:40:22Z",
"timestamp": "2026-08-12T22:40:22Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"Multi-modal"
],
"author": "hnyls2002",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34526",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix/ltx-vocoder-bf16-dtype-mismatch",
"text": "## Motivation LTX-2 and LTX-2.3 vocoder inference can fail when running with BF16 precision. The current implementation explicitly casts the vocoder input to FP32: ```python waveform = self.vocoder(hidden_states.float()) ``` When the vocoder weights and bias are loaded in `bfloat16`, the FP32 input causes a dtype mismatch during convolution, resulting in the following runtime error: ```text RuntimeError: Input type (float) and bias type (c10::BFloat16) should be the same ``` This PR removes the unnecessary FP32 cast and preserves the original input dtype, ensuring compatibility with BF16 inference. ## Modifications Updated the vocoder forward path in `ltx_2_vocoder.py`: ```diff - waveform = self.vocoder(hidden_states.float()) + waveform = self.vocoder(hidden_states) ``` This change: - Preserves the original dtype of `hidden_states`. - Fixes dtype mismatches between vocoder inputs and BF16 model parameters. - Prevents runtime failures during LTX-2 audio/video generation under BF16 inference. - Keeps dtype handling consistent with the framework's mixed-precision execution flow. ## Accuracy Tests ### Functional Validation Verified that LTX-2 and LTX-2.3 generation runs successfully with BF16 precision after this change. The previous dtype mismatch error is no longer reproducible. ### Output Quality Validation Generated videos were evaluated after the fix to ensure output quality remains unchanged. Since this change only removes an unnecessary FP32 cast and does not modify model weights or generation logic, no impact on --- ### CI States Latest PR Test (Base): :x: [Run #31658293077](https://github.com/sgl-project/sglang/actions/runs/31658293077) Latest PR Test (Extra): :white_check_mark: [Run #31666015337](https://github.com/sgl-project/sglang/actions/runs/31666015337) ",
"url": "https://github.com/sgl-project/sglang/pull/34526",
"createdAt": "2026-08-12T05:35:22Z",
"updatedAt": "2026-08-13T07:42:22Z",
"timestamp": "2026-08-13T07:42:22Z",
"metrics": {
"reactions": 0,
"comments": 2
},
"labels": [
"run-ci",
"diffusion",
"run-ci-extra"
],
"author": "nzr-niu",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34528",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts",
"text": "## Motivation On switch-free intra-node hosts — no NVLink, no multicast, every peer transfer crossing the CPU root complex — none of SGLang's custom all-reduce backends apply. `CustomAllreduce` and `QuickAllReduce` assume NVLink/multicast, `pymscclpp` assumes its own fabric, so on these machines **every per-layer reduction falls back to NCCL**. NCCL's ring is bandwidth-optimal but latency-poor at decode message sizes, and it shows up directly in TPOT. FlashInfer's `pcie_ipc` kernels ([flashinfer-ai/flashinfer#4393](https://github.com/flashinfer-ai/flashinfer/pull/4393)) target exactly this fabric: naive all-to-all peer writes collapse here, so the kernels *stage* their pushes to keep one outbound and one inbound stream per rank, and the 8-rank path uses a 4+4 island decomposition so the scarce cross-socket link carries the minimum. This affects the whole RTX PRO 6000 / RTX 6000D class, where SGLang is otherwise well supported. **Scope.** This is a decode-latency path, not a general NCCL replacement. It is sized so prefill chunks stay on NCCL (see *Workspace sizing*), because these kernels win by latency at small messages and lose to a ring at 200 MB. On a host that also wants prefill improved, this composes with a bandwidth-optimal backend rather than replacing it. ## Modifications Opt-in behind `SGLANG_ENABLE_PCIE_IPC_ALLREDUCE` (default off). +244/−0 across three files: - **`distributed/device_communicators/pcie_ipc_ar.py`** (new) — adapter between `GroupCoordinator` and `PcieIpcAllReduceWorkspace`. - **`distributed/parallel_state.py`** (+29) — construction, one `\"pcie_ipc\"` branch in `_resolve_outplace_all_reduce_method`, one dispatch arm. Built only on the TP group; other groups would just pin IPC buffers without issuing the reductions these kernels target. - **`environ.py`** (+10) — the gate plus `SGLANG_PCIE_IPC_MAX_NUMEL`. Two design points worth review: **No size heuristic lives in SGLang.** `should_pcie_ipc_ar()` delegates to `workspace.supports()`, which consults FlashInfer's own tuning table. A shape the kernels do not beat is reported unsupported and the caller keeps its NCCL path. This avoids a threshold knob that would need per-machine tuning and would drift from the kernel side. **The workspace is built on the first eligible tensor, not at group construction.** It cannot grow afterwards and costs ~`2 * world_size * max_numel * itemsize` per rank, so it must be sized for the largest reduction the model issues — one prefill chunk, `chunked_prefill_size * hidden`. `chunked_prefill_size` is in the server args but `hidden` is not known when the group is built, so construction is deferred to the first reduction, whose trailing dimension is exactly that. Ranks run the same reduction sequence, so they reach that call with the same shape and build the same workspace without extra exchange. Sizing it for a prefill chunk instead is the wrong trade, which is worth spelling out because an earlier revision of this PR did exactly that. These kernels win by latency at small messages; a prefill chunk is three orders of magnitude larger than a decode reduction, which is NCCL ring territory. Measured at 8 ranks, TP8, 8k context: | workspace | TTFT | TPOT | output tok/s | |---|---|---|---| | NCCL only | 1910 ms | 21.14 ms | 35.02 | | prefill-sized (`chunked_prefill_size * hidden`) | 3176 ms | 13.64 ms | 38.43 | | decode-sized (this PR) | **1849 ms** | **13.62 ms** | **48.05** | Routing prefill through the kernels costs 66% on TTFT and buys nothing on TPOT. The decode-sized workspace is also ~250x smaller, which matters at long context: at 128k with 4 concurrent requests the prefill-sized workspace regressed TPOT by 45% (122.33 ms against NCCL's 84.50 ms), and that regression disappears once it is sized for decode (84.60 ms). ## Accuracy GSM8K, 200 questions, same server and args as the perf runs: | arm | accuracy | invalid | |---|---|---| | NCCL | 0.960 | 0.000 | | PCIe-IPC, prefill-sized workspace | 0.955 | 0.000 | | PCIe-IPC, decode-sized (this PR) | 0.945 | 0.000 | The 0.015 spread is 3 questions out of 200, within binomial noise at this sample size (SE ~= 0.015). At the kernel level, 10 shapes from 1 to 16384 tokens at world 8 / hidden 6144 / bf16 were compared elementwise against NCCL with 0 mismatches; the maximum absolute difference is a constant 2.5e-1, one bf16 quantization step at that magnitude, i.e. accumulation order rather than corruption. ## Benchmarking Results One 8x RTX PRO 6000 Blackwell Server Edition host, GPUs 0-3 and 4-7 on separate NUMA nodes, no NVLink. Same tree, same client, same server args; the arms differ by one environment variable. ``` image lmsysorg/sglang:nightly-dev-cu13-20260812-c7c03ec5 sglang c54dc4582 + this branch flashinfer 0.6.15.post1 + flashinfer#4393 @ 090e6466 (the PR was force-pushed on 08-12; this is the rebased equivalent of the 6573c65 the first numbers were taken on) torch 2.13.0+cu130 sgl-kernel 0.4.6.post1 driver 595.58.03 server --tp 8 --quantization modelopt_fp4 --kv-cache-dtype fp8_e4m3 --chunked-prefill-size 16384 --mem-fraction-static 0.82 --disable-radix-cache --disable-custom-all-reduce --cuda-graph-max-bs 64 client bench_serving random, --random-range-ratio 1.0, --num-prompts 3x bs, OSL 256 ``` Mean values. KV pool was 342912 tokens in every arm; `#cached-token` was 0 on every prefill, so no TTFT is a prefix-cache artifact. **TTFT (ms)** | ISL / batch | NCCL | this PR | |---|---|---| | 8k, bs1 | 1909.68 | **1848.65** | | 8k, bs4 | 6324.39 | **6209.99** | | 128k, bs1 | **33384.58** | 33410.24 | | 128k, bs4 | **111272.74** | 111421.98 | **TPOT (ms)** | ISL / batch | NCCL | this PR | |---|---|---| | 8k, bs1 | 21.14 | **13.62** (-36%) | | 8k, bs4 | 30.30 | **25.23** (-17%) | | 128k, bs1 | 21.51 | **14.11** (-34%) | | 128k, bs4 | 84.50 | **84.60** | **Output throughput (tok/s)** | ISL / batch | NCCL | this PR | |---|---|---| | 8k, bs1 | 35.02 | **48.05** (+37%) | | 8k, bs4 | 72.82 | **80.93** (+11%) | | 128k, bs1 | 6.58 | **6.92** (+5%) | | 128k, bs4 | 7.07 | 7.06 | TPOT is where these kernels pay: -34% to -36% at bs1, and the ratio does not depend on context length, since a decode reduction's size is set by the batch. TTFT and the 128k bs4 cells are level with NCCL by construction -- those reductions are above the workspace bound and stay on NCCL. Before any of this was measured, all 56 ordered GPU pairs were verified to carry data intact. A sister host with the same driver returned zeroed buffers on every cross-device copy while `nvidia-smi topo -p2p r` reported all-OK; a bandwidth test does not catch that. ## Known limitations 1. **Depends on an unmerged, draft upstream PR.** flashinfer-ai/flashinfer#4393 is still draft, so `PcieIpcAllReduceWorkspace`'s signature can still change. This PR should not merge before it does. The `ImportError` path degrades to NCCL with a warning, so builds without the module are unaffected. 2. **TP4 coverage arrived mid-review.** Earlier revisions of this description said the kernels had no effect at `world_size == 4`: FlashInfer's policy table gated that branch on `hidden == 4096` exactly, so GLM's 6144 was rejected on every shape. Upstream has since removed that constraint (`remove hidden size constraint` in flashinfer#4393, closing [flashinfer#4463](https://github.com/flashinfer-ai/flashinfer/issues/4463)), and TP4×PP2 now works. Measured on the same host, against NCCL: | ISL / batch | TPOT | output tok/s | |---|---|---| | 8k, bs1 | 20.55 → **18.49** (-10%) | 37.19 → **40.28** (+8%) | | 8k, bs4 | 31.74 → **30.24** (-5%) | 87.34 → **90.26** (+3%) | | 128k, bs1 | 20.80 → **18.79** (-10%) | 11.08 → **11.33** (+2%) | | 128k, bs4 | 120.92 → 121.26 | 14.10 → 14.08 | The gain is smaller than at 8 ranks (-10% against -36%), which is what the fabric predicts: at 4 ranks NCCL has half the peers to reach and a less degraded baseline to beat. Kernel numerics at world 4 / hidden 6144: 6 shapes, 0 skipped, 0 mismatches. 3. **Not the best option at every point.** On the same host a bandwidth-optimal FP8-compressing ring beats these kernels on prefill TTFT (1381 ms vs 2239 at 8k bs1) while losing on decode. There is no single best all-reduce on this fabric, which is part of why this is opt-in rather than autodetected. 4. **Single host, single model, bf16 reductions only.** No world-size-2 coverage, no unit test yet. ## Checklist - [x] Format with `pre-commit run --all-files` - [x] Accuracy validation (GSM8K vs NCCL) - [ ] Upstream flashinfer#4393 merged — blocking - [x] Unit test (`test/registered/unit/distributed/test_pcie_ipc_ar.py`, CPU-only) --- ### CI States Latest PR Test (Base): :x: [Run #31689548390](https://github.com/sgl-project/sglang/actions/runs/31689548390) Latest PR Test (Extra): :x: [Run #31689548127](https://github.com/sgl-project/sglang/actions/runs/31689548127) ",
"url": "https://github.com/sgl-project/sglang/pull/34528",
"createdAt": "2026-08-12T05:59:57Z",
"updatedAt": "2026-08-13T10:06:13Z",
"timestamp": "2026-08-13T10:06:13Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "AliceChenyy",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34532",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Model] Add native LLaVA-OneVision implementation",
"text": "## Summary - Add native SGLang model class for LLaVA-OneVision (`llava-hf/llava-onevision-qwen2-7b-ov-hf` and siblings), wiring HF's `SiglipVisionModel` and `LlavaOnevisionMultiModalProjector` to SGLang's native `Qwen2ForCausalLM` and routing forward through `general_mm_embed_routine`. - Add companion multimodal processor that bypasses SGLang's classic `process_anyres_image` (written for CLIP + classic LLaVA) and calls HF's `LlavaOnevisionImageProcessor` directly. HF's OneVision preprocessor already applies the anyres_max recipe internally. Previously this architecture fell back to `TransformersMultiModalForCausalLM` and crashed at weight load with `ValueError: No module or parameter named 'model.vision_tower.vision_model' in TransformersMultiModalForCausalLM` (the fallback's attribute tree does not match the LLaVA-OneVision checkpoint layout). ## What the model file does - `hf_to_sglang_mapper` (`WeightsMapper`) absorbs two HF-side layout drifts in one table: - `transformers >= 4.52` — extra `model.*` top-level prefix (handled via `orig_to_new_prefix`). - `transformers >= 5.6` — `SiglipVisionModel` flattened, `vision_model` intermediate wrapper gone (handled via `orig_to_new_substr`, applied first so both drifts compose). - `pad_input_ids` expands each `![\"image\"](\"https://github.com/user-attachments/assets/aeb750a1-af83-4c1c-ae03-cda3cc1fdce2\")
### 8k1k (non-Dspark, disable-radix-cache) **This PR stack:** MI355×8, Kimi-K3 TP8, `rocm720-mi35x-20260811`, `--attention-backend aiter`, `--kv-cache-dtype fp8_e4m3`, Gluon decode (h12 `bh16bn128`), decode CUDA graph ON, `mem-fraction-static=0.99`, aiter `main` + #4480 + #4555. **Reference baseline:** [PR #33838](https://github.com/sgl-project/sglang/pull/33838) on `rocm720-mi35x-20260805` (triton/bf16, team golden — **0805**). We compare against 0805 rather than 0811 triton because 0811 image regresses ~10% TTT vs 0805 at high concurrency. GSM8K (this PR): **95.6%** | GSM8K (0805 ref): 96.2% | GSM8K (0811 triton ref): 95.6% | Concurrency | TP | TTT (tok/s) | TTT per GPU (tok/s/gpu) | Output Tput (tok/s) | Median E2EL (ms) | Median TTFT (ms) | Median TPOT (ms) | Median ITL (ms) | vs 0805 TTT/gpu | vs 0805 ITL | |------------:|---:|------------:|------------------------:|--------------------:|-----------------:|-----------------:|-----------------:|----------------:|----------------:|------------:| | 2 | 8 | 810.2 | 101.3 | 90.0 | 18,738.8 | 537.5 | 18.9 | 18.4 | 0.92× | 0.95× | | 4 | 8 | 1,654.5 | 206.8 | 182.4 | 19,757.5 | 536.5 | 21.1 | 19.6 | 1.04× | 0.95× | | 8 | 8 | 2,652.2 | 331.5 | 291.5 | 23,603.9 | 537.1 | 25.5 | 22.0 | 1.05× | 0.90× | | 16 | 8 | 4,145.6 | 518.2 | 464.8 | 31,279.1 | 541.1 | 32.6 | 25.2 | 1.07× | 0.87× | | 32 | 8 | 5,928.8 | 741.1 | 654.8 | 43,986.3 | 558.4 | 47.2 | 31.5 | **1.10×** | **0.85×** | **vs 0805 @ c=32:** TTT/gpu **+10%** (741 vs 673 tok/s/gpu), ITL p50 **−15%** (31.5 vs 37.2 ms), Output Tput **654.8 tok/s**. FP8 KV + Gluon decode does **not** regress 8k1k vs the 0805 golden; decode latency improves at high concurrency. **vs 0811 triton (secondary):** at c=32, Output Tput **654.8 vs 527.7 tok/s (+24%)** — recovers the 0811 image regression (~10% vs 0805) and adds Gluon decode uplift on top. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31685684597](https://github.com/sgl-project/sglang/actions/runs/31685684597) Latest PR Test (Extra): :x: [Run #31685684411](https://github.com/sgl-project/sglang/actions/runs/31685684411) ",
"url": "https://github.com/sgl-project/sglang/pull/34647",
"createdAt": "2026-08-13T01:21:47Z",
"updatedAt": "2026-08-13T14:52:19Z",
"timestamp": "2026-08-13T14:52:19Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"amd",
"jit-kernel"
],
"author": "chien-an-chen",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34648",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add codeowners for the DCP module",
"text": "## Motivation `python/sglang/srt/layers/dcp/` has no CODEOWNERS entry today, so DCP changes fall through to the broad `/python/sglang/srt/layers` owners. @thanhhao98 authored the consolidation that created the directory (#29365) and the A2A + FlashInfer-MNNVL comm backends / q-replicate Helix support (#21637); @kpham-sgl, @YAMY1234 and @Fridge003 are co-maintaining the module. ## Modifications Add one line to `.github/CODEOWNERS`, in the alphabetical `layers/` block: ``` /python/sglang/srt/layers/dcp @thanhhao98 @kpham-sgl @YAMY1234 @Fridge003 ``` This follows the existing convention for `layers/` subdirectories (`attention/mamba`, `attention/dsa`, `quantization/quark`), where the subdirectory entry lists the specific owning team rather than repeating the parent's owners. ## Accuracy Tests N/A — repository metadata only. ## Speed Tests and Profiling N/A — repository metadata only. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31659634509](https://github.com/sgl-project/sglang/actions/runs/31659634509) Latest PR Test (Extra): :x: [Run #31659634416](https://github.com/sgl-project/sglang/actions/runs/31659634416) ",
"url": "https://github.com/sgl-project/sglang/pull/34648",
"createdAt": "2026-08-13T01:52:48Z",
"updatedAt": "2026-08-13T02:22:27Z",
"timestamp": "2026-08-13T02:22:27Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "kpham-sgl",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34649",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Add thanhhao98 to CI_PERMISSIONS.json",
"text": "## Motivation Grant CI permissions to @thanhhao98 so they can trigger and rerun CI on their own PRs. They are an active contributor to DCP and speculative decoding — #21637, #29365, #29218, #31468 — and are proposed as the DCP module codeowner in #34648. ## Modifications Add one entry to `.github/CI_PERMISSIONS.json`, matching the standard `custom override` shape used for active contributors: ```json \"thanhhao98\": { \"can_tag_run_ci_label\": true, \"can_rerun_failed_ci\": true, \"can_rerun_stage\": true, \"cooldown_interval_minutes\": 0, \"reason\": \"custom override\" } ``` `python3 .github/update_ci_permission.py --sort-only` is a no-op on the result, and the `sort CI_PERMISSIONS.json` pre-commit hook passes. ## Accuracy Tests N/A — CI configuration only. ## Speed Tests and Profiling N/A — CI configuration only. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31659100773](https://github.com/sgl-project/sglang/actions/runs/31659100773) Latest PR Test (Extra): :x: [Run #31659100642](https://github.com/sgl-project/sglang/actions/runs/31659100642) ",
"url": "https://github.com/sgl-project/sglang/pull/34649",
"createdAt": "2026-08-13T01:52:58Z",
"updatedAt": "2026-08-13T02:02:04Z",
"timestamp": "2026-08-13T02:02:04Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "kpham-sgl",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34650",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "feat(diffusion): rebuild MiniMax-H3 AdaLN outputs on demand",
"text": "## Motivation Builds on #33991 (its four commits are included here and should be rebased away once it lands); only the last three commits are new. #33991 makes MiniMax-H3 drop the `adaln_proj` weights — 39.3% of the DiT, 13.0B parameters / 24.2 GiB — by precomputing their outputs into a sidecar file. That unlocks a large win: with those weights gone the DiT fits on-device at `tp_size=1`, so no layerwise offload is needed at all. The sidecar's problem is its identity: one artifact per `(mode, steps, flow_shift, audio_flow_shift, imgvid_cond_noise_aug, audio_cond_noise_aug)` combination. Distilled 4-step and 8-step variants multiply that set again. Worse, the H3 synthetic warmup runs `t2va` even on the `fl2va` partition, so its timestep plan never matches an `fl2va` sidecar and a single artifact always misses, failing the request outright rather than degrading. A request's whole timestep plan is settled before the denoise loop starts (`prepare_timestep_plan`), so the outputs can simply be rebuilt on demand and the artifact dropped entirely. ## Modifications - Rebuild AdaLN outputs from the checkpoint at the start of each request, behind `--minimax-h3-adaln-online`, off by default. One streaming pass fills every plan the request will look up, reading a layer at a time straight into GPU memory so the transient peak is one layer (496 MiB) rather than 24.2 GiB. - Reuse `MiniMaxH3AdalnCache` rather than adding a class: an empty slab with `plan_lengths=0` is invisible to `lookup()`, so storage, lookup and the model forward are untouched. Plans are memoized across requests. - Rebuild each plan at exactly the batch size it is consumed at. cuBLAS picks kernels by GEMM shape and not monotonically: against the runtime's `M==2`, `M==4/8/16/64/96` are bit-identical while `M==32` differs in 11760 of 96768 elements and `M==1` differs in 69. - Under `tp_size>1`, read only this rank's column shard and all-gather, mirroring `ColumnParallelLinear`. This is required for correctness, and cuts per-rank checkpoint reads to `1/tp`. - Two independent pre-existing fixes that `--text-encoder-cpu-offload` and `--vae-cpu-offload` need in order to work at all on H3, kept as separate commits: FSDP2 entry-point registration (also submitted as #34575) and an out-of-place latent reverse-normalisation in the H3 decode stage. ## Accuracy Tests Output is bit-identical (same mp4 md5) to resident `adaln_proj` weights across `t2va` / `fl2va` / `ref2va`, at `tp1/ul8` and `tp2/ul4`, 1344x768, 124 frames, 50 steps. The `tp2` case is compared against fully resident weights rather than against another cache. ## Speed Tests and Profiling 8x RTX PRO 5000 (sm_120), MiniMax-H3, `t2va`, 50 steps, 3 repetitions each, 1.72% noise floor: | | e2e | denoise stage | whole-card peak | |---|---|---|---| | `tp1/ul8` layerwise offload r35 | 135.80 s | 133.68 s | 70.34 GiB | | `tp1/ul8`, DiT fully resident | 128.41 s | 125.98 s | 60.06 GiB | `tp2/ul4` whole-card peak drops 59.43 -> 49.38 GiB. A rebuild pass costs 2.4 s at 10.9 GiB/s and is amortized over the request; with memoization repeat schedules pay nothing. Nsight (5-step capture, `cudaProfilerApi` range, 8 ranks): host-to-device copy-engine time falls from 35 498 ms with layerwise offload to 13 ms, since no weights stream during denoising. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31698615566](https://github.com/sgl-project/sglang/actions/runs/31698615566) Latest PR Test (Extra): :x: [Run #31698615429](https://github.com/sgl-project/sglang/actions/runs/31698615429) ",
"url": "https://github.com/sgl-project/sglang/pull/34650",
"createdAt": "2026-08-13T02:04:28Z",
"updatedAt": "2026-08-13T12:59:16Z",
"timestamp": "2026-08-13T12:59:16Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"documentation",
"run-ci",
"diffusion"
],
"author": "triple-mu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34651",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DCP] Share one pack kernel between both a2a backends, hoist fi_a2a send buffers",
"text": "## Motivation Follow-up to #34614, which fused the pack/unpack copies on the pynccl `a2a` path. That left the `fi_a2a` (FlashInfer MNNVL) path untouched, so it still paid four materializing copies per MLA layer per decode step plus two allocations and a zero-fill: ```python partial_o = out.view(B, N, H_pr, D).permute(0, 2, 1, 3).contiguous() # copy softmax_stats = torch.zeros(B, H_pr, N, 2, ...) # alloc + FillFunctor softmax_stats[..., 0] = lse_view # copy o_out, stats_out = decode_cp_a2a_alltoall(...) recv_o = o_out.permute(2, 0, 1, 3).contiguous() # copy recv_lse = stats_out[..., 0].permute(2, 0, 1).contiguous() # copy combined = dcp_lse_combine_triton(recv_o, recv_lse) ``` To be clear about provenance: #34614 is what created the asymmetry. Before it, both paths did ~4 copies + a combine; that PR improved one of them. This finishes the job rather than duplicating the fix. ## Modifications **One pack kernel for both transports.** `dcp_pack_a2a_send` now takes independently strided payload and LSE destinations instead of one interleaved buffer. The two transports want genuinely different layouts, for reasons rooted in each: - **pynccl** `all_to_all_single` chunks a flat buffer by byte offset, so the peer axis *must* be outermost. We interleave the LSE into the trailing fp32 word of each payload row to keep the exchange to a single collective. - **FlashInfer** wants `partial_o [B, H_pr, cp_size, D]` and `softmax_stats [B, H_pr, cp_size, S]` — peer axis inside the heads, payload and stats separate. Peer-axis placement is just a stride argument, so one kernel serves both and neither needs a layout copy. The receive side needs no work at all: `dcp_lse_combine_triton` already takes arbitrary strides, so `o_out` / `stats_out` are read in place. **Hoisted the `fi_a2a` send buffers out of the hot path.** The exchange does not write its inputs (`mutates_args=(\"workspace\",)`, verified empirically), so the send tensors are allocated once per `(h_per_rank, cp_size, head_dim, dtype)` and grown on demand. `softmax_stats` slots past the LSE stay zero after the one-time allocation, so the per-call `FillFunctor` disappears, and the addresses stay stable — a precondition if this path is ever CUDA-graph captured. Two allocations per layer per step also go away. ## Accuracy Tests **Verified against a torch oracle on 4x GB300 with real MNNVL**, all 4 ranks, both LSE bases, at `B=8 H_per_rank=12 D=512`: ``` FI_A2A_INIT_OK rank0..3 base_e=True max_err 7.8e-3 OK rank0..3 base_e=False max_err 7.8e-3 OK ``` That error is a bf16 payload against an fp32 reference — rounding, not drift. Batch sizes 4 / 16 / 8 / 16 to exercise the slice and grow paths: oracle match at every size, `softmax_stats` slot 1 verified still all-zero after each call including across the grow, and only 2 distinct buffer addresses over 4 calls (one growth, stable after). Unit tests: 24 passed + 5 subtests on aarch64, including a new case asserting the same kernel produces FlashInfer's split/peer-inside layout bit-for-bit against the permute+bitcast formulation it replaces. ## Speed Tests and Profiling Microbenchmark, 4x GB300, real MNNVL, `B=8 H_per_rank=12 D=512`: | | us/call | |---|---| | baseline | 219.0 | | + shared pack kernel (copies removed) | 171.3 | | + hoisted buffers | **167.8** | DeepSeek-V3.1, `cp_size=4`, 20 profiled decode steps, rank-0 trace: | run | total kernels | `direct_copy` | `FillFunctor` | pack | combine | |---|---|---|---|---|---| | baseline | 46342 | 4823 | 1203 | 0 | 1159 | | + shared pack kernel | 42865 | 187 | 1203 | 1159 | 1159 | | + hoisted buffers | **41706** | **187** | **44** | 1159 | 1159 | 4636 copies removed (exactly 4 x 1159, one call per attention layer per step) and 1159 fills removed; the 44 remaining fills are unrelated model code. The window between attention and the exchange: ``` before: attn -> copy 3.30 -> Fill 1.22 -> copy 1.63 -> a2a -> copy 2.91 after: attn -> PACK 1.57 -> a2a -> lse_combine ``` Prep before the exchange: **6.97 us -> 1.57 us**. Not claimed: the all-to-all kernel itself measures 6.78 / 4.70 / 6.59 us across the three runs. Same collective, same shape — that spread is run-to-run variance across separate server launches, as are the ITL differences. The kernel counts are the solid evidence; they are exact and reproducible. **No regression on the pynccl `a2a` path**, which #34614 already optimized. The generalization adds a second base-address computation and three stride arguments per program; both are amortized against a loop that moves `D/2` fp32 words, and for pynccl the two destinations are views into the same buffer, so the stores are byte-identical. CUDA-graph replay at the DCP8 shape, 3 runs each: | | us/replay (GPU) | |---|---| | `main` | 8.19, 8.19, 8.19 | | this PR | 8.19, 8.20, 8.19 | The hoist is the smaller half of the win (~3.5 us/call) and does not pay for itself on microseconds alone. The argument for it is removing 122 allocator calls per decode step from a fixed-shape path, plus address stability for graph capture; the fill removal is a bonus. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Follow-ups The `combine` kernel is now the only caller-side work left around the exchange. Folding it into the exchange itself needs the reduction to happen as peer data arrives, which only FlashInfer can do — filing that as a feature request upstream, along with a question about whether the fused kernel should consume a combined LSE or `(m, l)` pairs (the combined form is where the base-2 vs base-e ambiguity that caused #34240 comes from). `fi_a2a` requires MNNVL fabric, so CI cannot exercise this path; the evidence above is from a 4x GB300 devbox. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31674167839](https://github.com/sgl-project/sglang/actions/runs/31674167839) Latest PR Test (Extra): :x: [Run #31674167715](https://github.com/sgl-project/sglang/actions/runs/31674167715) ",
"url": "https://github.com/sgl-project/sglang/pull/34651",
"createdAt": "2026-08-13T02:13:41Z",
"updatedAt": "2026-08-13T06:32:36Z",
"timestamp": "2026-08-13T06:32:36Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"jit-kernel"
],
"author": "kpham-sgl",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34652",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] feat: publish an index of nightly comparison runs",
"text": "## Problem Runs are published as `diffusion-comparisons/![\"image\"](\"https://github.com/user-attachments/assets/bd98ae91-4183-4ff0-b772-e66b130539f0\")
`torch_npu.npu_transpose_batchmatmul` provides the numerically validated path for Kimi-K3, but it does not support shapes where the relevant dimension limit is reached. In particular, `B * K` must be smaller than `65536`. Large-prefill workloads can exceed this limit and fail at runtime. ## Changes - Use `torch_npu.npu_transpose_batchmatmul` when all shape constraints are satisfied, including `B * K < 65536`. - Fall back to the previous `torch.ops.npu.batch_matmul_transpose` implementation for unsupported large shapes. - Preserve the validated `torch_npu` path for normal Kimi-K3 workloads. - Replace the following Kimi-K3 environment variables with `ServerArgs` options: - `SGLANG_K3_SHARED_EXPERTS_ATTN_TP` → `--k3-shared-experts-attn-tp` - `SGLANG_K3_DENSE_MLP_ATTN_TP` → `--k3-dense-mlp-attn-tp` ## Accuracy gsm8k for k3 --98%(200). ![\"3a8b3cf6-4c9d-401c-9a06-563a7635e772\"](\"https://github.com/user-attachments/assets/76cde065-06e7-49dc-aacd-bd7df31117f5\")
## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31699853280](https://github.com/sgl-project/sglang/actions/runs/31699853280) Latest PR Test (Extra): :x: [Run #31699853172](https://github.com/sgl-project/sglang/actions/runs/31699853172) ",
"url": "https://github.com/sgl-project/sglang/pull/34715",
"createdAt": "2026-08-13T08:47:49Z",
"updatedAt": "2026-08-13T13:57:42Z",
"timestamp": "2026-08-13T13:57:42Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"deepseek",
"npu",
"run-ci"
],
"author": "McZyWu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34717",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[SM120] dspark: widen SWA index width to an instantiated sparse-MLA shape",
"text": "## Purpose DSPARK does not start on consumer Blackwell (SM120). CUDA graph capture aborts: ``` Check failed: (ok) is false: Unsupported sparse-MLA prefill configuration: model=DSV4 num_heads=16 topk=192 page_block_size=64 topk_extra=0 extra_page_block_size=0 ``` `build_dspark_swa_page_indices` derives its index width from `swa_window + block_size`, which lands on **192** for the DSPARK shape (128 sliding-window entries + the draft block). FlashInfer's SM120 sparse-MLA prefill matrix instantiates `topk` in `{128, 512, 1024, 2048}`, so the host gate rejects 192 — and it rejects it by aborting the process, not by falling back. `num_heads=16` itself is supported. ## What this does Widen an uninstantiated index width to the next instantiated one. The padded tail is `-1` and the kernel already masks per row through `topk_length`, so the extra columns are inert. `128` stays in the set deliberately: it is the only width the DSv4 dual (SWA extra-cache) dispatch instantiates, and widening it breaks the MTP path. ## Relationship to flashinfer-ai/flashinfer#4380 The exact-192 shape is the better long-term answer and now exists upstream — #4380 added `topk` 192/256 for DSV4 decode and single-cache prefill, and vLLM builds on it in vllm-project/vllm#51538 by carrying the true dense width (aligned to the kernel's 64-entry tile) instead of rounding. That path is not available here yet: | FlashInfer | prefill dispatch widths | | --- | --- | | 0.6.15.post1 (pinned by sglang) | 128, 512, 1024, 2048 | | v0.6.17 (latest stable) | 128, 512, 1024, 2048 | | nightly 0.6.18 | 128, **192**, **256**, 512, 1024, 2048 | sglang pins `flashinfer_python==0.6.15.post1`, so this change is what works on the pinned dependency. When the pin moves to a release carrying #4380, switching to the exact width is a one-line follow-up. I prototyped that follow-up (probe `_DECODE_DSV4_DISPATCH` at runtime, use 192 when present, fall back otherwise) and measured it — see below. It made no measurable difference, so it is not included here rather than carrying the extra complexity and its two fragile assumptions (intersecting across head counts, and using the decode table as a proxy because FlashInfer exposes no prefill capability query). ## Measurements 4x RTX 6000D (SM120), TP4, `deepseek-ai/DeepSeek-V4-Flash-0731`, `--moe-runner-backend flashinfer_mxfp4 --speculative-algorithm DSPARK`, `--mem-fraction-static 0.80`. ISL 8192 / OSL 1024. Rounded 512 (this PR) vs exact 192 (nightly-0.6.18 build), same tree, same FlashInfer: | bs | width | throughput (tok/s) | TPOT (ms) | ITL (ms) | | --- | --- | --- | --- | --- | | 1 | 192 | 91.00 | 10.87 | 9.82 | | 1 | 512 | 92.13 | 10.73 | 9.82 | | 8 | 192 | 313.41 | 18.00 | 10.62 | | 8 | 512 | 311.69 | 17.09 | 10.63 | | 32 | 192 | 520.8 / 522.2 | 37.7 / 39.0 | 21.4 / 22.4 | | 32 | 512 | 524.5 / 522.7 | 38.4 / 38.4 | 21.2 / 21.9 | Identical within run-to-run noise at every batch size (bs=32 was repeated twice per arm; the within-arm spread is as large as the between-arm difference). Accuracy, `sgl-eval run aime25 --n-repeats 16 --max-tokens 200000` (480 samples): | arch | pass@1 | spread | truncated | | --- | --- | --- | --- | | SM120, 4x RTX 6000D | **98.33%** | 86.7–100%, stdev 3.65% | 0 | | SM100, 4x B200 (reference, no patch needed) | 98.96% | 93.3–100%, stdev 2.01% | 0 | Without this change the SM120 server never reaches ready state, so there is no same-arch baseline to compare against. ## Regression check No effect off SM120: `_sm120_index_width` returns its argument unchanged when `is_sm120_supported()` is false, and arch detection is called inside the function so it does not pull CUDA into module import. On SM120 with `DeepSeek-V4-Flash` (fp8), FlashInfer 0.6.18, GSM8K 50q: | config | accuracy | TPOT (ISL 8K, bs1) | | --- | --- | --- | | deep_gemm MoE | 0.980 | 12.41 ms | | MTP-2 | 0.960 | 6.97 ms | Both match their pre-existing baselines. --- ### CI States Latest PR Test (Base): :x: [Run #31684742437](https://github.com/sgl-project/sglang/actions/runs/31684742437) Latest PR Test (Extra): :x: [Run #31684742223](https://github.com/sgl-project/sglang/actions/runs/31684742223) ",
"url": "https://github.com/sgl-project/sglang/pull/34717",
"createdAt": "2026-08-13T08:56:03Z",
"updatedAt": "2026-08-13T09:03:07Z",
"timestamp": "2026-08-13T09:03:07Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"jit-kernel"
],
"author": "AliceChenyy",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34721",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[rust] Unify MM feature transport on POSIX shm (drop inline mode)",
"text": "## Motivation The Rust MM worker had two feature transports: inline (rank 1: features ride the drain by value) and shm (TP>1: segment + `ShmPointerMMData` stub). This unifies on shm as the single transport for every topology and deletes the inline path, then removes the copies that made shm slower than inline. ## Modifications - **One transport**: the worker parks each item's features in its own POSIX segment; only segment names cross `take_mm`. `/dev/shm` exhaustion rejects the request (launch-time warning when the mount looks small); no inline fallback. - **Drain policy** (`stub_broadcast`): single-node TP>1 broadcasts the ~100 B stub and ranks map the segment in parallel; otherwise (tp1, multinode, `skip_tokenizer_init`) rank 0 takes ownership at the drain as a zero-copy `torch.frombuffer` view over the mapping. The unlink is synchronous: under a live mapping it is a ~0.01 ms name removal (page teardown lands at munmap, when the tensor dies). ## Benchmark Rank-1 handoff of one 1288x966 image (~30 MB f32), 50 iters, ms. *park* runs on MM worker threads; *wrap* runs on the scheduler loop (inflates every request's ITL): | mode | park (worker) | wrap (sched loop) | |------------------------------|---------------|-------------------| | inline (old rank-1) | 1.2 | ~0 | | shm + zero-copy view (new) | 11.0 | 0.08 | Scheduler-loop cost matches inline, and TP>1 broadcasts ~100 B instead of ~30 MB. The park gap is tmpfs page faults on fresh segments (~6 ms/30 MB) plus one memcpy (~1 ms), all on worker threads. --- ### CI States Latest PR Test (Base): :x: [Run #31689139800](https://github.com/sgl-project/sglang/actions/runs/31689139800) Latest PR Test (Extra): :x: [Run #31689139528](https://github.com/sgl-project/sglang/actions/runs/31689139528) ",
"url": "https://github.com/sgl-project/sglang/pull/34721",
"createdAt": "2026-08-13T09:40:53Z",
"updatedAt": "2026-08-13T10:12:15Z",
"timestamp": "2026-08-13T10:12:15Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "sherlockwu",
"state": "open",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:34722",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] [NPU] Optimize LTX-2/2.3 inference performance for NPU",
"text": " ## Motivation NPU compatibility and performance optimizations for LTX-2/2.3 inference. ## Modifications - Converted all `float64` computations to `float32` for NPU (NPU does not support float64 precision) - Use sglang's NPU-optimized `RMSNorm` instead of `torch.nn.RNSNorm` - Enable CUDA Triton kernels for NPU paths ## Accuracy Tests ## Speed Tests and Profiling Server: `SGLANG_CACHE_DIT_ENABLED=true SGLANG_CACHE_DIT_FN=2 SGLANG_CACHE_DIT_BN=1 SGLANG_CACHE_DIT_WARMUP=4 SGLANG_CACHE_DIT_RDT=0.4 sglang serve --model-path Lightricks/(LTX-2|LTX-2.3) --pipeline-class-name (LTX2Pipeline| LTX2TwoStagePipeline|LTX2TwoStageHQPipeline) --port 30000` T2V client: `python3 -m sglang.multimodal_gen.benchmarks.bench_serving --backend sglang-video --dataset vbench --task \"text-to-video\" --num-prompts 50 --max-concurrency 1` I2V client: `python3 -m sglang.multimodal_gen.benchmarks.bench_serving --backend sglang-video --dataset vbench --task \"image-to-video\" --num-prompts 20 --max-concurrency 1` ### LTX-2 ![\"pd_workflow_for_dense_overview\"](\"https://github.com/user-attachments/assets/8b0e4f64-d5bf-4668-9bfc-278026d07af3\")
![\"pd_workflow_for_dense_zoom_in\"](\"https://github.com/user-attachments/assets/ffede7b1-74c1-482d-8a14-b09b031eb190\")
## Accuracy Test This PR does not modify modeling_xx.py . ## Benchmark & Profiling - 80 GB (bf16) 2:4 sparsed model with sparse mm served by SGLANG with {(P1x4)x2}D1x4 scheme: ![\"80G_sparse_tensor_workflow\"](\"https://github.com/user-attachments/assets/fbd185f6-8713-43ee-980e-953f878f42cc\")
* The vllm sparse mm for bf16 inputs is poorly implemented in cutlass, optimization will be our next work * NCCL communication increase from 20 % to 40 % under P/D disaggregation scheme - compared to the baseline 144 GB 2:4 zero setting Qwen alike model without PD serving scheme: ![\"144G_zero_sparse_worload\"](\"https://github.com/user-attachments/assets/3669ca66-f124-42a9-aa2a-c56c2d0aadf6\")
The overal throughput: ![\"截屏2025-08-02](\"https://github.com/user-attachments/assets/99b79911-0d11-4b18-b915-b0866f78f66f\")
| P/D scheme | tp_p | dp_p | tp_d | dp_d | ISL | OSL | BS | latency | Input Tput | Output Tput | last token generation | TTFT（s） | | -------------------- | ---- | ---- | ---- | ---- | ---- | --- | --- | ------- | ---------- | ----------- | --------------------- | ------- | | (P1x4)D1x4 \\* | 4 | 1 | 4 | 1 | 2048 | 128 | 256 | 97.8 | 5628.17 | 7055.83 | 105.86 | 93.15 | | | 4 | 1 | 4 | 1 | 2048 | 128 | 128 | 49.6 | 5870.28 | 3310.74 | 36.14 | 44.66 | | {(P1x4)x2}D1x4 \\* \\* | 4 | 1 | 4 | 1 | 2048 | 128 | 256 | 90.29 | 6090.59 | 7779.22 | 35.92 | 86.08 | | | 4 | 1 | 4 | 1 | 2048 | 128 | 128 | 49.76 | 5781.88 | 3708.59 | 70.31 | 45.34 ## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci). - [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.",
"url": "https://github.com/sgl-project/sglang/pull/8708",
"createdAt": "2025-08-02T12:08:04Z",
"updatedAt": "2026-08-13T01:34:32Z",
"timestamp": "2026-08-13T01:34:32Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "yiakwy-xpu-ml-framework-team",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:8765",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Ability to use custom-all-reduce on systems with more than 2 PCIe GPUs via env var",
"text": " ## Motivation Allow custom all-reduce optimizations for systems without full NVLink support with more than 2 GPUs via using env variable `SGLANG_SKIP_NVLINK_CHECK=1`. ## Modifications When env variable is set `SGLANG_SKIP_NVLINK_CHECK=1` the check is bypassed considering full NVLink is available. ## Accuracy Test Skipped. ## Benchmark & Profiling Skipped. ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci). - [x] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.",
"url": "https://github.com/sgl-project/sglang/pull/8765",
"createdAt": "2025-08-04T12:23:05Z",
"updatedAt": "2026-08-13T01:34:33Z",
"timestamp": "2026-08-13T01:34:33Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "avtc",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:8807",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support GLM4.5 fused moe triton tuning",
"text": "## Motivation Support GLM4.5 and GLM4.5-Air fused moe triton tuning. ## Modifications - Change `benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py` - Change `glm4_moe.py` restoring to a specific model config `n_routed_experts` value limited ## Accuracy Test ``` Accuracy: 0.970 Invalid: 0.000 Latency: 19.836 s Output throughput: 1087.133 token/s ``` ## Benchmark & Profiling ## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci). - [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.",
"url": "https://github.com/sgl-project/sglang/pull/8807",
"createdAt": "2025-08-05T10:54:51Z",
"updatedAt": "2026-08-13T01:34:35Z",
"timestamp": "2026-08-13T01:34:35Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "lambert0312",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:8920",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "replace role literals with str for more flexible chat templating",
"text": " ## Motivation The role literals make experimenting with unique roles awkward, requiring code changes. This allows anyone to experiment with different roles, like say, multi agent settings with distinct roles for other agents. ## Modifications replace role: Literal with role: str ## Accuracy Test n/a ## Benchmark & Profiling n/a ## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci). - [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.",
"url": "https://github.com/sgl-project/sglang/pull/8920",
"createdAt": "2025-08-07T15:18:39Z",
"updatedAt": "2026-08-13T01:34:37Z",
"timestamp": "2026-08-13T01:34:37Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "dmahan93",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:8921",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Kernel] Fused RMS Norm Quant",
"text": "## Motivation Currently in Deepseek before every call to DeepGEMM we run a combination of RMSNorm followed by quantization, this merges it into a single kernel improving speed ## Modifications Adding 2 new kernels: - FusedRMSNormQuant - FusedRMSNormAddQuant It also integrates them into the RMSNorm layer, adds tests and integrates it into DeepSeek ## Accuracy Test With changes: ``` benchmark/mmlu# python3 bench_sglang.py --nsub 10 --port 42000 subject: abstract_algebra, #q:100, acc: 0.790 subject: anatomy, #q:135, acc: 0.837 subject: astronomy, #q:152, acc: 0.947 subject: business_ethics, #q:100, acc: 0.840 subject: clinical_knowledge, #q:265, acc: 0.913 subject: college_biology, #q:144, acc: 0.972 subject: college_chemistry, #q:100, acc: 0.640 subject: college_computer_science, #q:100, acc: 0.840 subject: college_mathematics, #q:100, acc: 0.760 subject: college_medicine, #q:173, acc: 0.844 Total latency: 14.270 Average accuracy: 0.856 ``` Previously: ``` subject: abstract_algebra, #q:100, acc: 0.790 subject: anatomy, #q:135, acc: 0.859 subject: astronomy, #q:152, acc: 0.941 subject: business_ethics, #q:100, acc: 0.860 subject: clinical_knowledge, #q:265, acc: 0.913 subject: college_biology, #q:144, acc: 0.979 subject: college_chemistry, #q:100, acc: 0.630 subject: college_computer_science, #q:100, acc: 0.860 subject: college_mathematics, #q:100, acc: 0.730 subject: college_medicine, #q:173, acc: 0.844 Total latency: 14.594 Average accuracy: 0.858 ``` ## Benchmark & Profiling With changes: ``` python -m sglang.bench_serving --backend sglang --model deepseek-ai/DeepSeek-R1-0528 --num-prompts 512 --sharegpt-output-len 500 --port 42000 benchmark_args=Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=42000, dataset_name='sharegpt', dataset_path='', model='deepseek-ai/DeepSeek-R1-0528', tokenizer=None, num_prompts=512, sharegpt_output_len=500, sharegpt_context_len=None, random_input_len=1024, random_output_len=1024, random_range_ratio=0.0, request_rate=inf, max_concurrency=None, output_file=None, output_details=False, disable_tqdm=False, disable_stream=False, return_logprob=False, seed=1, disable_ignore_eos=False, extra_request_body=None, apply_chat_template=False, profile=False, lora_name=None, prompt_suffix='', pd_separated=False, flush_cache=False, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256) Namespace(backend='sglang', base_url=None, host='0.0.0.0', port=42000, dataset_name='sharegpt', dataset_path='', model='deepseek-ai/DeepSeek-R1-0528', tokenizer=None, num_prompts=512, sharegpt_output_len=500, sharegpt_context_len=None, random_input_len=1024, random_output_len=1024, random_range_ratio=0.0, request_rate=inf, max_concurrency=None, output_file=None, output_details=False, disable_tqdm=False, disable_stream=False, return_logprob=False, seed=1, disable_ignore_eos=False, extra_request_body=None, apply_chat_template=False, profile=False, lora_name=None, prompt_suffix='', pd_separated=False, flush_cache=False, warmup_requests=1, tokenize_prompt=False, gsp_num_groups=64, gsp_prompts_per_group=16, gsp_system_prompt_len=2048, gsp_question_len=128, gsp_output_len=256) #Input tokens: 159002 #Output tokens: 256000 Starting warmup with 1 sequences... Warmup completed with 1 sequences. Starting main benchmark run... ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: not set Successful requests: 512 Benchmark duration (s): 36.25 Total input tokens: 159002 Total generated tokens: 256000 Total generated tokens (retokenized): 254109 Request throughput (req/s): 14.12 Input token throughput (tok/s): 4386.12 Output token throughput (tok/s): 7061.83 Total token throughput (tok/s): 11447.95 Concurrency: 508.18 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 35980.48 Median E2E Latency (ms): 36021.43 ---------------Time to First Token---------------- Mean TTFT (ms): 5110.08 Median TTFT (ms): 5404.20 P99 TTFT (ms): 8060.48 ---------------Inter-Token Latency---------------- Mean ITL (ms): 61.93 Median ITL (ms): 56.53 P95 ITL (ms): 61.21 P99 ITL (ms): 63.75 Max ITL (ms): 7336.25 ================================================== ``` Previously: ``` ============ Serving Benchmark Result ============ Backend: sglang Traffic request rate: inf Max request concurrency: not set Successful requests: 512 Benchmark duration (s): 36.66 Total input tokens: 159002 Total generated tokens: 256000 Total generated tokens (retokenized): 254223 Request throughput (req/s): 13.97 Input token throughput (tok/s): 4337.70 Output token throughput (tok/s): 6983.88 Total token throughput (tok/s): 11321.57 Concurrency: 508.32 ----------------End-to-End Latency---------------- Mean E2E Latency (ms): 36392.37 Median E2E Latency (ms): 36438.44 ---------------Time to First Token---------------- Mean TTFT (ms): 5158.23 Median TTFT (ms): 5443.19 P99 TTFT (ms): 8080.68 ---------------Inter-Token Latency---------------- Mean ITL (ms): 62.66 Median ITL (ms): 57.30 P95 ITL (ms): 61.61 P99 ITL (ms): 65.12 Max ITL (ms): 7459.16 ================================================== ``` NCU: ![\"image\"](\"https://github.com/user-attachments/assets/3a398fe4-7098-40ac-b3ca-1e2ab69274f5\")
## Checklist - [ ] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit Tests](https://docs.sglang.ai/references/contribution_guide.html#running-unit-tests-adding-to-ci). - [ ] Update documentation / docstrings / example tutorials as needed, according to [Writing Documentation](https://docs.sglang.ai/references/contribution_guide.html#writing-documentation-running-docs-ci). - [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to [Benchmark and Profiling](https://docs.sglang.ai/references/benchmark_and_profiling.html) and [Accuracy Results](https://docs.sglang.ai/references/accuracy_evaluation.html). - [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR. - [ ] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.",
"url": "https://github.com/sgl-project/sglang/pull/8921",
"createdAt": "2025-08-07T16:43:50Z",
"updatedAt": "2026-08-13T01:34:41Z",
"timestamp": "2026-08-13T01:34:41Z",
"metrics": {
"reactions": 2,
"comments": 4
},
"labels": [],
"author": "SzymonOzog",
"state": "closed",
"assignees": [
"Alcanderian",
"BBuf"
]
},
{
"id": "github:sgl-project/sglang:pull_request:9156",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[feat] add grammar sessions",
"text": " ## Motivation Enables persistent grammar for multi-turn conversations where the same grammar needs to be enforced across multiple requests, supporting agentic workflows that require consistent structured output formatting using the same grammar. ## Modifications - Add /create_grammar and /delete_grammar HTTP endpoints for grammar lifecycle management - Add grammar_id sampling parameter to reference existing grammar sessions - Implement grammar session storage and retrieval logic ## Accuracy Tests ## Benchmarking and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).",
"url": "https://github.com/sgl-project/sglang/pull/9156",
"createdAt": "2025-08-13T15:32:08Z",
"updatedAt": "2026-08-13T01:34:44Z",
"timestamp": "2026-08-13T01:34:44Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "nathanrchn",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:9179",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "support torch.compile(dynamic=True) to avoid recompile",
"text": "Currently, when both torch compile and CUDA graph are enabled, torch compile incurs a non-negligible overhead (~270s on deepseek r1). This patch supports using an environment variable to enable dynamic mode in torch.compile(), avoiding recompilation caused by inconsistent tensor sizes(~130s on deepseek r1). ## Motivation Speed up engine startup time (when torch.compile is enabled). ## Modifications Add an new ENV `SGLANG_ENABLE_DYNAMIC_TORCH_COMPILE` to enable dynamic torch.compile(). ## Accuracy Tests ## Benchmarking and Profiling Tested on DeepSeek R1 on 8*H20 with commands below: ``` python -m sglang.launch_server \\ --model-path /data/deepseek-r1/ \\ --trust-remote-code \\ --tp-size 8 \\ --quantization fp8 \\ --log-level info \\ --max-running-requests 64 \\ --mem-fraction-static 0.9 \\ --context-length 65535 \\ --enable-torch-compile \\ --torch-compile-max-bs 48 \\ --attention-backend flashinfer \\ --disable-shared-experts-fusion ``` Cost of cudagraph capturing(cuda graph capture + torch.compile()) | | cost(s) | | :----- | :--: | | dynamic=false | 270 | | dynamic=true | 130 | ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).",
"url": "https://github.com/sgl-project/sglang/pull/9179",
"createdAt": "2025-08-14T07:44:48Z",
"updatedAt": "2026-08-13T01:34:46Z",
"timestamp": "2026-08-13T01:34:46Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "tianyuzhou95",
"state": "closed",
"assignees": []
},
{
"id": "github:sgl-project/sglang:pull_request:9235",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "add note about race bug to docs",
"text": "documentation note. see https://github.com/sgl-project/sglang/issues/8056#issuecomment-3192019980",
"url": "https://github.com/sgl-project/sglang/pull/9235",
"createdAt": "2025-08-15T16:03:32Z",
"updatedAt": "2026-08-13T01:34:47Z",
"timestamp": "2026-08-13T01:34:47Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [],
"author": "frrad",
"state": "closed",
"assignees": []
}
],
"events": [
{
"id": "event:6335a820dc1725c0bc4a",
"signalId": "github:sgl-project/sglang:pull_request:34733",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34733",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[UT][NPU] Re-register NPU attention unit tests in base-a CI suites",
"text": " ## Motivation NPU unit tests under test/registered/unit/npu/ were registered to the stage-a-unit-test-npu suite, which no CI job dispatches, so they never ran on PR CI. Re-register them to base-a-unit-test-1-npu-a2 / base-a-unit-test-1-npu-a3 so they run on both 910B (a2) and 910C (a3) machines. ## Modifications - Re-register 4 existing NPU attention unit tests ( ascend_backend , ascend_dsv4_backend , ascend_torch_native_backend , mla_preprocess ) to base-a-unit-test-1-npu-a2 and base-a-unit-test-1-npu-a3 . - test/run_suite.py : add the two suite names to PER_COMMIT_SUITES[NPU] . - .github/workflows/pr-test-npu.yml : add base-a-unit-test-1-npu-a2 (910B runner) and base-a-unit-test-1-npu-a3 (910C runner) jobs, replacing the old base-a-test-1-npu-a2 job; update the finish gate list. ## Accuracy Tests N/A — registration/CI-only change, no model output affected. ## Speed Tests and Profiling N/A — no inference path changed. ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31698182285](https://github.com/sgl-project/sglang/actions/runs/31698182285) Latest PR Test (Extra): :x: [Run #31698182093](https://github.com/sgl-project/sglang/actions/runs/31698182093) ",
"url": "https://github.com/sgl-project/sglang/pull/34733",
"createdAt": "2026-08-13T12:00:25Z",
"updatedAt": "2026-08-13T13:47:25Z",
"timestamp": "2026-08-13T13:47:25Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"npu"
],
"author": "xdtbynd",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:33e021ef53548312594b",
"signalId": "github:sgl-project/sglang:pull_request:34729",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34729",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Retain SWA down to the last state checkpoint",
"text": "## Motivation A hybrid SWA + mamba model throws away most of its decode-region prefix reuse at the default `--mamba-track-interval`. Measured on one GPU with 32 prompts, counting how many get a decode-region cache hit on their second turn: ``` page_size track_interval prompts reusing the decode region 128 256 16/32 -> 32/32 128 512 5/32 -> 32/32 64 256 9/32 -> 32/32 ``` Counting tokens rather than prompts, on the first row, over the same 32 second-turn requests: ``` reused prefix 110208 -> 124416 tokens (+12.9%) recomputed 18435 -> 4227 tokens (-77.1%) mean reuse fraction 0.845 -> 0.965 ``` The gain is not deeper matches on requests that already hit, it is requests that were losing the whole generated region getting it back: 16 of 32 improved by 768 to 896 tokens each, none regressed, and the ones that already hit were at 98.5% reuse to begin with. The cause is that SWA frees out-of-window slots relative to the **tail**, while a prefix match lands on a **state checkpoint**, which sits behind the tail. `evict_threshold = pre_len - max(window, page)` therefore strands checkpoints that are still reachable in the tree: the match needs a full window of live SWA below the checkpoint, and part of that window has already been freed. `swa_evicted_seqlen` only moves forward, so nothing gets it back. Today the only way to get full reuse is to set the interval equal to the page size, which doubles checkpoint density and the mamba pool footprint. This change makes the two independent: keep the interval at 256 for the cheaper checkpoint storage and still reuse every decode-region prefix. ## Modifications `free_swa_out_of_window_slots` takes an optional `retain_floor` and promises not to free past it. It stays ignorant of mamba; the caller decides where the floor is. The floor is computed in one place, on the cache that knows about both components, and both call sites (decode-side eviction and the chunked-prefill insert path) ask it rather than each deriving it. `BasePrefixCache.swa_retain_floor` returns `None`, so a cache without a second state stream is unaffected. ## Accuracy `test_unified_radix_cache_kl_hybrid_bitexact` asserts prefill and decode score every token identically at a `kl_div` floor of 1e-9, which makes it the right instrument here: the change **increases** how much state gets reused, so a wrong retention floor shows up as a nonzero KL on exactly the prompts that newly reuse. Same tree, same diff, the only variable being whether the floor is applied. `hits` is the number of prompts whose second turn hit the decode region, `nonzero` counts per-prompt KL above zero: ``` floor off floor on SM90 (H200) 16/32 hits, 0 nonzero 32/32 hits, 0 nonzero SM100 (B200) 17/32 hits, 0 nonzero 32/32 hits, 0 nonzero ``` Every prompt that newly reuses reads exactly 0, so the retained window is correct rather than merely present. The grid above repeats this at `page_size` 64, where the page size no longer coincides with the mamba chunk size, and reuse is still complete with the floor on. On memory: peak device usage is unchanged (134.2 GB against 130.3 GB, and the direction is noise), but that metric does not answer the question, because the pools are preallocated from `mem-fraction-static` and holding more slots inside them does not move the device peak. What the change actually costs is SWA pool occupancy, which I did not measure. The analytic bound is the checkpoint spacing, and the probe that located this measured the extra retention at 127 to 255 tokens per request against a 511-token window, so on the config above it is under 1% of the SWA pool. Worth a reviewer's judgement rather than my assertion. A cache with no mamba component is unaffected, checked at the seam rather than by sampling: `BasePrefixCache.swa_retain_floor` returns `None`, the unified override early-returns `None` when mamba is off, `retain_floor` is read in exactly one guarded branch, the original threshold line is untouched, and both call sites go through `swa_retain_floor`. With `retain_floor=None` the helper is byte-identical to before. ## TODO - [ ] The prefill region is not covered yet. `req.mamba_last_track_seqlen` is cleared at the end of `cache_unfinished_req` once the checkpoint has been handed to the tree, so at the first decode step the floor reads `None` and no extra retention happens. Covering it needs the floor to come from the tree rather than from the request. - [ ] Move the SWA pool sizing floor from `window` to `window + interval`, so the extra retention cannot bite under pool pressure.",
"url": "https://github.com/sgl-project/sglang/pull/34729",
"createdAt": "2026-08-13T10:42:13Z",
"updatedAt": "2026-08-13T13:41:49Z",
"timestamp": "2026-08-13T13:41:49Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [
"run-ci"
],
"author": "ispobock",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:2e346deed6332b0d1cbb",
"signalId": "github:sgl-project/sglang:pull_request:34728",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34728",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[XPU][test] Add cache_salt=None to _make_req in test_lmcache_radix_cache.py",
"text": "## Summary - Adds `cache_salt=None` to the `SimpleNamespace` request stand-in in `_make_req` at `test/registered/xpu/test_lmcache_radix_cache.py:64`. - Fixes `AttributeError: 'types.SimpleNamespace' object has no attribute 'cache_salt'` on `stage-b-test-1-gpu-xpu`, introduced when #30827 added `cache_salt=req.cache_salt` to `RadixCache.cache_finished_req` (`python/sglang/srt/mem_cache/radix_cache.py:485`). - #30827 patched the sibling stand-in in `test/registered/unit/mem_cache/test_swa_eviction_boundary.py` (this file's docstring says it mirrors that one) but missed the XPU sibling — one-line parity fix. ## Failing test ``` File \"test/registered/xpu/test_lmcache_radix_cache.py\", line 192, in test_store_then_load_back_through_match_prefix tree.cache_finished_req(req, kv_len_to_handle=len(token_ids)) File \"python/sglang/srt/mem_cache/storage/lmcache/lmc_radix_cache.py\", line 446, in cache_finished_req super().cache_finished_req(...) File \"python/sglang/srt/mem_cache/radix_cache.py\", line 485, in cache_finished_req cache_salt=req.cache_salt, AttributeError: 'types.SimpleNamespace' object has no attribute 'cache_salt' ``` ## Blast radius Observed on 11+ `stage-b-test-1-gpu-xpu` runs across many unrelated PRs since #30827 merged (2026-08-12 23:14Z) — e.g. runs [31660770721](https://github.com/sgl-project/sglang/actions/runs/31660770721), [31664149492](https://github.com/sgl-project/sglang/actions/runs/31664149492), [31662709715](https://github.com/sgl-project/sglang/actions/runs/31662709715), [31662394459](https://github.com/sgl-project/sglang/actions/runs/31662394459), [31661559239](https://github.com/sgl-project/sglang/actions/runs/31661559239), [31659060228](https://github.com/sgl-project/sglang/actions/runs/31659060228), [31658690490](https://github.com/sgl-project/sglang/actions/runs/31658690490), [31656310028](https://github.com/sgl-project/sglang/actions/runs/31656310028), [31655236650](https://github.com/sgl-project/sglang/actions/runs/31655236650), [31654038170](https://github.com/sgl-project/sglang/actions/runs/31654038170), [31651905553](https://github.com/sgl-project/sglang/actions/runs/31651905553). ## Test plan - [ ] Re-run `stage-b-test-1-gpu-xpu` (`test/registered/xpu/test_lmcache_radix_cache.py::test_store_then_load_back_through_match_prefix`) and confirm it passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31697254447](https://github.com/sgl-project/sglang/actions/runs/31697254447) Latest PR Test (Extra): :x: [Run #31697254114](https://github.com/sgl-project/sglang/actions/runs/31697254114) ",
"url": "https://github.com/sgl-project/sglang/pull/34728",
"createdAt": "2026-08-13T10:30:24Z",
"updatedAt": "2026-08-13T13:41:01Z",
"timestamp": "2026-08-13T13:41:01Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"run-ci"
],
"author": "arathi-hlab",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:4b28e677b8da35168d0b",
"signalId": "github:sgl-project/sglang:pull_request:34592",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34592",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[GDN] Honor configured linear-attn verify backend in the kernel dispatcher",
"text": "# Motivation `GDNKernelDispatcher` derives its verify kernel purely from whether the decode or prefill backend selected FlashInfer, silently overriding an explicitly configured `--linear-attn-verify-backend`. The server logs end up contradicting themselves: ``` Linear attention kernel backend: decode=triton, prefill=flashinfer, verify=triton GDN kernel dispatcher: decode=TritonGDNKernel, extend=FlashInferGDNKernel, verify=FlashInferGDNKernel ``` This matters on SM90: the FlashInfer MTP verify path (`flashinfer.gdn_decode.gated_delta_rule_mtp`) asserts a fp32 SSM state, so any GDN model served with `--mamba-ssm-dtype bfloat16` crashes at startup as soon as NEXTN speculative decoding is enabled — and there is currently no way to force the Triton verify kernel (which handles bf16 state fine) because the flag is ignored: ``` AssertionError: initial_state must be float32, got torch.bfloat16 ``` # Modifications `GDNKernelDispatcher` takes the configured verify backend; an explicit `triton` choice now wins, while the existing auto rule (FlashInfer verify when the selected FlashInfer kernel supports MTP verify) is unchanged when the flag is unset. With the fix, `--speculative-algorithm NEXTN` + `--mamba-ssm-dtype bfloat16` + `--linear-attn-verify-backend triton` compose. Measured on a GDN hybrid 27B (FP8) on 1x H200, 4096-in/1024-out: bs=64 2092.7 -> 2143.4 output tok/s vs the previous best non-composable configs, bs=1 266.3 with accept length 2.5-3.1, GSM8K-500 accuracy unchanged (0.954) vs the fp32-state gate. # Checklist - [x] Format your code with pre-commit - [x] Benchmark + accuracy evidence above 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31689642600](https://github.com/sgl-project/sglang/actions/runs/31689642600) Latest PR Test (Extra): :x: [Run #31689642504](https://github.com/sgl-project/sglang/actions/runs/31689642504) ",
"url": "https://github.com/sgl-project/sglang/pull/34592",
"createdAt": "2026-08-12T15:35:05Z",
"updatedAt": "2026-08-13T13:38:59Z",
"timestamp": "2026-08-13T13:38:59Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "BBuf",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:bf2d33f2551b9e9801b6",
"signalId": "github:sgl-project/sglang:pull_request:34744",
"event": "discovered",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34744",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "fix: support Qwen3_5MoeForCausalLM in fused MoE tuning utils",
"text": "## Problem `get_model_config` in `benchmark/kernels/fused_moe_triton/common_utils.py` does not recognize the `Qwen3_5MoeForCausalLM` architecture (used by Qwen3.8). It falls through to the default Mixtral branch, which reads `num_local_experts` from a config object that has no such attribute: ``` AttributeError: 'Qwen3_5MoeTextConfig' object has no attribute 'num_local_experts' ``` This crashes `tuning_fused_moe_triton.py` (and the sep variant) for Qwen3.8-family models. ## Fix Add `Qwen3_5MoeForCausalLM` to the Qwen3 branch so `num_experts // ep_size`, `num_experts_per_tok`, and `moe_intermediate_size` are read correctly, matching the serving-side support in `sglang/srt/models/qwen3_5_text.py`. ## Verification - Reproduced the crash by running `tuning_fused_moe_triton.py --model Qwen3.8-2.4T-A95B-FP8 --tp-size 8 --dtype fp8_w8a8 --tune` on 8x NVIDIA H20 (triton 3.7.1). - With this one-line change the tuning run proceeds normally. - A companion PR will add the tuned H20 `fp8_w8a8` configs for Qwen3.8's E=512 shape. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31705857436](https://github.com/sgl-project/sglang/actions/runs/31705857436) Latest PR Test (Extra): :x: [Run #31705857175](https://github.com/sgl-project/sglang/actions/runs/31705857175) ",
"url": "https://github.com/sgl-project/sglang/pull/34744",
"createdAt": "2026-08-13T13:37:26Z",
"updatedAt": "2026-08-13T13:38:51Z",
"timestamp": "2026-08-13T13:38:51Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "TobyMint",
"state": "open",
"assignees": [],
"change": "new"
}
},
{
"id": "event:a198079d256f88e491b1",
"signalId": "github:sgl-project/sglang:pull_request:34743",
"event": "discovered",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34743",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Benchmark] Fix per-run speculative accept length errorr in `Serving Benchmark Result` when dp_size>1",
"text": " ## Motivation `bench_serving` currently obtains speculative accept length from: ```python server_info[\"internal_states\"][0][\"avg_spec_accept_length\"] ``` However, internal_states[0] contains only one scheduler's state, not the aggregate result across all DP ranks. For example, a decode service with multiple DP ranks run `curl -sS http://10.13.3.250:60001/server_info | jq '[.internal_states[] | .avg_spec_accept_length]` may return: ``` [ null, null, 2.9035714285714285, 3.671818181818182, null, 4.643529411764706, null, null, null, null, null, null, null, null, null, null ] ``` Reading only internal_states[0] produces no accept length, even though other DP schedulers processed speculative requests and reported valid values. Therefore, when decode dp_size > 1, the benchmark should aggregate request-level speculative counters across the current benchmark run instead of selecting a single scheduler state. ## Modifications This PR adds per-request speculative decoding counters to SGLang's OpenAI-compatible responses: ```text sglext.spec_verify_ct sglext.spec_completion_tokens ``` The counters are available for both streaming and non-streaming Chat Completions and Completions responses. They are optional and are emitted only when speculative verification data is available. When dp_size > 1, bench_serving calculates the accept length from successful requests in the current benchmark: ``` accept_length = sum(spec_completion_tokens) / sum(spec_verify_ct) ``` This is a weighted aggregation based on the number of verification forwards, rather than an arithmetic mean of per-rank averages. when deployments with dp_size == 1, he existing /server_info behavior is preserved. For PD deployments, the decode-side DP size is obtained from `server_info[\"decode\"][0][\"dp_size\"]` When --output-details is enabled, the result also includes: ``` spec_verify_cts spec_completion_tokens ``` These fields allow the reported accept length to be reproduced directly from the benchmark output. ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31705873625](https://github.com/sgl-project/sglang/actions/runs/31705873625) Latest PR Test (Extra): :x: [Run #31705873185](https://github.com/sgl-project/sglang/actions/runs/31705873185) ",
"url": "https://github.com/sgl-project/sglang/pull/34743",
"createdAt": "2026-08-13T13:31:21Z",
"updatedAt": "2026-08-13T13:39:46Z",
"timestamp": "2026-08-13T13:39:46Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "zhanxxxxxxx",
"state": "open",
"assignees": [],
"change": "new"
}
},
{
"id": "event:b338adb22aa0a8b1fda5",
"signalId": "github:sgl-project/sglang:pull_request:31320",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:31320",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NPU] [Diffusion] support distributed inference pipeline for GLM-Image",
"text": "## Summary This PR adds heterogeneous GLM-Image serving: a batched external AR server feeds a pool of independent batch-1 DiT denoisers. ![\"Image\"](\"https://github.com/user-attachments/assets/d78a7458-6077-4258-9cb0-23fc3d403830\")
* NCCL c10d timeout: ``` 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Scheduler hit an exception: Traceback (most recent call last): 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 384, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.capture() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 492, in capture 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] ) = self.capture_one_batch_size(bs, forward) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 663, in capture_one_batch_size 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] run_once() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py\", line 652, in run_once 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] logits_output_or_pp_proxy_tensors = forward( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py\", line 120, in decorate_context 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return func(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 2257, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] hidden_states = self.model( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 2128, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] hidden_states, residual = layer( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 1939, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] hidden_states = self.mlp( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 465, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self.forward_deepep(hidden_states, forward_batch) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/models/deepseek_v2.py\", line 620, in forward_deepep 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] final_hidden_states = self.experts( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1773, in _wrapped_call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._call_impl(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py\", line 1784, in _call_impl 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return forward_call(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/ep_moe/layer.py\", line 431, in forward 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] dispatch_output = self.dispatch( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/ep_moe/layer.py\", line 497, in dispatch 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self.deepep_dispatcher.dispatch( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/two_batch_overlap.py\", line 974, in dispatch 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return self._execute(\"dispatch\", **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/two_batch_overlap.py\", line 971, in _execute 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return getattr(self._inners[tbo_subbatch_index or 0], name)(**kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 831, in dispatch 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] self.dispatch_a(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 851, in dispatch_a 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] inner_state = self._get_impl(forward_batch).dispatch_a( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 492, in dispatch_a 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] buffer = self._get_buffer() 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 609, in _get_buffer 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return DeepEPBuffer.get_deepep_buffer( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/root/eric.hc/public/sglang/python/sglang/srt/layers/moe/token_dispatcher/deepep.py\", line 185, in get_deepep_buffer 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] cls._buffer = Buffer( 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/deep_ep/buffer.py\", line 87, in __init__ 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] device_ids = all_gather_object(local_device_id) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/deep_ep/buffer.py\", line 68, in all_gather_object 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] dist.all_gather_object(object_list, obj, group) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return func(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py\", line 3168, in all_gather_object 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] all_gather(object_size_list, local_size, group=group) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py\", line 81, in wrapper 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] return func(*args, **kwargs) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] File \"/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py\", line 3879, in all_gather 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] work = group.allgather([tensor_list], [tensor], opts) 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] torch.distributed.DistBackendError: [10] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: wait timeout after 600000ms, keys: /default_pg/0//3//cuda//0 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] Exception raised from doWait at /pytorch/torch/csrc/distributed/c10d/TCPStore.cpp:597 (most recent call first): 2025-08-28 21:19:34 ERROR 125765 [ DP10 TP10 EP10 scheduler.py:2655] frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string![\"20260421-003446_4bac9823\"](\"https://github.com/user-attachments/assets/f4990105-17bc-4351-8cc8-1ed0572dab14\")
## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31676886716](https://github.com/sgl-project/sglang/actions/runs/31676886716) Latest PR Test (Extra): :white_check_mark: [Run #31676886686](https://github.com/sgl-project/sglang/actions/runs/31676886686) ",
"url": "https://github.com/sgl-project/sglang/pull/23274",
"createdAt": "2026-04-20T16:41:12Z",
"updatedAt": "2026-08-13T11:48:27Z",
"timestamp": "2026-08-13T11:48:27Z",
"metrics": {
"reactions": 0,
"comments": 6
},
"labels": [
"run-ci",
"diffusion",
"run-ci-extra"
],
"author": "Dragonliu2018",
"state": "closed",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:8631d052bfec6146033c",
"signalId": "github:sgl-project/sglang:pull_request:34732",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34732",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "K3 pp",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/34732",
"createdAt": "2026-08-13T11:47:06Z",
"updatedAt": "2026-08-13T11:47:30Z",
"timestamp": "2026-08-13T11:47:30Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"deepseek",
"speculative-decoding",
"npu",
"jit-kernel"
],
"author": "JinyanYi",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:0580971867cfc3f39691",
"signalId": "github:sgl-project/sglang:pull_request:27010",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:27010",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[HiCache] Fix PP inconsistency with HiCache L3 (#22607)",
"text": "## Motivation When deploying HiCache and PP, multiple PP ranks go diverge and eventually cause sglang crash. The detailed reason was discussed in issue #22607. ## Modifications This patch introduces two kinds of synchronization mechanisms to fix the issue. I will illustrate this idea with an example of pp_size=2 and tp_size=1. 1. **pp_sync** across scheduler threads. pp_sync is a new name introduced by this patch. PP0's scheduler thread determine the number of completion events in its local queues (ack_load_queue, ack_prefetch_queue, etc). Then, PP1 must consume the **exact** same number of events from its queues. This prevents divergence. 2. **all_reduce** across prefetch threads. Consider a case that PP0 and PP1 have different lengths on their L3 storage for a given sequence. They have to reach an agreement on the length, by perform all_reduce(MIN). This behavior is consistent with TP. This patch also changes how `check_prefetch_progress` works. Before this patch, it simply does all_reduce(MIN) to determine the final tokens prefetchted. However, this would not work on PP, as we cannot do that in the scheduler thread. This patch adds a new background thread `prefetch_sync_thread`, which does the all_reduce work instead. ## Accuracy Tests Added `test_unified_radix_cache_hicache_pp_kl.py`. I have also tested SWE-bench verified with DSv4 flash model. I ran two groups: - Group A: The baseline, commit 5747ed3b, without this PR, TP=8, L3 enabled. Resolved 356/500. - Group B: This PR, PP=4, TP=2, L3 enabled. Resolved 353/500. I analyzed the difference, and found 351 cases successful in both group, 5 are successful only in group A, and 2 are successful only in group B. The 7 cases that are successfuly in one group but failed in another, are due to container exits after a long time. Here are commands for anyone has interested in reproducing this test. Hints to make L3 are in-use: - Use large number of workers (128), leading to large number of running requests. - Limit number of tokens in L1 and L2. ```sh # Group A sglang serve \\ --model=deepseek-ai/DeepSeek-V4-Flash \\ --tp-size 8 \\ --moe-runner-backend flashinfer_mxfp4 \\ --disable-cuda-graph \\ --mem-fraction-static 0.85 \\ --reasoning-parser deepseek-v4 \\ --tool-call-parser deepseekv4 \\ --host 0.0.0.0 \\ --port 30000 \\ --max-total-tokens=500000 \\ --swa-full-tokens-ratio=0.4 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"512gb\", \"protocol\": \"tcp\"}' # Group B sglang serve \\ --model=deepseek-ai/DeepSeek-V4-Flash \\ --pp-size 4 \\ --tp-size 2 \\ --moe-runner-backend flashinfer_mxfp4 \\ --disable-cuda-graph \\ --mem-fraction-static 0.85 \\ --reasoning-parser deepseek-v4 \\ --tool-call-parser deepseekv4 \\ --host 0.0.0.0 \\ --port 30000 \\ --max-total-tokens=500000 \\ --swa-full-tokens-ratio=0.4 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"512gb\", \"protocol\": \"tcp\"}' # SWE bench agent python -m minisweagent.run.benchmarks.swebench \\ --subset verified \\ --split test \\ -c swebench.yaml -c environment.container_timeout=24h \\ -o /root/swe_test/result \\ -m \"openai/deepseek-ai/DeepSeek-V4-Flash\" \\ -w 128 # SWE evaluation python -m swebench.harness.run_evaluation \\ --predictions_path /root/swe_test/result/preds.json \\ --dataset_name princeton-nlp/SWE-Bench_Verified \\ --split test \\ --max_workers 32 \\ --run_id test-mini-swe-agent \\ --timeout 1200 ``` ## Speed Tests and Profiling I tested with L1 only, L1+L2 and L1+L2+L3. The size of L1 and L2 are configured not to hold all tokens. L3 is large enough to hold all tokens. `request-rate` is set to `1` so that L3 prefetch are performed and not limited by budget. ```sh # L1 sglang serve \\ --model=Qwen/Qwen3-32B \\ --tp-size=2 \\ --pp-size=4 \\ --max-total-tokens=$((256*1024)) \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen25 \\ --page-size=64 \\ --enable-metrics # L2 sglang serve \\ --model=Qwen/Qwen3-32B \\ --tp-size=2 \\ --pp-size=4 \\ --max-total-tokens=$((256*1024)) \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen25 \\ --page-size=64 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-mem-layout=page_first # L3 sglang serve \\ --model=Qwen/Qwen3-32B \\ --tp-size=2 \\ --pp-size=4 \\ --max-total-tokens=$((256*1024)) \\ --reasoning-parser qwen3 \\ --tool-call-parser qwen25 \\ --page-size=64 \\ --enable-metrics \\ --enable-hierarchical-cache \\ --hicache-mem-layout=page_first \\ --hicache-storage-backend=mooncake \\ --hicache-storage-backend-extra-config '{\"master_server_address\": \"127.0.0.1:50051\", \"local_hostname\": \"localhost\", \"metadata_server\": \"http://127.0.0.1:8080/metadata\", \"global_segment_size\": \"256gb\", \"protocol\": \"tcp\"}' # Benchmark python benchmark/hicache/bench_multiturn.py \\ --num-clients=128 \\ --num-rounds=10 \\ --disable-auto-run \\ --model=Qwen/Qwen3-32B \\ --enable-round-barrier \\ --request-rate 1 ``` ``` L1 Per-round metrics: Round 0: Average TTFT = 0.14s, Cache Hit Rate = 0.000000 (128 requests, 128 clients) Round 1: Average TTFT = 0.17s, Cache Hit Rate = 0.470588 (128 requests, 128 clients) Round 2: Average TTFT = 0.18s, Cache Hit Rate = 0.653846 (128 requests, 128 clients) Round 3: Average TTFT = 0.26s, Cache Hit Rate = 0.630580 (128 requests, 128 clients) Round 4: Average TTFT = 0.77s, Cache Hit Rate = 0.363459 (128 requests, 128 clients) Round 5: Average TTFT = 1.22s, Cache Hit Rate = 0.239976 (128 requests, 128 clients) Round 6: Average TTFT = 1.44s, Cache Hit Rate = 0.146925 (128 requests, 128 clients) Round 7: Average TTFT = 1.94s, Cache Hit Rate = 0.143266 (128 requests, 128 clients) Round 8: Average TTFT = 2.01s, Cache Hit Rate = 0.055469 (128 requests, 128 clients) Round 9: Average TTFT = 8.92s, Cache Hit Rate = 0.035112 (128 requests, 128 clients) L2 Per-round metrics: Round 0: Average TTFT = 0.15s, Cache Hit Rate = 0.000000 (128 requests, 128 clients) Round 1: Average TTFT = 0.17s, Cache Hit Rate = 0.470588 (128 requests, 128 clients) Round 2: Average TTFT = 0.18s, Cache Hit Rate = 0.653846 (128 requests, 128 clients) Round 3: Average TTFT = 0.19s, Cache Hit Rate = 0.742857 (128 requests, 128 clients) Round 4: Average TTFT = 0.18s, Cache Hit Rate = 0.795455 (128 requests, 128 clients) Round 5: Average TTFT = 0.23s, Cache Hit Rate = 0.830189 (128 requests, 128 clients) Round 6: Average TTFT = 0.19s, Cache Hit Rate = 0.854839 (128 requests, 128 clients) Round 7: Average TTFT = 0.70s, Cache Hit Rate = 0.652729 (128 requests, 128 clients) Round 8: Average TTFT = 1.16s, Cache Hit Rate = 0.474805 (128 requests, 128 clients) Round 9: Average TTFT = 1.86s, Cache Hit Rate = 0.379213 (128 requests, 128 clients) L3 Per-round metrics: Round 0: Average TTFT = 0.19s, Cache Hit Rate = 0.000000 (128 requests, 128 clients) Round 1: Average TTFT = 0.19s, Cache Hit Rate = 0.470588 (128 requests, 128 clients) Round 2: Average TTFT = 0.25s, Cache Hit Rate = 0.653846 (128 requests, 128 clients) Round 3: Average TTFT = 0.20s, Cache Hit Rate = 0.742857 (128 requests, 128 clients) Round 4: Average TTFT = 0.23s, Cache Hit Rate = 0.795455 (128 requests, 128 clients) Round 5: Average TTFT = 0.25s, Cache Hit Rate = 0.830189 (128 requests, 128 clients) Round 6: Average TTFT = 0.21s, Cache Hit Rate = 0.854839 (128 requests, 128 clients) Round 7: Average TTFT = 0.25s, Cache Hit Rate = 0.873239 (128 requests, 128 clients) Round 8: Average TTFT = 0.26s, Cache Hit Rate = 0.887500 (128 requests, 128 clients) Round 9: Average TTFT = 0.29s, Cache Hit Rate = 0.898876 (128 requests, 128 clients) ``` It shows that, L1 and L2 hit ratios drop at round 4 and 7 respectively. L3 may hold all rounds. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31661559415](https://github.com/sgl-project/sglang/actions/runs/31661559415) Latest PR Test (Extra): :x: [Run #31661559287](https://github.com/sgl-project/sglang/actions/runs/31661559287) ",
"url": "https://github.com/sgl-project/sglang/pull/27010",
"createdAt": "2026-06-02T04:46:51Z",
"updatedAt": "2026-08-13T11:46:59Z",
"timestamp": "2026-08-13T11:46:59Z",
"metrics": {
"reactions": 3,
"comments": 35
},
"labels": [
"high priority",
"hicache",
"run-ci",
"run-ci-extra",
"release-highlight"
],
"author": "stepinto",
"state": "open",
"assignees": [
"whybeyoung",
"xiezhq-hermann",
"ispobock",
"hzh0425",
"ShangmingCai"
],
"change": "updated"
}
},
{
"id": "event:0654e5af33b04582316e",
"signalId": "github:sgl-project/sglang:pull_request:34415",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34415",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "K3 pp",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/34415",
"createdAt": "2026-08-11T11:54:40Z",
"updatedAt": "2026-08-13T11:45:22Z",
"timestamp": "2026-08-13T11:45:22Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"deepseek",
"speculative-decoding",
"npu",
"jit-kernel"
],
"author": "JinyanYi",
"state": "closed",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:34b645ee1c89439bcee9",
"signalId": "github:sgl-project/sglang:pull_request:30929",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:30929",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Support decode radix cache on DeepSeek-V4 (hybrid-SWA, SWA-tail prealloc)",
"text": "## Motivation `--disaggregation-decode-enable-radix-cache` is unconditionally rejected for **every** hybrid-SWA model in `build_kv_cache`: ``` ValueError: --disaggregation-decode-enable-radix-cache is incompatible with sliding window attention (SWA) models ``` But `disaggregation/decode.py` already implements a full **SWA-tail-prealloc decode-radix path** (`DecodeTransferQueue._uses_swa_tail_prealloc` → `_swa_tail_len` / `alloc_extend_swa_tail`) for pools whose allocator exposes `alloc_extend_swa_tail` with `page_size > 1` (e.g. `SWAKVPool` / `DeepSeekV4TokenToKVPool`): decode receives the full KV for full-attention layers and only the sliding-window tail for SWA layers, which is compatible with the prefix-match-and-lock allocation path. The blanket guard is stale — even `server_args` help documents decode radix as incompatible only with `--enable-hisparse`, speculative decoding, and the fake transfer backend, **not SWA**. As a result, decode-side prefix reuse is unavailable on DeepSeek-V4 and other SWA-tail-capable models, forcing redundant KV transfers. ## Modifications 1. **`mem_cache/kv_cache_builder.build_kv_cache`** — only reject SWA models whose allocator lacks the SWA-tail-prealloc capability (`hasattr(allocator, \"alloc_extend_swa_tail\") and page_size > 1`). Mamba/SSM stays rejected. 2. **`mem_cache/unified_radix_cache.UnifiedRadixCache.cache_unfinished_req`** — fix a fatal assert on hybrid-SWA decode radix. `match_prefix()` gates device indices on **all** components, so the SWA validator collapses `device_indices` to length 0 at the first out-of-window tombstone for a long prefix whose full-attention KV is entirely device-resident (`new_prefix_len=![\"image\"](\"https://github.com/user-attachments/assets/4bd83cf9-90ed-4de8-a336-10546b6fcfcd\")
```bash SGLANG_MLA_DECODE_TUNE=1 \\ # 0 for the baseline arm SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 \\ AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1 \\ python3 -m sglang.launch_server \\ --model-path $MODEL \\ --served-model-name moonshotai/Kimi-K3 \\ --trust-remote-code \\ --tp-size 8 \\ --dtype bfloat16 \\ --kv-cache-dtype fp8_e4m3 \\ --prefill-attention-backend aiter \\ --decode-attention-backend triton \\ --page-size 64 \\ --mem-fraction-static 0.85 \\ --max-running-requests 32 \\ --cuda-graph-max-bs-decode 32 \\ --chunked-prefill-size 32768 \\ --max-prefill-tokens 32768 \\ --mamba-full-memory-ratio 0.3 \\ --mamba-ssm-dtype bfloat16 \\ --enable-int8-mamba-checkpoint \\ --mamba-track-interval 1024 \\ --reasoning-parser kimi_k3 \\ --tool-call-parser kimi_k3 \\ --enable-cache-report \\ --host 0.0.0.0 --port $PORT ``` ```bash # aiperf 0.11.0. --warmup-request-count 32, not the default 3, is what gets all 8 # prefixes resident before measurement. aiperf profile \\ --model 'moonshotai/Kimi-K3' \\ --tokenizer $MODEL --tokenizer-trust-remote-code \\ --url \"http://localhost:$PORT\" \\ --endpoint-type chat --streaming --use-server-token-count \\ --num-prefix-prompts 8 \\ --prompt-prefix-length 63240 \\ --synthetic-input-tokens-mean 4760 --synthetic-input-tokens-stddev 0 \\ --output-tokens-mean 350 --output-tokens-stddev 0 \\ --extra-inputs 'ignore_eos:true' \\ --extra-inputs 'min_tokens:350' \\ --extra-inputs 'max_tokens:350' \\ --warmup-request-count 32 \\ --sweep-type zip \\ --concurrency 6,12,24,32 \\ --request-count 60,72,96,128 \\ --random-seed 42 ``` ### Short context: 8192 in / 1024 out MLA attention is 7.5% of the decode step here (2.68 of 35.72 ms at concurrency 32). bf16 KV, `page_size=1`, Triton on both ends, radix off, so this is not the sweep above with a different workload. Both arms ran back to back in one session. With the decode Triton attention kernel optimized, we got 2-12% ITL improvement as well as 2-8% TTT boost. ![\"image\"](\"https://github.com/user-attachments/assets/b0de2a23-7351-427f-af52-834bcccc5f9b\")
```bash SGLANG_MLA_DECODE_TUNE=1 \\ SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 \\ AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1 AITER_SITUV2_A4W4=0 \\ SGLANG_K3_FLYDSL_AR_NORM=1 \\ python3 -m sglang.launch_server \\ --model-path $MODEL \\ --served-model-name moonshotai/Kimi-K3 \\ --trust-remote-code \\ --tp-size 8 \\ --dtype bfloat16 \\ --attention-backend triton \\ --mem-fraction-static 0.85 \\ --cuda-graph-max-bs-decode 256 \\ --disable-radix-cache \\ --reasoning-parser kimi_k3 \\ --tool-call-parser kimi_k3 \\ --host 0.0.0.0 --port $PORT ``` ```bash for C in 2 4 8 16 32; do python3 -m sglang.benchmark.serving \\ --backend sglang \\ --host 127.0.0.1 --port $PORT \\ --model $MODEL --tokenizer $MODEL \\ --dataset-name random \\ --random-input-len 8192 \\ --random-output-len 1024 \\ --random-range-ratio 1.0 \\ --num-prompts $(( C * 8 )) \\ --max-concurrency $C \\ --warmup-requests $(( C * 2 )) \\ --seed 42 done ``` ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31659060438](https://github.com/sgl-project/sglang/actions/runs/31659060438) Latest PR Test (Extra): :x: [Run #31659060242](https://github.com/sgl-project/sglang/actions/runs/31659060242) ",
"url": "https://github.com/sgl-project/sglang/pull/34580",
"createdAt": "2026-08-12T13:43:44Z",
"updatedAt": "2026-08-13T10:46:41Z",
"timestamp": "2026-08-13T10:46:41Z",
"metrics": {
"reactions": 2,
"comments": 3
},
"labels": [
"amd",
"run-ci",
"jit-kernel"
],
"author": "amd-danli103",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:4d3d4abbbd9d619cb29b",
"signalId": "github:sgl-project/sglang:pull_request:31574",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:31574",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[EPD] Batch embedding cache host-device range copies",
"text": " ## Motivation The paged multimodal embedding cache may store one embedding across multiple non-contiguous pinned-host page runs. Loading from or storing to this pool currently submits one `Tensor.copy_` operation per run from Python. As the cache becomes fragmented, repeated Python dispatch and CUDA copy submission reduce effective H2D and D2H throughput. Existing KV transfer kernels are tied to KV layouts and index-based gather/scatter semantics, so they are not a good fit for copying contiguous embedding ranges. ## Modifications - Add `transfer_embedding_ranges_direct`, which accepts source starts, destination starts, and lengths for multiple embedding ranges. - Submit the ranges with `cudaMemcpyBatchAsync` on supported CUDA runtimes. - Fall back to individual `cudaMemcpyAsync` calls when the batch API or stream configuration is unsupported. - Use the new operation for embedding-cache host-device transfers while preserving the Python `Tensor.copy_` loop as a compatibility fallback. - Separate storage-transfer buffer construction from host-device transfer-plan construction and add unit coverage for fragmented H2D/D2H plans. ## Accuracy Tests ### CUDA correctness test Added `test_transfer_embedding_ranges_direct` to `test_kvcacheio.py`. It compares both host-to-device and device-to-host range copies against equivalent PyTorch `copy_` operations. ### End-to-end EPD test ![\"image\"](\"https://github.com/user-attachments/assets/8f4c3957-ee84-49f3-8f39-4ad1a3afc988\")
## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31685684597](https://github.com/sgl-project/sglang/actions/runs/31685684597) Latest PR Test (Extra): :x: [Run #31685684411](https://github.com/sgl-project/sglang/actions/runs/31685684411) ",
"url": "https://github.com/sgl-project/sglang/pull/34647",
"createdAt": "2026-08-13T01:21:47Z",
"updatedAt": "2026-08-13T09:39:54Z",
"timestamp": "2026-08-13T09:39:54Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"amd",
"jit-kernel"
],
"author": "chien-an-chen",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:8e912baaa3eb9c93da5c",
"signalId": "github:sgl-project/sglang:pull_request:34400",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34400",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Check paged KV capacity before kernel launch",
"text": "## Summary - Check paged KV capacity before launching the extend/decode Triton allocator kernels. - Preserve the existing release-page merge behavior before the capacity check. - Add regression tests covering OOM, successful allocation, and release-page merging. ## Motivation The old implementation launched the allocation kernel and only then checked whether enough free pages were available. On an A100 with two free pages, a prefill or decode request requiring three pages returned `None` but still launched the kernel once. The kernel could read beyond the logical free-page list. This is reachable in normal serving when the KV cache is full and no more evictable pages are available. See issue #34399 for the reproducible online path and evidence. ## Validation ### Unit and allocator tests ```text 138 passed, 2 warnings, 33 subtests passed in 29.19s ``` The test command covered the new paged allocator regression tests, multi-ended allocator, paged free, SWA, HiSparse, DCP layout, KV page invariants, and decode bookkeeping tests. ### A100 validation With `NVIDIA A100-PCIE-40GB`, PyTorch `2.9.1+cu128`, and `CUDA_LAUNCH_BLOCKING=1`: ```text extend_oom= True kernel_launches= 0 free_unchanged= True decode_oom= True kernel_launches= 0 free_unchanged= True ``` The same unmodified-mainline inputs launched one kernel in each path. ### OOM-path microbenchmark After warming the successful kernel path, 100 synchronous prefill OOM calls measured: ```text main: 126.14 us/call patch: 59.37 us/call ``` This is a single local failure-path measurement, not an end-to-end throughput benchmark. Successful allocation behavior is unchanged. ## Scope This patch is intentionally limited to the paged allocator capacity-check ordering. It does not change the Triton kernel interface, page numbering, or normal successful allocation behavior. Fixes #34399 --- ### CI States Latest PR Test (Base): :x: [Run #31687561391](https://github.com/sgl-project/sglang/actions/runs/31687561391) Latest PR Test (Extra): :x: [Run #31687561086](https://github.com/sgl-project/sglang/actions/runs/31687561086) ",
"url": "https://github.com/sgl-project/sglang/pull/34400",
"createdAt": "2026-08-11T09:30:29Z",
"updatedAt": "2026-08-13T09:39:53Z",
"timestamp": "2026-08-13T09:39:53Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "waizuichougou",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:cc6abcc21b19586a191b",
"signalId": "github:sgl-project/sglang:pull_request:34542",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34542",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[MiniMax-M3] Overlap shared and routed experts",
"text": " ## Motivation This PR is split from [#33957](https://github.com/sgl-project/sglang/pull/33957) and keeps only the MiniMax-M3 shared/routed overlap and the TRT-LLM MoE PDL limit change. MiniMax-M3's unfused shared and routed expert branches run sequentially during CUDA Graph execution. We observed that a 7,680-token FlashInfer TRT-LLM MXFP8 routed-MoE call can stall when PDL is enabled during multi-stream CUDA Graph replay. The existing 8,192-token PDL limit leaves this failing shape enabled and is not forwarded by the FP8 wrapper paths. ## Modifications - Run the standalone shared expert and routed MoE on separate CUDA streams during CUDA Graph execution. - Lower the TRT-LLM MoE PDL limit from 8,192 to 4,096 tokens and apply it to the FP8 wrapper paths. - Add a manual single-GPU routed-MXFP8 CUDA Graph reproducer for the PDL stall. ## Accuracy Tests | Configuration | GSM8K | LongBench v2 | |---|---:|---:| | Baseline | 0.9726 | 0.5484 | | This PR | 0.9718 | 0.6236 | ## Speed Tests and Profiling The serving benchmark uses TP4 with fixed 8K input / 1K output requests. ```bash SGLANG_DISABLE_MSA=true sglang serve \\ --model-path \"$MODEL_PATH\" \\ --host 0.0.0.0 --port 8000 \\ --tp 4 --dtype bfloat16 --trust-remote-code \\ --moe-runner-backend flashinfer_trtllm_routed \\ --attention-backend trtllm_mha \\ --fp8-gemm-backend auto \\ --cuda-graph-backend-prefill tc_piecewise \\ --disable-radix-cache ``` TTFT and TPOT are P50 latency in milliseconds; Peak interactivity is measured in tokens/s. | C | TTFT base | TTFT PR | TPOT base | TPOT PR | Peak interactivity base | Peak interactivity PR | |---:|---:|---:|---:|---:|---:|---:| | 1 | 407.30 | 393.53 | 5.97 | 5.29 | 167.62 | 189.11 | | 2 | 580.51 | 576.84 | 6.60 | 5.92 | 151.54 | 168.98 | | 4 | 912.22 | 903.30 | 7.58 | 6.89 | 131.96 | 145.16 | | 8 | 1602.95 | 1573.13 | 9.03 | 8.31 | 110.69 | 120.26 | | 16 | 3008.79 | 2962.64 | 11.62 | 10.82 | 86.07 | 92.42 | | 32 | 5829.11 | 5718.22 | 15.63 | 14.74 | 63.98 | 67.86 | | 64 | 11492.14 | 11275.94 | 20.19 | 19.45 | 49.53 | 51.42 | ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: [Run #31676678168](https://github.com/sgl-project/sglang/actions/runs/31676678168) Latest PR Test (Extra): :white_check_mark: [Run #31676683039](https://github.com/sgl-project/sglang/actions/runs/31676683039) ",
"url": "https://github.com/sgl-project/sglang/pull/34542",
"createdAt": "2026-08-12T08:01:31Z",
"updatedAt": "2026-08-13T09:38:58Z",
"timestamp": "2026-08-13T09:38:58Z",
"metrics": {
"reactions": 0,
"comments": 1
},
"labels": [
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "rogeryoungh",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:5a787f96d86cfeb2e29b",
"signalId": "github:sgl-project/sglang:issue:34709",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:issue:34709",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] benchmark/endpoint.py forks the server process after the parent initializes the accelerator (breaks XPU, latent on CUDA)",
"text": "### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Describe the bug launch_or_reuse_server starts the server with multiprocessing.Process (python/sglang/benchmark/endpoint.py:62), i.e. the platform default start method — fork on Linux. By that point the CLI parent has already initialized the accelerator in ServerArgs.__post_init__, so the child inherits a dead device context: - XPU: server_args.py:3594 → get_device_memory_capacity() → get_xpu_memory_capacity() → torch.xpu.mem_get_info() (utils/common.py:764) - CUDA: server_args.py:3677 → _handle_environment_variables() → get_device_sm() → torch.cuda.get_device_capability() (utils/common.py:611), reached whenever SGLANG_OPT_FP8_WO_A_GEMM is on — it defaults to True (environ.py:1133) launch_server itself forces spawn (entrypoints/engine.py:1703), but that runs inside the already-forked child, so it cannot help. The crash surfaces as soon as the child touches the device — on a multimodal model the warmup request's image preprocessing does image.to(device) — and is reported only as Server process exited during startup (exit code -9). On CUDA the same fork happens but usually goes unnoticed: the forked child is the HTTP/tokenizer-manager process, which does no device work for text-only models, and the scheduler processes it creates are spawned. ### Reproduction Steps as below ``` # NVIDIA python3 -m sglang.benchmark.one_batch_server --model-path Qwen/Qwen3.5-9B \\ --trust-remote-code --batch-size 1 --input-len 128 --output-len 8 --tp-size 4 #Error : RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method #INTEL python3 -m sglang.benchmark.one_batch_server --model-path Qwen/Qwen3.5-9B \\ --trust-remote-code --batch-size 1 --input-len 128 --output-len 8 --tp-size 4 --device xpu #Error : RuntimeError: Cannot re-initialize XPU in forked subprocess. To use XPU with multiprocessing, you must use the 'spawn' start method ``` ### Environment Nvidia Environment - sglang : 0.0.0.dev16528+gef3043a51 torch : 2.13.0+cu130 | cuda: 13.0 | hip: None | xpu: None python : 3.12.3 | Linux-5.15.0-1071-nvidia-x86_64-with-glibc2.39 device : 8 x NVIDIA A100-SXM4-40GB driver : 580.95.05 | mem GB: - 40GB Intel Environment - sglang : 0.5.18.dev364+g667e18d99 torch : 2.12.0+xpu | cuda: None | hip: None | xpu: 20250302 python : 3.12.13 | Linux-6.14.0-37-generic-x86_64-with-glibc2.41 device : 8 x Intel(R) Arc(TM) Pro B60 Graphics driver : 1.14.36300+8 | mem GB: 22.7",
"url": "https://github.com/sgl-project/sglang/issues/34709",
"createdAt": "2026-08-13T08:08:27Z",
"updatedAt": "2026-08-13T09:36:51Z",
"timestamp": "2026-08-13T09:36:51Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "dayanandav",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:3f982cf2c650405495e0",
"signalId": "github:sgl-project/sglang:pull_request:33684",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:33684",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Weight Cache] Support static DP/EP layouts",
"text": "## Motivation Weight-cache daemons and engines must construct identical model-parallel layouts before CUDA IPC can safely map a cached MoE shard. The existing daemon launcher rejected `dp_size > 1`, preventing static DP/EP deployments from using the daemon-backed weight cache. This change adds static DP/EP layout support to the CUDA IPC weight-cache path. It is the implementation foundation for the DP/EP roadmap in #33522. ## Modifications - Propagate static DP/EP, MoE DP, attention DP, MoE A2A, and DeepEP-mode configuration from the engine into weight-cache daemons. - Initialize daemon-side model-parallel and MoE state with the same static layout as the engine. - Extend `CacheConfig` with MoE DP/EP shard identity and layout fields, so daemon/client compatibility validation rejects incompatible shards. - Fix static Mooncake EP dispatch to use an all-active rank mask when Elastic EP state is absent. - Route Qwen3 MoE through the existing DeepEP-class backend predicate so Mooncake uses the EP execution path. - Add CPU coverage for MoE shard-rank calculation, extended cache fingerprints, and daemon DeepEP-mode forwarding. The implementation supports static layouts. Dynamic expert placement, EPLB, redundant experts, and recovery lifecycle ownership remain roadmap work. ## Accuracy Tests - `PYTHONPATH=python python -m pytest -q test/registered/unit/model_loader/test_weight_cache_protocol.py` - `29 passed` - Eight-GPU Qwen3-30B-A3B-FP8 daemon IPC serving with `TP=DP=EP=8`, Mooncake low-latency A2A, DP attention, and DP LM head. - 8 daemons exported IPC handles. - 8 schedulers logged `Loaded model via IPC`. - A deterministic completion generated `Paris` for `The capital of France is`. ## Speed Tests and Profiling N/A. This change establishes static DP/EP correctness and does not claim a new performance result. ## Checklist - [x] Format modified Python files with Black; CI will run the repository pre-commit hooks. - [x] Add unit tests according to [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). --- ### CI States Latest PR Test (Base): :x: [Run #31676640818](https://github.com/sgl-project/sglang/actions/runs/31676640818) Latest PR Test (Extra): :x: [Run #31676640548](https://github.com/sgl-project/sglang/actions/runs/31676640548) ",
"url": "https://github.com/sgl-project/sglang/pull/33684",
"createdAt": "2026-08-05T09:10:25Z",
"updatedAt": "2026-08-13T09:36:11Z",
"timestamp": "2026-08-13T09:36:11Z",
"metrics": {
"reactions": 0,
"comments": 4
},
"labels": [
"run-ci"
],
"author": "UNIDY2002",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:af3f45ae12801d8329d5",
"signalId": "github:sgl-project/sglang:issue:34719",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:issue:34719",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "issue",
"title": "[Bug] Scheduler crashes with AttributeError ('list' object has no attribute 'tolist') on mixed batches with token_ids_logprob — prefill and decode paths, v0.5.14–v0.5.17",
"text": "## Summary The scheduler process dies with ``` AttributeError: 'list' object has no attribute 'tolist' ``` whenever a request using `token_ids_logprob` shares a batch with a request that does not. One scoring client can take the whole server down. This affects the released versions **v0.5.14, v0.5.16, and v0.5.17** (verified by reading each tag's source), in **both** the prefill and decode result paths of `scheduler_components/batch_result_processor.py`. ## Mechanism `next_token_token_ids_logprobs_val` holds one entry per request in the batch. Requests that asked for token-ids logprobs contribute GPU tensors; co-batched requests that did not contribute plain `[]` entries (in decode, `get_token_ids_logprobs_raw` appends a bare `[]` for `token_ids is None`; the prefill delayed-D2H path produces plain lists as well). The consumer then calls `.tolist()` unconditionally on every entry: ```python # batch_result_processor.py, v0.5.17 — same in v0.5.14/v0.5.16 if logits_output.next_token_token_ids_logprobs_val: logits_output.next_token_token_ids_logprobs_val = [ v.tolist() for v in logits_output.next_token_token_ids_logprobs_val ] ``` The same unguarded pattern appears twice — in the prefill-path `move_logprobs_to_cpu` and in `_normalize_decode_outputs` — and `next_token_top_logprobs_val` / `next_token_top_logprobs_idx` carry the identical latent bug at both sites. Solo scoring requests survive (batch of one, all tensors), so light testing passes; the first *mixed* batch in production kills the scheduler. The HTTP server keeps answering `/health` with 503, so under Kubernetes the pod sits dead until a liveness probe restarts it, and the next such request kills it again — a crash loop. ## Reproduction Server: `sglang serve ![\"image\"](\"https://github.com/user-attachments/assets/903594e3-2b84-4e89-ae50-1a54072c31b2\")
## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :hourglass_flowing_sand: [Run #31679685134](https://github.com/sgl-project/sglang/actions/runs/31679685134) Latest PR Test (Extra): :x: [Run #31679684932](https://github.com/sgl-project/sglang/actions/runs/31679684932) ",
"url": "https://github.com/sgl-project/sglang/pull/34597",
"createdAt": "2026-08-12T16:37:28Z",
"updatedAt": "2026-08-13T09:00:46Z",
"timestamp": "2026-08-13T09:00:46Z",
"metrics": {
"reactions": 1,
"comments": 0
},
"labels": [
"amd",
"deepseek",
"run-ci"
],
"author": "1am9trash",
"state": "closed",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:b61f0ee2fac2a928d5c1",
"signalId": "github:sgl-project/sglang:pull_request:27770",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:27770",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[P/D disagg] Decode-side radix cache for SWA hybrid models (unified radix tree)",
"text": " This PR enables device-resident decode-side radix reuse for hybrid SWA models on the experimental unified radix tree. It keeps full-attention KV reusable while transferring a fresh SWA window, reducing repeated P/D KV transfer without introducing HiCache or Mamba support. ### How This Was Implemented - Caps the reused full-attention prefix at the SWA window boundary and allocates only the fresh SWA tail. - Caches an unfinished request's computed prefix, deduplicates overlapping full-attention KV, then repoints the live request to the cache-owned slots. - Carries SWA lock boundaries through rejection paths, including when the cap reduces a matched prefix to zero. - Requires the unified radix tree and rejects hierarchical cache, Mamba/SSM, DSA, and SWA-compress variants. - Adds registered device-only gpt-oss-20b P/D coverage. ### Before and After | Unfinished-request state | Before | After | |---|---|---| | Full-attention KV is device-resident but the old SWA window is tombstoned | The normal full+SWA match can return zero indices after insertion has deduplicated the request's full KV. Repointing then fails `new_prefix_len <= len(new_indices)`; continuing would leave the request referencing freed duplicate slots. | This one lifecycle path requests the full device-resident match, rewrites the request table to the cache-owned full-KV slots, and transfers its lock to the deepest full-resident node. Normal scheduler matching remains SWA-window-safe. | ### Validation - `PYTHONDONTWRITEBYTECODE=1 .venv/bin/python test/registered/unit/mem_cache/test_decode_radix_lock_ref.py -v` — 6 passed. - `PYTHONDONTWRITEBYTECODE=1 .venv/bin/python test/registered/unit/mem_cache/test_unified_radix_cache_unittest.py` — 1,818 ran; 758 skipped; OK. - Mocked `build_kv_cache` smoke — device-only path succeeds; SWA + hierarchical cache raises. - `.venv/bin/pre-commit run --files ...` — passed. - Targeted `/rerun-group disaggregation` — SWA decode-radix passed on 8×H20; hybrid-attention passed on 8×H200. ## Motivation Extend the decode-side radix cache (P/D disaggregation, #19746) to **sliding-window-attention (SWA) hybrid models** such as gpt-oss, which are currently rejected at startup. With this, a decode worker can **reuse a cached prefix's full-attention KV across requests/turns** and request only the delta from the prefill worker, instead of receiving the entire prefix KV every time. The feature is gated to the **experimental unified radix tree** (`SGLANG_ENABLE_UNIFIED_RADIX_TREE`). The default `SWARadixCache` and the Mamba/SSM pools remain unsupported on this path, as do DSA (DeepSeek-V4) and SWA-compress (Gemma/MiMo) variants. > Supersedes #26218, which took the same feature through the default `SWARadixCache`. We're standardizing on the unified tree for SWA decode-radix, so that path is dropped here. ## Background: why SWA needs special handling A hybrid SWA model keeps **two** KV caches with different lifetimes: - **Full-attention layers** — keep KV for every token, as long as it's cached. Large, sticky. - **Sliding-window layers** — keep KV for only the last *W* tokens; older slots recycle as the window advances. Bounded. So on the decode worker the prefix's **full-attention** KV can be reused directly, but its **sliding-window** KV cannot — by reuse time the original writer's window has slid past it. Such a node is a **tombstone** (full KV present, SWA recycled). This is normal and harmless for correctness: a reuser only needs the window for its *own* current position, which it gets fresh. ## What this PR does - **`kv_cache_builder.py`** — allow SWA + `--disaggregation-decode-enable-radix-cache` **only when the unified tree is active**; otherwise raise (DSA / SWA-compress still blocked even under unified). - **`decode.py`** — decode-side SWA mechanics: cap full-attention prefix reuse at the sliding-window start so the window lands in the fresh delta; `alloc_extend_swa_tail` on radix hits; mark the non-window prefix `swa_evicted_seqlen` (SWA tombstone); SWA-aware budget/eviction accessors; thread the SWA lock uuid through prefix lock/release. - **`unified_radix_cache.py` + `swa_component.py` + `base_prefix_cache.py`** — two `UnifiedRadixCache` fixes this path needs: 1. **SWA pool leak.** `SWAComponent.prepare_for_caching_req` only forwarded `swa_evicted_seqlen` when `is_finished=True`; on the unfinished path the out-of-window prefix tokens were inserted as *live* SWA and counted as `swa_evictable`, so at idle `available + evictable` exceeds the pool. Forward it on both paths so they become tombstones (`0` on the aggregated path → unchanged there). 2. **`cache_unfinished_req` repoint.** The dedup/repoint operates on full-attention cards but was sized by the SWA-window-safe match. On a reused, tombstoned prefix the window-safe match collapses to `0` while the cards number in the hundreds, tripping `assert new_prefix_len <= len(new_indices)`. Add `MatchPrefixParams.return_full_match`: `_match_prefix_helper` returns the deepest device-resident node, `_match_post_processor` returns the full cards + that node. The unified `inc_lock_ref` already skips tombstones, so no lock change is needed. No `swa_radix_cache.py` changes — the old path is intentionally not supported. ## Usage ``` --disaggregation-mode decode \\ --disaggregation-transfer-backend {nixl|mooncake} \\ --disaggregation-decode-enable-radix-cache \\ --page-size 64 # with SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 in the env ``` ## Test `test/manual/disaggregation/test_disaggregation_decode_radix_cache_swa.py` (gpt-oss-20b; manual because the model isn't in the CI pool) launches with the unified tree via `extra_*_env` and runs the shared decode-radix mixin (multi-turn cache-hit + 2-pass gsm8k). Validated on `openai/gpt-oss-20b`, 1P1D on 2×L40S, `--page-size 64 --attention-backend triton`, strict idle + per-iteration memory checks: | Check | Result | |---|---| | Multi-turn prefix reuse (decode cache hit) | cached tokens grow across rounds (0 → 1792 → 3584) | | gsm8k, 500 ex, 2 passes (2nd hits the cache) | 0.532 / 0.572 | | SWA-pool leak / `cache_unfinished_req` assert | **0 / 0** | ## Deep dive Writeup of the two `UnifiedRadixCache` bugs hit while enabling this path — exact errors, root causes, fix walkthrough, before/after validation: https://gist.github.com/ishandhanani/ed3cd2082381f31d848a7f6fcac977c6 --- ### CI States Latest PR Test (Base): :x: [Run #31675095098](https://github.com/sgl-project/sglang/actions/runs/31675095098) Latest PR Test (Extra): :x: [Run #31675095021](https://github.com/sgl-project/sglang/actions/runs/31675095021) ",
"url": "https://github.com/sgl-project/sglang/pull/27770",
"createdAt": "2026-06-10T07:28:25Z",
"updatedAt": "2026-08-13T09:45:06Z",
"timestamp": "2026-08-13T09:45:06Z",
"metrics": {
"reactions": 3,
"comments": 38
},
"labels": [
"hicache",
"run-ci",
"run-ci-extra"
],
"author": "ishandhanani",
"state": "open",
"assignees": [
"hzh0425",
"ShangmingCai"
],
"change": "updated"
}
},
{
"id": "event:8103d08f8fa2eefd3c17",
"signalId": "github:sgl-project/sglang:pull_request:34498",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34498",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[ROCm] Direct-write a8w8 bmm output to eliminate o_proj transpose copy",
"text": "Here's the draft PR description for PR 34498 (direct-write). The label in your data says \"fold_quant\" but I've relabeled it direct_write to match this PR. Title: [ROCm] Direct-write a8w8 bmm output to eliminate o_proj transpose copy This PR removes a redundant memory copy in the per-channel FP8 attention output path on AMD gfx95 (validated on MI355X with Kimi-K2.7-Code-MXFP4). By emitting the absorbed a8w8 BMM output already in the layout the o_proj epilogue needs, the downstream `flatten` becomes a free view instead of a transpose copy, improving throughput by up to ~3.3% with no accuracy change. ## Motivation For per-channel FP8 `kv_b_proj` on gfx95, the absorbed value BMM (`batched_gemm_a8w8_a_per_token_group_prequant_w_per_batched_tensor_quant`) writes its output in `(heads, tokens, vdim)` layout (`YQ=None, transpose_bm=False`). The o_proj epilogue then runs `attn_output.transpose(0, 1).flatten(1, 2)`, where the `.transpose(0, 1)` forces a full memory copy — visible in the profile as a standalone `direct_copy_kernel bf16`. ## Change Preallocate `_bmm_buf` in `(tokens, heads, vdim)` layout and pass `YQ=_bmm_buf, transpose_bm=True` so the GEMM writes directly in the final layout. The downstream `flatten(1, 2)` is then a free view — the transpose copy is gone. This mirrors the existing uint8 / MXFP4 `_bmm_buf` fast path directly above it. No numerical change (identical values, different write layout). ## Accuracy test Results (MI355X / gfx950, Kimi-K2.7-Code-MXFP4) **Accuracy (GSM8K)** — neutral, within run-to-run noise: Results (MI355X / gfx950, Kimi-K2.7-Code-MXFP4) | parallel | 100 | 300 | 600 | 1200 | | --- | --- | --- | --- | --- | | original | 0.951 | 0.953 | 0.942 | 0.950 | | direct_write | 0.946 | 0.947 | 0.950 | 0.951 | ## Speed Tests and Profiling **Throughput / GPU (tok/s)** — consistent improvement: Results (MI355X / gfx950, Kimi-K2.7-Code-MXFP4) | Conc | 4 | 8 | 16 | 32 | 64 | 128 | | --- | --- | --- | --- | --- | --- | --- | | original | 906.82 | 1491.25 | 2128.38 | 2955.05 | 3925.4 | 5048.96 | | direct_write | 936.7 | 1522.67 | 2199.21 | 2990.27 | 3989.75 | 5112.88 | | Δ | +3.30% | +2.11% | +3.33% | +1.19% | +1.64% | +1.27% | original profiling: ![\"Image\"](\"https://github.com/user-attachments/assets/f3d40c9c-7607-48da-b170-e52f76ce7fa1\")
### Reproduction `v0.5.17-rocm720-mi35x-20260811 ` ### Environment `v0.5.17-rocm720-mi35x-20260811 `",
"url": "https://github.com/sgl-project/sglang/issues/34611",
"createdAt": "2026-08-12T17:56:26Z",
"updatedAt": "2026-08-13T06:38:33Z",
"timestamp": "2026-08-13T06:38:33Z",
"metrics": {
"reactions": 0,
"comments": 3
},
"labels": [],
"author": "functionstackx",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:779759414d2bdac35ff7",
"signalId": "github:sgl-project/sglang:pull_request:34687",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34687",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Main transport",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :x: **Missing `run-ci` label** -- add it to run CI tests. Latest PR Test (Extra): :x: **Blocked** -- `run-ci` is required first. ",
"url": "https://github.com/sgl-project/sglang/pull/34687",
"createdAt": "2026-08-13T06:35:35Z",
"updatedAt": "2026-08-13T06:35:47Z",
"timestamp": "2026-08-13T06:35:47Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"npu"
],
"author": "longxin9715",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:1960c37353adfb6fbc41",
"signalId": "github:sgl-project/sglang:pull_request:34471",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34471",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[diffusion] Support LTX-2.5",
"text": " ## Motivation ## Modifications ## Accuracy Tests ## Speed Tests and Profiling ## Checklist - [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review and Merge Process 1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci` 4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31632222955](https://github.com/sgl-project/sglang/actions/runs/31632222955) Latest PR Test (Extra): :x: [Run #31632222552](https://github.com/sgl-project/sglang/actions/runs/31632222552) ",
"url": "https://github.com/sgl-project/sglang/pull/34471",
"createdAt": "2026-08-11T23:21:25Z",
"updatedAt": "2026-08-13T06:35:45Z",
"timestamp": "2026-08-13T06:35:45Z",
"metrics": {
"reactions": 1,
"comments": 1
},
"labels": [
"documentation",
"run-ci",
"diffusion"
],
"author": "AgainstEntropy",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:08fd20771da6985f61b4",
"signalId": "github:sgl-project/sglang:pull_request:34686",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34686",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[NIXL] Add the missing Prefill bootstrap timeout",
"text": "- RFC: [PD disaggregation: single protocol layer, per-backend transport #33861](https://github.com/sgl-project/sglang/issues/33861) - Staged implementation plan and PR tracking: [PD shared-protocol implementation plan #34510](https://github.com/sgl-project/sglang/issues/34510) ## Background RFC #33861 proposes gradually consolidating the duplicated PD request/room protocol logic in Mooncake, NIXL, and Mori into a single common protocol layer, while keeping third-party engine-specific behavior in each backend Transport. Before extracting the common protocol layer, Step 1 of the implementation plan in #34510 aligns clear, non-controversial semantic gaps through small, independent, backend-local PRs. This PR addresses the first gap: the missing bootstrap timeout in the NIXL Prefill Sender. The bootstrap timeout covers the following case: > Prefill has created the Sender/room for a request, but Decode destination metadata never arrives. The Sender should not remain in `KVPoll.Bootstrapping` indefinitely; it should transition to `KVPoll.Failed` after the existing `SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT` deadline. ## Current Problem `CommonKVSender` already provides `_check_bootstrap_timeout()`: ```python # python/sglang/srt/disaggregation/common/conn.py def _check_bootstrap_timeout(self) -> Optional[KVPoll]: if self.init_time is None: return None elapsed = time.time() - self.init_time if elapsed < self.kv_mgr.bootstrap_timeout: return None self.kv_mgr.record_failure( self.bootstrap_room, f\"Request {self.bootstrap_room} timed out after {elapsed:.1f}s \" f\"in KVPoll.Bootstrapping\", ) self.kv_mgr.update_status(self.bootstrap_room, KVPoll.Failed) return KVPoll.Failed ``` This helper: 1. Computes the bootstrap wait time from the Sender's `init_time`; 2. Returns `None` while the request remains within the deadline; 3. Records a failure reason after the deadline; 4. Updates the room to `KVPoll.Failed`; 5. Returns `KVPoll.Failed`. However, the current NIXL Sender has two missing pieces: 1. `NixlKVSender.__init__()` does not record the start of the Prefill bootstrap deadline; 2. `NixlKVSender.poll()` does not call the existing helper while the room is in `KVPoll.Bootstrapping`. NIXL currently records `_transfer_start_time` only for actual KV/state transfer latency: ```python if self._transfer_start_time is None and ( len(kv_indices) > 0 or state_indices is not None ): self._transfer_start_time = time.perf_counter() ``` That timer starts when the first meaningful KV/state chunk is submitted. It does not include the bootstrap phase spent waiting for Decode metadata, so it cannot replace `init_time`. Similarly, the `init_time` set by `NixlKVReceiver.send_metadata()` belongs to the Decode Receiver waiting timeout. It is not the Prefill Sender bootstrap deadline. As a result, if Decode destination metadata never arrives, a NIXL Prefill room can remain in `KVPoll.Bootstrapping` indefinitely. ## Existing Behavior in the Other Backends ### Mooncake Mooncake records the bootstrap start time when creating the Sender: ```python # python/sglang/srt/disaggregation/mooncake/conn.py super().__init__( mgr, bootstrap_addr, bootstrap_room, dest_tp_ranks, pp_rank, req_has_disagg_prefill_dp_rank, ) self.conclude_state = None self.init_time = time.time() self._init_trace_ctx() ``` Its `poll()` calls the common helper while the room remains in `KVPoll.Bootstrapping`: ```python # python/sglang/srt/disaggregation/mooncake/conn.py elif status == KVPoll.Bootstrapping: timeout_result = self._check_bootstrap_timeout() if timeout_result is not None: return timeout_result ``` Mooncake therefore cannot wait indefinitely for missing Decode metadata. ### Mori Mori also records the bootstrap start time when creating the Sender: ```python # python/sglang/srt/disaggregation/mori/conn.py super().__init__( mgr, bootstrap_addr, bootstrap_room, dest_tp_ranks, pp_rank, req_has_disagg_prefill_dp_rank, ) self.transfer_statuses = [] self.pending_infos = None self.conclude_state = None self.status_notified = False self.init_time = time.time() ``` Mori does not call `_check_bootstrap_timeout()` directly. Instead, it performs the equivalent check inline in its own `poll()`: ```python # python/sglang/srt/disaggregation/mori/conn.py if status == KVPoll.Bootstrapping: elapsed = time.time() - self.init_time if elapsed >= self.kv_mgr.bootstrap_timeout: reason = ( f\"Request {self.bootstrap_room} timed out after {elapsed:.1f}s \" \"in KVPoll.Bootstrapping\" ) sent_status, _ = self._finalize_failure(reason) return sent_status return status ``` Mori uses an inline implementation because its Sender currently owns backend-specific terminalization. In addition to updating the local room state, `_finalize_failure()`: - Records the Mori failure reason; - Sets `conclude_state`; - Uses `_notify_lock/status_notified` to emit the terminal status at most once; - Notifies Decode through the Mori control channel when destination information is already available. The common `_check_bootstrap_timeout()` helper only records a local failure and updates the Manager status. It does not understand Mori's remote notification or terminal-once state. Mori therefore implements the same deadline semantics while retaining its backend-local failure finalization. This PR only aligns NIXL with the bootstrap deadline already implemented by Mooncake and Mori. It does not change Mori's terminalization behavior. ## Changes This PR only changes `NixlKVSender`. ### 1. Record the bootstrap start time when creating the Sender ```python # python/sglang/srt/disaggregation/nixl/conn.py super().__init__( mgr, bootstrap_addr, bootstrap_room, dest_tp_ranks, pp_rank, req_has_disagg_prefill_dp_rank, ) self.init_time = time.time() ``` ### 2. Call the existing timeout helper while Bootstrapping ```python # python/sglang/srt/disaggregation/nixl/conn.py status = self.kv_mgr.check_status(self.bootstrap_room) if status == KVPoll.Bootstrapping: timeout_result = self._check_bootstrap_timeout() if timeout_result is not None: return timeout_result ``` The timeout check runs only when `status == KVPoll.Bootstrapping`. Once enough Decode metadata has arrived and the room transitions to `WaitingForInput`, this deadline no longer applies. ## Behavior After This Change Before: ```text Create NixlKVSender → request_status[room] = Bootstrapping → Decode metadata never arrives → poll() returns Bootstrapping indefinitely ``` After: ```text Create NixlKVSender → init_time = current time → request_status[room] = Bootstrapping → Decode metadata does not arrive before the deadline → _check_bootstrap_timeout() → record_failure(...) → request_status[room] = Failed → poll() returns Failed ``` The deadline continues to use the existing environment variable: ```text SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=300 ``` Users can continue to relax the deadline through the existing environment variable. This PR adds no new configuration. ## Testing To keep the implementation PR diff minimal, the CPU regression test is currently stored on a dedicated branch in the fork: ```text branch: https://github.com/jambow0320/sglang/tree/rfc-pd-test path: test/registered/unit/disaggregation/rfc-test/test_nixl_sender_bootstrap_timeout.py ``` Test scenario: ```text Sender creation time: 10s Current poll time: 20s bootstrap_timeout: 5s Decode metadata: missing ``` Assertions: - `sender.init_time == 10.0`; - `sender.poll() == KVPoll.Failed`; - `request_status[room] == KVPoll.Failed`; - The failure reason contains `timed out`. Test results: ```text Test from the dedicated test branch + source from this PR: 1 passed The same test + source before this fix: 1 failed Failure: sender.init_time is None ``` --- ### CI States Latest PR Test (Base): :x: [Run #31674263919](https://github.com/sgl-project/sglang/actions/runs/31674263919) Latest PR Test (Extra): :x: [Run #31674263615](https://github.com/sgl-project/sglang/actions/runs/31674263615) ",
"url": "https://github.com/sgl-project/sglang/pull/34686",
"createdAt": "2026-08-13T06:33:15Z",
"updatedAt": "2026-08-13T06:34:12Z",
"timestamp": "2026-08-13T06:34:12Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [],
"author": "jambow0320",
"state": "closed",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:f6af4742edccfea17459",
"signalId": "github:sgl-project/sglang:pull_request:34651",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34651",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[DCP] Share one pack kernel between both a2a backends, hoist fi_a2a send buffers",
"text": "## Motivation Follow-up to #34614, which fused the pack/unpack copies on the pynccl `a2a` path. That left the `fi_a2a` (FlashInfer MNNVL) path untouched, so it still paid four materializing copies per MLA layer per decode step plus two allocations and a zero-fill: ```python partial_o = out.view(B, N, H_pr, D).permute(0, 2, 1, 3).contiguous() # copy softmax_stats = torch.zeros(B, H_pr, N, 2, ...) # alloc + FillFunctor softmax_stats[..., 0] = lse_view # copy o_out, stats_out = decode_cp_a2a_alltoall(...) recv_o = o_out.permute(2, 0, 1, 3).contiguous() # copy recv_lse = stats_out[..., 0].permute(2, 0, 1).contiguous() # copy combined = dcp_lse_combine_triton(recv_o, recv_lse) ``` To be clear about provenance: #34614 is what created the asymmetry. Before it, both paths did ~4 copies + a combine; that PR improved one of them. This finishes the job rather than duplicating the fix. ## Modifications **One pack kernel for both transports.** `dcp_pack_a2a_send` now takes independently strided payload and LSE destinations instead of one interleaved buffer. The two transports want genuinely different layouts, for reasons rooted in each: - **pynccl** `all_to_all_single` chunks a flat buffer by byte offset, so the peer axis *must* be outermost. We interleave the LSE into the trailing fp32 word of each payload row to keep the exchange to a single collective. - **FlashInfer** wants `partial_o [B, H_pr, cp_size, D]` and `softmax_stats [B, H_pr, cp_size, S]` — peer axis inside the heads, payload and stats separate. Peer-axis placement is just a stride argument, so one kernel serves both and neither needs a layout copy. The receive side needs no work at all: `dcp_lse_combine_triton` already takes arbitrary strides, so `o_out` / `stats_out` are read in place. **Hoisted the `fi_a2a` send buffers out of the hot path.** The exchange does not write its inputs (`mutates_args=(\"workspace\",)`, verified empirically), so the send tensors are allocated once per `(h_per_rank, cp_size, head_dim, dtype)` and grown on demand. `softmax_stats` slots past the LSE stay zero after the one-time allocation, so the per-call `FillFunctor` disappears, and the addresses stay stable — a precondition if this path is ever CUDA-graph captured. Two allocations per layer per step also go away. ## Accuracy Tests **Verified against a torch oracle on 4x GB300 with real MNNVL**, all 4 ranks, both LSE bases, at `B=8 H_per_rank=12 D=512`: ``` FI_A2A_INIT_OK rank0..3 base_e=True max_err 7.8e-3 OK rank0..3 base_e=False max_err 7.8e-3 OK ``` That error is a bf16 payload against an fp32 reference — rounding, not drift. Batch sizes 4 / 16 / 8 / 16 to exercise the slice and grow paths: oracle match at every size, `softmax_stats` slot 1 verified still all-zero after each call including across the grow, and only 2 distinct buffer addresses over 4 calls (one growth, stable after). Unit tests: 24 passed + 5 subtests on aarch64, including a new case asserting the same kernel produces FlashInfer's split/peer-inside layout bit-for-bit against the permute+bitcast formulation it replaces. ## Speed Tests and Profiling Microbenchmark, 4x GB300, real MNNVL, `B=8 H_per_rank=12 D=512`: | | us/call | |---|---| | baseline | 219.0 | | + shared pack kernel (copies removed) | 171.3 | | + hoisted buffers | **167.8** | DeepSeek-V3.1, `cp_size=4`, 20 profiled decode steps, rank-0 trace: | run | total kernels | `direct_copy` | `FillFunctor` | pack | combine | |---|---|---|---|---|---| | baseline | 46342 | 4823 | 1203 | 0 | 1159 | | + shared pack kernel | 42865 | 187 | 1203 | 1159 | 1159 | | + hoisted buffers | **41706** | **187** | **44** | 1159 | 1159 | 4636 copies removed (exactly 4 x 1159, one call per attention layer per step) and 1159 fills removed; the 44 remaining fills are unrelated model code. The window between attention and the exchange: ``` before: attn -> copy 3.30 -> Fill 1.22 -> copy 1.63 -> a2a -> copy 2.91 after: attn -> PACK 1.57 -> a2a -> lse_combine ``` Prep before the exchange: **6.97 us -> 1.57 us**. Not claimed: the all-to-all kernel itself measures 6.78 / 4.70 / 6.59 us across the three runs. Same collective, same shape — that spread is run-to-run variance across separate server launches, as are the ITL differences. The kernel counts are the solid evidence; they are exact and reproducible. **No regression on the pynccl `a2a` path**, which #34614 already optimized. The generalization adds a second base-address computation and three stride arguments per program; both are amortized against a loop that moves `D/2` fp32 words, and for pynccl the two destinations are views into the same buffer, so the stores are byte-identical. CUDA-graph replay at the DCP8 shape, 3 runs each: | | us/replay (GPU) | |---|---| | `main` | 8.19, 8.19, 8.19 | | this PR | 8.19, 8.20, 8.19 | The hoist is the smaller half of the win (~3.5 us/call) and does not pay for itself on microseconds alone. The argument for it is removing 122 allocator calls per decode step from a fixed-shape path, plus address stability for graph capture; the fill removal is a bonus. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Follow-ups The `combine` kernel is now the only caller-side work left around the exchange. Folding it into the exchange itself needs the reduction to happen as peer data arrives, which only FlashInfer can do — filing that as a feature request upstream, along with a question about whether the fused kernel should consume a combined LSE or `(m, l)` pairs (the combined form is where the base-2 vs base-e ambiguity that caused #34240 comes from). `fi_a2a` requires MNNVL fabric, so CI cannot exercise this path; the evidence above is from a 4x GB300 devbox. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- ### CI States Latest PR Test (Base): :x: [Run #31674167839](https://github.com/sgl-project/sglang/actions/runs/31674167839) Latest PR Test (Extra): :x: [Run #31674167715](https://github.com/sgl-project/sglang/actions/runs/31674167715) ",
"url": "https://github.com/sgl-project/sglang/pull/34651",
"createdAt": "2026-08-13T02:13:41Z",
"updatedAt": "2026-08-13T06:32:36Z",
"timestamp": "2026-08-13T06:32:36Z",
"metrics": {
"reactions": 0,
"comments": 0
},
"labels": [
"jit-kernel"
],
"author": "kpham-sgl",
"state": "open",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:d888a69a82b818918dd2",
"signalId": "github:sgl-project/sglang:pull_request:34653",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:34653",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "Enable unified cache out-of-window slot freeing by default",
"text": "## Motivation `SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS` releases SWA slots that have already left the sliding window while a chunked prefill is being cached, instead of holding them until the request finishes. It has been opt-in since it landed; this makes it the default. **This is a default flip, so it changes behavior for every server running the unified radix cache with an SWA component.** The single consumer is the chunked-prefill caching path in `unified_radix_cache.py`, gated behind `SGLANG_ENABLE_UNIFIED_RADIX_TREE`; passing `False` restores the old behavior. The risk worth calling out is not the freeing itself but what it does to prefix reuse: how much SWA data stays resident decides how deep a later request can match, since a hit needs a full window of live SWA below the match point. Freeing earlier could in principle shorten that and silently cost cache hits. ## Accuracy `test_unified_radix_cache_kl_hybrid_bitexact.py` is the sharpest check available for this: it asserts prefill and decode score every token identically (`kl_div` floor 1e-9, measured exactly 0), and its hicache case additionally asserts the reused prefix length per request. One H200 (SM90, same architecture as the `1-gpu-large` CI runner), both fixes in, flag defaulted on: ``` SELFCHECK flag_effective True test_logprobs_match ... avg_kl_div=0.0 ok test_prefill_cache_hit ... avg_kl_div=0.0 ok test_decode_cache_hit ... avg_kl_div=0.0 ok test_multiturn_decode_cache_hit_branching ... avg_kl_div=0.0 ok Ran 4 tests in 427.750s OK ``` The per-request `cached_tokens` assertion in the multi-turn case passes unchanged, so the reusable prefix does not shrink. Two things this does **not** cover, and I would rather say so than imply otherwise: only the hybrid SWA + mamba configuration was measured, not the SWA-only or mamba-only models in the same directory, and the reason it originally shipped opt-in is not recorded anywhere I could find (the introducing commit carries no rationale). Running the full `unified_radix_tree/` suite on this PR would close the first gap. --- ### CI States Latest PR Test (Base): :white_check_mark: [Run #31661339455](https://github.com/sgl-project/sglang/actions/runs/31661339455) Latest PR Test (Extra): :white_check_mark: [Run #31661339489](https://github.com/sgl-project/sglang/actions/runs/31661339489) ",
"url": "https://github.com/sgl-project/sglang/pull/34653",
"createdAt": "2026-08-13T02:33:37Z",
"updatedAt": "2026-08-13T06:31:44Z",
"timestamp": "2026-08-13T06:31:44Z",
"metrics": {
"reactions": 0,
"comments": 13
},
"labels": [
"run-ci",
"bypass-fastfail",
"run-ci-extra"
],
"author": "ispobock",
"state": "closed",
"assignees": [],
"change": "updated"
}
},
{
"id": "event:483e34c9cf72d54e6507",
"signalId": "github:sgl-project/sglang:pull_request:33048",
"event": "changed",
"observedAt": "2026-08-13T13:48:00.446149Z",
"changedFields": [],
"signal": {
"id": "github:sgl-project/sglang:pull_request:33048",
"source": "github",
"group": "ai-infrastructure",
"project": "sgl-project/sglang",
"kind": "pull_request",
"title": "[Bugfix] Hold references to fire-and-forget tasks in disaggregation",
"text": "## Motivation Five `asyncio.create_task()` calls in `srt/disaggregation` throw the returned task away. The event loop only keeps a **weak** reference to a running task, so one whose sole reference was the `create_task()` expression can be garbage collected before it finishes — [the asyncio docs call this out explicitly](https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task): > Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn't referenced elsewhere may get garbage collected at any time, even before it's done. `MMEncoder` in `encode_server.py` already handles this correctly — `self.background_tasks` at line 346, used at 1060, 2022 and 2069. This PR applies that same pattern to the classes in the package that miss it. | site | task | what a collection costs | |---|---|---| | `encode_server.py:2929` `DPDispatcher.start` | `_result_listener` | dispatcher stops consuming worker results | | `encode_server.py:2930` | `_worker_watchdog` | dead ranks are never detected | | `encode_server.py:2931` | `_cleanup_stale_mappings` | stale mooncake mappings accumulate | | `common/conn.py:1721` `CommonKVBootstrapServer._run_server` | `_cleanup_expired_entries` | `room_to_dp_rank` grows without bound | | `encode_receiver.py:1668` `MMReceiverBase` | the encode dispatch | the encode never happens; the paired `_recv_mm_data` just times out | The first four are process-lifetime loops, which is the worst case for this bug — nothing else ever holds them. `encode_receiver.py` deserves a note: the code creates the encode task and then awaits `_recv_mm_data` on a socket, so the task is awaited *indirectly*, never through the task object. A local binding would only cover the happy path — if `wait_for` raises `TimeoutError` the frame unwinds while the encode may still be in flight — so this uses the same instance-level set as the rest. ## Modifications - `DPDispatcher`: add `self.background_tasks`, register the three loops from `start()`. - `CommonKVBootstrapServer`: add `self._background_tasks`, register the cleanup loop. - `MMReceiverBase`: add `self._background_tasks`, register the encode dispatch. Each uses `set.discard` in the done callback so a double-callback can't raise. No control flow, scheduling order or awaited behaviour changes — the tasks were fire-and-forget before and remain so; they simply can no longer be collected while still running. ## Accuracy Tests Not applicable — no kernel or model forward code is touched. ## Speed Tests and Profiling Not applicable. The added work is one `set.add` per task and one `discard` on completion. ## Checklist - [x] Format your code according to the pre-commit config — run with the pinned versions: ```console $ isort --check-only