551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
Draft for Miles Video processing <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31641949255](https://github.com/sgl-project/sglang/actions/runs/31641949255)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->:x: [Run #31641948993](https://github.com/sgl-project/sglang/acti
## Motivation With a reasoning parser configured, `usage.reasoning_tokens` counts from the first output token until the end-of-thinking delimiter appears — it never checks whether a thinking block was actually opened. Any request where the model answers without thinking gets its output billed as reasoning while `reasoning_content` is empty. Easiest reproduct
## Motivation `usage.reasoning_tokens` counts from the first output token until the end-of-thinking delimiter, without checking that a thinking block was ever opened. ```bash curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "inkling-small", "messages": [{"role": "user", "content": "What is 1+3? Answer with ju
### Motivation DSpark compact ragged target-verify CUDA graphs crash on the first replay of a mid-size token tier. Reported in #34384: with graph sizes `[1, 32, 192]` and verify width 6 on 4×H20 (TP4), the first `bs=32` target replay faults with `CUDA error: an illegal memory access` (`dspark_verify.py:404` → `full_cuda_graph_backend.py:150`). The mechanism:
Nightly on this branch: https://github.com/sgl-project/sglang/actions/runs/31648935157 ## Restore `ModelEvalMetrics` Dropped in #34523 as unreferenced, but `test/registered/eval/test_vlms_mmmu_eval.py` imports it — `nightly-test-2-gpu-large` fails at import on main ([run](https://github.com/sgl-project/sglang/actions/runs/31647986897/job/94285941166)). <!--
## Motivation `--load-format remote_instance` and `--model-loader-extra-config` cannot currently be combined. `RemoteInstanceModelLoader.__init__` rejects any non-empty `model_loader_extra_config`: ``` File "sglang/srt/model_loader/loader.py", line 3085, in __init__ raise ValueError( ValueError: Model loader extra config is not supported for load format Load
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Motivation The ROCm 7.2 DeepSeek-V4 nightly benchmarks report throughput but do not assert it, so a performance regression passes CI. This PR only adds gates to those existing benchmarks. GPT-OSS and Kimi-K3 coverage is separate in [#34645](https://github.com/sgl-project/sglang/pull/34645). ## Modifications Add direct `
## Motivation On ROCm/AMD, `torch.cuda.mem_get_info()` (hipMemGetInfo) can report **more free GPU memory than physically exists**, causing KV-cache pool sizing to over-allocate and OOM. Root cause is in the amdgpu/KFD driver memory accounting: registering the **same** GPU buffer as an RDMA memory region (`ibv_reg_mr`, PeerDirect / GPUDirect) across **multipl
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Motivation Stop scheduling Grok-1/Grok-2 on MI30x to free runner capacity. Keep all MI35x Grok coverage—including both accuracy and perf steps—completely unchanged. Keep every test file unchanged for manual runs and future re-enablement. ## Modifications Change only the two workflows that directly define Grok jobs: - `.
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Motivation For the reported ROCm 7.2 + Triton 3.7 regressions, DeepSeek-V4 is the only affected model with a performance benchmark today. GPT-OSS and Kimi-K3 have no throughput coverage. This PR adds those missing benchmarks independently of [#34640](https://github.com/sgl-project/sglang/pull/34640), which now contains
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This PR introduces an initial runtime adapter for [Attention on Heads (AoH)](https://ar
Enable 12-head MLA aiter fp8 Gluon decode (batched `bh16bn128`) on gfx950 for Kimi-K3 TP8 (12 local heads). **aiter runtime dependencies** (container/image, not in this repo): - [ROCm/aiter#4480](https://github.com/ROCm/aiter/pull/4480) (required): fp8 Gluon `bh16bn128`, batch>1, 64-bit KV base. - [ROCm/aiter#4555](https://github.com/ROCm/aiter/pull/4555) (r
## Motivation `python/sglang/srt/layers/dcp/` has no CODEOWNERS entry today, so DCP changes fall through to the broad `/python/sglang/srt/layers` owners. @thanhhao98 authored the consolidation that created the directory (#29365) and the A2A + FlashInfer-MNNVL comm backends / q-replicate Helix support (#21637); @kpham-sgl, @YAMY1234 and @Fridge003 are co-main
## Motivation Grant CI permissions to @thanhhao98 so they can trigger and rerun CI on their own PRs. They are an active contributor to DCP and speculative decoding — #21637, #29365, #29218, #31468 — and are proposed as the DCP module codeowner in #34648. ## Modifications Add one entry to `.github/CI_PERMISSIONS.json`, matching the standard `custom override`
## Motivation Follow-up to #34614, which fused the pack/unpack copies on the pynccl `a2a` path. That left the `fi_a2a` (FlashInfer MNNVL) path untouched, so it still paid four materializing copies per MLA layer per decode step plus two allocations and a zero-fill: ```python partial_o = out.view(B, N, H_pr, D).permute(0, 2, 1, 3).contiguous() # copy softmax_s
## Problem Runs are published as `diffusion-comparisons/<date>_<run_id>.json`, so any consumer that wants the history must first enumerate the directory. From a browser the only way to do that is the contents API, and its anonymous budget is **60/hour shared per egress IP** — viewers behind a shared proxy get permanent 403s. The practical consequence is that
## Summary - Add a Decode Context Parallelism page under Advanced Features covering MLA KV striping, LSE merge, communication backends, and compositions with DPA, DSpark, PD, and HiCache L2. - Register the page in the docs sidebar and overview, and document `--dcp-size`, `--dcp-comm-backend`, and `--dcp-replicate-q-proj` in server arguments. ## Test plan - [
## Motivation #34582 reports a 100% reproducible two-node startup hang in the first DP-attention MLP-sync `all_gather_into_tensor`. The reported stack identifies the collective call, but not its backend. Under the reported overlap-scheduler configuration, the source selects `tp_group.cpu_group` unless `SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH`
Follow-up docs for #33829: converts the dots.note.omni serving recipe into the standard cookbook MDX format. <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:white_check_mark: [Run #31705004596](https://github.com/sgl-project/sglang/actions/runs/31705004596)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- sl
## Motivation Fixes #34631. Channel-framed reasoning formats such as Muse Glimmer emit an answer-channel header after the reasoning end marker. Guided decoding currently resumes as soon as `<|eom|>` is accepted, so the JSON grammar constrains the model-written `<|start|>assistant to=user<|message|>` header instead of starting at the answer body. ## Modificat
Code style refactor for rust tokenizer manager multimodal-relevant code. <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31674783703](https://github.com/sgl-project/sglang/actions/runs/31674783703)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->:x: [Run #31674783584](ht
## Motivation Covers the part of #34604 that #34609 leaves open. That PR fixes the `string indices must be integers` class (106 of ~190 daily failures); this one covers the other two: - 84x `Tool call parsing error: unexpected character: line 1 column 1 (char 0)` - 1x `Tool call parsing error: Input is a zero-length, empty document` ## Root cause The report
## Summary - add LingBot Video MoE and SANA-Video to the public diffusion cookbook, navigation, README, and compatibility matrix - remove the retired LTX `snapshot` alias and the unused diffusion benchmark `--backend` flag, then refresh affected examples - make nightly diffusion results record the exact checked-out commit - reduce redundant fallback logs for
## Motivation #34582 reports a 100% reproducible two-node startup hang in the first DP-attention MLP-sync `all_gather_into_tensor`. The reported stack identifies the collective call, but not its backend. Under the reported overlap-scheduler configuration, the source selects `tp_group.cpu_group` unless `SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH`
## Motivation #34582 reports a 100% reproducible two-node startup hang in the first DP-attention MLP-sync `all_gather_into_tensor`. The reported stack identifies the collective call, but not its backend. Under the reported overlap-scheduler configuration, the source selects `tp_group.cpu_group` unless `SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH`
Targets #34585 (`qwen38`). Ports the single `flashinfer_backend.py` hunk from #33997 (Bump FlashInfer to 0.6.17): FlashInfer 0.6.17's `plan()` takes an extra trailing `uniform_q_len` argument, so `fast_prefill_plan` has to pass it or the positional arg list no longer matches. Nothing else from #33997 is included. <!-- pr-states:start --> --- ### CI States La
## Modifications Add the following IDs into `CI_PERMISSIONS.json` file. @MeowMiaoJ and @siju-samuel : Intel xpu dev. @nzr-niu and @xinguozhu-2026 : CPU model enabling, env & doc maintenance. Also corrected @yuchengliu1 since the ID was wrong in initial submission. ## Checklist - [X] Format your code according to the [Format code with pre-commit](https://docs
## Motivation Enable the Ascend MegaMoE fused dispatch/GEMM/combine path for ModelSlim W4A8 Kimi-K3. The existing NPU FuseEP modes cover the W8A8 paths, while the validated W4A8 deployment needs a separate MegaMoE weight ABI and operator backend. ## Modifications - Add `--fuseep-mode 3` and keep the deprecated environment-variable compatibility in sync. - Ro
## Motivation Restores KTransformers CPU-expert offload for DeepSeek-V4 on the Ascend NPU backend. This lets a DeepSeek-V4-Flash MoE model serve with GPU-resident dense/attention layers and CPU-resident (KTransformers-managed) expert weights on a single Ascend 910B card, using a W8A8-int8 checkpoint (the officially released fp8 checkpoint's compute path is h
## Motivation `NPUCompressedTensorsW8A8Int8DynamicMoE.apply_weights` reads `layer.w13_weight_offset` / `layer.w2_weight_offset` via direct attribute access, but `create_weights` never registers these parameters for symmetric (offset-free) int8 checkpoints. Any symmetric W8A8-int8 MoE checkpoint on this NPU quantization path hits a deterministic `AttributeErr
## Motivation Fixes #34677. When a parallel tool call batch contains a name that was not offered, the streaming state machine discards or corrupts the valid calls around it, while the non-streaming path in the same class handles the identical output correctly. `parse_base_json` skips just the bad entry and continues. The streaming branch does this instead: `
## Motivation A single `POST /v1/chat/completions` whose `regex` begins with a NUL byte kills the whole server. xgrammar's regex converter appends its own NUL terminator and indexes past it, raising **SIGSEGV** rather than an exception — and a signal is below the interpreter, so the `except RuntimeError` in `dispatch_regex` cannot catch it. The scheduler die
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This PR adds a new sglang-native CUTLASS **MXFP4A8 (W4A8)** grouped-GEMM MoE backend fo
## Motivation Fixes the startup failure half of #34683: on SM100, launching `google/gemma-4-12B-it` with `--enable-deterministic-inference` and no explicit `--attention-backend` fails at argument resolution with ``` ValueError: Currently only ['ascend', 'fa3', 'fa4', 'flashinfer', 'triton'] attention backends are supported for deterministic inference, but yo
## Motivation MiMo emits XML-like tool calls, but its streaming parser currently buffers the entire `<tool_call>... </tool_call>` block before returning a tool call. As a result, OpenAI-compatible clients receive long function arguments only after generation finishes instead of as streaming deltas. ## Modifications - Replace full-block buffering with a state
- RFC: [PD disaggregation: single protocol layer, per-backend transport #33861](https://github.com/sgl-project/sglang/issues/33861) - Staged implementation plan and PR tracking: [PD shared-protocol implementation plan #34510](https://github.com/sgl-project/sglang/issues/34510) ## Background RFC #33861 proposes gradually consolidating the duplicated PD reques
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> #14636 changed how deepst
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Every real image request to Kimi-K3 crashes the scheduler with `KeyError: 'backend'`, i
## Motivation `dsv3_router_gemm` is a single-purpose kernel: it only accepts exactly 256 or 384 experts with a hidden dim that is a multiple of 1024, and it is capped at 16 tokens. The tiny GEMM added for Kimi-K3 solves the same problem — a skinny `x[m, k] @ w[n, k].T` with a handful of rows — for a strictly larger set of shapes. Keeping both means two kerne
## Motivation When the DiT FP8 attention path is enabled (`SGLANG_DIFFUSION_AITER_FP8_ATTN=1`), the AITER attention backend quantizes q/k/v with `aiter.per_tensor_quant`. That function is a PyTorch reference implementation: it upcasts the activation to fp32 and then runs `abs` / `max` / `div` / `cast` as four more eager kernels. For the Wan2.2 self-attention
## Description Enable OpenAI-compatible logprob responses when serving with DSpark speculative decoding. ### Changes - Allow DSpark requests with `return_logprob` to pass scheduler admission validation. - Compute and propagate accepted-token logprobs and top logprobs through the DSpark verify/accept path. - Re-enable the DSpark logprob coverage in the regist
## Motivation `CommonKVManager` provides a decode-side heartbeat checker that fails rooms associated with an unavailable prefill node. Mooncake and NIXL start this checker during decode initialization, but Mori only starts its decode message thread. As a result, Mori decode workers do not run the shared prefill health checks. This addresses the "Mori Decode
## Overview Subtask 1 of the [auto_tuner roadmap](#13363): unified CLI to auto-tune MoE kernel configs. ## Usage \\n ## What it tunes **Triton fused MoE kernel** - 6 params, 1920 configs: | Parameter | Values | |-----------|--------| | BLOCK_SIZE_M | 16, 32, 64, 128, 256 | | BLOCK_SIZE_N | 32, 64, 128, 256 | | BLOCK_SIZE_K | 64, 128, 256 | | GROUP_SIZE_M | 1
## Motivation `input_ids` currently ride the (tm to) scheduler ring as a raw int64 column: widened into the frame on the Rust side, memcpy'd into a `PyBytes` at `recv_requests` (GIL-held, on the scheduler loop), then copied again into `array("q")`. This PR moves `input_ids` to the same transport the MM feature buffers already use: the ring carries only the m
## Problem `StandardDispatcher` decides whether to remap `topk_ids` into the rank-local expert range from `get_moe_runner_backend()`. Its own comment states the invariant correctly — only the AITER runner keeps global expert IDs — but the *configured* backend is not necessarily the backend of the runner this layer got. A quantization method may build whichev
## Problem `python/sglang/jit_kernel/csrc/lora/moe_lora_align_kernel.cu` does not compile on ROCm, so MoE LoRA cannot run there: ``` error: use of undeclared identifier 'cudaDevAttrMaxSharedMemoryPerBlockOptin' ninja: build stopped: subcommand failed ``` The AOT build hipifies its sources, so they can spell CUDA names directly. The JIT path hands `.cu` files
## Motivation The model gateway injects the DP rank selected by DP-aware routing into the request body. The OpenAI Responses API currently drops these fields while parsing `ResponsesRequest`, so the selected DP rank does not reach the scheduler. This makes `/v1/responses` inconsistent with the existing Completions and Chat Completions paths addressed by #146
# Add distributed exact prompt Top-N logprobs ## Summary Extend the distributed prompt-logprob path to `top_logprobs_num > 0` without gathering the full prompt vocabulary logits. Before this PR: ```text TP-local logits [rows, vocab / TP] | +-- full-vocab AllGather --> [rows, vocab] | +-- global Top-N +-- normalize selected values ``` With this PR: ```text TP
## Problem `stage-c-test-large-8-gpu-amd-mi35x-rocm720` shard 0 fails on `test_qwen35_fp8_ar_fusion_mi35x.py`. The surfaced error is misleading: ``` AssertionError: GSM8K benchmark failed: ... File "benchmark/gsm8k/bench_sglang.py", line 131, in main preds.append(get_answer_value(states[i]["answer"])) KeyError: 'answer' ``` That is a *symptom*. What actually