551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Fixes [#33632](https://github.com/sgl-project/sglang/issues/33632). On SM120, ModelOpt
Let every pod in a multi-node LWS deployment expose the same Rust HTTP port range while keeping DP rank identity global. This gives the load balancer one shared port configuration for the whole group. Pods with `node_rank > 0` skip the dummy health server because their local dp-rank-0 rust listener now owns the base port. tested on: - DP omitted, so dp_size=
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This patch is 1/N of the original https://github.com/sgl-project/sglang/pull/32796 to e
## Summary Default Kimi-K3 + DSPARK (pure-TP, SM100/SM103) decode attention backend from `trtllm_mla` to `cutedsl_mla`. Applies only when the user has not explicitly set an attention backend; explicit `--attention-backend` keeps priority. Non-DSPARK default is unchanged (`trtllm_mla`), DCP path already defaults to `cutedsl_mla`. ## Motivation For MTP/spec ve
## Motivation Depends on https://github.com/sgl-project/sglang/pull/34140. Until that PR merges, this draft includes its prerequisite commit; GitHub will drop the duplicated diff after merge. ROCm stochastic tree verification calls top-p renormalization over 154,880 probabilities per row. The existing path uses general-purpose `torch.topk(..., 32)`, includin
## Motivation DeepSeek-V4's `sqrtsoftplus` gate has four routing implementations, and each was missing a different piece of the numerics its reference gate provides (`inference/model.py` `Gate.forward`, `F.softplus`). Same class as flashinfer-ai/flashinfer#3803. 1. **Triton router** (default, layers 3-42) computed `log(1.0 + exp(x))`, which in fp32 rounds to
## Problem Under wide expert-parallel disaggregated decode with DP attention + MTP speculative decoding, a decode rank can receive **zero requests** in a round while still being a non-idle (`target_verify`) rank. The existing `is_idle()` guards on the attention layers do not cover this "non-idle but empty" case: the GDN linear-attention and full-attention la
Makes `--language-only` mean one thing: serve the language half of a VLM, with no local vision tower. Where image features come from is decided by encoder registration (`--encoder-urls` or the `EncoderBootstrapServer`), not by this flag. - The vision tower is now actually skipped, not built-and-left-empty, across the VLMs that support it. - A multimodal requ
## Motivation AMD multimodal-gen jobs spend most of their runner time not testing. Measured on mi300 in [run 31443692177](https://github.com/sgl-project/sglang/actions/runs/31443692177), every job pays a container pull of 27–97 min plus a dependency install of 23–98 min, so 100–155 min of fixed cost before a single test executes. Across the eight multimodal
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Motivation Every AMD CI job re-acquires the `rocm/sgl-dev` image from scratch, and the setup that precedes it re-fetches the whole git remote. On the [nightly CI dashboard](https://michaelzhang-ai.github.io/sglang-ci/nightly-ci/) this is the `Setup docker` step, and it dominates the wall clock of the AMD nightlies. Meas
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Exact token ids behind a
## Motivation The current Triton download method may cause dependency installation issues in some environments, especially when building on different architectures or using customized Python environments. This change aims to improve the reliability and flexibility of the Triton package download process. ## Modifications - Modified the Triton download method.
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Kimi-K3 routes 16 of 896
Here's the draft PR description for PR 34498 (direct-write). The label in your data says "fold_quant" but I've relabeled it direct_write to match this PR. Title: [ROCm] Direct-write a8w8 bmm output to eliminate o_proj transpose copy This PR removes a redundant memory copy in the per-channel FP8 attention output path on AMD gfx95 (validated on MI355X with Kim
This PR fuses the per-token FP8 activation quantization into the preceding RMSNorm for per-channel FP8 attention projections on AMD gfx95 (validated on MI355X with Kimi-K2.7-Code-MXFP4). By emitting a pre-quantized (fp8, scale) tuple straight from the norm, it removes the standalone per-token quant kernel before each attention projection, improving throughpu
# [JIT Kernel] Migrate moe_topk_softmax from AOT to JIT Test and benchmark on an **H100 80GB (sm_90), CUDA 13.0, torch 2.12.0+cu130**. **Size comparision** of the compiled files(.so) : Kernel | Variants | JIT| AOT 1-arch | AOT ÷ JIT -- | -- | -- | -- | -- moe_topk_softmax | 3 dtypes | 379.69 KiB | 777.65 KiB | 2.048× - JIT: one instantiated runtime variant.
## Wait longer for the driver to release GPU memory - Reproduced in two nightly runs: the class after `test_ring_2_5_1t` (1T params, ~122 GiB per GPU) fails `setUpClass` with `GPU 0 uses 122.22 GiB (no other compute processes)` — the server process had exited well inside the 30s window but the driver had not returned the memory yet - Raise `_GPU_IDLE_TIMEOUT
## Motivation On switch-free intra-node hosts — no NVLink, no multicast, every peer transfer crossing the CPU root complex — none of SGLang's custom all-reduce backends apply. `CustomAllreduce` and `QuickAllReduce` assume NVLink/multicast, `pymscclpp` assumes its own fabric, so on these machines **every per-layer reduction falls back to NCCL**. NCCL's ring i
## Summary - Add native SGLang model class for LLaVA-OneVision (`llava-hf/llava-onevision-qwen2-7b-ov-hf` and siblings), wiring HF's `SiglipVisionModel` and `LlavaOnevisionMultiModalProjector` to SGLang's native `Qwen2ForCausalLM` and routing forward through `general_mm_embed_routine`. - Add companion multimodal processor that bypasses SGLang's classic `proc
## Motivation Under layerwise DiT offload, `--dit-layerwise-resident-layers N` pins the contiguous prefix `0..N-1` and streams the tail. Every streamed layer therefore sits next to another streamed layer, so the entire weight stream is crammed into the last `(n-N)/n` of the denoising step while the copy engines idle through the head. That concentration costs
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This PR is split from [#33957](https://github.com/sgl-project/sglang/pull/33957) and ke
## Motivation On ROCm with `SGLANG_USE_1STAGE_ALLREDUCE=1`, `should_custom_ar()` accepts an input of any size, and `_all_reduce_impl()` routes anything that does not fit the staging buffer to `self.register_buffer(inp)` + `deterministic_all_reduce_reg()`. That registration happens on every call and is never released: `CustomAllreduce::register_buffer()` does
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Add SGLang support for the **Mamba family of pure state-space models**, none of which c
## Summary Intel SYCL runtime's persistent kernel cache mishandles torch 2.13 XPU `aten.topk` on the pinned Intel graphics stack (compute-runtime 26.05 / IGC 2.28). Reloading the cached kernel segfaults inside `libsycl`, crashing `test_biased_grouped_topk` with SIGSEGV. Setting `SYCL_CACHE_PERSISTENT=0` in the CI container avoids the broken reload path. ## R
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Qwen3.5-style conditional-generation models keep language-model attributes in the neste
## Summary Runs MiniMax-H3 text-to-video-and-audio on one 24GB GPU and cuts denoise wall clock by **1.34× with exact attention**, up to **2.48×** with opt-in approximate attention backends. 1344×768 · 107 frames · 20 NFE · same process: warmup (seed 0) → timed (seed 42), timed only: | Config | timed e2e | Denoise | vs BF16 | PSNR vs BF16 | | --- | --- | ---
## Motivation `wan-ti2v` (Wan2.2-TI2V-5B, 704p/121f) was the largest remaining eager-vs-compile gap in the H100/H200 benchmark: eager 56.76 s vs compile 51.61 s denoise on H100 (−9.1%). A 5-step trace (99.5% GPU busy) shows why: the TI2V **per-token** adaLN path burns ~14% of every step in avoidable memory traffic — - `(scale_shift_table.unsqueeze(0) + temb.
## Summary - add opt-in Pi0.5 prompt token buckets so nearby prompt lengths reuse the same prefix and action CUDA graph signatures - replace unbounded action graph storage with a capacity-bounded LRU, and give prefix graphs bounded/LRU behavior without causing default recapture churn - make masked Prefix/Action CUDA graph capture safe, including attention ma
# Motivation `GDNKernelDispatcher` derives its verify kernel purely from whether the decode or prefill backend selected FlashInfer, silently overriding an explicitly configured `--linear-attn-verify-backend`. The server logs end up contradicting themselves: ``` Linear attention kernel backend: decode=triton, prefill=flashinfer, verify=triton GDN kernel dispa
## Summary - add `MergedReplicatedLinear` and pack Pi0.5 non-TP Q/K/V and gate/up projections - hoist invariant denoise layout work and reuse the sinusoidal scaling vector - make masked prefix/action CUDA graphs capturable and replay-safe - bound both graph caches; exact mode falls back to eager at capacity, while opt-in prompt buckets use bounded LRU evicti
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Under `--enable-unified-m
## Motivation DeepEP/MoRI dispatch outputs carry `topk_ids`/`topk_weights` directly and do not expose `.topk_output`. `Mxfp4MoEMethod.apply` reads `dispatch_output.topk_output` unconditionally before branching, so combining an a2a backend with the mxfp4 MoE method raises `AttributeError` — even though the runner branches that actually execute in that configu
## Motivation With DP attention enabled, `chunked_prefill_size` is divided by `dp_size` during argument normalization (`server_args.py`, the "DP attention is enabled. chunked prefill size is adjusted" path). The MoE however runs **after** the DP gather and sees the sum of all DP ranks' tokens again. The MoRI/pplx buffer-capacity validations compare only the
## Motivation Partially addresses #34604 (the `string indices must be integers` class, 106 of ~190 daily parser failures we see in production with Kimi-K3). With `tool_choice=required`/named, the json_schema constraint asks the model for an **array** of `{name, parameters}` objects — but models frequently emit a **single object** instead. `orjson.loads` then
## Motivation Grant CI permissions to a new contributor so they can label and rerun their own runs instead of waiting on a maintainer. Same shape as the other entries: label, rerun-failed, rerun-stage, no cooldown. <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:white_check_mark: [Run #31624945168](https://github
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation ```cosmos3_guardrails.py``` currently hardcodes ```cuda``` for idle device and the ```o
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> > **Stacked on #34602.** The first 5 commits are that PR; this PR's own change is > the last 10 commi
## Summary Make diffusion `performance_mode=auto` apply residency decisions per component instead of letting one explicit memory option freeze the whole residency pass. This matters for commands that explicitly disable DiT layerwise offload. On a high-memory GPU such as B300, that explicit choice should not preserve an unrelated implicit `dit_cpu_offload=Tru
## Summary Speed up FLUX.2 eager inference with two bit-exact fusions: - LayerNorm + modulation at the single- and double-stream AdaLN sites - packed SwiGLU without materializing two sliced operands Both paths retain the original expressions for `torch.compile`. The eager fast paths use per-signature first-use verification and fall back safely when layout, d
## Summary Reduce HunyuanVideo eager overhead with three scoped paths: - a lossless Triton kernel that applies image RoPE while packing image/text Q, K and V into contiguous attention inputs - a `quality=high` strided QK RMSNorm path that avoids input contiguous copies - the existing quality-gated fused linear+GELU epilogue for the Hunyuan MLP The QKV/RoPE p
## Summary Enable breakable CUDA graph for Cosmos3 Nano and preserve request warmup for request-only dimensions such as frame count. The Cosmos-specific prompt padder keeps `max_text_seq_len` consistent with the padded text mask. Warmup captures the requested text buckets before serving and forwards the current request context through the graph runner. ## B3
## Summary Retune the fused Q/K head LayerNorm launch for B300 / SM103 from 16 rows per program to 32 rows per program, while leaving the independent SM120 and Hopper choices unchanged. The production GLM-Image shape is bit-exact under both launch configurations. The B300 sweep selected `ROWS=32, num_warps=1`. ## B300 result GLM-Image, 1024px, 30 steps, nati
## Motivation `RayEngine` names each `SchedulerActor`, but the name format is inlined in `_create_scheduler_actor`. A process outside the engine that wants handles to those schedulers — e.g. an RL trainer that supplied the placement group and wants to push weights straight into the scheduler actors — has no supported way to address them, and ends up scanning
## Motivation SGLang can have at most **one** request mid-prefill at a time. A long prompt therefore monopolizes the prefill batch for its entire prefill, and every queued request — however small, however long it has waited — waits it out. **Measured in production** (Sference fleet, SGLang backends): - **Kimi-K3** (`chunked_prefill_size=8192`): with 642K–912
## Motivation On DeepSeek-V4 decode a good part of each attention layer goes to kernels that only move or re-lay-out data rather than compute anything. Three of them run on every layer of every step: 1. On CSA layers (`compress_ratio` 4) compress, norm and rope are three separate launches. Each is 4-8 us, which is small enough that dispatch rather than the w
## Motivation This PR follows #34373 and fixes the Qwen3.5 MTP hang that remains in its symmetric-memory path. #34373 improves performance by moving host staging and the metadata exchange off the WAR-fenced schedule stream and onto a private stream. Symmetric memory itself is not the main end-to-end speedup. However, its GPU barrier is still a rank rendezvou
## Motivation `meta-models/Muse-Glimmer-30B-GGUF` has renamed its artifacts to a canonical scheme that encodes the quant type ([hub discussion #5](https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF/discussions/5/files)), and Meta asked downstream repos to conform: | Old | New | |---|---| | `muse-glimmer-30B-kquant-17gb.gguf` | `Muse-Glimmer-30B-KQuant-
## Motivation SGLang drops generated-token logprobs in the Python Engine API path when prompt/input logprobs are not requested. `convert_logprob_style()` currently returns when `input_token_logprobs_val` is `None`, even if valid output logprobs are present. Consequently, downstream consumers such as Dynamo receive no logprobs for output-only requests like `l
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Motivation `run_suite.py` splits the mm_gen `2-gpu` suite into `total_partitions - len(STANDALONE_FILES["2-gpu"])` parametrized partitions plus one partition per standalone file, and bails when that subtraction goes negative: ```python parametrized_partitions = args.total_partitions - len(standalone_files) if parametriz
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Motivation #34204 swapped the AMD PR gate to ROCm 7.2 and, in `pr-test-amd-extra.yml`, made the job display names suppress the suffix for `rocm720` and add `-rocm700` for ROCm 7.0. That inverts the convention both callers use for their own jobs: `pr-test-amd-rocm720.yml` suffixes every job `-rocm720`, and `pr-test-amd.y