551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
## Motivation ROCm EAGLE verification currently routes every HIP request through greedy `argmax`, even when `temperature`, `top_p`, or `top_k` request stochastic sampling. ROCm also lacks the CUDA target-only tree-sampling kernel and its top-k/top-p renormalization operators. While validating a portable tree verifier with GLM-5.2-FP8, EAGLE top-k > 1 exposed
## Motivation Follow-up to #27926: DSV4 still emits FP32 activation scales, so DeepGEMM runs `transpose_and_pack_fp32_into_ue8m0` before every `fp8_einsum` (61 launches, ~114 µs per BS1 decode step). ## Modification Emit DeepGEMM's TMA-aligned packed UE8M0 `int32` scale layout directly from the DSV4 quant kernel. Quantized outputs and einsum numerics are unc
Implements #34295. `--torchao-config` has raised `ImportError` for every accepted value since the torchao pin moved to 0.17.0, so there is nothing working to deprecate. This removes the flag, `torchao_utils.py`, the layered-loading torchao branch, the manual tests, the docs, and the torchao dependency. <!-- pr-states:start --> --- ### CI States Latest PR Tes
The scheduler drops requests from the waiting queue in three places without recording anything: a full queue, a preemption by a higher-priority request, and a waiting timeout. None are observable. The tokenizer-side sglang:num_aborted_requests_total counter does not cover them, because its only increment site counts aborts the frontend initiates, and these t
Under `is_split_kv` the guard around the KV prologue is a runtime compare, so `n_block_first` and `page_idx` were first bound inside an scf.if region and became live-outs with no type on the skip path. CuTe DSL 4.6 rejects that with `TYPE_UNSTABLE_JOIN`; both are side-effect-free, so they move above the guard.
## Motivation [CVE-2026-15969](https://github.com/advisories/GHSA-359v-m36h-r94v) is an unauthenticated RCE via `/load_lora_adapter_from_tensors`: `SafeUnpickler.find_class()` used prefix allowlists + a deny-list, so reflective chains (`builtins.__import__` + `getattr`, `operator.attrgetter` + `pickletools.sys`, `sglang.srt.utils.common.dynamic_import`, `io_
## Motivation LingBot's realtime causal attention has only ever had one way to split work across ranks — Ulysses — and the causal path explicitly raises `NotImplementedError` for `ring_degree > 1`. That reads like an unfinished feature. It isn't. Ulysses is the right default here, for a structural reason, and this PR makes that claim reproducible rather than
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Expose `flashinfer_cute_dsl` as fp8 gemm backend since it achieves better perf for muse
## Motivation LTX-2 and LTX-2.3 vocoder inference can fail when running with BF16 precision. The current implementation explicitly casts the vocoder input to FP32: ```python waveform = self.vocoder(hidden_states.float()) ``` When the vocoder weights and bias are loaded in `bfloat16`, the FP32 input causes a dtype mismatch during convolution, resulting in the
## Motivation Two failures hit a hybrid-SSM (Nemotron-H) deployment running DP attention over a concurrency sweep (`--tp 4 --ep 4 --dp 4 --enable-dp-attention`, one server per concurrency level). **1. `--max-running-requests` smaller than `attn_dp_size` gives every worker zero slots.** The per-DP-worker share `max_running_requests // attn_dp_size` floors to
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Reenable breakable CUDA graph for NemotronH(revert #33130). The issue has been fixed on
[by Codex] ## Summary - remove the stochastic negative-penalty integration assertion based on whole-completion vocabulary diversity - add deterministic unit coverage showing that negative frequency penalties increase a repeated token's logit - add deterministic unit coverage showing that negative presence penalties increase a present token's logit once ## Wh
## Motivation Running Nemotron-H with DP attention enabled together with a CUDA-graph runner backend (breakable CUDA graph / torch.compile piecewise) crashes at graph replay. ## Modifications `python/sglang/srt/models/nemotron_h.py`: in the `is_dp_attention_enabled()` branch of `NemotronHMambaDecoderLayer.forward`, route the mixer through the same split-op d
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation With out-of-window SWA slot freeing now enabled by default in [#34653](https://github.c
## Motivation Follow-up to #34240. That PR removed two no-op launches from the DCP MLA verify path; profiling the same window showed the rest of it is the a2a buffer plumbing — four elementwise copies per MLA layer, per decode step, all on the critical path between the attention epilogue and the NCCL all-to-all: | kernel | what it moves | |---|---| | `direct
## Summary Fuse ERNIE-Image Q/K RMSNorm with its full-width rotate-half RoPE cache. The generic JIT QKNorm+RoPE kernel gains an explicit full-width NeoX cache mode. It remains orthogonal to the Cosmos pack-KV path: full-width caches are rejected for pack-KV, while existing cache layouts retain their behavior. ERNIE hoists a concatenated full-width cos/sin ca
## Motivation On RTX Pro 6000 with Qwen3.5 speculative NEXTN serving, using the model from [Qwen/Qwen3.5-35B-A3B-GPTQ-Int4](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4), I launched the server with: ```bash SGLANG_MAMBA_CONV_DTYPE=float16 python3 -m sglang.launch_server \ --model-path ~/autodl-tmp/models/Qwen3.5-35B-A3B-GPTQ-Int4/ \ --host 0.0.0.0 \
## Motivation The B300 test suite should use the standard CUDA CI dependency installation path instead of maintaining a Kimi-K3-specific wrapper. ## Modifications - Point the `8-gpu-b300` runner config to `scripts/ci/cuda/ci_install_dependency.sh`. - Remove the unused `kimi_k3_install` YAML anchor. - Delete `scripts/ci/cuda/ci_install_kimi_k3.sh`. ## Validat
Reverts sgl-project/sglang#33623 Breaks CI https://github.com/sgl-project/sglang/actions/runs/31650215286/job/94292793773#step:15:2133 and seems like a real regression given full 2048 tokens <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31653682855](https://github.com/sgl-project/sglang/actions/runs/31
## Motivation Builds on #33991 (its four commits are included here and should be rebased away once it lands); only the last three commits are new. #33991 makes MiniMax-H3 drop the `adaln_proj` weights — 39.3% of the DiT, 13.0B parameters / 24.2 GiB — by precomputing their outputs into a sidecar file. That unlocks a large win: with those weights gone the DiT
## Motivation MiniMax-H3 is the only joint video+audio model served here, and it has no nightly coverage — regressions in it stay invisible until someone runs it by hand. ## Harness change it needs first `_build_sglang_payload` forwards a fixed whitelist, and H3 does not fit it in either direction. It derives its shape from `target` and **rejects** an explic
## Motivation The numbers I put in these docstrings when the file landed (#34607) were all measured on B200, but CI runs `1-gpu-large`, which is SM90. Prefill and decode take different fa4 kernels on the two architectures, and the corrupted state surfaces in a different case on each, so the docstrings describe something a reader will not reproduce on the run
Reverts sgl-project/sglang#33661 ## Why #33661 adds `is_in_breakable_cuda_graph()` to `fallback_to_flashinfer_impl` and relaxes the prefill runner's MHA-companion restriction from an arch allowlist to `not is_cuda()`. Together these pin the absorbed MLA path inside capture, and prefill loses the path it wants: - BCG capture cannot take the MHA companion, so
## Motivation `UnifiedRadixTreeTestMixin` bundles an MMLU case that no consumer trusts. Of the seven files using the mixin, two skip it in CI (one of them with the reason "mmlu eval not stable enough"), four lowered the threshold to 0.4 or 0.7, and the remaining two sit on the 0.8 default. One of those two just went red on it: ``` AssertionError: 0.796875 no
## Motivation `test/registered/rl/test_fp32_lm_head.py` fails deterministically on the AMD `stage-b-test-1-gpu-small-amd` shard (`linux-mi300-1gpu-sglang`, partition 7): ``` RuntimeError: gemm input type at::BFloat16 and output type float is not supported for ROCm ``` The FP32 LM head fast path added in #32370 selects `torch.mm(..., out_dtype=torch.float32)`
## Motivation Part of #34510 Step 1. Related to #17845. `CommonKVSender` already implements the Prefill bootstrap timeout. Mooncake starts the deadline when its sender is created and checks it while the request is bootstrapping. NIXL does neither, so a request can remain in `KVPoll.Bootstrapping` indefinitely if Decode destination metadata never arrives. ##
The swapAB path (weight on gemm-M, tokens on gemm-N) was only used for `m <= 64`. It stays faster up to `m = 256`, so the old crossover sent m in [96, 256] to a slower kernel. Before: - `m <= 64` -> swapAB 128x32x128 - `m > 64` -> non-swapAB 128x128x128 After: - `m <= 128` -> swapAB 128x32x128 - `128 < m <= 256` -> swapAB 128x64x128 - `m > 256` -> non-swapAB
## Motivation SGLang-Diffusion currently exposes overlapping CPU-offload and layerwise-offload flags with different scopes. In particular, an explicit `--dit-layerwise-offload false` can still inherit automatic DiT component offload, even though callers use that spelling to request a resident DiT. The DiT/VAE booleans also implicitly control model-specific h
## Summary - declare `backend: gpu` in Kimi-K3's regular-serving deferred preprocessing metadata - add regression coverage for the producer contract and the resulting multimodal item metadata ## Root cause The regular tokenizer-side deferred path stores CHW `uint8` images for owner-side GPU preprocessing, but its metadata omitted `backend`. `KimiK3ForConditi
## Motivation `test/registered/unit/server_args/test_resolution_is_reproducible.py` is failing on the GPU registrations (seen on `base-b-test-1-gpu-small`, `1-gpu-5090`): ``` FAIL: test_a_resolution_does_not_leak_into_the_next (intermediate='multimodal') File "test/registered/unit/server_args/test_resolution_is_reproducible.py", line 343 self.assertEqual(aft
## Motivation The `test_disaggregation_dsv4` PD test (and other PD tests sharing the disaggregation fixture) intermittently fail at server startup with: ``` torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 35061, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use ... File
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation Support CORSMiddleware settings like allow_methods, allow_headers and others so users can lau
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation Sometimes we need to set proxy to download model from huggingface, like `export https_proxy=h
## Motivation Currently running RedHatAI/Pixtral-Large-Instruct-2411-hf-FP8-dynamic fails with the following exception: ```[2025-06-19 23:46:36 TP1] Scheduler hit an exception: Traceback (most recent call last): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2527, in run_scheduler_process scheduler = Scheduler(server_args, port_ar
## Motivation Fix issue #3876 ## Modifications Replace deprecated cafile parameter with SSL context in urllib.request.urlopen ## Checklist - [x] Format your code according to the [Code Formatting with Pre-Commit](https://docs.sglang.ai/references/contribution_guide.html#code-formatting-with-pre-commit). - [ ] Add unit tests as outlined in the [Running Unit T
Add `other` field to ChatCompletionRequest to preserve unknown parameters when routing to workers. Updates request adapter to merge custom fields and adds test coverage for parameter preservation.
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> ## Modifications <!
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> Support BurstGPT fo
## Motivation Addresses [sglang#18427](https://github.com/sgl-project/sglang/issues/18427) — remove duplicated documentation between `sglang/docs` and `sgl-cookbook`. A companion PR to `sgl-cookbook` (see [https://github.com/sgl-project/sgl-cookbook/pull/141]) merges all model-specific content from `sglang/docs` into the cookbook. **This PR handles the `sgla
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation The existing MoE expert distribution recorder on "per-token" mode introduces too much o
## Motivation The `cache_aware_zmq` policy prices workers by a router-side in-flight counter. That counter reflects what this router dispatched, not what the engine has queued: it cannot see traffic from other router replicas or from direct clients, and for streaming responses it stays held for the full response rather than for the time the request occupies
## Motivation `mla_kv_pack_quantize_fp8` is a purely memory-bound op (fused `cat(k_nope, broadcast k_pe)` + FP8 quantize for K, FP8 quantize for V). The `_v1_flat_kernel` path spent a meaningful fraction of its time on per-element address arithmetic (`// num_heads`, `% num_heads`, two-term strided offsets) rather than HBM traffic, and the `_pick_kernel` heur
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Exact token ids behind a
## Summary MXFP4 KV cache for DeepSeek V4 on Hopper (SM90/H20), end to end: codec + memory pool + fused decode attention kernel + backend integration. The decode kernel is a JIT-compiled port of the FlashMLA three-stage split-KV design (scheduler metadata kernel + persistent WGMMA main kernel + combine kernel), covering all 44 layers (SWA + C4/C128 + attn_si
Automated sync of the LMSYS SGLang blog cards in `docs_new/index.mdx`. 🤖 Generated with GitHub Actions <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:white_check_mark: [Run #30594717790](https://github.com/sgl-project/sglang/actions/runs/30594717790)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-
## Motivation compressed-tensors FP8 W8A8 quantized models (e.g. RedHatAI's `*-FP8-dynamic` family — `Apertus-8B-Instruct-2509-FP8-dynamic`, `Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic`, `granite-4.0-h-small-FP8-dynamic`, `NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic`) currently fail to serve on XPU. The very first failure happens at layer-construction time,
## Motivation MiMo-V2.5 declares its audio tokenizer encoder as causal and uses hybrid attention: causal sliding-window attention `(128, 0)` alternating with causal full attention. After the TP-aware `VisionAttention` refactor in [#31343](https://github.com/sgl-project/sglang/pull/31343), `AudioEncoderAttention` continued storing `causal` and `window_size`,
## Motivation This PR supersedes #32662 with a smaller, dependency-free milestone for the KV Indexer proposed in #31458. The Indexer is an experimental metadata-only service that tracks which SGLang worker may reuse each content-addressed KV-cache block. KV data remains owned by the workers. The end-to-end path is: ```text SGLang worker └─ ZMQ component-awar
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Add npu unit test for dsv4 <!-- Describe the purpose and goals of this pull request. --