551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
Align toml file <!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifi
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation https://github.com/sgl-project/sglang/pull/30797 follow up. GDN prefill currently rebui
## Motivation Two fixes so SGLang runs efficiently under **NVIDIA Confidential Computing (CC)**, where the GPU operates behind a bounce buffer: 1. **Per-step D2H result readback stalls the scheduler.** Under CC, `cudaMemcpyAsync` (D2H) is forced synchronous — it blocks at issue — so the per-step readback of `next_token_ids` / logprobs serializes onto the sch
## Motivation This PR addresses the long-prefill optimization proposed in #24644 and #24645. Previously, dLLM prefill and decode shared the same `block_size` (typically 32). As a result, a long prompt could advance by only one decode block per scheduler round, causing excessive scheduling overhead and underutilizing the GPU. This PR decouples the maximum pre
## Motivation Follow-up to #31468, several paths construct a fresh cuda `Event()` every decode step: the WAR `read_done` in `DecodeCudaGraphRunner.execute()` (twice per DFlash step: draft + verify runners), the same in `EagleDraftExtendCudaGraphRunner`, and the scheduler's `copy_done` (both the overlap and non-overlap spec paths). Each costs a `cudaEventCrea
update pytorch to 2.13 for xpu and update other relation. The oneAPI version should be 2026.0 after updated. ## Tests python3 run_suite.py --hw xpu --suite stage-a-test-1-gpu-xpu and python3 run_suite.py --hw xpu --suite stage-b-test-1-gpu-xpu <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31715088591](
Fixes #31766. ## Motivation A prefill worker caches one outbound ZeroMQ PUSH socket and one monitor PAIR socket for every decode endpoint it contacts. Endpoint churn can therefore grow the process FD count until libzmq raises `ZMQError: Too many open files`. In the reported failure, that exception escapes the Mooncake transfer loop as a fatal worker error an
## Motivation Enable XPU-specific optimized kernels for the DeepSeek V4 MHC (Multi-Head Compression) path on Intel XPU hardware. The existing CUDA-path kernels (deep_gemm, CUDA-first device selection) are either unavailable or suboptimal on XPU, causing incorrect device dispatch or missing performance opportunities. ## Modifications deepseek_v4.py: On XPU, d
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation When the model is deployed with pure DP and dp-attention, **--enable-dp-lm-head will su
Co-authored-by:@JackChuang ## Summary This PR is part of the roadmap tracked in https://github.com/sgl-project/sglang/issues/25746. This PR ports the Q8KV8 sparse MLA prefill path to the DeepSeek-V4 runtime backend and adds a runtime dispatch path via `--dsv4-prefill-backend flashmla_sparse_q8`. When the KV cache uses `fp8_e4m3`, DeepSeek-V4 can run the FP8
## Motivation KV-aware routers consume `BlockStored` / `BlockRemoved` to track which blocks live on which tier. On hybrid models the unified radix tree keeps several independent KV components on one node — full attention, sliding window, mamba state — and evicts and restores them independently. A `BlockStored` today only says "this block is on GPU". That is
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Add npu unit test for asc
[by Codex] ## Summary - accept both `torch.int32` and `torch.int64` router IDs in `PackTopkIds` - keep the existing in-kernel conversion to int32, avoiding a temporary cast allocation during CUDA-graph capture - add focused coverage for both input dtypes, multiple shapes, exact reference parity, CUDA-graph capture/replay, and invalid input This fixes a Qwen3
## Motivation **What the MoE activation costs today.** The triton fused-MoE path runs three steps per layer: an up-GEMM that writes `intermediate_cache1` at the full gate+up width `N`, a standalone `silu_and_mul` that reads that buffer and writes `intermediate_cache2` at width `N/2`, and a down-GEMM. The middle step is pure data movement dressed as compute —
## Motivation Ascend 950 (Atlas A5) is a new-generation NPU whose numeric capabilities differ substantially from the currently supported 910B/910C (Atlas A3) parts: it natively supports FP8-e4m3 KV cache, E8M0-scaled MXFP8 GEMM, and MXFP4 grouped matmul, and the corresponding sparse-attention / KV-compression kernels take different signatures and buffer layo
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Adaptation of Minimax M3 for NPU platforms ## Modifications 1、Define the complete atten
## Summary Follow-up to the dense-linear bpreshuffle scale no-copy. Several DeepSeek-V4 sites **pre-quantize** an activation and hand a `(fp8, scale)` tuple to a downstream Linear; those scales are emitted row-major and then relaid out with `materialize_bpreshuffle_fp8_scale` — a relayout **copy per site, per layer** on MI355X (gfx950). This PR eliminates th
## Motivation `fp8_scaled_mm` is the last large CUTLASS GEMM still in the AOT wheel — the blockwise FP8 GEMM already moved to `sglang.kernels.jit`. This brings the per-row/per-column one across too. While porting: SM120 ran a single `128x128x128` tile for every M, which wastes most of a 128-row tile on decode-shaped batches. It now buckets on M. ## Modificat
## Motivation DSpark block-diffusion speculative decoding was silently falling back to greedy decoding on AMD ROCm. The sampler kernels (`top_k_renorm_prob`, `top_k_top_p_sampling`, etc.) were only imported when `is_cuda() or is_musa()`, so on ROCm the import was skipped and DSpark produced greedy output with no error or warning. ## Modifications `python/sgl
## Motivation Classic EAGLE/NEXTN rejection sampling draws each draft proposal with `fast_sample()`. Although target sampling and verify-side rejection coins support request-local seeds, this proposal draw used the global PyTorch RNG (`exponential_()`), so `--enable-deterministic-inference` still changed outputs with batch shape and CUDA graph replay. This i
## Motivation Add support for Nemotron 3.5 including DFlash and DSpark spec-dec ## Modifications <!-- Detail the changes made in this pull request. --> ## Accuracy Tests <!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. --> ## Speed Tests and Profiling <!-- If this pull request
## Motivation The GLM-5.2 MXFP4 checkpoint keeps four attention projections in BF16: - fused `q_a + kv_a` - `q_b_proj` - `o_proj` This PR adds an opt-in FP8 path for those projections on gfx950. PTPC means per-token activation scaling and per-channel weight scaling. BF16 remains the default and fallback path. ## Modifications - Cache bpreshuffled FP8 project
Replaces #33521, which was automatically closed when its base branch was deleted. Rebasing to main as requested in the review. ## Motivation Kimi-K3 MLA computes QKV-A and the TP-local output gate from the same hidden states. They currently run as separate GEMMs. Fuse them to reduce projection cost while keeping the gate output TP-local. ## Modifications - M
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This is a follow-up to #32890, which introduced the TRT-LLM ragged attention zero-KV fi
## Motivation Fixes #34149. Related prior report: #34112. While testing chunked-prefill cancellation on latest `main`, we independently reproduced the same user-visible symptom reported in #34112: a cancelled request can expose one token and then finish with `abort`. The #34112 attachment used SGLang v0.5.9. Its negative batch-level output IDs were old Futur
## Motivation SGLang issue #20865 asks contributors to improve unit-test coverage for core runtime components that don't require a GPU, specifically calling out `test_reasoning_parser.py` as a starting point. The reasoning parser's streaming path (`parse_streaming_increment`) has several edge cases that were not previously covered: malformed/nested tags, Uni
## Motivation Every `multimodal-gen-test-2-gpu-amd` shard on both `pr-test-amd` and `pr-test-amd-rocm720` fails before running a single test (e.g. [run 31327057626](https://github.com/sgl-project/sglang/actions/runs/31327057626)): ``` Error: total_partitions (3) must be >= standalone files (7) ``` `run_suite.py` reserved one whole partition per standalone fi
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Set SGLANG_USE_SGL_XPU de
## Motivation Part of #20865 (Improve Unit Test Coverage). `srt/parser/reasoning_parser.py` had 76% unit-test coverage. Several code paths were untested: - `Apertus2509Detector.detect_and_parse` / `detect_and_parse_block_sequence` (only the `force_nonempty_content` swap was covered) - `Apertus2509Detector.parse_streaming_increment` (tool blocks inside reason
## Problem Encoders loaded with `--text-encoder-cpu-offload` go through FSDP2 `shard_model()`. Parameters that `_fsdp_shard_conditions` does not match land in the catch-all root group, and FSDP2 only unshards around the wrapped module's own `forward`. An encoder whose pipeline entry point is a custom method never triggers that hook, so those parameters stay
## Motivation During decode, MiniMax-M3 recomputes its **lightning indexer** (block-score + top-k block selection) on **every** sparse layer. On MI350x/TP4 that indexer + sparse-attn bucket is ~**3.5 ms of a ~16 ms decode step (~22%)** — one of the biggest single decode levers on this model. Because the selected blocks (`topk_idx`) are highly correlated betw
## Motivation The cache-aware-zmq router infers a worker's load from a router-side in-flight counter. That counter measures what *this router* dispatched — it misses traffic from other routers and direct clients, and for streaming responses it stays held for the whole response rather than the time the request occupies the scheduler. The engine already builds
- Snapshot `req.prefix_indices` on the cache-disabled path in `SWARadixCache` and `UnifiedRadixCache`, matching what `ChunkCache` and `MambaRadixCache` already do there - The row it aliased is mutable (the radix caches rewrite it during rematch) and is `int32`, while `prefix_indices` is `int64` everywhere else <!-- pr-states:start --> --- ### CI States Lates
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation When running the gpt-oss-120b-bf16 model, the service failed to start with the error: o
## Motivation This PR adds support for LongCat-AudioDiT, a state-of-the-art (SOTA) diffusion-based text-to-speech (TTS) model that directly operates on the waveform latent space. * Huggingface: https://huggingface.co/meituan-longcat/LongCat-AudioDiT-1B * GitHub: https://github.com/meituan-longcat/LongCat-AudioDiT ## Modifications ### Serve - http_server.py:
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Modify low errors ## Modifications <!-- Detail the changes made in this pull request. -
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation The hard-coded `mem_fraction_static *= 0.85` when attention_backend=aiter and context_l
## Motivation Load-aware routers currently infer a worker's load from a router-side in-flight counter. That counter measures what *this router* has dispatched, not what the engine actually has queued — it misses traffic from other routers and direct clients, and for streaming responses it stays held for the whole response rather than the time the request occ
## Motivation Fixes #22117. `natfii/Qwen3.6-27B-VLM-NVFP4-MTP` fails during TP=2 checkpoint loading with a shape assertion: ```text AssertionError: param_data.shape == loaded_weight.shape ``` This checkpoint uses ModelOpt FP4/NVFP4 quantization and contains packed weights and scale parameters in Qwen GDN and attention submodules. Qwen3.6 VLM reuses the Qwen3
## Summary - Add a block-split gather path for small Kimi K2.5 MLA batches and a tiled per-loc gather path for larger batches. - Tune the get-buffer dispatch for BF16 Kimi K2.5 rows (`nope=512`, `rope=64`) with PDL/GDC support where available. - Add registered correctness tests covering set/get behavior and the new dispatch boundary. ## Benchmark Local bench
Add intel_xpu to DETERMINISTIC_ATTENTION_BACKEND_CHOICES <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31669990908](https://github.com/sgl-project/sglang/actions/runs/31669990908)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->:x: [Run #31669990721](https://github.com
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation When running Qwen3.5-397B NVFP4 (https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4)
## Motivation <!-- Describe the purpose and goals of this pull request. --> Enable release/resume_memory_occupation on Intel XPU using the upstream torch_memory_saver package (Level Zero VMM backend). Replaces in-repo implementation with pip-installable upstream dependency. Features: - Pause/resume physical GPU memory while preserving virtual addresses - Tag
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation For DeepSeek-V4 models, DSV4PoolConfigurator sizes the c4 compressed-state pool using t
## Motivation This change prevents unbounded Prometheus cardinality growth caused by arbitrary unknown HTTP paths. One real-world example is vulnerability scanners, which can probe tens of thousands of unique random endpoints. When the raw request path is used as the endpoint metric label, every unique unknown path creates a new Prometheus time series for th
## Summary This PR adds GLM-Image distributed serving: the public head batches external AR generation and dispatches each AR-complete request to an independent batch-1 denoiser that runs request preparation, DiT, and VAE decoding. <img width="1466" height="1142" alt="GLM-Image heterogeneous topology" src="https://github.com/user-attachments/assets/b079ec92-4
## Summary Five independent, small fixes uncovered while enabling additional models. - **`utils/hf_transformers/common.py`**: `get_rope_config()` raised `KeyError: 'rope_theta'` whenever a model's `rope_parameters` dict existed but didn't contain a `rope_theta` key (e.g. `baidu/ERNIE-4.5-VL-28B-A3B-PT`, `allenai/Olmo-3-7B-Instruct`). Now falls back to `confi
## Motivation Long-context decode is memory-bandwidth-bound on the per-step KV read; verifying K drafted tokens amortizes that read across K+1 tokens, so the optimal K depends on ctx as well as BS. A batch-only palette has to pick one K per BS and either leaves ctx-amortization on the table for long sequences or over-drafts short ones. This is the SGLang sid
Makes **EAGLE / EAGLE3** work with decode context parallelism (`--dcp-size > 1`). Stacked on #21637. DFlash × DCP already worked (see #33912, `_dflash_draft_cell_size`); it gets one shared fix here, not new support. **Root cause:** the target KV pool is sharded across DCP ranks, but the draft pool is **replicated**. Draft-side code that reads the DCP topolog
## Motivation Inspired by SGLang's work on breaking graphs into piecewise CUDA graphs, we've built a small, standalone pure-Python package -- piecewise-cuda-graphs (https://github.com/meta-pytorch/piecewise-cuda-graphs). Our goal is to make it easily reusable across the PyTorch community, so that projects don't each have to maintain their own version. This P