551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
## Motivation For W4AFP8 MoE on the DeepEP normal-dispatch path, the dispatcher previously sent BF16 hidden states over the all-to-all and re-quantized them on the receiving side using the checkpoint's static activation scale. This doubles the communication volume (BF16 instead of FP8) and re-does a quantization that the weights were already calibrated again
## Motivation A hybrid SWA + mamba model throws away most of its decode-region prefix reuse at the default `--mamba-track-interval`. Measured on one GPU with 32 prompts, counting how many get a decode-region cache hit on their second turn: ``` page_size track_interval prompts reusing the decode region 128 256 16/32 -> 32/32 128 512 5/32 -> 32/32 64 256 9/32
Use an N32 tile for the gfx950 Lq576/Lv512 extend-attention specialization on Triton 3.7+ to eliminate register spilling and recover Kimi-K3 prefill performance, while preserving the existing Triton 3.6 configuration. <!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This PR adds Transfer capability to allow additional control inputs into the pipeline.
## Motivation SGLang now uses `sglang serve` as the common entry point for both autoregressive and diffusion runtimes. Ecosystem projects also build on SGLang, but publishing a second console script named `sglang` would make installation and uninstallation order decide which executable survives. This PR makes `sglang serve` an extensible frontend while keepi
Enable long context length <!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation When setting a large context length (e.g., for LLaMA 4), launching
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation Currently, SGLang always requesting huggingface hub for checking whether `hf_quant_config.jso
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation fix potential memory corruption when sorted_token_ids is not divisible by block size during f
## Motivation Fixes #17646 When a user configures top_k with a value exceeding int32 range (e.g., 2147483648), the server crashes with: RuntimeError: value cannot be converted to type int32 without overflow This happens because the top_ks tensor is created with dtype=torch.int32, and values beyond 2147483647 cause an overflow. While this is an outlier config
## Motivation Update qwen_vl's `MultimodalSpecialTokens` to support video&audio token's regex ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/deve
## Summary Fix an off-by-one bug in `PrefillAdder.add_req_state()` where the `insert_sort` path fails to append elements that are larger than all existing elements in `self.req_states`. ## Bug In `schedule_policy.py` lines 662-667, the sorted insertion logic uses a `for` loop to find the correct position: ```python # Before (buggy) i = 0 for i in range(len(s
## Motivation Bare `except:` clauses catch all exceptions including `SystemExit` and `KeyboardInterrupt`, which can mask critical errors and prevent clean process shutdown. This follows the same pattern as #22947. ## Modifications Replace two bare `except:` clauses in `model_runner.py` with specific exception types: 1. **EAGLE config access** (line 389): `ex
## Summary Fix an output-logprob cursor off-by-one when overlap scheduling retracts an in-flight request before its prefill result is processed. The fix is deliberately retraction-specific. Non-retracted prefill-only requests with `max_new_tokens=0` retain the existing behavior of returning their first logprob. ## Root cause Under overlap scheduling and retr
## What sglang-side support for **RDT (Ray Direct Transport / NIXL) weight sync** — lets the [miles](https://github.com/radixark/miles) trainer push RL weights to rollout engines via a zero-copy RDMA pull instead of NCCL broadcast. Paired with the miles-side PR (radixark/miles#1313), which has the perf write-up. ## Changes - **`ray/scheduler_actor.py`** — ad
Superseded by #31447. --- Targets `release/v0.5.12`. Enables Confidential Computing (CC) inference on B300 (validated with Qwen3.5-397B-A17B-FP8, TP4). **CC perf fixes** — auto-enabled when NVIDIA CC is detected (`is_confidential_compute()`), byte-identical off-CC: - **Async D2H copy worker** — run the per-step D→H token readback on a dedicated daemon thread
## Summary Fuse the deferred MoE finalize into the next layer's input-RMSNorm via flashinfer `kMoEFinalizeARResidualRMSNorm` (pattern 7): weighted expert combine + shared-expert add + all-reduce + residual + RMSNorm in one kernel, while keeping `residual_out` available for Eagle3 aux capture. Gated by `SGLANG_ENABLE_MOE_DEFERRED_FINALIZE`. Builds on #28343.
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Currently supports offline inference for CUDA devices with auto-round quantized models,
Enable the LoRA functionality on XPU (in addition to CUDA/ROCm), and enable the corresponding unit tests. Source changes: - backends (triton/chunked/torch): use torch.device(self.device) instead of a hard-coded "cuda". - lora_moe_runners: route XPU to the pure-torch _naive_moe_lora_align_block_size fallback. - rotary_embedding base.py / mrope.py: guard the X
This PR adds support for NGRAM on XPU target. JH...! <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #31673353486](https://github.com/sgl-project/sglang/actions/runs/31673353486)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->:x: [Run #31673353388](https://github.com/sgl
## Summary Part of [ROCm/sglang-ci#349](https://github.com/ROCm/sglang-ci/issues/349). Registers the **diffusion QKNorm+RoPE** kernel benchmark for AMD. `diffusion/bench_qknorm_rope.py` compared two providers: `split` (JIT QKNorm + **FlashInfer** RoPE, CUDA-only) and `fused` (SGL JIT). Guard the `split` provider behind `is_hip()` so ROCm benchmarks the fully
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> Optimize MiniMax-M2.7 on
# [AMD] DeepSeek-V4: fix shared-experts fusion for top-6 ## Summary Enabling shared-experts fusion (`--enforce-shared-experts-fusion`) for DeepSeek-V4 on MI355X (gfx950) crashed at startup. Two independent issues in the fused topk / append path assume DeepSeek-V3 conventions (fp32 correction bias, power-of-two topk) that DeepSeek-V4 (bf16 correction bias, **
> [!IMPORTANT] > **Depends on https://github.com/MoonshotAI/checkpoint-engine/pull/96** > (adds Intel XPU support to checkpoint-engine). That PR must merge first > `pyproject_xpu.toml` tracks checkpoint-engine's upstream `main`, which only > resolves to XPU-capable code once #96 lands. ## Motivation SGLang's checkpoint-engine integration enables fast in-plac
Co-authored with Ryan Lucas (MIT) and Kayhan Behdin (LinkedIn). ## Motivation Compressing reasoning models with standard pruning does more damage than compressing a conventional LLM, and it can make the model **slower**. One-shot pruning picks weights by minimizing a layer-wise reconstruction error against a calibration activation matrix `X`: ``` min_{W'} ||
## Issue Description Fixes #32378. ## Cause When running `python3 -m sglang.compile_deep_gemm`, the script starts a temporary SGLang server process using `multiprocessing.Process` to precompile DeepGEMM kernels and then terminates the server via `kill_process_tree(proc.pid)`. In Python 3.12, `multiprocessing.resource_tracker` tracks shared memory segments an
## Summary - replace token-bucket-based concurrency limiting with lifecycle-owned admission leases - enforce a hard per-Pod in-flight limit, bounded waiter count, and per-request absolute queue deadline - hold leases through the complete response body lifecycle, including streaming responses and client disconnects - keep local QPS rate limiting independent f
## Motivation Qwen3.6 hybrid models require both Full KV and the matching Mamba state when a HiCache prefix is reused on Ascend. PR #32275 introduced the correct Torch-based Ascend host-transfer fallback for these Mamba state sidecars. This PR preserves that fallback and extends the integration in two areas found during Qwen3.6 NPU validation: 1. FIA exposes
## Summary Add a HIP/aiter fused `mhc_post` + next-layer `mhc_pre` path for DeepSeek-V4 on gfx95, dispatched across the attention/MoE boundary alongside the existing TileLang and Triton fused paths. On MI355X (gfx950) this is a **+0.8%–1.8% output-throughput win across concurrency 4–64 with accuracy preserved**. ## What changed - **`python/sglang/srt/models/
## Motivation Fix transcription for decoder-only speech/audio models that were already present Adds /v1/audio/transcriptions support for three decoder-only speech/audio LMs, so they serve real ASR instead of falling back to the Whisper adapter: - Qwen/Qwen2-Audio-7B-Instruct - zai-org/GLM-ASR-Nano-2512 - ibm-granite/granite-speech-3.3-8b ## Modifications **N
This PR add support for XPU with DFLASH algo in speculative decoding. JH...! <!-- pr-states:start --> --- ### CI States Latest PR Test (Base): <!-- slot:pr-test:start -->:x: [Run #30608529938](https://github.com/sgl-project/sglang/actions/runs/30608529938)<!-- slot:pr-test:end --> Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->:x: [Run #30608529765
## Motivation `AITerImpl.forward` requests the softmax LSE and then discards it on the same line via `output, _ =`. Minor cleanup: ask for the output only. Ring attention is the only consumer of the LSE, and it is already restricted to the FA and SageAttention backends, so nothing reads this value today. ## Modifications `return_lse=True` -> `False` in the B
## Motivation On Qwen3.5 `fused_qkvzba_split_reshape_cat_contiguous_kernel` is a bit-exact identity copy. The checkpoint stores `in_proj_qkv` and `in_proj_z` separately, so `MergedColumnParallelLinear` already emits `[all_q | all_k | all_v | all_z]` and every load/store offset in the kernel maps to itself. Its four outputs are plain column slices: ```python
## Motivation In Qwen3.5 GDN layers the input projection is issued as two separate GEMMs, `in_proj_qkvz` and `in_proj_ba`. Both read the same activation, so on checkpoints that quantize both to the *same* scheme they can be served by one wider GEMM. Keeping them apart costs an extra kernel launch, an extra activation-quantization pass and an extra epilogue p
Reduce GDN prefill launches for benchmarked BF16/FP32 GPU shapes while preserving the original path on unsupported platforms, dtypes, layouts, and launch-bound wide shapes. <!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join
## Summary The OpenAI client docs already show pointing `base_url` at a local SGLang server. This PR adds a short tip that the same `base_url` pattern works with any OpenAI-compatible multi-model gateway when you are not self-hosting SGLang, using [DaoXE](https://daoxe.com) (`https://api.daoxe.com/v1`) as one concrete example. Docs only — no runtime behavior
_forward_metadata poisons padded request rows with state index -1, and the decode kernel already skips them (fused_recurrent: `if idx >= 0`). The chunked extend kernel had no such guard, so -1 reached `initial_state + index * stride_init_state` for both the initial-state load and the in-place final-state store, addressing before the state pool. boundary_chec
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Reduce scheduler overhead for HiCache with pipeline parallelism by batching write and l
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <img width="1196" height="233" alt="image" src="https://github.com/user-attachments/ass
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Restructure NPU CI output/log persistence so runs from different workflows are distingu
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Summary PR [#32022](https://github.com/sgl-project/sglang/pull/32022) previously fixed this issue
## Motivation Cache-DiT (DBCache) has been wired up for MiniMax-H3 for a while (`_CUSTOM_BLOCK_ADAPTER_SPECS` already maps `MiniMaxH3DiTModel`), but it never actually cached anything: enabling `SGLANG_CACHE_DIT_ENABLED=1` gave a 1.00x speedup and byte-identical output, with no error and no warning. ## Root cause `CachedBlocks_Pattern_3_4_5` retains the block
`--file-storage-path` is parsed into `server_args.file_storage_path` but nothing reads it. The `file` HiCache storage backend (`HiCacheFile`) only looks at the `SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR` env var and otherwise falls back to `/tmp/hicache`, so the flag is silently ignored and L3 lands in `/tmp` no matter what you pass. On a box where `/tmp` is t
When an evicted prefix is reloaded from the host KV tier, the transfer runs the per-token kernel (`transfer_kv_per_layer_mla`) at ~4.3 GB/s no matter how the bytes are laid out, because it pays a per-token indexing cost for every token. The host pool makes that worse: it hands out free slot ids first-fit and appends freed ids on free, so after a few fill/evi
Follows #32701, which added `free_segment` to the paged allocator so a per-request free needs no device sync. That migration deliberately left the hybrid SWA allocator on the legacy `free()` path, so SWA configurations still pay a sync on every free. This extends the same idea to the SWA side, and composes with the free-group batching from #33475 rather than
## Summary - keep Kimi-K3 CPU-transport image payloads as per-image uint8 tensors - defer resize, normalization, patchification, and H2D until DP load balancing selects the vision owner rank - activate the deferred path only for the resolved `cpu` transport; CUDA IPC and CUDA VMM continue to use their existing GPU transport paths - accept only PIL images or
## Motivation Qwen3.5 hybrid models use recurrent GDN layers alongside full-attention layers. Multi-item scoring already gives the full-attention layers an item-branching mask, but the GDN prefill path previously processed the packed suffix as one continuous recurrent sequence. As a result, later items could inherit convolution and SSM state from earlier ite
## Summary - add an explicit MiniMax H3 AdaLN sidecar-cache format and CUDA builder - skip original AdaLN checkpoint keys only when a matching cache is selected - reject timestep embeddings absent from the cache rather than changing the inference path - document the opt-in CLI and cookbook workflow ## Motivation The MiniMax H3 model card notes that its AdaLN
On gfx942 (CDNA3, MI325X) the DSA bf16 decode kernel overflows the 64 KB per-block LDS limit under dp-attention. `sparse_mla_fwd_decode_partial` sizes its shared buffers by heads-per-block. With dp-attention each rank runs all 64 heads, so `H_per_block = 64` and `Q_buf` alone is 64 x 512 x 2 B = 64 KB, which hits the LDS ceiling before the other shared buffe
The admission knobs we have today (`--max-running-requests`, `--max-queued-requests`) are count-based. They can't tell a 4k prompt from a 500k one. So when a burst of long-context requests lands, the server admits them by count, over-commits prefill memory, and OOM-crashes instead of shedding. On a GLM-5.2-FP8 tp8/dp8 run, 8 concurrent 500k-token prompts tak
WIP for #32950... :D ## Motivation the cute-dsl swap-AB/split-K kernels are a lot faster than the persistent cutlass one for dense mxfp8 gemms, so it should be selectable and the default on sm10x ## Modifications <!-- Detail the changes made in this pull request. --> adds an alias in the cli (/srt/server_args.py) for `--fp8-gemm-backend=flashinfer_cutedsl`,