551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
## Motivation Following PR #10567 ("Opt fused triton moe: add tma for down proj kernel"), which added TMA support for the down-projection (second MoE GEMM), this PR extends TMA support to the up-projection (first MoE GEMM, a.k.a. gate_up). Runtime couples up/down TMA via c_sorted = down_moe_use_tma. Since c_sorted is a tl.constexpr, different values produce
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Cosmos3 Omni model has 2 transformer towers stacked on top of each other for video gene
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation This PR adds a test case to verify that the `--dllm-fdfo` (First-Done-First-Out) schedu
## Motivation This PR integrates exported FlashInfer CAKE KDA kernels into SGLang and adds an explicit `cake` backend for recurrent KDA prefill and decode: ```bash --mamba-ssm-dtype bfloat16 --linear-attn-backend cake ``` The shared flag selects CAKE for both phases. Existing per-phase flags remain authoritative when a mixed configuration is requested, and t
## Summary This PR adds MiniMax-M3 DSpark speculative decoding support with MiniMax-M3 as the target model and MiniMax-M3-DSpark as the draft model. MiniMax-M3-DSpark is a draft model and cannot be served standalone. When it is paired with MiniMax-M3, several integration points are required across auxiliary hidden-state capture, VL wrapper forwarding, target
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation The chunked-prefill scripted-runtime suite under `test/manual/chunked_prefill/` was wri
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Refactor PD related code and embed the pd lb to decode worker. ## Modifications <!-- De
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Merge dots.note.omni model ## Modifications This PR completes the SGLang integration of
## Motivation DSpark currently assumes the autoregressive draft-block convention: a gamma-row block emits gamma drafts and the verify window is gamma + 1 rows. DFlash-style mask-filling draft heads use a different block layout — slot 0 carries the anchor token and slots 1..gamma-1 are the predictions — so a gamma-row block emits gamma-1 real drafts and the f
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Fixes #32750. The existing PD + DSpark path works without Pipeline Parallelism because
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det
<html><head></head><body><h2><span>Motivation</span></h2><p><span>This PR is a follow-up to </span><a href="https://github.com/sgl-project/sglang/pull/30658"><span>#30658</span></a><span>.</span></p><p><span>PR #30658 defers host-pool free-list concatenation for </span><code><span>HostKVCache</span></code><span>-based pools. However, </span><code><span>Logic
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation EAGLE with `topk=1` only needs the token with the largest logit. On the existing ROCm p
## Motivation Tail-Optimized LRU (T-LRU, [NeurIPS 25](https://arxiv.org/abs/2510.15152)) keeps only as much of a conversation cached as its next prefill needs to stay under a TTFT SLO budget, and frees the rest of the tail before falling back to recency order, so newer requests can be admitted. On agentic traces the tail is what a plain LRU protects at the e
## Motivation On SM120/SM121 `server_args.py` disables a block of features with the rationale *"SM120 lacks tcgen05/TMEM: disable features that depend on DeepGEMM"*. For `wo_a` (the first stage of DeepSeek-V4's o-projection) the consequence is that the FP8 weights already present in the checkpoint are **dequantised to bf16 at load** and the projection then r
## Motivation On sm120/sm121 the SM120 block in `server_args.py` sets `SGLANG_OPT_USE_TILELANG_MHC_PRE=False`, so DeepSeek-V4's `hc_pre` falls through to `hc_pre_torch_impl` — an fp32 `F.linear` of shape `[M, 16384] × [16384, 24]`. cuBLAS serves that with `cutlass_80_simt_sgemm`, i.e. **plain CUDA cores, no tensor cores at all**. On 2× DGX Spark (GB10 / sm_1
## Framing PR #27408 introduced SGLang's native sampling-mask response and its serving/disaggregation plumbing. This PR is standalone, general serving hardening for that shared sampling primitive. It is not RL-specific, and it does not duplicate endpoint exposure or top-p-only admission work. When those request paths are present, they use the same bounded pr
## Motivation A decode batch is processed for token-ids logprobs whenever **any** request in it asks for them. Requests that did not ask contribute a `None` entry to `token_ids_logprobs_list`, and `get_token_ids_logprobs_raw` turns that entry into a bare `[]`: ```python # python/sglang/srt/layers/logprob_processor.py if stage == LogprobStage.DECODE: for i, t
## Motivation Two use-after-free seams in the overlap scheduler. Both are silent, in-range corruption: the caching allocator hands a freed block to a same-stream reallocation while the **forward stream's** already-enqueued kernels still read it, so the server keeps running and emits wrong tokens. Found while running speculative-decoding workloads on a downst
## Motivation Under overlap scheduling, the FutureMap relay is written from two streams into the same pool-indexed rows: - the in-flight forward's **tail** relay writes on the forward stream — `on_publish` / the non-spec publish and `_relay_forward_payload` -> `stash` all execute inside `forward_stream_ctx`; - **batch-prep staging** on the schedule stream, w
## Motivation `MultiLayerEagleWorkerV2.forward_batch_generation` routes on ```python if batch.forward_mode.is_extend() or batch.is_extend_in_batch: ``` `is_extend_in_batch` is the **global** dp-attention stamp (`layers/dp_attention.py`), so a rank whose **local** batch is not an extend can be diverted into the target-prefill branch while the scheduler nulled
Server boot on a cold cache is mostly download, not load: `test_glm_46` has been seen at 818s wall while sglang reported 129s of actual weight loading. That puts network variance inside every per-test launch timeout — the same file ran 461s / 557s / 697s across three nightly runs. Fetching up front (20 min, advisory, never fails the job) moves that into a st
## Summary - standardize diffusion model pages with shared capability tags and concise capability, selection, and boundary guidance - simplify the MiniMax-H3 command picker to topology, checkpoint, and request-mode choices while documenting attention, quantization, caching, and encoder scheduling as independent feature overlays - add a reusable diffusion aut
`--attention-backend` is one field of three: a launch that sets only `--prefill-attention-backend` or `--decode-attention-backend` leaves the base field at `None`. Seven decisions read that base field alone and therefore answered from a field the operator never set. `attention_backends()` is the pair with the base-field fallback already applied, so each site
A callee that takes `server_args` keeps the supplied-instance contract, so no ratchet counts its reads — and that is right for the *object*. What it does not cover is what the object will carry once `ServerArgs` holds the user's raw input: a callee reading a field **resolution fills in** starts seeing the CLI default instead of the effective value. Measured,
## Problem `_local_jit_source_hash` decided whether a cached `.so` was still valid by hand-rolling a preprocessor out of two regexes. It follows `"..."` includes only when they sit next to their own file, and `<...>` only under `sgl_kernel/`. Measured on this tree: | | | |---|---| | Files the walker reaches for a CUTLASS kernel | **8** | | Files the compiler
## Motivation Addresses the prefill-replacement stall in #33789. When a prefill is replaced behind a bootstrap address that stays reachable (typically a k8s Service), the decode never notices: its cached rank endpoints still point at the dead process, so every request after the swap dies on the waiting timeout until the decode is restarted. Reproduced on 2x
## Motivation ROCm JIT compilation currently drops `-use_fast_math`, so ordinary `expf` in the DeepSeek V4 C4 and C128 compression kernels keeps full FP32 range reduction and overflow/underflow handling. These softmax exponentials are always max-shifted and therefore non-positive, making the ROCm fast device exponential appropriate without exposing positive-
# Motivation Qwen3.6-27B (and other GDN hybrid checkpoints with `num_v_heads / num_k_heads == 3`, e.g. 48 v-heads / 16 k-heads at head dim 128) currently misses two GDN fast paths, and single-token dense decode on Hopper leaves measurable DRAM bandwidth on the table: 1. **The fused qkvzba split/cat kernel is gated to ratios `[1, 2, 4]`.** The gate exists bec
# [quantization] Add tuned Triton tile configs for channelwise FP8 GEMM on L40S ## Summary The per-token / per-channel FP8 linear path in `apply_fp8_linear` dispatches to CUTLASS `fp8_scaled_mm` whenever the weight shape is CUTLASS-compatible, and to the Triton `scaled_mm_kernel` (with a coarse M-based tile heuristic) otherwise. On **L40S / SM89** an offline
## Motivation #12903 enabled LoRA with NGRAM speculative decoding. This extends it to **EAGLE / NEXTN / EAGLE3, DFLASH and DSPARK with multiple adapters co-batched**, one of the LoRA items in #11762. Adapters apply to the target model only; one shared draft runs unadapted. Speculation stays lossless per adapter (verify samples from the adapted target), so on
## Motivation Reimplementation of #33043 on `main`, as that PR instructed. It targeted the `kimi-k3` branch and was closed with: > **This PR needs to be reimplemented on main after Kimi K3 is merged into main.** … the disaggregation transfer paths it touches differ between the two branches. Original author: @yhyang201. The design below is theirs; this PR por
close Issue #34211 ## Motivation Fixes startup failure when serving ModelSlim-quantized Qwen3.5 NEXTN checkpoints on NPU, where the MTP (draft) module is stored unquantized (all `mtp.*` entries in `quant_model_description.json` are `FLOAT`) while the main model is `W8A8_DYNAMIC`. Additionally fixes a runtime crash on the first inference request due to Ascend
## Motivation Decode Context Parallelism (DCP) currently runs only on CUDA and AMD HIP. `initialize_model_parallel` rejects `--dcp-size > 1` on every other platform, so Intel XPU users cannot shard the KV cache by token position at all. This PR enables DCP on Intel XPU through the existing Triton attention backend. DCP shards the KV cache across TP ranks by
# Summary On Ascend 950 (A5), a mixed chunked-prefill batch currently goes through a single `npu_fused_infer_attention_score` (FIA) call that has to cover two very different shapes at once: long causal prefill queries and one-token decode queries. The kernel tiles for the worst case, so the decode part pays the prefill's cost. This PR splits that into two in
## Summary * On ROCm the DSA indexer runs `wk` and `weights_proj` as two separate GEMMs. Folding them into one `wk_weights_proj` GEMM needs no fused kernel, only the concatenated weight, but it was gated on CUDA. Split `use_dsa_indexer_fusion` into `use_dsa_indexer_fusion` (still CUDA-only: the `fused_q_indexer_rope_first_quant` / `fused_k_indexer_norm_rope_
## Summary - add strict SHA-256 media snapshots and processor artifact identities - add a bounded, byte-accounted CPU LRU with per-key single-flight - expose `mm_content_hashes` and OpenAI `image_url.content_hash` without changing `mm_hashes` semantics - add model-auto cache sizing (Kimi-K3: 256 MiB total, `0` disables) and an explicit trusted-hash gate - cl
## Summary - Check paged KV capacity before launching the extend/decode Triton allocator kernels. - Preserve the existing release-page merge behavior before the capacity check. - Add regression tests covering OOM, successful allocation, and release-page merging. ## Motivation The old implementation launched the allocation kernel and only then checked whether
# Compute exact input logprobs from TP-sharded vocab logits without full-vocabulary TP gather ## Summary This change adds an opt-in-safe distributed path for prompt/input logprobs. Each tensor-parallel rank keeps its vocabulary-sharded logits and contributes: - one row maximum; - one rescaled exponential sum; - selected scores for requested target and explic
## Motivation #34364 handed the ROCm 7.2 image's Triton to AITER's `install_triton.sh`, so an unversioned helper script in another repo decides what the image ships. AITER does not claim that role — its `requirements.txt` says torch and triton are *"provided by the base image ... their provenance is governed by the base image, not this file"*, and the only t
## Summary Stacked on #34398. - adds a model-independent per-media artifact coordinator for strict identity, option-aware keys, trusted/untrusted hashes, partial hits, batching, single-flight, cancellation, flush generations, and CPU-only retention - reduces Kimi-K3 to a typed adapter: the wrapped processor owns one `KimiK3PreprocessConfig`, K3 implements on
## Motivation KV-aware routing depends on an accurate view of the KV blocks available on each worker replica. Previously, sgl-router built this view only from live `BlockStored` and `BlockRemoved` events. If the Router started after the Worker cache had already been populated, or if the Router restarted and lost its in-memory index, the existing KV placement
## Summary Stacked on #34404; review `fd40b11ac0cb0..63cf4f894c57c` for the PR-3-only diff. - acquires per-image embedding-cache leases on the request's final DP route before processor dispatch, and omits features only when every required TP rank has pinned the same entries - transfers lease ownership to the admitted request; consume, cancel, abort, flush, t
## Motivation Add disaggregated inference (DI) and MTP (EAGLE speculative decoding) deployment commands for Qwen3.5-FP8 on AMD MI355X to the cookbook page, and bump the MI355X docker image to v0.5.16. This aligns with InferenceX PR [SemiAnalysisAI/InferenceX#2557](https://github.com/SemiAnalysisAI/InferenceX/pull/2557) which adds the corresponding benchmark
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det
## Motivation Reported in #34000 (Bug 2). `PipelineExecutor.__init__` seeds `component_residency_manager = None`, and every `_execute_stages` run enters `_component_residency_request` → `begin_component_residency_request`, which dereferences it unguarded. Of the three forward entry points in `ComposedPipelineBase`, two install the manager and one does not: |
## Motivation `Qwen3_5GatedDeltaNet.forward` only takes the fused Triton split/reshape/cat path when the head-group ratio `num_v_heads // num_k_heads` is 1, 2 or 4. Layouts with a ratio of 8 fall back to the eager sequence (`fix_query_key_value_ordering` -> `.contiguous()` -> `torch.cat`), which costs four separate data-movement kernels per GDN layer. Measur
## Motivation On ROCm, running the Wan VAE decode with spatial-shard parallel decode (`--num-gpus 2 --enable-cfg-parallel`) crashes as soon as the decoding stage starts: ``` RuntimeError: The size of tensor a (39) must match the size of tensor b (43) at non-singleton dimension 3 # under torch.compile the same failure surfaces during tracing: # Attempting to
add_common_sglang_args_and_parse hardcoded choices=["auto", "cuda", "rocm", "cpu"], so --device xpu was rejected even though ServerArgs accepts it. It also offered "rocm", which DeviceConfig always refuses: PyTorch drives AMD GPUs through the "cuda" device type. This change as part of #24922 ## Motivation Any test or benchmark using the shared CLI parser fai