551 signals · 636 observation events
SGLang is a high-performance serving framework for large language models and multimodal models.
v0.5.17Prereleases and milestones indicate public plans; they are not delivery commitments.
### Checklist - [x] I have searched related issues but cannot find a duplicate. - [x] The bug has not been fixed in the latest version. ### Describe the bug Hi @HaiShaw, On realistic Qwen3.5 397B FP4 AgentX workloads, MI355X (gfx950) is significantly behind the B200/B300 SGLang MTP frontiers at comparable P90 E2E normalized interactivity. <img width="1190" h
## Describe the bug Running Kimi-K3 with `--tool-call-parser kimi_k3` in production (agentic coding workload, Claude-Code-style tool loop), the tool call parser raises on a significant fraction of model outputs. Over a 24h window with moderate traffic we counted **~190 parser failures**: - 106x `Tool call parsing error: string indices must be integers` (Type
## Summary `flashinfer_trtllm` online NVFP4 MoE becomes numerically invalid after upgrading the aligned FlashInfer trio beyond `0.6.16rc4`. On Blackwell, the newly exposed tile-192 TRTLLM_GEN BMM path produces non-finite MoE output; the registered 200-example GSM8K test consequently scores `0.0`. This is a dependency/integration regression, not a feature-spe
## Summary For a reasoning format where the **model writes its own channel header**, `ReasonerGrammarBackend` re-arms the grammar at the token that follows `think_end_token` — but at that point the model still owes the header of the answer channel. Those header tokens get mask-constrained, so the model is forced to emit `{` in a position where it has never s
## Bug Description When DP-aware routing is enabled, the model gateway selects a DP rank and injects the rank into the serialized request body. For the OpenAI Responses API, `ResponsesRequest` does not declare the DP rank fields. Pydantic therefore ignores the injected fields during request parsing, and the selected rank is not forwarded to `GenerateReqInput
### Checklist - [x] I searched related issues but found no solution. - [x] The bug occurs on SGLang 0.5.16, build commit `fdebc938f7f4d16fe6b9f55dcd9a767cf0899ea1`. - [x] Environment information and a reproduction configuration are included below. - [x] This is a bug report, written in English. ### Describe the bug Under sustained Kimi-K3 traffic with hybrid
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
## Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version (reproduced at 983dfd6a9a1e). - [x] Environment info and a minimal reproducible demo are included below. - [x] Not a general question. - [x] English. ## Describe the bug Two related problems, found while investigating output reproducibility for `g
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [x] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [x] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
## Summary The scheduler process dies with ``` AttributeError: 'list' object has no attribute 'tolist' ``` whenever a request using `token_ids_logprob` shares a batch with a request that does not. One scoring client can take the whole server down. This affects the released versions **v0.5.14, v0.5.16, and v0.5.17** (verified by reading each tag's source), in
### Checklist - [x] I searched related issues but found no solution. - [x] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
## Checklist - [x] I searched related issues but found no solution. - [x] The bug persists on the latest `main` (`74c032234201`). - [x] Environment information and a deterministic reproduction are included below. - [x] This report is written in English. ## Describe the bug In heterogeneous-TP disaggregated serving with a staging buffer, the decode-side `Deco
### Checklist - [x] I searched related issues but found no solution. - [ ] The bug persists in the latest version. - [ ] Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback. - [ ] If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/d
### Checklist - [x] If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed. - [x] Please use English. Otherwise, it will be closed. ### Motivation The DeepSeek router (MoE gate) GEMM produces **fp32** logits on every code path *except* deterministic
Two small typo fixes: 1. **bench_one_batch.py**: "United Kindom" → "United Kingdom" 2. **bench_serving.py**: "Fail to load tokenizer config" → "Failed to load tokenizer config"
## Summary - Fix 7 instances of duplicate words across the codebase: - `for for` → `for` (3 instances) - `with with` → `with` (3 instances) - `from from` → `from` (1 instance) ## Test plan - [ ] Verify no functional changes, comments/docstrings only 🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary - Fix typo: `datastrucutre` → `datastructure` - Fix grammar: `consider simplify` → `consider simplifying` - Fix missing apostrophe: `dont` → `don't` (2 instances) - Fix grammar: `Avoid to divide zero` → `Avoid division by zero` (3 instances, matching 9+ existing uses in codebase) - Fix possessive: `it's own` → `its own` - Fix grammar: `allows to f
## Summary Replace SHA256 with xxhash64 for radix cache block hashing in `hash_page()` and `get_hash_str()`. xxhash64 is ~7.7x faster than SHA256 for non-cryptographic hash operations. The radix cache hashing is called on every KV cache store/remove operation, making this a meaningful hot-path optimization. ## Changes - `python/sglang/srt/mem_cache/radix_cac
## Summary - Coalesce compatible KV store and removal events at the cache-event queue tail before scheduler publication. - Preserve store-chain semantics and metadata barriers for medium, LoRA ID, block size, event kind, and clear events. - Update cache and manual event tests to consume multi-block payloads. ## Motivation SGLang emitted a separate KV event f
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Encoder Decoder models produces garbage output on the intel_xpu backend (page_size > 1)
## Motivation Each scheduler rank embeds its own `MooncakeDistributedStore` client, and `MooncakeStore.__init__ `passes all of them the same ssd_offload_path. Mooncake's SSD tier assumes one client per storage location, and a shared directory corrupts silently: - bucket ids are generated per process (`timestamp<<12` seed with a plain atomic increment, `moonc
## Motivation DeepSeek V4 Prefill context parallelism requires every CP rank to access the complete attention history. The ordinary layout keeps persistent KV replicated across CP ranks and limits long-context capacity. Following the Shared-KV direction introduced in #31435, this PR stores each attention page on one CP owner while exposing one logical addres
## Summary This PR adds self-contained LLaDA2.2-flash support for block-routing MoE and JointThresholdInDel (JTI) on Ascend NPU. Model-forward graph capture continues to use SGLang's existing NPU dLLM graph path; this PR adds graph capture only for the fixed-width JTI token-update step. It incorporates only the block-routing contract needed from #31768 and t
## Motivation Load-aware routing needs fresh, serving-stack-authoritative signals. Polling worker endpoints adds fan-out, creates inconsistent views, and can miss short-lived queue changes across data-parallel ranks. This PR adds the serving-side (Worker-process) Load Reporter only. It continuously pushes per-DP-rank load snapshots to external Routers over l
## Motivation Add an opt-in Helion backend (e.g. used for Inkling [silu_mul fusion](https://github.com/sgl-project/sglang/blob/main/python/sglang/kernels/ops/moe/inkling_moe.py)) for Kimi Delta Attention packed decode and prefill while preserving the existing default backends, public contracts, state mutations, and fallback behavior. The decode kernel delive
## Motivation MoRI EP prefill with `--enable-two-batch-overlap` on MI355X aborts with `HSA_STATUS_ERROR_OUT_OF_RESOURCES`. The traceback blames whichever kernel needed scratch at that moment (for us, aiter `per_1x32_mx_quant_hip`), which is a victim, not the cause. The MoRI EP TBO path calls `record_stream(comm_stream)` on every dispatch/combine tensor. That
## Motivation SGLang currently routes SM12x FA4 through an SM80-derived forward path. That path uses the older mainloop and does not expose the SM12x TMA dataflow, paged KV, SplitKV, or relative/shared-bias support. This PR adds an architecture-owned SM12x implementation and keeps its kernel, scheduler, and runtime policy under `sglang.kernels.ops.attention.
# [AMD] DeepSeek-V4: route decode wo_a bf16 batched matmul to aiter `batched_gemm_bf16` ## Summary On the DeepSeek-V4 ROCm decode path, the MLA output-absorb (`wo_a`) bf16 GEMM runs `torch.einsum("tgd,grd->tgr", o, wo_a)`, which dispatches to a **rocBLAS/Tensile `Cijk_*` batched GEMM** — the single largest kernel in the DSV4 decode attention region (~30 µs/s
`/rerun-failed-ci` only reruns runs whose conclusion is `failure` or `skipped`, so a workflow left `cancelled` (manual cancel, lost runner, cancel race) can never be restarted from a PR comment — you have to click "Re-run all jobs" in the UI. It also iterated over every run at the head SHA, and GitHub can have several runs of the same workflow at one commit,
## Motivation The multimodal embedding cache (`SGLANG_VLM_CACHE_SIZE_MB`) and the GPU feature-transport pools (`SGLANG_MM_FEATURE_CACHE_MB`, used by `--mm-feature-transport cuda_ipc`/`cuda_vmm`) allocate GPU memory only **after** the KV pool is sized: the embedding cache fills lazily per rank at runtime, and the feature pools are created on `base_gpu_id` whe
## Motivation `ModelOptNvFp4A16LinearMethod.apply()` calls `apply_fp4_marlin_linear()` unconditionally, there is no capability branch, so this is the path every dense NVFP4 W4A16 layer takes on SM120 today. But the two NVFP4 Marlin tests in `test_gptq_marlin.py` are skip-gated to SM80/SM86/SM90, so that path has never been asserted correct on Blackwell. I ve
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det
Moves the GB300 MMMU-Pro evals and all mmlu evals onto sgl-eval, and deletes the NeMo-Skills glue that the former needed. **MMMU-Pro.** Replaces the per-process NeMo-Skills venv bootstrap and its stdout-regex score parsing with the existing sgl-eval shim. Sampling is pinned in the two GB300 tests to what `ns eval --benchmarks=mmmu-pro:1` sent implicitly (tem
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation <!-- Explain the purpose of this PR and the goals it aims to achieve. --> ## Modifications <!
<!-- Thank you for your contribution! We appreciate it. The following guidelines will help improve your pull request and facilitate feedback. If anything is unclear, don't hesitate to submit your pull request and ask the maintainers for assistance. --> ## Motivation ` python3 -m sglang.launch_server --model meta-llama/Llama-3.1-8B-Instruct --pp-size 4 --port
## Summary Adds comprehensive CPU-only unit tests for `srt/model_loader/utils.py` and `srt/model_loader/weight_utils.py`, covering **86 test cases** in **1046 lines**. Closes #20865 (partial) ### `test_utils.py` — 38 tests | Module under test | What's covered | |---|---| | `set_default_torch_dtype` | Context manager restore, nesting, exception safety | | `_i
## Summary Fixes #22518 - **Bug**: Embedding/reward models (prefill-only) permanently deadlock after processing one full batch. When requests fill `max_running_requests`, `batch_is_full` is set to `True`. For prefill-only models, completed requests are removed via `filter_batch()` on `running_batch` (line 2306-2307), but `batch_is_full` is never reset. This
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation Enables TRT-LLM's fmha_v2 prefill attention backend for SM90/120. This backend is more
## Motivation This PR adds support for LongCat-Image, a new text-to-image model based on a Flux-like MMDiT architecture with a Qwen2.5-VL (7B) text encoder. * Huggingface: https://huggingface.co/meituan-longcat/LongCat-Image * GitHub: https://github.com/meituan-longcat/LongCat-Image ## Modifications New components: - `LongCatImageTransformer2DModel`: Flux-li
## Motivation In `NixlBackendSelection.create_backend()`, if `agent.get_plugin_list()` (line 101) raises an exception, the `except` block at line 158-161 references `plugin_list` and `initparams` — variables that were never assigned. This turns the original exception into an `UnboundLocalError`, hiding the real failure. ## Modifications Use `locals().get()`
## Summary _check_vocab_boundary_finish crashes with TypeError when new_accepted_tokens is None and vocab_size is also None (e.g. embedding requests). Add explicit None check to prevent the error. 🤖 Generated with Claude Code
## Summary - Simplify the Python wrapper for `tree_speculative_sampling_target_only` by allocating the scratch `draft_probs` tensor internally. - Remove caller-side `draft_probs` allocation/passing from DFlash and EAGLE speculative sampling. - Update the direct speculative sampling test call to match the simplified API. Addresses the `draft_probs` cleanup it
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation When Mooncake HiCache storage is attached at runtime via `PUT /hicache/storage-backend`
## Motivation Intermittent GPU memory access fault during EAGLE multi-layer speculative decode on hybrid sliding-window models (a mix of full-attention and SWA layers, e.g. MiMo-V2.5-Pro). It only shows up with cuda graph enabled and only when the SWA kv pool is smaller than the full pool. With `--disable-cuda-graph` the model runs clean, which is the tell t
## Motivation SGLang already supports ModelSlim W4A4_MXFP4 quantization for dense linear layers on Ascend NPU, but the corresponding MoE path is still missing. As a result, MoE models whose expert weights are exported with the W4A4_MXFP4 scheme cannot be loaded and executed through the ModelSlim quantization backend. The Ascend MoE implementation has also be
## Motivation Env variable `SGLANG_AUTO_NUMA_BIND` remained declared and documented ([here on main branch](https://github.com/sgl-project/sglang/blob/main/docs_new/docs/references/environment_variables.mdx#numa--cpu)), but its gate was accidentally dropped during the automatic NUMA configuration refactor, leaving the variable ineffective. ## Modifications -
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation The EPD encoder had grown into a single 4464-line `python/sglang/srt/disaggregation/enc
<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. --> ## Motivation <!-- Describe the purpose and goals of this pull request. --> ## Modifications <!-- Det