SGLang v0.5.19 expands model support, inference performance and hardware reach

SGLang v0.5.19 is a wide-ranging inference-infrastructure release built from 786 pull requests by 214 contributors. Its changes span model coverage, request scheduling, caching, distributed execution, quantization, speculative decoding, multimodal processing, diffusion serving and hardware-specific kernels. The release also carries migration work that operators should review before upgrading, including new defaults, renamed settings, a required FlashInfer version and tighter request limits.

The release notes report numerous benchmark results, but those figures are tied to particular models, accelerators, batch sizes, context lengths and configurations. They should be read as measurements of the documented test cases, not universal guarantees for every deployment.

Highlights

Model support grows through nine listed entries: eight autoregressive entries and one diffusion entry. The autoregressive additions are Qwen3.8 (2.4T-A95B), Qwen3.8-27B, dots3.note, Ling-3.0-flash, Ling-3.0-tiny, Spark2.5, MiniCPM-SALA and Granite 4.2. LongCat-Image-Edit & Edit-Turbo is the diffusion entry.

Cookbook work accompanies the implementations. It includes deployment guidance for GLM-5.3 and PaddleOCR-VL; Kimi-K3 on Ascend A3; Kimi-K2.7-Code-MXFP4 on MI355X; Qwen3.5 MXFP4 on MI355X using an FP8 KV cache or HiCache host-memory tier; and MiniMax-H3 on one 24 GB GPU or DGX Spark. Ling-3.0-flash receives DGX Spark guidance, while Qwen3.8-27B measurements cover RTX 5090, RTX PRO 6000 and DGX Spark.

Scheduler & Runtime

Beam search is a prominent user-facing addition. Requests can pass beam_width and receive the n best sequences rather than one sample. The current implementation cannot be combined with speculative decoding, disaggregation, DP attention or HiCache. Runtime changes also cover mixed chunk prefill, configurable decode intervals, load publication for routers, tokenizer context handling, constrained generation, audio input in chat completions, tool-argument schema resolution, weight-version tracking, KV sizing and more detailed profiling annotations.

Rust Server

The Rust server gains a process-local in-memory KV indexer with router integration, end-to-end latency metadata, synchronized sampling and scheduler wire schemas, derived server addresses, signed environment values and a configurable HTTP/2 connection window. A gateway dependency update addresses blocked dependents.

Speculative Decoding

Speculative decoding receives new algorithms, extensibility and correctness fixes. DFlash2 combines local convolution with a candidate selector and is reported at 3.43x the no-spec baseline at batch 1 and about 24% ahead of DFlash at concurrency 64. DSpark support for LFM2 and LFM2-MoE produced 1.05x to 2.42x faster decoding on LFM2.5 targets using one H100. Nemotron 3.5 Lightning support recorded GSM8K results from 94.6% to 95.8% across MTP, DFlash and DSpark on GB300.

The opt-in KDA fused-accept path, enabled with SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1, reduces MTP verify-and-commit time by 45% to 63% on Kimi-Linear shapes while retaining bit-identical output. The release also supports multi-adapter LoRA with several speculative methods, custom draft worker classes and extensible draft attention backends. Fixes address state divergence, startup failures, graph memory, initialization and duplicate sampling. Quantizing Gemma-4 MTP bridge projections raised reported FP8 MTP acceptance from 0% to 60%.

Piecewise & Breakable CUDA Graph

CUDA graph work adds pipeline parallelism to full-prefill graphs and coordination across DP-attention ranks. Proxy tensors yielded up to 2.48x faster pipeline-parallel prefill for 2K-token forwards on Qwen3.5-397B using GB300. Reducing idle DP work cut median time to first token by 18.9% and raised QPS by 12.9%. Command-A-Plus decode TPOT was reported 2.28x faster at batch 1, reaching as much as 9,152 tokens per second on four GB300s.

Attention Backends

Decode context parallelism now operates on the default Blackwell trtllm_mla path. At 128K input, the release notes say plain tensor parallelism stops scaling near 680 tokens per second on eight B200s, while DCP continues scaling with concurrency. DSA ragged-prefill top-k moves to a v2 kernel that is 1.3x to 1.8x faster on B200. Other reported gains include an fmha_v2 kernel about 15% faster than FA3 at kernel level, Q8KV8 sparse MLA prefill improvements of 4.4% to 7.5% from 3.5K to 160K input on H20, and DP-prefill improvements of 4.9% to 6.8%.

MoE & Expert Parallelism

DeepEP v2 introduces an ElasticBuffer MoE all-to-all backend, selected with --moe-a2a-backend deepep_v2 for FP8 DeepSeek-V3/V4 and Qwen3-MoE. Fixed-size buffers allow decode under CUDA graphs across nodes, with performance described as comparable to the classic backend. Hopper systems can use FP8 activations with MXFP4 experts through --flashinfer-mxfp4-moe-precision fp8. The documented W4A8 kernel is 1.63x to 2.08x faster than W4A16, while DeepSeek-V4-Flash output throughput rises about 12%—11.7% in the detailed result—with no GSM8K accuracy change. This path requires FlashInfer 0.6.18.

Additional MoE work covers W4A4 MegaMoE, default backend selection for DeepSeek-V4 FP4 checkpoints, shared-expert fusion, SSD serving, overlapping shared and routed experts, native W4AFP8 schemas, topology synchronization and routing correctness. A requantization kernel reaches up to 10.6x improvement at 256 experts in its specified test.

Quantization

Quantization support expands across NVFP4, MXFP4, MXFP8, FP8 and compressed-tensors formats. A CuTe DSL NVFP4 W4A16 mode moved Qwen3-30B-A3B GSM8K from 0.965 to 0.980 and throughput from 1,662 to 1,858 tokens per second. MXFP8 KV caches can participate in prefill/decode transfer and CPU offload. A routing fix restores SM90 FP8 decode from 190 to about 215 tokens per second after a regression.

Parallelism & Disaggregation

LayerNorm sequence parallelism is enabled with --enable-layernorm-sp. Each tensor-parallel rank normalizes its own share of prefill tokens, reducing Qwen3-8B prefill by 3.5% on H100 and 5.6% on B200; the saving grows with tensor-parallel degree. Support is currently limited to dense Qwen3 models. Destination-contiguous RDMA blocks reduce documented prefill/decode transfer time by 28% to 39% for Kimi-Linear-48B on eight B300s. A DP-attention prefix fix restored Qwen3-30B-A3B GSM8K from 70.4 to 94.2.

HiCache & Radix Cache

The unified radix tree becomes the default cache for every model, including full-attention-only configurations. Prefill/decode workers can reuse cached prefixes for SWA hybrid models, L3 storage can be attached or detached while a server is running, and pipeline-parallel HiCache L3 state is kept consistent across ranks. Other work adds Rust TreeCore, external-linker contracts, a Mooncake backend, buffer-only host-memory operation and unified-memory sub-pools. One test increased cache survival from 35.7% to 67.9% and reduced replay end-to-end time by 37.9%.

Multimodal

PaddleOCR-VL processing now overlaps page preprocessing, packs the vision transformer and uses prefill CUDA graphs. On an H200 with a 1080p page, reported time to first token fell from 235 ms to 120 ms and request throughput rose 1.84x. The release also consolidates multimodal processor execution, fixes Pixtral multi-image handling and cache mismatches, reduces transient embedding memory, corrects Qwen3-VL MoE ordering and addresses Kimi-K3 vision preprocessing, transcription and audio-understanding problems.

Model Support & Optimizations

Beyond the nine headline entries, model-specific work adds Granite SWA support and repairs fallback or topology behavior for DeepSeek-VL2, BailingMoeV3, Kimi-K3, Qwen3.8-MXFP4 and DeepSeek-V4.

Kernel Library

New H200 MoE configurations improve reported throughput by 3.8% for Qwen3.5 FP8 and 7.1% for Qwen3.6 FP8. A GB300 Triton configuration raises GLM-4.5 FP8 decode throughput by 21.9%. Shared FLA and MoE kernels receive broader shape limits, and kernel wheels now declare their PyTorch ABI dependency.

Configuration System

The configuration system separates raw input from resolution. Constructing ServerArgs no longer resolves values automatically, so programs building their own record must call resolve_once() before reading resolved fields. The launcher performs that step for the engine. The broader refactor reorganizes argument choices, published readers and per-model declarations.

SGLang-Diffusion

Diffusion changes cover MiniMax-H3, Cosmos3, Pi0.5, LongCat, SANA-Video, LingBot-Video, LTX-2.5, Wan, Qwen-Image, FLUX.2 and Hunyuan. MiniMax-H3 can run on one 24 GB GPU; documented denoising is 1.34x faster with exact attention and up to 2.48x with approximate attention. That approximate path is a quality-related tradeoff rather than a universally equivalent optimization. Checkpoint support expands across GGUF, pruned safetensors, Comfy NVFP4 and native Diffusers components.

Pi0.5 end-to-end time falls 9% in eager execution and 68% with bounded graphs on H200. Cache-DiT with layerwise offload is reported 1.84x faster at 30.1 dB PSNR on a 24 GB RTX 4090 D; Cache-DiT, CFG gating and attention overrides are explicitly lossy per-request accelerations. Numerous fused kernels reduce denoising or end-to-end time for video and image pipelines. Hunyuan VAE now defaults to tiled decode; the reported outcome is an 18% end-to-end increase for Hunyuan, while FastHunyuan avoids VAE-decode out-of-memory failures on four H200s.

Local & Desktop AI

Desktop-oriented fixes cover NVFP4 diffusion on SM120 hardware, including RTX PRO 6000 and RTX 50-series devices, plus a FlashMLA buffer correction. MLX support moves to Torch 2.13 and MLX 0.32 or newer with a redesigned tensor bridge, alongside a startup-crash fix.

AMD / ROCm

AMD systems receive a persistent work-centric Lean attention kernel for long or uneven decode batches. On MI355X, it reaches up to 1.52x throughput and up to 3.62x lower median inter-token latency. It is automatically enabled where its gate predicts a benefit and can be disabled with --enable-lean-attention=False or SGLANG_DISABLE_LEAN_ATTENTION=1.

For disaggregated GLM-5.2 on eight MI355Xs, fused top-k seed remapping reduces decode TPOT from 23.16 ms to 7.94 ms, summarized as roughly 23 ms to 8 ms. DeepSeek-V4 top-k becomes 2.66x to 2.97x faster, with up to 1.07x end-to-end improvement. A Qwen3.5 context-attention kernel drops from 14.24 ms to 4.04 ms at 70K input, with end-to-end latency down 24.6% to 28.5%. The AMD section also contains MoRI, AITER, speculative decoding, collective, correctness, image-build and diffusion compatibility work.

NPU / Ascend

Ascend support expands for DeepSeek-V4 operations, MXFP4 W4A4 MoE quantization, KDA causal convolution, streaming sessions and diffusion. MiniMax-H3 Laser attention is reported 1.26x to 1.45x faster than Ascend flash attention, while a 14-denoiser distributed GLM-Image pipeline is 3.7x faster than monolithic batching. Several fixes address diffusion regressions, KV allocation and operator input handling.

CPU / Intel / XPU

Intel XPU gains INT4 dense linear support for AWQ and GPTQ, SYCL implementations for DeepSeek-V4 operations, prefill-only model support and compressed-tensors FP8 W8A8. A fused Qwen3.5 GDN kernel cuts documented prefill from 165 ms to 59 ms. CPU changes enable ERNIE models and repair NUMA binding, DP attention, bidirectional masking, speculative verification and GPTQ INT4 weight handling.

Security

The release bounds CUDA memory used by fast image preprocessing and restricts stop-pattern inputs. Requests may contain at most 32 stop strings and 32 stop regular expressions, each no larger than 256 bytes; exceeding those limits returns HTTP 400. CI commands also change: /rerun-stage is removed, while /rerun-test requires commenter trust based on zero cooldown or repository write access.

Dependencies

FlashInfer moves to—and for affected features requires—version 0.6.18. Other recorded versions include sgl-deep-ep 0.1.2, sgl-deep-gemm 0.1.7, mooncake 0.3.13, tilelang 0.1.12 and compressed-tensors 0.18.0. Images include a CUDA 13.4 preview for initial Rubin support and ROCm 10 builds for gfx942, gfx950 and gfx1250.

Breaking Changes & Upgrade Notes

• The unified radix tree is the default, and SGLANG_ENABLE_UNIFIED_RADIX_TREE is deprecated.
• Spark3 becomes Spark2.5 across configuration and model classes; the tool-call parser changes from spark to spark25.
• DeepSeek-V4 FP4 expert checkpoints select FlashInfer MXFP4 automatically on SM90, SM100 and SM120 unless an explicit backend is supplied.
• W4A4 MegaMoE moves to --enable-w4a4-megamoe; its older environment variables only warn.
• FlashInfer 0.6.18 is mandatory for the FlashInfer DSA top-k backend and CuTe DSL NVFP4 W4A16 mode, with no fallback to older APIs.
• ROCm 7.2.4 images use Python 3.12, Torch 2.11 and Triton 3.7.
• SGLANG_USE_SGL_XPU is removed; XPU defaults to sgl-kernel MoE, while --moe-runner-backend triton restores the Triton path.
• Diffusion Cache-DiT, CFG gating and DiT attention selection become per-request parameters; related environment variables remain server defaults but are slated for deprecation.
• kill_process_tree waits for process reaping by default in CLI, benchmark and test teardown flows.

New Contributors

The release notes identify 51 first-time contributors, while the overall release credits 214 contributors across 786 pull requests. The first-contribution list spans work in speculative decoding, model support, XPU and NPU enablement, Rust serving, multimodal processing, ROCm, kernels, configuration and runtime fixes. The contributor summary names yangweigbh and TarangKhanna alongside 49 other contributors.

Contributors

The scale of the contributor list reflects how broadly v0.5.19 reaches across SGLang’s serving stack. Individual benchmark claims remain configuration-specific, and several capabilities require explicit flags, particular model families or particular hardware. Operators should therefore validate the release against their own workloads and complete the documented migration steps before adopting new defaults.

Source: SGLang Releases, v0.5.19 release notes.

Definition. SGLang v0.5.19 is an inference-infrastructure release that broadens model and hardware support while updating scheduling, caching, distributed execution, quantization and speculative decoding.