Engineering

SpecForge

❖ Today, we are introducing a major update to SpecForge. The new runtime separates target-model inference from draft-model training, supports a broader family of speculative decoding algorithms, and unifies online, offline, and disaggregated workflows behind one typed training entry point. Alongside the release, we are publishing more draft models, covering different speculative decoding methods and different target models.

Agents-A1

❖ Agents-A1 is a family of long-horizon Mixture-of-Experts agentic models (35B and 4B) that reaches trillion-parameter-level performance by scaling the agent horizon. We investigate agent-horizon scaling from two perspectives: scaling long-horizon trajectories and scaling heterogeneous agent abilities.

VeRL-Omni

❖ VeRL-Omni is a general RL training framework focused on multimodal generative models, built on top of verl (Volcano Engine Reinforcement Learning). It is designed to provide easy, fast, and stable RL training for diffusion and omni-modality models. verl is the open-source version of HybridFlow — a Flexible and Efficient RLHF Framework.

TorchTPU

❖ TorchTPU is an open-source, native PyTorch backend for Google Cloud TPUs, developed in collaboration between Google and the PyTorch team at Meta. With TorchTPU, developers can run existing PyTorch workloads on TPU hardware with minimal code changes and scale training and inference workloads from a single chip to multi-slice TPU pods.

CUDA Graphs in SGLang

❖ CUDA Graphs promise to remove kernel-launch overhead, but realizing that benefit in an inference engine requires graphing as much of the workload as possible without sacrificing compatibility, startup time, or memory. In SGLang, we refactored CUDA Graph support around a common runner/backend interface, making different capture strategies reusable across execution paths.

❖ DeepSeek Sparse Attention (DSA) was a breakthrough for long-context models, reducing core attention complexity from O(L²) to O(Lk), but its lightning indexer remained an expensive O(L²) tax at every layer. IndexCache eliminates up to 75% of indexer computations in DSA through cross-layer index reuse — achieving up to 1.82× prefill speedup and 1.48× decode speedup with negligible quality degradation.

TurboQuant

❖ TurboQuant compresses vectors by rotating them into a predictable distribution, then snapping to a precomputed grid. Step through the animation below to see every stage of TurboQuant: normalize, rotate, quantize, reconstruct. Hover over any point for exact coordinates. On step 0, click to add or remove points.

❖ This is the introduction of Attention Residuals (AttnRes), a drop-in replacement for standard residual connections in Transformers that enables each layer to selectively aggregate earlier representations via learned, input-dependent attention over depth.