Agentic Inference with NVFP4 KV Cache
❖ The KV cache is a fundamental building block of the modern LLM inference system. The context from multiple conversation rounds in agent sessions is cached as keys and values (KV) in GPU memory, allowing the model to reuse them during subsequent decoding steps. When generating each new token, the new query token attends to the relevant past KV tokens from the KV cache. As context windows grow, storing and reading KV cache puts increasing pressure on GPU memory capacity and bandwidth.
TorchNeuron
❖ TorchNeuron is an open-source PyTorch extension that provides new native backend integration for AWS Trainium. The implementation includes support for eager mode for rapid iteration and experimentation, torch.compile for just-in-time compilation, and standard distributed processing APIs. TorchNeuron enables ecosystem compatibility and supports custom kernel development through the Neuron Kernel Interface (NKI) for performance optimization and research applications.
PyTorch on TPU
❖ PyTorch on Cloud TPU lets you train, fine-tune, and serve open-source models on Cloud TPU hardware using standard PyTorch semantics, distributed APIs, and the ecosystem tools you already use, including Hugging Face, vLLM, SGLang, llm-d, TorchTitan, TorchRec, and Ray. This ecosystem provides libraries, frameworks, and tools for every stage of the machine learning lifecycle, from core runtime execution and custom kernels to production inference, large-scale training, and cluster orchestration.
Miles
❖ We present Miles v0.1, a full-stack production-ready system for frontier post-training, the successor to our first Miles release [1]. Building upon slime’s clean design, Miles optimizes every stage in the RL training loop around a simple principle: verified, clean, and customizable everywhere. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and developers alike. In this blog post, we will walk through Miles end-to-end.
Deep Researcher
❖ The NVIDIA Deep Researcher Agent Blueprint is a deployable research backend built on the NVIDIA NeMo Agent Toolkit and LangChain Deep Agents. Teams can self-host the application boundary and connect deployment-owned models, data sources, authentication, policy controls, storage, and observability. It provides both quick, cited answers and in-depth, report-style research, plus benchmarks and evaluation harnesses for measuring quality.
Speculators
❖ Speculators is a library for training speculative decoding draft models that deploy directly to LLM inference engines like vLLM. Speculative decoding is a lossless technique that speeds up LLM inference by using a smaller, faster draft model (“the speculator”) to propose tokens, which are then verified by the larger base model, reducing latency without compromising output quality.
SpecForge
❖ Today, we are introducing a major update to SpecForge. The new runtime separates target-model inference from draft-model training, supports a broader family of speculative decoding algorithms, and unifies online, offline, and disaggregated workflows behind one typed training entry point. Alongside the release, we are publishing more draft models, covering different speculative decoding methods and different target models.
Agents-A1
❖ Agents-A1 is a family of long-horizon Mixture-of-Experts agentic models (35B and 4B) that reaches trillion-parameter-level performance by scaling the agent horizon. We investigate agent-horizon scaling from two perspectives: scaling long-horizon trajectories and scaling heterogeneous agent abilities.