VeRL-Omni

VeRL-Omni is a general RL training framework focused on multimodal generative models, built on top of verl (Volcano Engine Reinforcement Learning). It is designed to provide easy, fast, and stable RL training for diffusion and omni-modality models.

Why VeRL-Omni

Multimodal generative RL training differs from text-only LLM RL not only in model structure, but also in I/O patterns, compute characteristics, and runtime bottlenecks. As this space grows, it deserves a dedicated training repository that can evolve quickly around its own constraints.

verl/HybridFlow

verl is the open-source version of HybridFlow: A Flexible and Efficient RLHF Framework paper.

verl is flexible and easy to use with:

• Easy extension of diverse RL algorithms: The hybrid-controller programming model enables flexible representation and efficient execution of complex post-training dataflows. Build RL dataflows such as GRPO, PPO in a few lines of code.

• Seamless integration of existing LLM infra with modular APIs: Decouples computation and data dependencies, enabling seamless integration with existing LLM frameworks, such as FSDP, Megatron-LM, vLLM, SGLang, etc

• Flexible device mapping: Supports various placement of models onto different sets of GPUs for efficient resource utilization and scalability across different cluster sizes.

• Ready integration with popular HuggingFace models

verl is fast with:

• State-of-the-art throughput: SOTA LLM training and inference engine integrations and SOTA RL throughput.

• Efficient actor model resharding with 3D-HybridEngine: Eliminates memory redundancy and significantly reduces communication overhead during transitions between training and generation phases.

Scope

VeRL-Omni targets RL post-training for three families of generative models:

1. Diffusion generative models for image, video, and audio — e.g., Qwen-Image, Wan2.2.
2. Unified multimodal understanding + generation models — e.g., BAGEL, HunyuanImage-3.0.
3. Omni-modality models that jointly handle text, image, audio, and video — e.g., Qwen3-Omni.

What we focus on

• Fast multi-modal rollout: Adopt vLLM-Omni and SGLang-Omni backends, and accelerate generation via rollout routing, rollout batching, embed caching optimizations, and more.
• Flexible & async multi-reward serving: Support multi-reward serving (HPSv3, GenRM-OCR, UnifiedReward, etc.), HTTP scorer, and asynchronous reward computation to overlap the rollout phase.
• Modular training backends: Selectable VeOmni and FSDP2 backends with combinable parallelism (USP/TP/DP) for distributed training.
• Stability: Boost stability and speed in diffusion RL pipelines via rollout correction to skip logP recomputation, and achieve reproducible E2E training with deterministic RL. Reward, rollout and actor update are composable and extensible, via Hydra configs.
• Efficient and convergent training recipes: On our reference Qwen-Image FlowGRPO setup, VeRL-Omni achieves ~25% higher end-to-end throughput than the diffusers-based flow_grpo implementation, driven by vLLM-Omni and SGLang-Omni rollout, FSDP2 trainer, overlapped reward computation (asynchronous), etc.

Getting Started 🚀

Visit our documentation to learn more.

• Installation
• Quickstart

Model and Algorithm Support 🎨

Ascend NPU Support 💠

VeRL-Omni now supports Ascend NPU. For instructions on how to install and get started with FlowGRPO training on Ascend NPU, please refer to our Ascend NPU Quickstart Guide.

Roadmap 🗺

Future work is tracked in VeRL-Omni Q3 Roadmap

Citation 📚

If you find the project helpful, please cite and star ⭐

@misc{verlomni_github,
  title        = {{VeRL-Omni: Easy, Fast, and Stable RL Training for Diffusion and Omni-Modality Models}},
  author       = {Yongxiang Huang and Cheung Kawai and Jingan Zhou and Yingshu Chen and {openYuanrong Team} and Xibin Wu},
  year         = {2026},
  howpublished = {\url{https://github.com/verl-project/verl-omni}},
  urldate      = {2026-04-28}
}