Alibaba Qwen introduced QwenGyre, an elastic reinforcement learning framework designed to accelerate multi-hour extreme-long-horizon agent training. Developed with USTC and Tsinghua University, QwenGyre addresses GPU idling and trajectory redundancy using an Elastic Scheduler and Trajectory Processor. On the flagship Qwen 3.8 2.4T model, QwenGyre improved NL2RepoBench accuracy from 52.5% to 58.5% in 48 training steps while achieving up to 1.85x speedups over colocated setups.
QwenGyre elastic RL framework
- ▪Alibaba Qwen, in collaboration with the University of Science and Technology of China and Tsinghua University, introduced QwenGyre, an elastic reinforcement learning framework for training extreme-long-horizon agents.
- ▪Alibaba Qwen evaluated the QwenGyre framework on GPU clusters using the Qwen 3.6 122B model across 24 to 32 nodes and the Qwen 3.8 2.4T model across 192 nodes.
- ▪QwenGyre trains coding agents through unmodified black-box harnesses like Claude Code 2.1.220, using Megatron for model training and SGLang for rollout serving across multi-node GPU clusters.
Extreme-long-horizon agent training challenges
- ▪Extreme-long-horizon agent tasks require multi-hour runtimes spanning hundreds of tool interactions and consume between 700,000 and 1,000,000 tokens per rollout across context windows.
- ▪Applying online reinforcement learning to extreme-long-horizon agent executions causes GPU idling from severe runtime variance and creates trajectory redundancy from complex non-linear agent branching.
Elastic scheduler GPU allocation
- ▪QwenGyre's Elastic Scheduler recorded average role switching times of 8.52 seconds from rollout to training and 3.46 seconds from training to rollout.
- ▪In the QwenGyre framework, satellite GPU cells migrate key-value cache blocks over RDMA and pull authoritative weights from a core cell via Mooncake's Transfer Engine when joining training batches.
- ▪QwenGyre's Elastic Scheduler dynamically transfers GPUs between rollout generation and model training without interrupting active agent executions.
Trajectory processor deduplication
- ▪QwenGyre's Trajectory Processor organizes branching agent histories into prefix-shared trajectory trees, deduplicating shared targets and evaluating partial progress even when executions time out.
- ▪QwenGyre caps retained trajectories at 5 per execution, cutting forward-backward training time by 25.2% while matching uncapped score performance.
Benchmark performance gains
- ▪QwenGyre achieved end-to-end training speedups of 1.21x to 1.85x over colocated setups and 1.38x to 1.78x over asynchronous setups across benchmark evaluations.
- ▪Applied to the Qwen 3.8 2.4T model on NL2RepoBench, QwenGyre increased accuracy from 52.5% to 58.5% in 48 training steps, completing the steps in 75.42 hours compared to 134.55 hours for asynchronous training.
Debatable claims
- ▪Dynamic GPU reallocation between rollout and training improves infrastructure efficiency in long-horizon agent learning
- ▪Alibaba Qwen should train long-horizon coding agents through unmodified black-box harnesses
Story comments
Loading comments…