EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
Journal:
arXiv
Published Date:
Jun 27, 2025
Abstract
Recent advances in reinforcement learning (RL) have significantly enhanced
the reasoning capabilities of large language models (LLMs). Group Relative
Policy Optimization (GRPO), an efficient variant of PPO that lowers RL's
computational cost, still faces limited exploration, low sample efficiency and
instability, constraining its performance on complex reasoning tasks. To
address these limitations, we introduce EFRame, an Exploration-Filter-Replay
framework that systematically augments GRPO along three critical dimensions.
EFRame performs additional rollouts to explore high-quality trajectories,
applies online filtering to eliminate low-quality samples that introduce noise
and variance, and leverages experience replay to repeatedly exploit rare but
informative samples. EFRame establishes a complete and stable learning cycle,
guiding the model through a structured transition from exploration to
convergence. Our experiments across a variety of reasoning benchmarks
demonstrate that EFRame not only improves the robustness and efficiency of
training, but also enables access to deeper reasoning capabilities that remain
unattainable under vanilla GRPO. Furthermore, EFRame not only enables
fine-grained categorization of training samples for deeper insight into their
contributions, but also introduces an efficient and precise mechanism for
entropy control, which is critical for balancing exploration and convergence in
RL training. Our code is available at https://github.com/597358816/EFRame.
