Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
In brief
- TreeOPSD assigns implicit credit to intermediate steps via a tree structure and terminal rewards.
- The method bounds policy drift relative to the behavior policy using a finite target.
- Empirical results: 4.97-point improvement on Qwen3-30B-A3B-Thinking-2507 in Deep Research.
TreeOPSD, a new posterior distillation method, improves credit assignment in long-horizon agentic reinforcement learning tasks. It avoids using critics or step-level annotations while preventing the current policy from drifting too far from the behavior policy.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error