PAR: Progress-Aligned Recurrence for Length Generalization in State Tracking
In brief
- PAR aligns intermediate predictions with actual operations to enhance length generalization.
- A Qwen2.5-0.5B model achieves 95.5% accuracy on sequences 16 times longer than training.
- Norm anchoring and prefix distillation reduce extrapolation errors significantly.
Researchers introduce Progress-Aligned Recurrence (PAR), a method to improve length generalization in recurrent models for sequential state tracking. According to the announcement, PAR trains the model to predict the state after k iterations by aligning these predictions with actual operations, while normalizing recurrent states to their initial norm. In tests, a Qwen2.5-0.5B model trained on sequences of 32 moves achieves 95.5% accuracy on sequences of 512 moves, or 16 times the training length.
Results show that norm anchoring reduces the extrapolation factor from 16 to less than 2.2, while replacing aligned targets with final-answer supervision limits it to 8. PAR also enables selecting when to answer without a provided stopping depth, and prefix self-distillation induces the correspondence between short prefixes and final answers when intermediate labels are unavailable.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error