Improving GRPO for safety alignment in large reasoning models
In brief
- GRPO suffers from two critical safety alignment failures: advantage collapse and unsafe behavior diversification in large reasoning models.
- Beacon restores safety learning signals using trajectory filtering and safe reference supervision.
- Experiments across eight benchmarks reveal a 32.83-point average reduction in safety errors.
Researchers introduce Beacon, a method to enhance safety in large reasoning models (LRMs) by addressing two key weaknesses of GRPO: advantage collapse and unsafe behavior diversification. Tests across eight benchmarks show an average 32.83-point reduction in safety errors.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error