Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization
In brief
- Lean compilation does not ensure semantic faithfulness in translated statements.
- Translation errors account for up to 29.5 percentage points difference between compilation and acceptance.
- Recent models (GPT-6 Astra, Claude Opus 5.5) significantly reduce translation errors.
A study measures the gap between successful compilation and semantic faithfulness in translating mathematical statements into Lean. Results show translation errors remain common even when compilation passes. Recent models reduce these errors, but automated judges struggle to identify them accurately.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error