Competence-based evaluation of logical reasoning in large language models
In brief
- CBE assesses whether models consistently apply a logical skill across similar problems, not just accuracy on isolated tasks.
- Even strong models fail to generalize reasoning when problem formulations vary, despite good performance on bounded tasks.
- Techniques like chain-of-thought or GRPO improve results but do not fully resolve the competence gaps identified.
Large language models (LLMs) show impressive progress, but their logical reasoning remains inconsistent, especially under linguistic variations. A new evaluation method, competence-based evaluation (CBE), is introduced to measure how reliably models apply a given competence across a family of problems. Tests reveal weaknesses even in advanced models.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error