LLM-based agents fail formal planning validation in most cases
In brief
- Agents achieve 78% offline success but only 40% in real-world scenarios.
- Unsolvable tasks are detected in under 10% of episodes.
- Complex prompting strategies use 5 to 22 times more tokens with no clear advantage.
A study evaluates LLM-based agents' planning abilities using formal PDDL models. Results reveal limited performance, especially in real-world conditions, with excessive token usage for minimal gains.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error