ToMStories: Evaluating Theory of Mind in Language Models
In brief
- ToMStories evaluates models' inference of psychological states (emotions, motivations) in synthetic stories.
- Models from 1B to 72B parameters reach human-level performance on multiple-choice questions.
- Open-ended responses reveal gaps in implicit interpretation, highlighting current limitations.
A study introduces ToMStories, a dataset of 6,000 synthetic narratives assessing language models' ability to infer implicit or explicit psychological states of characters. Results show top models match human performance on multiple-choice questions but struggle with open-ended responses.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error