Can LLMs predict the outcomes of machine learning experiments?
In brief
- LLMs predict ML experiment outcomes better than chance, but accuracy remains low.
- Claude Opus 5.5 explains 49% of outcome variance, far from perfect prediction.
- Models consistently underestimate the magnitude of experimental changes.
A study assesses how well large language models can anticipate the impact of machine learning experiment ideas. Results show predictions are better than random but limited, with persistent gaps across benchmarks.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error