Harness-Delta Attribution: Dissecting Gains From Harness Evolution
In brief
- Harness-Delta Attribution (HDA) breaks down AI agent gains into three components: overfitting, inference compute, and generalizable improvement.
- In 21 tested cases, 54% of gains come from overfitting and 14% from additional compute, leaving little room for real improvement.
- Gains observed on a dataset often fail to transfer to other tasks, limiting practical utility.
A new evaluation method, Harness-Delta Attribution, assesses gains from agent systems optimized through structural evolution (prompts, tools, orchestration). It separates real improvement from overfitting or increased inference compute. Results show most observed gains do not generalize beyond the training dataset.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error