StreamEval: Benchmark for Interactive Streaming Video Generation Models
In brief
- StreamEval tests 5 tasks of real-time interactive video generation with 300 test cases.
- Scores reveal major gaps in reliable request execution despite strong visual quality.
- State consistency and content preservation remain persistent challenges for these models.
Researchers introduce StreamEval, a unified benchmark to evaluate interactive streaming video generation models in real time. According to the announcement, the framework tests five tasks: avatar dialogue, avatar performance, world generation, instruction editing, and reference editing. StreamEval includes 300 scripted test cases lasting 15 to 300 seconds, featuring incremental multimodal requests and explicit requirements for state preservation and content retention.
Results from 37 distinct models show that high visual quality and identity retention do not ensure reliable request execution or sustained state consistency. Request execution remains a major challenge, with top scores of 36.90 for Following among seven avatar-performance models, 25.93 for Following in text-conditioned world generation, and 15.00 for Edit Success among three reference-editing models.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error