Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
In brief
- GUI agents must verify transitions between screens, not just their content.
- LGWM directly predicts the next screen's representation after an action, without a decoder or semantic annotation.
- The model reaches 0.987 AUC with 17 ms latency, outperforming existing approaches in efficiency.
Researchers introduce a method to secure GUI agents by verifying transitions between screens rather than their content alone. According to the study, a legitimate screen can be reused in a malicious context, making screen-only verifiers ineffective. Their model, LGWM, directly predicts the representation of the next screen after an action, without a decoder or semantic annotation, trained on 1.85 million real transitions.
Evaluation on RSWT-BENCH, a benchmark where each login screen appears in both a legitimate and a hijacked transition, shows LGWM achieves 0.987 AUC with a 17 ms decision time, outperforming closed models and generative GUI world models by about 10 AUC points at 1,000 times lower latency. The residual signal also distinguishes harmful violations from benign mismatches.
This summary gives you the essentials. The full article, its figures and quotes are on Hugging Face Daily Papers.
Read the original articleAI-generated summary of the source, checked by rules. How we write · Report an error