GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay
GameHorizon Suite is a new framework for evaluating AI models in video games, focusing on their ability to handle tasks over different time horizons. It provides a standardized way to measure how well AI can understand visual information, plan actions, and control gameplay, addressing limitations of previous datasets.
Reading in Plain English mode — toggle below to switch
GameHorizon Suite is a new framework for evaluating AI models in video games, focusing on their ability to handle tasks over different time horizons. It provides a standardized way to measure how well AI can understand visual information, plan actions, and control gameplay, addressing limitations of previous datasets.
Imagine an AI playing chess. A short-horizon task might be capturing a single pawn, while a multi-horizon task involves planning several moves ahead to checkmate the opponent. GameHorizon helps evaluate AI's ability to excel at both short-term tactics and long-term strategic planning in complex game environments.
How GameHorizon Suite Works
- 1GameHorizon-Annotator: A scalable, automated pipeline annotates gameplay videos with multi-horizon instructions, linking specific actions to long-term goals.
- 2GameHorizon-Data: This pipeline creates a large dataset of 5,000 hours from 21 AAA games, featuring temporally aligned videos, player actions, and multi-horizon instructions from 100 expert players.
- 3GameHorizon-Bench: This benchmark offers both offline and online evaluation. The offline track uses thousands of standardized questions to provide reproducible evaluation across three primary tasks and diagnostic variants. The online track tests if offline scores reflect actual gameplay capabilities and identifies failures in long-horizon gameplay.
E[Rₜ | Sₜ, Aₜ]
This represents the expected total reward an AI agent will receive from time $t$ onwards, given its current state $S_t$ and the action $A_t$ it takes. It's a core concept in reinforcement learning for planning over horizons.
An AI model playing a strategy game might be given a short-horizon instruction like 'build a barracks' and a long-horizon instruction like 'conquer the enemy base'. GameHorizon evaluates if the AI can successfully execute both, understanding how the short-term action contributes to the long-term goal.
GameHorizon provides a comprehensive, standardized suite for evaluating AI's multi-horizon planning and execution capabilities in complex video game environments, revealing significant differences in model performance.