On the estimation and validity of AI time horizons---a statistical look at the METR plot
METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. O
Reading in Plain English mode — toggle below to switch
METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. O
This paper refines the methodology for evaluating AI capabilities, particularly in code generation and task completion, by improving the accuracy and interpretability of time horizon metrics. Understanding these refined metrics helps ML engineers better assess and compare the practical utility and progress of different AI models.