WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding, the accuracy of predictions about component changes after budgeted experimentation. Agents inspect code, select measurements, and submit a response surface, a table predicting scores for every configuration of component setting
Reading in Plain English mode — toggle below to switch
AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding, the accuracy of predictions about component changes after budgeted experimentation. Agents inspect code, select measurements, and submit a response surface, a table predicting scores for every configuration of component setting
This work introduces a novel benchmark for evaluating the experimental understanding of AI agents, which is crucial for developing more autonomous and effective research agents. As ML engineers, understanding how to design and evaluate agents that can reliably predict experimental outcomes is becoming increasingly important for automating complex tasks and accelerating scientific discovery.