When Agents Slow Down: Understanding LLM Agents' Test-Time Strategies via Elo-per-token Analysis
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for intermediate submissions, making progress observable throughout long trajectori
Reading in Plain English mode — toggle below to switch
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for intermediate submissions, making progress observable throughout long trajectori
This research introduces a novel method, Elo-per-token analysis, to rigorously evaluate LLM agents' performance and resource allocation over time. Understanding these test-time strategies is crucial for ML engineers to design more efficient and effective agents, especially in complex, open-ended tasks.