Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark
Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level
Reading in Plain English mode — toggle below to switch
Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level
This paper introduces a novel benchmark, SWE-Flux, specifically designed to evaluate LLMs' ability to reason about dynamic code execution at a repository level. This is crucial for developing more capable code generation and debugging tools, as it moves beyond static code analysis to assess understanding of runtime behavior.