Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness
Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-verify pipeline catches incorrect claims well but says little about whether a response contains all the information it should. Measuring factual completeness, the missing half of factuality, is harder: i
Reading in Plain English mode — toggle below to switch
Evaluating the factuality of long-form generations has focused predominantly on precision, measuring whether the claims a model makes are correct. The dominant decompose-search-verify pipeline catches incorrect claims well but says little about whether a response contains all the information it should. Measuring factual completeness, the missing half of factuality, is harder: i
This research introduces a novel framework and benchmark for evaluating factual completeness in open-ended generations, addressing a critical gap in current LLM evaluation methods. For ML engineers, understanding and applying such comprehensive evaluation rubrics is crucial for developing more reliable and trustworthy AI systems, especially in applications requiring high factual accuracy.