How Does Response Length Affect Long-Form Factuality
Journal:
arXiv
Published Date:
May 29, 2025
Abstract
Large language models (LLMs) are widely used for long-form text generation.
However, factual errors in the responses would undermine their reliability.
Despite growing attention to LLM factuality, the effect of response length on
factuality remains underexplored. In this work, we systematically investigate
this relationship by first introducing an automatic and bi-level long-form
factuality evaluation framework, which achieves high agreement with human
annotations while being cost-effective. Using this framework, we conduct
controlled experiments and find that longer responses exhibit lower factual
precision, confirming the presence of length bias. To explain this phenomenon,
we empirically examine three hypotheses: error propagation, long context, and
facts exhaustion. Our results reveal that facts exhaustion, where the model
gradually exhausts more reliable knowledge, is the primary cause of factual
degradation, rather than the other two hypotheses.
