How AI agents search.

Journal: PloS one
Published Date:

Abstract

Large Language Models (LLMs) have evolved from relying primarily on knowledge stored in their parametric memory to using non-parametric memory mechanisms such as Retrieval-Augmented Generation (RAG) to provide and ground knowledge. Recently, LLMs have begun to perform retrieval autonomously within a reasoning loop, an approach referred to as Agentic RAG. This shift introduces new querying behaviours, with implications for the design of retrieval systems that have yet to be evaluated. In this work, we introduce a large-scale dataset (ASQ) capturing how Agentic RAG systems perform multi-step search and reasoning. ASQ contains about 1M agentic search traces collected from open-source Agentic RAG systems over 3 corpora, 5 agent variants, and 6 retrieval settings. Our analysis of ASQ shows that, within this controlled benchmark setting, a single search query can trigger long chains of retrieval calls (up to 186) and follow query reformulation dynamics different from those observed in historical human-search data. Our results also reveal substantial behavioural variation across agent and retrieval configurations: changes in re-ranking and retrieval depth do not consistently affect search workloads in terms of search-call volume and trace length across agents. To our knowledge, publicly available production logs do not expose the agent, dataset, and retrieval configuration associated with each trace. ASQ provides a reusable resource for further analysing agent search behaviour and supporting future information retrieval studies on agentic workloads under controlled settings.

Authors

Keywords

No keywords available for this article.