Med.ai ASK: an agentic system for biomedical question answering.

OBJECTIVE: Intelligent agent-driven research co-pilots, leveraging advances in generative AI, are transforming how scientists access biomedical knowledge. This paper presents Med.ai ASK, an agentic question-answering system designed to address biomedical inquiries through dynamic retrieval augmentation and tool-driven reasoning. We aim to develop a system capable of parsing the nuance in biomedical scientists' research questions to provide reliable, grounded responses that are more accurate than other generative AI solutions. MATERIALS AND METHODS: We adopt the ReAct framework's tool-calling architecture and leverage atomic reasoning from Self-Discover to build Med.ai ASK. It selectively queries multiple biomedical knowledge bases and employs map-reduce tools for vector database retrieval, alongside external API and NER tool integration. We ingested 44 million biomedical documents from diverse sources. The agent is evaluated on a range of biomedical question-answering datasets. RESULTS: Human evaluation on an internal dataset shows strong performance and stability. Ratings from a large language model are aligned with human assessments, supporting its use in further experiments. Automatic evaluations indicate superior performance in long-form answers regarding accuracy, faithfulness, factuality, and reduced hallucinations. For short-form and multiple-choice answers, performance is competitive with state-of-the-art systems. The agent's detailed answers are more interpretable than other systems attributed to its agentic design. The agent effectively selects tools based on question type and is deployed in a production-level chat platform with over 1600 users and 25 000 answered questions. CONCLUSION: Med.ai ASK dynamically orchestrates biomedical information retrieval tools to deliver robust interpretative, accurate, and factual answers, which is crucial in the biomedical domain.