Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
Journal:
arXiv
Published Date:
Jun 25, 2025
Abstract
Retrieval-Augmented Generation (RAG) systems are emerging as a key approach
for grounding Large Language Models (LLMs) in external knowledge, addressing
limitations in factual accuracy and contextual relevance. However, there is a
lack of empirical studies that report on the development of RAG-based
implementations grounded in real-world use cases, evaluated through general
user involvement, and accompanied by systematic documentation of lessons
learned. This paper presents five domain-specific RAG applications developed
for real-world scenarios across governance, cybersecurity, agriculture,
industrial research, and medical diagnostics. Each system incorporates
multilingual OCR, semantic retrieval via vector embeddings, and domain-adapted
LLMs, deployed through local servers or cloud APIs to meet distinct user needs.
A web-based evaluation involving a total of 100 participants assessed the
systems across six dimensions: (i) Ease of Use, (ii) Relevance, (iii)
Transparency, (iv) Responsiveness, (v) Accuracy, and (vi) Likelihood of
Recommendation. Based on user feedback and our development experience, we
documented twelve key lessons learned, highlighting technical, operational, and
ethical challenges affecting the reliability and usability of RAG systems in
practice.
