SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture
Journal:
arXiv
Published Date:
Jun 18, 2025
Abstract
Language Models (LMs) are indispensable tools shaping modern workflows, but
their global effectiveness depends on understanding local socio-cultural
contexts. To address this, we introduce SANSKRITI, a benchmark designed to
evaluate language models' comprehension of India's rich cultural diversity.
Comprising 21,853 meticulously curated question-answer pairs spanning 28 states
and 8 union territories, SANSKRITI is the largest dataset for testing Indian
cultural knowledge. It covers sixteen key attributes of Indian culture: rituals
and ceremonies, history, tourism, cuisine, dance and music, costume, language,
art, festivals, religion, medicine, transport, sports, nightlife, and
personalities, providing a comprehensive representation of India's cultural
tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic
Language Models (ILMs), and Small Language Models (SLMs), revealing significant
disparities in their ability to handle culturally nuanced queries, with many
models struggling in region-specific contexts. By offering an extensive,
culturally rich, and diverse dataset, SANSKRITI sets a new standard for
assessing and improving the cultural understanding of LMs.
