Leveraging Interview-Informed LLMs to Model Survey Responses: Comparative Insights from AI-Generated and Human Data
Journal:
arXiv
Published Date:
May 28, 2025
Abstract
Mixed methods research integrates quantitative and qualitative data but faces
challenges in aligning their distinct structures, particularly in examining
measurement characteristics and individual response patterns. Advances in large
language models (LLMs) offer promising solutions by generating synthetic survey
responses informed by qualitative data. This study investigates whether LLMs,
guided by personal interviews, can reliably predict human survey responses,
using the Behavioral Regulations in Exercise Questionnaire (BREQ) and
interviews from after-school program staff as a case study. Results indicate
that LLMs capture overall response patterns but exhibit lower variability than
humans. Incorporating interview data improves response diversity for some
models (e.g., Claude, GPT), while well-crafted prompts and low-temperature
settings enhance alignment between LLM and human responses. Demographic
information had less impact than interview content on alignment accuracy. These
findings underscore the potential of interview-informed LLMs to bridge
qualitative and quantitative methodologies while revealing limitations in
response variability, emotional interpretation, and psychometric fidelity.
Future research should refine prompt design, explore bias mitigation, and
optimize model settings to enhance the validity of LLM-generated survey data in
social science research.
