Evaluation of Gender Bias in the Evaluation of Synthetic Cardiovascular Disease Cases with Open Source LLMs

To systematically evaluate gender bias in open-source large language models (LLMs) for cardiovascular diagnostic decision-making using controlled synthetic case vignettes. We generated 500 synthetic cardiovascular cases with randomly assigned gender (male/female, equal distribution) and age (45-80 years), keeping all other clinical variables identical. Two structured prompts simulated sequential cardiovascular evaluation stages: initial chest discomfort presentation and post-stress-test evaluation. Three open-source LLMs were evaluated via local Ollama API: Gemma-2b, Phi, and TinyLLaMA. Primary outcomes included coronary artery disease (CAD) likelihood ratings (low/intermediate/high), diagnostic certainty (low/intermediate/high), and test usefulness scores (1-10 scale). Statistical analysis included chi-square tests, Mann-Whitney U tests, and logistic/linear regression with multiple comparison adjustments. Power analysis determined minimum detectable effects of 12.5% for individual models and 7.2% for pooled data. Evaluation of 1,500 model responses (500 cases × 3 models) revealed minimal gender-related differences. Only one statistically significant finding emerged: Gemma-2b assigned higher diagnostic certainty to female patients in initial presentations (58% vs. 48%, p=0.031, adjusted p=0.092). No other gender-based differences reached significance after multiple-comparison adjustment. Effect sizes were consistently small across all comparisons (Cohen’s h: 0.01-0.18; Cliff’s delta: -0.11 to 0.12). Substantial inter-model variability was observed, with Gemma-2b and Phi demonstrating assertive diagnostic patterns while TinyLLaMA showed conservative tendencies. Parsing quality exceeded 95% for all models. Open-source LLMs demonstrated largely gender-neutral outputs in controlled cardiovascular scenarios, contrasting with documented biases in human clinicians and commercial LLMs. The isolated gender effect in Gemma-2b was modest and clinically insignificant. More concerning was substantial inter-model variability in diagnostic confidence and test recommendations, highlighting the critical importance of rigorous model benchmarking before clinical deployment. These preliminary findings suggest that open-source LLMs may offer advantages for equitable healthcare applications, but broader validation across diverse clinical contexts and real-world constraints remains essential.