A Scalable Predictive Modeling Approach to Duplicate Detection in Pharmacovigilance: Algorithm Development and Validation Study.

BACKGROUND: Adverse event reports referring to the same case but treated as independent can negatively impact statistical analysis and may mislead clinical assessment. Pharmacovigilance relies on large databases of adverse event reports to discover potential new causal associations. Their size necessitates computational methods to identify duplicates at scale. The current state-of-the-art is statistical record linkage which outperforms rule-based approaches. In particular, vigiMatch is in routine use for VigiBase, the World Health Organization global database of adverse event reports, and represents the first statistical duplicate detection approach in pharmacovigilance deployed at scale. Originally developed for both drugs and vaccines, its application to vaccines has been limited due to inconsistent performance across countries. OBJECTIVE: This study aims to advance the state-of-the-art for duplicate detection in large-scale pharmacovigilance databases and achieve more consistent performance across adverse event reports from different countries. METHODS: This paper extends vigiMatch from probabilistic record linkage to predictive modeling, refining features for drugs, vaccines, and adverse events using country-specific reporting rates, extracting dates from free text, and training separate support vector machine classifiers for drugs and vaccines. Recall was evaluated using 5 independent reference sets. Precision was assessed by annotating random selections of report pairs classified as duplicates. RESULTS: Precision for the new method was 92% for vaccines and 54% for drugs, compared with 41% for a previous generation method. Recall ranged from 80% to 85% across test sets for vaccines and from 40% to 86% for drugs, compared with 24%-53% for the comparator method. CONCLUSIONS: Predictive modeling, use of free text, and country-specific features advance state-of-the-art for duplicate detection in pharmacovigilance.