HybridDeepSynergy: A hybrid deep learning model integrating CNN, LSTM, and attention mechanisms for cancer drug synergy prediction.

A variety of AI-based approaches have been employed to analyze complex genomic datasets. Predicting the synergy of drug combinations is a critical step toward optimizing cancer treatment by identifying the most effective drug pairs. This study presents HybridDeepSynergy, a novel hybrid deep learning model that integrates Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Transformer attention mechanisms to predict drug synergy across diverse drug combinations and cancer cell lines. The model is designed to enhance precision medicine and cancer treatment outcomes. HybridDeepSynergy leverages CNNs to capture local feature interactions, LSTMs to model sequential dependencies, and attention mechanisms to extract long-range relationships within the data. The model was trained and evaluated on a comprehensive dataset containing numerous drug combinations, using five established synergy scoring models: Bliss Independence (BLISS), Zero Interaction Potency (ZIP), Loewe Additivity (LOEWE), Highest Single Agent (HSA), and General Synergy (S). Our model demonstrated superior performance compared to existing approaches, achieving a lower Root Mean Squared Error (RMSE = 3.911) and Mean Absolute Error (MAE = 2.922), along with higher coefficients of determination (R2 = 0.953), Pearson correlation (0.917), and Spearman correlation (0.886). These results confirm its predictive efficiency and consistency across multiple synergy scoring models. Furthermore, the incorporation of attention mechanisms provides interpretability by highlighting significant features associated with drug resistance. Future work will focus on incorporating additional cancer datasets, enhancing model predictive capabilities, and validating the approach in clinical settings to support personalized medicine. The findings suggest that HybridDeepSynergy has the potential to substantially improve treatment strategies for cancer and may be applicable to other disease contexts.