Identification of non-hydroxamate histone deacetylase 8 inhibitors using deep learning-based screening of two-dimensional molecular images.

Histone deacetylase 8 (HDAC8) is a zinc-dependent enzyme that deacetylates proteins and is involved in cancer development, making its inhibition a potential anticancer strategy. Most reported HDAC8 inhibitors contain a hydroxamate zinc-binding group; however, hydroxamate moieties have been associated with potential mutagenicity, motivating the search for non-hydroxamate inhibitors. In this study, we applied a two-dimensional (2D) molecular image-based deep learning approach to identify non-hydroxamate HDAC8 inhibitors. We first assessed DEEPScreen, a reported convolutional neural network-based platform that uses a 2D molecular image model; however, the original architecture exhibited limited predictive performance for HDAC8 (F1-score < 0.7). Incorporating a ResNet architecture into DEEPScreen improved the classification accuracy of HDAC8 inhibitors (F1-score = 0.84) and outperformed conventional descriptor-based methods. Using the trained ResNet model, we screened the Osaka University library and experimentally assessed the resulting virtual hit compounds. This workflow identified a non-hydroxamate HDAC8 inhibitor hit S29 (61.6% inhibition at 30 μM; IC50 = 14.1 μM). Subsequent straightforward structural optimization revealed that the α-methyl substituent on the ester moiety was not essential for HDAC8 inhibition and resulted in the identification of compound 1, a more potent non-hydroxamate HDAC8 inhibitor (72% inhibition at 30 μM; IC50 = 7.8 μM). This study demonstrates that 2D image-based deep learning-integrated with experimental validation and structural optimization-can facilitate the discovery of non-hydroxamate HDAC8 inhibitors and expand the accessible chemical space for this target.