Multimodal Deep Learning Model to Estimate CT-based Body Composition Measures Using Chest radiographs and Clinical Data

Body composition metrics such as visceral fat volume, subcutaneous fat volume and skeletal muscle volume, are important predictors for cardiovascular disease, diabetes, and cancer prognosis. Recent advances in artificial intelligence have enabled automatic calculation of body composition metrics from CT scans and MRIs. In this study, we explore using deep learning to estimate body composition metrics from chest radiographs and a small set of variables that are easy to obtain in the clinical environment. A retrospective cohort of patients with concurrent non-contrast abdominal CT scan and chest radiograph was selected using the Truveta dataset. TotalSegmentator was used to delineate various tissue components in the CT scan. Volumetric and mid-l3 level body composition measures including subcutaneous and visceral fat volume, skeletal muscle and bone related indices, and aortic calcification scores were calculated. A multitask, multimodal deep learning model using chest radiographs and select clinical variables was trained to estimate the ground truth body composition metrics. Three data fusion strategies were compared in this process: early, intermediate, and late. Our final cohort consisted of 1,118 patients. Mean age of imaging was 67 years old, mean height was 1.67 meters and mean weight was 78 kgs. The late fusion multimodal model outperformed both unimodal clinical-only and imaging-only models, in addition to the early and intermediate fusion models. It achieved a Pearson correlation of 0.85 in prediction of subcutaneous fat volume and 0.76 and 0.72 in estimating visceral fat volume and vertebral bone volume in the hold-out test set. While previous studies have used artificial intelligence to automate calculation body composition metrics in CT scans and MRIs, we introduce a novel strategy that utilizes simpler imaging modalities. Our models can be used in future studies to calculate body composition metrics for larger cohorts of patients using only a chest radiograph and readily available clinical variables.