Hybrid noise-resistant technique for malware classification.

Visualization-based malware detection has recently gained significant attention for binary and multiclass malware classification using machine learning and deep learning techniques. However, existing visualization-based frameworks still face several important limitations, including insufficient robustness evaluation, limited cross-dataset validation, restricted malware diversity, and difficulty distinguishing visually similar and noise-sensitive malware families. In many cases, the visual similarity between malware classes and the presence of perturbations negatively affect feature extraction quality, leading to degraded classification performance and reduced generalization capability. To address these challenges, this study proposes a novel hybrid malware representation framework that integrates Convolutional Autoencoder (CAE)-based latent structural learning with Local Binary Pattern (LBP)-based texture feature extraction for robust malware classification. To the best of our knowledge, this study represents one of the first comprehensive investigations of hybrid latent-texture representation learning within a memory-forensics malware visualization setting while jointly addressing robustness, perturbation resilience, scalability, and cross-dataset generalization through a unified evaluation framework. The proposed framework combines global hierarchical representations learned through CAE with fine-grained local texture descriptors extracted using LBP to improve the discrimination of visually similar malware families and enhance robustness against noisy visualization conditions. The extracted features are subsequently evaluated using multiple machine learning classifiers, where XGBoost achieved the highest performance with an accuracy of 99.90%, precision of 99.79%, recall of 99.92%, and F1-score of 99.85%. To comprehensively evaluate the proposed framework, extensive experiments are conducted using both a memory-forensics malware dataset and the large-scale BODMAS dataset containing 134,435 PE malware samples spanning 581 malware families. The experimental evaluation incorporates cross-validation, ablation analysis, robustness assessment under multiple perturbation conditions, and feature-space visualization analysis. The results demonstrate that the proposed CAE+LBP framework consistently outperforms standalone feature extraction approaches and conventional end-to-end CNN models while maintaining strong robustness and cross-dataset generalization capability across diverse malware distributions and noisy conditions.