Context-Aware Weakly Supervised Image Manipulation Localization with SAM Refinement
Journal:
arXiv
Published Date:
Mar 26, 2025
Abstract
Malicious image manipulation poses societal risks, increasing the importance
of effective image manipulation detection methods. Recent approaches in image
manipulation detection have largely been driven by fully supervised approaches,
which require labor-intensive pixel-level annotations. Thus, it is essential to
explore weakly supervised image manipulation localization methods that only
require image-level binary labels for training. However, existing weakly
supervised image manipulation methods overlook the importance of edge
information for accurate localization, leading to suboptimal localization
performance. To address this, we propose a Context-Aware Boundary Localization
(CABL) module to aggregate boundary features and learn context-inconsistency
for localizing manipulated areas. Furthermore, by leveraging Class Activation
Mapping (CAM) and Segment Anything Model (SAM), we introduce the CAM-Guided SAM
Refinement (CGSR) module to generate more accurate manipulation localization
maps. By integrating two modules, we present a novel weakly supervised
framework based on a dual-branch Transformer-CNN architecture. Our method
achieves outstanding localization performance across multiple datasets.
