Vision Foundation Model Embedding-Based Semantic Anomaly Detection
Journal:
arXiv
Published Date:
May 12, 2025
Abstract
Semantic anomalies are contextually invalid or unusual combinations of
familiar visual elements that can cause undefined behavior and failures in
system-level reasoning for autonomous systems. This work explores semantic
anomaly detection by leveraging the semantic priors of state-of-the-art vision
foundation models, operating directly on the image. We propose a framework that
compares local vision embeddings from runtime images to a database of nominal
scenarios in which the autonomous system is deemed safe and performant. In this
work, we consider two variants of the proposed framework: one using raw
grid-based embeddings, and another leveraging instance segmentation for
object-centric representations. To further improve robustness, we introduce a
simple filtering mechanism to suppress false positives. Our evaluations on
CARLA-simulated anomalies show that the instance-based method with filtering
achieves performance comparable to GPT-4o, while providing precise anomaly
localization. These results highlight the potential utility of vision
embeddings from foundation models for real-time anomaly detection in autonomous
systems.
