The Rising Threat to Emerging AI-Powered Search Engines
Journal:
arXiv
Published Date:
Feb 7, 2025
Abstract
Recent advancements in Large Language Models (LLMs) have significantly
enhanced the capabilities of AI-Powered Search Engines (AIPSEs), offering
precise and efficient responses by integrating external databases with
pre-existing knowledge. However, we observe that these AIPSEs raise risks such
as quoting malicious content or citing malicious websites, leading to harmful
or unverified information dissemination. In this study, we conduct the first
safety risk quantification on seven production AIPSEs by systematically
defining the threat model, risk level, and evaluating responses to various
query types. With data collected from PhishTank, ThreatBook, and LevelBlue, our
findings reveal that AIPSEs frequently generate harmful content that contains
malicious URLs even with benign queries (e.g., with benign keywords). We also
observe that directly query URL will increase the risk level while query with
natural language will mitigate such risk. We further perform two case studies
on online document spoofing and phishing to show the ease of deceiving AIPSEs
in the real-world setting. To mitigate these risks, we develop an agent-based
defense with a GPT-4o-based content refinement tool and an XGBoost-based URL
detector. Our evaluation shows that our defense can effectively reduce the risk
but with the cost of reducing available information. Our research highlights
the urgent need for robust safety measures in AIPSEs.
