Effective Red-Teaming of Policy-Adherent Agents
Journal:
arXiv
Published Date:
Jun 11, 2025
Abstract
Task-oriented LLM-based agents are increasingly used in domains with strict
policies, such as refund eligibility or cancellation rules. The challenge lies
in ensuring that the agent consistently adheres to these rules and policies,
appropriately refusing any request that would violate them, while still
maintaining a helpful and natural interaction. This calls for the development
of tailored design and evaluation methodologies to ensure agent resilience
against malicious user behavior. We propose a novel threat model that focuses
on adversarial users aiming to exploit policy-adherent agents for personal
benefit. To address this, we present CRAFT, a multi-agent red-teaming system
that leverages policy-aware persuasive strategies to undermine a
policy-adherent agent in a customer-service scenario, outperforming
conventional jailbreak methods such as DAN prompts, emotional manipulation, and
coercive. Building upon the existing tau-bench benchmark, we introduce
tau-break, a complementary benchmark designed to rigorously assess the agent's
robustness against manipulative user behavior. Finally, we evaluate several
straightforward yet effective defense strategies. While these measures provide
some protection, they fall short, highlighting the need for stronger,
research-driven safeguards to protect policy-adherent agents from adversarial
attacks
