Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy
Journal:
arXiv
Published Date:
Feb 17, 2025
Abstract
Model merging is a widespread technology in large language models (LLMs) that
integrates multiple task-specific LLMs into a unified one, enabling the merged
model to inherit the specialized capabilities of these LLMs. Most task-specific
LLMs are sourced from open-source communities and have not undergone rigorous
auditing, potentially imposing risks in model merging. This paper highlights an
overlooked privacy risk: \textit{an unsafe model could compromise the privacy
of other LLMs involved in the model merging.} Specifically, we propose PhiMM, a
privacy attack approach that trains a phishing model capable of stealing
privacy using a crafted privacy phishing instruction dataset. Furthermore, we
introduce a novel model cloaking method that mimics a specialized capability to
conceal attack intent, luring users into merging the phishing model. Once
victims merge the phishing model, the attacker can extract personally
identifiable information (PII) or infer membership information (MI) by querying
the merged model with the phishing instruction. Experimental results show that
merging a phishing model increases the risk of privacy breaches. Compared to
the results before merging, PII leakage increased by 3.9\% and MI leakage
increased by 17.4\% on average. We release the code of PhiMM through a link.
