Beyond Natural Language Perplexity: Detecting Dead Code Poisoning in Code Generation Datasets
Journal:
arXiv
Published Date:
Feb 27, 2025
Abstract
The increasing adoption of large language models (LLMs) for code-related
tasks has raised concerns about the security of their training datasets. One
critical threat is dead code poisoning, where syntactically valid but
functionally redundant code is injected into training data to manipulate model
behavior. Such attacks can degrade the performance of neural code search
systems, leading to biased or insecure code suggestions. Existing detection
methods, such as token-level perplexity analysis, fail to effectively identify
dead code due to the structural and contextual characteristics of programming
languages. In this paper, we propose DePA (Dead Code Perplexity Analysis), a
novel line-level detection and cleansing method tailored to the structural
properties of code. DePA computes line-level perplexity by leveraging the
contextual relationships between code lines and identifies anomalous lines by
comparing their perplexity to the overall distribution within the file. Our
experiments on benchmark datasets demonstrate that DePA significantly
outperforms existing methods, achieving 0.14-0.19 improvement in detection
F1-score and a 44-65% increase in poisoned segment localization precision.
Furthermore, DePA enhances detection speed by 0.62-23x, making it practical for
large-scale dataset cleansing. Overall, by addressing the unique challenges of
dead code poisoning, DePA provides a robust and efficient solution for
safeguarding the integrity of code generation model training datasets.
