Improving robustness by action correction via multi-step maximum risk estimation.
Journal:
Neural networks : the official journal of the International Neural Network Society
Published Date:
Dec 27, 2024
Abstract
Certifying robustness against external uncertainties throughout the control process to reduce the risk of instability is very important. Most existing approaches based on adversarial learning use a fixed parameter to adjust the intensity of adversarial perturbations and design these perturbations in a greedy manner without considering future implications. However, they often lead to severe vulnerabilities when attack budgets vary dynamically or under foresighted attacks. To address the problem, an algorithm that optimizes reinforcement learning policies by extending the focus beyond fixed-strength greedy attacks and estimating the maximum risk value, referred to as Multi-Step Maximum Risk-Aware Robust Deep Reinforcement Learning (MMRAR-RL), is proposed. MMRAR-RL operates in two stages: risk assessment and policy improvement. In the risk assessment stage, the adversary adaptively allocates the attack budget based on the agent's potential subsequent trajectories, planning multi-step perturbations to craft more powerful attacks. MMRAR-RL defines the multi-step perturbation value loss under these dynamic budgets as the difference between the original action-value function and the expected cumulative discounted returns under disturbances. The multi-step perturbation value loss characterizes action risk and directly estimates the maximum risk of a policy through a novel maximal risk Bellman operator. In the policy improvement stage, MMRAR-RL updates the policy based on the maximum action risk value function and a nominal loss function, thereby enhancing robustness against dynamic and foresighted attacks. Experiments demonstrate that MMRAR-RL achieves state-of-the-art performance under strong adversarial conditions, effectively tolerating action perturbations.