Stabilizing iterative pruning with local adjustments and global scaling learning rate.

Iterative pruning, while effective for reducing model complexity, often introduces instability into the importance estimation of weights. This instability arises from the structural imbalance created during pruning: pruned weights are typically zeroed out or masked, while unpruned weights retain their previously optimized states. Such imbalance leads to fluctuating importance scores across iterations, rendering pruning decisions unreliable and hampering retraining efficiency. To address this challenge, we propose LAGA (Local Adjustments and Global Scaling), a dynamic learning rate adaptation strategy designed to stabilize importance estimation in iterative pruning. LAGA adjusts the learning rate of each weight according to its pruning status across two dimensions: locally, it smooths temporal sensitivity signals and compensates reactivation dynamics; globally, it scales the learning rate in proportion to model sparsity to account for structural compression. These mechanisms collectively rebalance training dynamics and recover stable importance trajectories during pruning. Extensive experiments demonstrate that LAGA significantly improves both the stability of importance evaluation and the performance of pruned models. On CIFAR-100, pruning ViT-B/16 by 60% yields a 4.71% higher Top-1 accuracy than when using AdamW. On CIFAR-100, pruning DeiT-S by 40% yields a 0.68% Top-1 improvement, surpassing the unpruned baseline. Consistent gains across ViT-B/16, DeiT-S, and Swin-B on CIFAR-100 and ImageNet-1K confirm the robustness and generalizability of LAGA across diverse pruning scenarios.