Efficient reinforcement learning for urban drainage control via a neural network-based model using truncated and parallel rollouts.

Reinforcement learning (RL) has been increasingly applied to train real-time control agents for urban drainage networks, yet its training is computationally expensive, due to extensive rollout simulations required to generate control experiences. While fast surrogate models can be utilized to accelerate RL training, errors accumulate over multiple auto-regressive prediction steps and introduce bias into the experience data used for RL training, resulting in unreliable control policies. Here, we propose a truncated and parallel rollout framework for efficient and reliable RL training using a neural network-based model (NNM), which performs rollout simulations in limited horizons to contain error accumulation and generate control experiences in parallel. The proposed rollout framework was demonstrated in NNM-based RL of two sewer networks to develop control agents, with their training efficiency and control performance benchmarked against RL supported by a physics-based model (PBM). We found that NNM-based RL preserved reliable training to produce control policies, which remained effective under PBM evaluations and achieved comparable control performance to PBM-based RL, while reducing the agent training time by over 30-fold with a GPU-enabled parallel computing workflow after the NNM was developed. The truncated rollout horizon of NNM-based RL was carefully selected to balance compounding model errors with delayed hydraulic responses. We demonstrated that the truncated and parallel rollout framework makes RL more feasible and cost-effective for real-time control of catchment-scale drainage networks.