On the approximation capability of shallow and deep neural networks having smooth activations with respect to the Sobolev norm.

In this paper, we investigate the simultaneous approximation of the functions and their derivatives by neural networks having smooth non-polynomial activation functions and a fixed depth, which is motivated by the physics-informed machine learning. We start by proving that the neural networks with smooth non-polynomial activation functions and with only one hidden layer having width O(Nd) can approximate any Ws,p-regular function with rate O(Nk-s) in the Wk,p-norm. We then extend this result to the networks having more than one hidden layers by using the mathematical induction.