Collective mutual information maximization to unify passive and positive approaches for improving interpretation and generalization.

The present paper aims to propose a simple method to realize mutual information maximization for better interpretation and generalization. To train neural networks and obtain better performance, neurons should impartially consider as many input patterns as possible. Simultaneously, and especially for ease of interpretation, they should represent characteristics specific to certain input patterns as faithfully as possible. This contradiction can be solved by introducing mutual information between neurons and input patterns. However, because of the complicated computational procedures associated with mutual information maximization, it has been difficult to apply mutual information maximization to actual problems. Though many simplified methods have been developed so far, they have not necessarily been applied with success, in particular to large-scale practical problems. To further aid simplification, here we propose a new computational method to realize mutual information. One of the main characteristics of this new method is the consideration of multiple neural networks when defining mutual information, thereby simplifying the method. In addition, learning is also simplified by using the indirect, independent, and fast learning of the potential method. This method was applied to two well known data sets: the Australian credit data and on-line popularity data set. The experimental results showed that mutual information could be increased via the present method. In addition, mutual information maximization was accompanied by an increase in generalization and interpretation performance, mainly due to the simple internal representations.