Generalization in a two-layer neural network with multiple outputs

Abstract

We study generalization in a fully connected two-layer neural network with multiple output nodes. Similar to the learning of fully connected committee machine, the learning is characterized by a discontinuous phase transition between the permutation symmetric phase and the permutation symmetry breaking phase. We find that the learning curve in the permutation symmetric phase is universal, irrespective of the number of output nodes. The first-order phase transition point, i.e., the critical number of examples required for perfect learning, is inversely proportional to the number of outputs. The replica calculation shows good agreement with Monte Carlo simulation.