On-line learning of a mixture-of-experts neural network

Abstract

The on-line learning of a mixture-of-experts system is studied in the framework of statistical physics. The time dependence of the overlap-order parameters during training is calculated analytically in the thermodynamic limit. When the number of training examples is small each expert is in a symmetric state. As the number of time steps approaches a critical point, the symmetric state begins to disintegrate. This symmetry-breaking behaviour is accounted for by means of a gating network. In the symmetric state the gating network has little effect on the learning, but when the symmetry is broken the gating network assigns the experts to appropriate subspaces in the input space. A generalization curve shows a plateau between the symmetric- and broken-symmetry states. We also find that the learning curves show different behaviours depending on the stiffness of the gating function.