On-line learning from a finite training set: A solvable model

¹ Institut für Theoretische Physik, Julius-Maximilians-Universität Würzburg D-97074 Würzburg, Germany
² Neural Computing Research Group, Aston University - B4 7ET Birmingham, UK

Received: 24 June 1999
Accepted: 12 November 1999

Abstract

We discuss the problem of on-line learning from a finite training set with feedforward neural networks. Defining a modified learning rule, which randomly chooses inputs and weights to be updated, the dynamics of learning can be treated within a diffusion approximation in the thermodynamic limit. No assumption on the generation of data is made. Explicit results for the stationary distribution and relaxation times can be found for a network with linear transfer function. Assuming self-averaging of the diffusion term, a general relation between on-line learning and batch learning with an effective temperature is established.