On-line learning from finite training sets

¹ Department of Physics, University of Edinburgh - Edinburgh EH9 3JZ, UK
² Neural Computing Research Group, Aston University - Birmingham B4 7ET, UK

Received: 23 January 1997
Accepted: 3 April 1997

Abstract

We analyse on-line (gradient descent) learning of a rule from a finite set of training examples at non-infinitesimal learning rates η, calculating exactly the time-dependent generalization error for a simple model scenario. In the thermodynamic limit, we close the dynamical equation for the generating function of an infinite hierarchy of order parameters using “within-sample self-averaging”. The resulting dynamics is non-perturbative in η, with a slow mode appearing only above a finite threshold . Optimal settings of η for given final learning time are determined and the results are compared with offline gradient descent.