Volume 119, Number 6, September 2017
|Number of page(s)||7|
|Published online||29 November 2017|
Spectral dynamics of learning in restricted Boltzmann machines
1 LRI, AO team, Bât 660 Université Paris Sud - Orsay Cedex 91405, France
2 Inria Saclay - Tau team, Bât 660 Université Paris Sud - Orsay Cedex 91405, France
Received: 10 August 2017
Accepted: 30 October 2017
The restricted Boltzmann machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution. This let us show how in the linear regime, in which the RBM is found to operate at the beginning of the training, the statistical properties of the data drive the selection of the unstable modes of the weight matrix. A set of equations characterizing the non-linear regime is then derived, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the RBM.
PACS: 02.30.Zz – Inverse problems / 02.70.Hm – Spectral methods / 89.75.-k – Complex systems
© EPLA, 2017
Current usage metrics show cumulative count of Article Views (full-text article views including HTML views, PDF and ePub downloads, according to the available data) and Abstracts Views on Vision4Press platform.
Data correspond to usage on the plateform after 2015. The current usage metrics is available 48-96 hours after online publication and is updated daily on week days.
Initial download of the metrics may take a while.