publication . Conference object . Preprint . Article . 2017

Equivalence of Equilibrium Propagation and Recurrent Backpropagation

Benjamin Scellier; Yoshua Bengio;
Open Access
  • Published: 22 Nov 2017
  • Publisher: Cognitive Computational Neuroscience
Abstract
<jats:p> Recurrent backpropagation and equilibrium propagation are supervised learning algorithms for fixed-point recurrent neural networks, which differ in their second phase. In the first phase, both algorithms converge to a fixed point that corresponds to the configuration where the prediction is made. In the second phase, equilibrium propagation relaxes to another nearby fixed point corresponding to smaller prediction error, whereas recurrent backpropagation uses a side network to compute error derivatives iteratively. In this work, we establish a close connection between these two algorithms. We show that at every moment in the second phase, the temporal de...
Subjects
arXiv: Computer Science::Neural and Evolutionary Computation
free text keywords: Computer Science - Learning, Arts and Humanities (miscellaneous), Cognitive Neuroscience, Equivalence (measure theory), Backpropagation, Applied mathematics, Computer science, Supervised training, Recurrent neural network, Fixed point, Mean squared prediction error, Artificial neural network, Algorithm, Computation

L. B. Almeida. A learning rule for asynchronous perceptrons with feedback in a combinatorial environment. volume 2, pages 609-618, San Diego 1987, 1987. IEEE, New York.

F. Crick. The recent excitement about neural networks. Nature, 337(6203):129-132, 1989. [OpenAIRE]

G. E. Hinton and J. L. McClelland. Learning representations by recirculation. In D. Z. Anderson, editor, Neural Information Processing Systems, pages 358-366. American Institute of Physics, 1988.

J. J. Hopfield. Neurons with graded responses have collective computational properties like those of two-state neurons. 81, 1984. [OpenAIRE]

F. J. Pineda. Generalization of back-propagation to recurrent neural networks. 59:2229-2232, 1987. [OpenAIRE]

B. Scellier and Y. Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11, 2017. [OpenAIRE]

Any information missing or wrong?Report an Issue