Natural gradient matrix momentum

Part of book or chapter of book English OPEN
Scarpetta, Silvia ; Rattray, Magnus ; Saad, David (1999)
  • Publisher: IEEE
  • Subject:
    acm: MathematicsofComputing_NUMERICALANALYSIS

Natural gradient learning is an efficient and principled method for improving on-line learning. In practical applications there will be an increased cost required in estimating and inverting the Fisher information matrix. We propose to use the matrix momentum algorithm in order to carry out efficient inversion and study the efficacy of a single step estimation of the Fisher information matrix. We analyse the proposed algorithm in a two-layer network, using a statistical mechanics framework which allows us to describe analytically the learning dynamics, and compare performance with true natural gradient learning and standard gradient descent.
  • References (12)
    12 references, page 1 of 2

    [l] Saad D (ed) 1998 On-Line Learning in Neural Networks, Publications of the Newton Institute, Cambridge University Press, Cambridge.

    [2] Amari S 1998 'Natural gradient works efficiently in learning' Neural Comp. 10 251.

    [3] Rattray M and Saad D 1998 'Transient and Asymptotics od Natural Gradient Learning', Proc. of the Int. Conf. on Artificial Neural Networks, ed Niklasson, Bodbden and Ziemke (London, UK: Springer-Verlag) 165; Rattray M Saad D and Amari S 1998, Phys. Rev. Lett.,

    [4] Orr G B and Leen T K 1997 Advances in Neural Information Processing Systems, vol 9 , ed Mozer, Jordan and Petsche (Cambridge, MA: MIT Press) 606.

    (5) Rattray M and Saad D 1998 'The Dynamics of Matrix Momentum', Proc. of the Int. Conf. on Artificial Neural Networks, ed Niklasson, Bodbden and Ziemke (London, UK: Springer-Verlag) 183.

    [6] Saad D and Solla S A 1995 Phys. Rev. Lett. 74 4337, Phys. Rev. E 5 2 4225.

    [7] Yang H Y and Amari S 1997 Advances in Neural Information Processing Systems, vol 10, ed Mozer, Jordan and Petsche (Cambridge, MA: MIT Press) 385.

    [8] Cybenko Math. Control Signals and Systems 2 303.

    [9] Prugel-Bennett A 1996, unpublished notes.

    [lo] Saad D and Rattray M 1997 Phys. Rev. Lett. 79 2578; Rattray M and Saad D 1998 Phys. Rev. E 58 6379.

  • Metrics
    0
    views in OpenAIRE
    0
    views in local repository
    9
    downloads in local repository

    The information is available from the following content providers:

    From Number Of Views Number Of Downloads
    Aston Publications Explorer - IRUS-UK 0 9
Share - Bookmark