publication . Part of book or chapter of book . Preprint . 2016

alternating optimization method based on nonnegative matrix factorizations for deep neural networks

Sakurai, Tetsuya; Imakura, Akira; Inoue, Yuto; Futamura, Yasunori;
Open Access
  • Published: 15 May 2016
  • Publisher: Springer International Publishing
Abstract
The backpropagation algorithm for calculating gradients has been widely used in computation of weights for deep neural networks (DNNs). This method requires derivatives of objective functions and has some difficulties finding appropriate parameters such as learning rate. In this paper, we propose a novel approach for computing weight matrices of fully-connected DNNs by using two types of semi-nonnegative matrix factorizations (semi-NMFs). In this method, optimization processes are performed by calculating weight matrices alternately, and backpropagation (BP) is not used. We also present a method to calculate stacked autoencoder using a NMF. The output results of...
Subjects
arXiv: Computer Science::Neural and Evolutionary Computation
free text keywords: Nonnegative matrix, Matrix (mathematics), Pattern recognition, Non-negative matrix factorization, Computation, Backpropagation, Artificial intelligence, business.industry, business, Deep neural networks, Autoencoder, Computer science, Computer Science - Learning, Computer Science - Neural and Evolutionary Computing, Statistics - Machine Learning
Related Organizations
Download fromView all 2 versions
http://arxiv.org/pdf/1605.0463...
Part of book or chapter of book
Provider: UnpayWall
http://link.springer.com/conte...
Part of book or chapter of book
Provider: Crossref
17 references, page 1 of 2

1. Bengio,Y., Lamblin, P., Popovici, D., and Larochelle, H.: Greedy layer-wisetraining of deep networks. In Proc. Advances in Neural Information Processing Systems 19 153-160 (2006).

2. Ciresan, D.C., Meier, U., Masci, J., Gambardella, L.M., and Schmidhuber, J.: Flexible, high performance convolutional neural networks for image classification, Proc. 22nd International Joint Conference on Artificial Intelligence, 1237-1242 (2011).

3. Ding, D., Li, T., and Jordan, M. I.: Convex and semi-nonnegative matrix factorizations. IEEE Transactions on Pattern Analysis and Machine Intelligence 32, 45-55 (2010).

4. Glorot, X. and Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks, in International conference on artificial intelligence and statistics, 249-256 (2010).

5. Glorot, X., Bordes, A., and Bengio., Y.: Deep sparse rectifier neural networks. In Proc.14th International Conference on Artificial Intelligence and Statistics 315-323 (2011). [OpenAIRE]

6. Hinton, G.E., Deng, L., Yu, D., Dahl, G.E., Mohamed, A., Jaitly, N., Senior, A., and Vanhoucke, V.: Deep neural networks for acoustic modeling in speech recognition, IEEE Signal Processing Magazine 29,82-97 (2012).

7. Kingma, D. P. and Ba, J.: ADAM: A Method for Stochastic Optimization, The International Conference on Learning Representations (ICLR), San Diego, 2015

8. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 1 7 (2009).

9. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., and Jackel, L.D.: Backpropagation applied to handwritten zip code recognition, Neural Computation 1, 541-551 (1989). [OpenAIRE]

10. LeCun, Y.: The MNIST database of handwritten digits, http://yann.lecun.com/exdb/mnist.

11. Lee, D.D., and Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401, 788-791 (1999).

12. LeCun, Y., Bottou, L., Bengio, Y., Huffier, P.: Gradient-based learning applied to document recognition, In Proc. IEEE 86, 2278-2324 (1998).

13. Nair, V., and Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines, In Proc. ICML (2010).

14. Paatero, P., and Tapper, U.: Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5 111-126 (1994). [OpenAIRE]

15. Rumelhart, D. E., Hinton, G. E., and Williams, R. J.: Learning representations by back-propagating errors. Nature 323, 533-536 (1986). [OpenAIRE]

17 references, page 1 of 2
Powered by OpenAIRE Research Graph
Any information missing or wrong?Report an Issue