Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank

Preprint English OPEN
Zhao, Liang; Liao, Siyu; Wang, Yanzhi; Li, Zhe; Tang, Jian; Pan, Victor; Yuan, Bo; (2017)
  • Subject: Computer Science - Computer Vision and Pattern Recognition | Statistics - Machine Learning | Computer Science - Learning
    acm: ComputingMethodologies_COMPUTERGRAPHICS | ComputingMethodologies_IMAGEPROCESSINGANDCOMPUTERVISION | GeneralLiterature_MISCELLANEOUS
    arxiv: Quantitative Biology::Neurons and Cognition | Computer Science::Neural and Evolutionary Computation

Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achi... View more
  • References (23)
    23 references, page 1 of 3

    [1] Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3):930-945, 1993.

    [2] Dario Bini, Victor Pan, and Wayne Eberly. Polynomial and matrix computations volume 1: Fundamental algorithms. SIAM Review, 38(1):161-164, 1996.

    [3] Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEEE International Conference on Computer Vision, pages 2857-2865, 2015.

    [4] George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems (MCSS), 2(4):303-314, 1989.

    [5] Olivier Delalleau and Yoshua Bengio. Shallow vs. deep sum-product networks. In Advances in Neural Information Processing Systems, pages 666-674, 2011.

    [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.

    [7] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, pages 1269-1277, 2014.

    [8] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.

    [9] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.

    [10] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. 4(2):251-257, 1991.

  • Metrics
    No metrics available
Share - Bookmark