Low-cost representation for restricted Boltzmann machines

Article English OPEN
Tran, S. ; Garcez, A. (2014)

This paper presents a method for extracting a low-cost representation from restricted Boltzmann machines. The new representation can be considered as a compression of the network, requiring much less storage capacity while reasonably preserving the network's performance at feature learning. We show that the compression can be done by converting the weight matrix of real numbers into a matrix of three values {-1, 0, 1} associated with a score vector of real numbers. This set of values is similar enough to Boolean values which help us further translate the representation into logical rules. In the experiments reported in this paper, we evaluate the performance of our compression method on image datasets, obtaining promising results. Experiments on the MNIST handwritten digit classification dataset, for example, have shown that a 95% saving in memory can be achieved with no significant drop in accuracy.
  • References (10)

    1. Geo rey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Comput., 14(8):1771{1800, August 2002.

    2. Geo rey E. Hinton, Simon Osindero, and Yee-Whye Teh. A Fast Learning Algorithm for Deep Belief Nets. Neural Comp., 18(7):1527{1554, July 2006.

    3. Hugo Larochelle and Yoshua Bengio. Classi cation using discriminative restricted boltzmann machines. In Proceedings of the 25th international conference on Machine learning, ICML '08, page 536543, New York, NY, USA, 2008. ACM.

    4. Honglak Lee, Chaitanya Ekanadham, and Andrew Y. Ng. Sparse deep belief net model for visual area v2. In Advances in Neural Information Processing Systems. MIT Press, 2008.

    5. Leo de Penning, Artur S. d'Avila Garcez, Lus C. Lamb, and John-Jules Ch Meyer. A neural-symbolic cognitive agent for online learning and reasoning. In IJCAI, pages 1653{1658, 2011.

    6. Gadi Pinkas. Reasoning, nonmonotonicity and learning in connectionist networks that capture propositional knowledge. Arti cial Intelligence, 77(2):203 { 247, 1995.

    7. Matthew Richardson and Pedro Domingos. Markov logic networks. Mach. Learn., 62(1-2):107136, February 2006.

    8. Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. In Parallel Distributed Processing: Volume 1: Foundations, pages 194{281. MIT Press, Cambridge, 1986.

    9. Son Tran and Artur Garcez. Logic extraction from deep belief networks. In ICML 2012 Representation Learning Workshop, Edinburgh, July 2012.

    10. Son N. Tran and Artur d'Avila Garcez. Knowledge extraction from deep belief networks for images. In IJCAI-2013 Workshop on Neural-Symbolic Learning and Reasoning, 2013.

  • Software (1)
  • Metrics
    No metrics available
Share - Bookmark