publication . Article . Preprint . 2016

A topological insight into restricted Boltzmann machines

Mocanu, Decebal Constantin; Mocanu, Elena; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio;
Open Access English
  • Published: 20 Apr 2016 Journal: Machine Learning, volume 104, issue 2-3, pages 243-270 (issn: 0885-6125, eissn: 1573-0565, Copyright policy)
  • Publisher: Springer Nature
Abstract
Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on sy...
Subjects
arxiv: Computer Science::Neural and Evolutionary Computation
free text keywords: Computer Science - Artificial Intelligence, Computer Science - Social and Information Networks, Software, Computer Science - Neural and Evolutionary Computing, Artificial Intelligence
Related Organizations
Funded by
EC| INTER-IoT
Project
INTER-IoT
Interoperability of Heterogeneous IoT Platforms
  • Funder: European Commission (EC)
  • Project Code: 687283
  • Funding stream: H2020 | RIA
47 references, page 1 of 4

Ackley H, Hinton E, Sejnowski J (1985) A learning algorithm for boltzmann machines. Cognitive Science pp 147-169

Ammar HB, Mocanu DC, Taylor M, Driessens K, Tuyls K, Weiss G (2013) Automatically mapped transfer between reinforcement learning tasks via three-way restricted boltzmann machines. In: Blockeel H, Kersting K, Nijssen S, elezn F (eds) Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, vol 8189, Springer Berlin Heidelberg, pp 449-464, DOI 10.1007/978-3-642-40991-2 29 [OpenAIRE]

Ba J, Caruana R (2014) Do deep nets really need to be deep? In: Advances in Neural Information Processing Systems 27, pp 2654-2662

Barabasi AL, Albert R (1999) Emergence of scaling in random networks. Science 286(5439):509-512, DOI 10.1126/science.286.5439.509

Bengio Y (2009) Learning deep architectures for ai. Found Trends Mach Learn 2(1):1-127, DOI 10.1561/2200000006

Brgge K, Fischer A, Igel C (2013) The flip-the-state transition operator for restricted boltzmann machines. Machine Learning 93(1):53-69, DOI 10.1007/s10994-013-5390-3

Carreira-Perpinan MA, Hinton GE (2005) On contrastive divergence learning. In: 10th Int. Workshop on Artificial Intelligence and Statistics (AISTATS)

Clauset A, Shalizi CR, Newman MEJ (2009) Power-law distributions in empirical data. SIAM Review 51(4):661-703, DOI 10.1137/070710111

Del Genio CI, Gross T, Bassler KE (2011) All scale-free networks are sparse. Phys Rev Lett 107:178,701, DOI 10.1103/PhysRevLett.107.178701

Desjardins G, Courville A, Bengio Y, Vincent P, Delalleau O (2010) Tempered Markov Chain Monte Carlo for training of restricted Boltzmann machines. In: Teh YW, Titterington M (eds) Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, May 13-15, 2010, Chia Laguna Resort, Sardinia, Italy, pp 145-152

Dieleman S, Schrauwen B (2012) Accelerating sparse restricted boltzmann machine training using non-gaussianity measures. In: Bengio Y, Bergstra J, Le Q (eds) Deep Learning and Unsupervised Feature Learning, Proceedings, p 9 [OpenAIRE]

Gehler PV, Holub AD, Welling M (2006) The rate adapting poisson model for information retrieval and object recognition. In: Proceedings of the 23rd International Conference on Machine Learning, ACM, ICML '06, pp 337-344, DOI 10.1145/1143844.1143887 [OpenAIRE]

Germain M, Gregor K, Murray I, Larochelle H (2015) MADE: Masked Autoencoder for Distribution Estimation. In: Proceedings of the 32nd International Conference on Machine Learning, JMLR.org, JMLR Proceedings, vol 37, pp 881-889

Hagberg AA, Schult DA, Swart PJ (2008) Exploring network structure, dynamics, and function using NetworkX. In: Proceedings of the 7th Python in Science Conference (SciPy2008), Pasadena, CA USA, pp 11-15

Hakimi SL (1962) On realizability of a set of integers as degrees of the vertices of a linear graph. I. J Soc Indust Appl Math 10:496-506

47 references, page 1 of 4
Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue
publication . Article . Preprint . 2016

A topological insight into restricted Boltzmann machines

Mocanu, Decebal Constantin; Mocanu, Elena; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio;