publication . Contribution for newspaper or weekly magazine . Conference object . Preprint . 2016

Sequential neural models with stochastic layers

Marco Fraccaro; Søren Kaae Sønderby; Ulrich Paquet; Ole Winther;
Open Access English
  • Published: 24 May 2016
  • Country: Denmark
Abstract
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the sta...
Subjects
free text keywords: Statistics - Machine Learning, Computer Science - Learning
Related Organizations
28 references, page 1 of 2

[1] E. Archer, I. M. Park, L. Buesing, J. Cunningham, and L. Paninski. Black box variational inference for state space models. arXiv:1511.07367, 2015.

[2] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, and Y. Bengio. Theano: new features and speed improvements. arXiv:1211.5590, 2012.

[3] J. Bayer and C. Osendorfer. Learning stochastic recurrent networks. arXiv:1411.7610, 2014. [OpenAIRE]

[4] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent. Modeling temporal dependencies in highdimensional sequences: Application to polyphonic music generation and transcription. arXiv:1206.6392, 2012.

[5] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. arXiv:1511.06349, 2015.

[6] K. Cho, B. Van Merriënboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.

[7] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014. [OpenAIRE]

[8] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. In NIPS, pages 2962-2970, 2015.

[9] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39(1), 1977.

[10] S. Dieleman, J. Schlüter, C. Raffel, E. Olson, S. K. Sønderby, D. Nouri, E. Battenberg, and A. van den Oord. Lasagne: First release, 2015.

[11] A. Doucet, N. de Freitas, and N. Gordon. An introduction to sequential Monte Carlo methods. In Sequential Monte Carlo Methods in Practice, Statistics for Engineering and Information Science. 2001.

[12] O. Fabius and J. R. van Amersfoort. Variational recurrent auto-encoders. arXiv:1412.6581, 2014. [OpenAIRE]

[13] Z. Gan, C. Li, R. Henao, D. E. Carlson, and L. Carin. Deep temporal sigmoid belief networks for sequence modeling. In NIPS, pages 2458-2466, 2015.

[14] D. Geiger, T. Verma, and J. Pearl. Identifying independence in Bayesian networks. Networks, 20:507-534, 1990.

[15] K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: A recurrent neural network for image generation. In ICML, 2015. [OpenAIRE]

28 references, page 1 of 2
Abstract
How can we efficiently propagate uncertainty in a latent state representation with recurrent neural networks? This paper introduces stochastic recurrent neural networks which glue a deterministic recurrent neural network and a state space model together to form a stochastic and sequential neural generative model. The clear separation of deterministic and stochastic layers allows a structured variational inference network to track the factorization of the model's posterior distribution. By retaining both the nonlinear recursive structure of a recurrent neural network and averaging over the uncertainty in a latent path, like a state space model, we improve the sta...
Subjects
free text keywords: Statistics - Machine Learning, Computer Science - Learning
Related Organizations
28 references, page 1 of 2

[1] E. Archer, I. M. Park, L. Buesing, J. Cunningham, and L. Paninski. Black box variational inference for state space models. arXiv:1511.07367, 2015.

[2] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, and Y. Bengio. Theano: new features and speed improvements. arXiv:1211.5590, 2012.

[3] J. Bayer and C. Osendorfer. Learning stochastic recurrent networks. arXiv:1411.7610, 2014. [OpenAIRE]

[4] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent. Modeling temporal dependencies in highdimensional sequences: Application to polyphonic music generation and transcription. arXiv:1206.6392, 2012.

[5] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from a continuous space. arXiv:1511.06349, 2015.

[6] K. Cho, B. Van Merriënboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.

[7] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014. [OpenAIRE]

[8] J. Chung, K. Kastner, L. Dinh, K. Goel, A. C. Courville, and Y. Bengio. A recurrent latent variable model for sequential data. In NIPS, pages 2962-2970, 2015.

[9] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39(1), 1977.

[10] S. Dieleman, J. Schlüter, C. Raffel, E. Olson, S. K. Sønderby, D. Nouri, E. Battenberg, and A. van den Oord. Lasagne: First release, 2015.

[11] A. Doucet, N. de Freitas, and N. Gordon. An introduction to sequential Monte Carlo methods. In Sequential Monte Carlo Methods in Practice, Statistics for Engineering and Information Science. 2001.

[12] O. Fabius and J. R. van Amersfoort. Variational recurrent auto-encoders. arXiv:1412.6581, 2014. [OpenAIRE]

[13] Z. Gan, C. Li, R. Henao, D. E. Carlson, and L. Carin. Deep temporal sigmoid belief networks for sequence modeling. In NIPS, pages 2458-2466, 2015.

[14] D. Geiger, T. Verma, and J. Pearl. Identifying independence in Bayesian networks. Networks, 20:507-534, 1990.

[15] K. Gregor, I. Danihelka, A. Graves, and D. Wierstra. DRAW: A recurrent neural network for image generation. In ICML, 2015. [OpenAIRE]

28 references, page 1 of 2
Any information missing or wrong?Report an Issue