publication . Other literature type . Preprint . Conference object . 2016

Neural Semantic Encoders.

Tsendsuren Munkhdalai; Hong Yu;
Open Access English
  • Published: 14 Jul 2016
Abstract
We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated...
Subjects
free text keywords: Article, Computer Science - Learning, Computer Science - Computation and Language, Statistics - Machine Learning, Artificial intelligence, business.industry, business, Natural language processing, computer.software_genre, computer, Computer science, Encoder
Communities
Digital Humanities and Cultural Heritage
36 references, page 1 of 3

[1] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.

[2] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.

[3] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.

[4] Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a foreign language. In NIPS, 2015a.

[5] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.

[6] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In ICML 2015, 2015.

[7] Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In NIPS 2015, pages 1819-1827, 2015. [OpenAIRE]

[8] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.

[9] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In NIPS 2015, pages 2431-2439, 2015. [OpenAIRE]

[10] Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. CoRR, abs/1506.07285, 2016.

[11] Scott Reed and Nando de Freitas. Neural programmer-interpreters. In ICLR 2016, 2016.

[12] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.

[13] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-1543, 2014.

[14] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015.

[15] Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. CoRR, abs/1603.06021, 2016.

36 references, page 1 of 3
Abstract
We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE can also access multiple and shared memories. In this paper, we demonstrated the effectiveness and the flexibility of NSE on five different natural language tasks: natural language inference, question answering, sentence classification, document sentiment analysis and machine translation where NSE achieved state-of-the-art performance when evaluated...
Subjects
free text keywords: Article, Computer Science - Learning, Computer Science - Computation and Language, Statistics - Machine Learning, Artificial intelligence, business.industry, business, Natural language processing, computer.software_genre, computer, Computer science, Encoder
Communities
Digital Humanities and Cultural Heritage
36 references, page 1 of 3

[1] Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990.

[2] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.

[3] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.

[4] Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. Grammar as a foreign language. In NIPS, 2015a.

[5] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.

[6] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In ICML 2015, 2015.

[7] Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. Learning to transduce with unbounded memory. In NIPS 2015, pages 1819-1827, 2015. [OpenAIRE]

[8] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.

[9] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In NIPS 2015, pages 2431-2439, 2015. [OpenAIRE]

[10] Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. CoRR, abs/1506.07285, 2016.

[11] Scott Reed and Nando de Freitas. Neural programmer-interpreters. In ICLR 2016, 2016.

[12] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.

[13] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-1543, 2014.

[14] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015.

[15] Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. A fast unified model for parsing and sentence understanding. CoRR, abs/1603.06021, 2016.

36 references, page 1 of 3
Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue
publication . Other literature type . Preprint . Conference object . 2016

Neural Semantic Encoders.

Tsendsuren Munkhdalai; Hong Yu;