publication . Preprint . Conference object . 2017

Deep learning evaluation using deep linguistic processing

Alexander Kuhnle; Ann Copestake;
Open Access English
  • Published: 05 Jun 2017
Abstract
We discuss problems with the standard approaches to evaluation for tasks like visual question answering, and argue that artificial data can be used to address these as a complement to current practice. We demonstrate that with the help of existing 'deep' linguistic processing technology we are able to create challenging abstract datasets, which enable us to investigate the language understanding abilities of multimodal deep learning models in detail, as compared to a single performance value on a static and monolithic dataset.
Subjects
free text keywords: Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Computer Vision and Pattern Recognition, Computer Science - Learning, Natural language processing, computer.software_genre, computer, Artificial intelligence, business.industry, business, Computer science, Deep learning, Deep linguistic processing
Communities
Digital Humanities and Cultural Heritage
35 references, page 1 of 3

Adi, Y., E. Kermany, Y. Belinkov, O. Lavi, and Y. Goldberg (2017). Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. Proceedings of the International Conference on Learning Representations 2017 (ICLR 2017). [OpenAIRE]

Agrawal, A., D. Batra, and D. Parikh (2016). Analyzing the behavior of visual question answering models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, pp. 1955-1960.

Andreas, J., M. Rohrbach, T. Darrell, and D. Klein (2016). Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016.

Antol, S., A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015). VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, ICCV 2015.

Arthur, P., G. Neubig, and S. Nakamura (2016). Incorporating discrete translation lexicons into neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Austin, Texas, USA. [OpenAIRE]

Bellemare, M. G., Y. Naddaf, J. Veness, and M. Bowling (2013). The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research 47(1), 253-279. [OpenAIRE]

Brockman, G., V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba (2016). OpenAI Gym.

Copestake, A. (2009). Slacker semantics: Why superficiality, dependency and avoidance of commitment can be the right way to go. In Proceedings of the 12th Meeting of the European Chapter of the Association for Computational Linguistics, Athens, Greece, pp. 1-9.

Copestake, A., G. Emerson, M. W. Goodman, M. Horvat, A. Kuhnle, and E. Muszyn´ska (2016). Resources for building applications with Dependency Minimal Recursion Semantics. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC-16), Portorozˇ, Slovenia, pp. 1240-1247. European Language Resources Association (ELRA).

Copestake, A., D. Flickinger, C. Pollard, and I. A. Sag (2005). Minimal Recursion Semantics: An introduction. Research on Language and Computation 3(4), 281-332.

Flickinger, D. (2000). On building a more efficient grammar by exploiting types. Natural Language Engineering 6(1), 15-28.

Goyal, Y., T. Khot, D. Summers-Stay, D. Batra, and D. Parikh (2016). Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. CoRR abs/1612.00837.

Hodosh, M. and J. Hockenmaier (2016). Focused evaluation for image description with binary forcedchoice tasks. In Proceedings of the 5th Workshop on Vision and Language, Berlin, Germany. [OpenAIRE]

Hu, R., J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko (2017). Learning to reason: End-to-end module networks for visual question answering. ArXiv e-prints.

Jabri, A., A. Joulin, and L. van der Maaten (2016). Revisiting Visual Question Answering Baselines, pp. 727-739. Cham: Springer International Publishing.

35 references, page 1 of 3
Powered by OpenAIRE Open Research Graph
Any information missing or wrong?Report an Issue
publication . Preprint . Conference object . 2017

Deep learning evaluation using deep linguistic processing

Alexander Kuhnle; Ann Copestake;