<script type="text/javascript">
<!--
document.write('<div id="oa_widget"></div>');
document.write('<script type="text/javascript" src="https://www.openaire.eu/index.php?option=com_openaire&view=widget&format=raw&projectId=undefined&type=result"></script>');
-->
</script>

COPY SCRIPT

For further information contact us at helpdesk@openaire.eu

Attention in Natural Language Processing

descriptionPublicationkeyboard_double_arrow_right Article , Preprint 01 Oct 2021Embargo end date: 01 Jan 2019 Italy Publisher:Institute of Electrical and Electronics Engineers (IEEE)Journal:IEEE Transactions on Neural Networks and Learning Systems, volume 32, pages 4,291-4,308 (issn: 2162-237X, eissn: 2162-2388,

Authors: Andrea Galassi; Marco Lippi; Paolo Torroni;

doi: 10.1109/tnnls.2020.3019893 , 10.48550/arxiv.1902.02181

pmid: 32915750

arXiv: http://arxiv.org/abs/1902.02181

handle: 11380/1215137 , 11585/663866 , 2158/1356460

Attention in Natural Language Processing

- Summary
- Subjects
- Metrics

Abstract

Attention is an increasingly popular mechanism used in a wide range of neural architectures. The mechanism itself has been realized in a variety of formats. However, because of the fast-paced advances in this domain, a systematic overview of attention is still missing. In this article, we define a unified model for attention architectures in natural language processing, with a focus on those designed to work with vector representations of the textual data. We propose a taxonomy of attention models according to four dimensions: the representation of the input, the compatibility function, the distribution function, and the multiplicity of the input and/or output. We present the examples of how prior information can be exploited in attention models and discuss ongoing research efforts and open challenges in the area, providing the first extensive categorization of the vast body of literature in this exciting domain.

18 pages, 8 figures

Country

Italy

Related Organizations

University of Modena and Reggio Emilia
Italy
Alma Mater Studiorum University of Bologna
Italy
University of Florence
Italy

Keywords

I.2, FOS: Computer and information sciences, Computer Science - Machine Learning, Computer Science - Computation and Language, Natural language processing (NLP); neural attention; neural networks; review; survey, Computation and Language Computer Science Artificial Intelligence Learning Machine Learning Natural language processing (NLP) neural attention neural networks review survey, Computer Science - Artificial Intelligence, Natural language processing (NLP); neural attention; neural networks; review; survey;, Machine Learning (stat.ML), I.7, Machine Learning (cs.LG), I.2; I.7, Artificial Intelligence (cs.AI), Statistics - Machine Learning, 68T50, 68T05, 68T07, Computation and Language (cs.CL)

Impact byBIP!

	citations This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).	343
	popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.	Top 0.1%
	influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).	Top 1%
	impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.	Top 0.1%