publication . Other literature type . Conference object . 2019

Analysis of Positional Encodings for Neural Machine Translation

Rosendahl, Jan; Tran, Viet Anh Khoa; Wang, Weiyue; Ney, Hermann;
Open Access
  • Published: 02 Nov 2019
  • Publisher: Zenodo
Abstract
In this work we analyze and compare the behavior of the Transformer architecture when using different positional encoding methods. While absolute and relative positional encoding perform equally strong overall, we show that relative positional encoding is vastly superior (4.4% to 11.9% BLEU) when translating a sentence that is longer than any observed training sentence. We further propose and analyze variations of relative positional encoding and observe that the number of trainable parameters can be reduced without a performance loss, by using fixed encoding vectors or by removing some of the positional encoding vectors.
Subjects
Medical Subject Headings: mental disorderspsychological phenomena and processes
Download fromView all 3 versions
Zenodo
Other literature type . 2019
Provider: Datacite
Zenodo
Other literature type . 2019
Provider: Datacite
ZENODO
Conference object . 2019
Provider: ZENODO
Powered by OpenAIRE Research Graph
Any information missing or wrong?Report an Issue