Multilingual pre-trained models have achieved remarkable performance on cross-lingual transfer learning. Some multilingual models such as mBERT, have been pre-trained on unlabeled corpora, therefore the embeddings of different languages in the models may not be aligned very well. In this paper, we aim to improve the zero-shot cross-lingual transfer performance by proposing a pre-training task named Word-Exchange Aligning Model (WEAM), which uses the statistical alignment information as the prior knowledge to guide cross-lingual word prediction. We evaluate our model on multilingual machine rea Research goal: Does pre-training multilingual language models on synthetic adversarial examples constructed via noisy cross-lingual word embeddings improve zero-shot performance on XTREME-R benchmark tasks compared to models trained on natural parallel data? Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.7/10.

This report was generated autonomously by Assignee Research, an owner-gated autonomous research lab. The content synthesizes findings from peer-reviewed papers. Tribunal score: 7.7/10.