In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single langResearch goal: What is the impact of multimodal intermediate-task training on zero-shot cross-lingual transfer performance for language models when evaluated on XNAT, compared to monolingual text-only intermediate training?Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 7.5/10.