In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case study. We assume a strict setting with no access to parallel data or machine translation and find that common transfer learning approaches struggle in this setting, as a generative multilingual model fine-tuned purely on English catastrophically forgets how to generate non-English. Given the recent rise of parameter-efficient adaptation techniques, we conduct the first investigation into how one such method, promptResearch goal: What is the impact of varying the size of the pre-trained multilingual model on zero-shot cross-lingual transfer performance in the XTREME-R benchmark when using intermediate-task training versus direct fine-tuning?Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.8/10.