Native (L1) and to some extent non-native (L2) speakers have shown processing advantages for idioms compared to novel literal phrases, and there is limited evidence that this advantage also extends to memory in L1 children. This study investigated whether these advantages generalize to recognition memory in adults. It employed a learning paradigm to test whether there is a recognition memory advantage for idioms compared to literal phrases in adult L1 and L2 learners considering both form and meaning recognition. Additionally, we asked whether the presence of unfamiliar vocabulary interferes with phrasal learning by looking at recall of such unfamiliar words. When encountering new idioms, L2 learners often must cope with both figurative meaning and unfamiliar vocabulary. While single word meaning need not interfere with idiomatic meaning, it is a building block for the meaning of literal phrases. In Experiment 1, L2 learners showed equal recall for the form and meaning of literal and idiomatic phrases in which either all words were highly familiar, or one word was unfamiliar. However, unfamiliar words decreased overall recognition and were also remembered significantly better in literal compared to idiomatic phrases. In Experiment 2, L1 speakers also showed no recall differences between phrase types, but they displayed a trending increase in recognition in the presence of unfamiliar words. We conclude that there is no inherent recognition memory advantage for idioms based on figurativeness alone, and word- and phrasal meaning interact differently in learner groups.