RNA modifications are critical to the functional diversity and regulatory complexity of the transcriptome. With increasing frequency, direct nanopore RNA sequencing is applied to identify RNA modifications de novo. Here, we directly compare the MS2 phage genome RNA modification profiles determined using nanopore to orthogonal LC-MS/MS assays. The results reveal very different views of the modification landscape, suggesting caution when calling new RNA modifications using nanopore alone.