High-throughput "omics" data, including genomic, transcriptomic, and epigenetic data, have become increasingly produced and have contributed in recent years to the advances in cancer research. In particular, multimodal omics data get now employed in addition to clinical data to stratify patients according to their clinical outcomes. Despite some recent work on benchmarking multi-modal integration strategies for cancer survival prediction, there is still a need for the standardization of the results of model performances and for the consecutive exploration of the relative performance of statistical and deep learning models. Here, we propose a unique benchmark, SurvBoard, which standardizes several important experimental design choices to enable comparability between cancer survival models that incorporate multi-omics data. By designing several benchmarking scenarios, SurvBoard allows for the comparison of single-cancer models and models trained on pan-cancer data; SurvBoard also makes it possible to investigate the added value of using patient data with missing modalities. Additionally, in this work, we point out several potential pitfalls that might arise during the preprocessing and validation of multi-omics cancer survival models and address them in our benchmark. We compare statistical and deep learning models revealing that statistical models often outperform deep learning models, particularly in terms of model calibration. Finally, we offer a web service that enables quick model evaluation against our benchmark (https://www.survboard.science/). All code and other resources are available on GitHub: https://github.com/BoevaLab/survboard/.