Peptide-based materials hold great promise as immunotherapeutic agents for the treatment of many malignant cancers. Extensive studies have focused on the development of peptide-based cancer vaccines and delivery systems by mimicking the functional domains of proteins with highly specific immuno-regulatory functions or tumor cells fate controls. However, a systemic understanding of the interactions between the different peptides and immune systems remains unknown. This review describes the role of peptides in regulating the functions of the innate and adaptive immune systems and provides a comprehensive focus on the design, categories, and applications of peptide-based cancer vaccines. By elucidating the impacts of peptide length and formulations on their immunogenicity, peptide-based immunomodulating agents can be better utilized and dramatic breakthroughs may also be realized. Moreover, some critical challenges for translating peptides into large-scale synthesis, safe delivery, and efficient cancer immunotherapy are posed to improve the next-generation peptide-based immunotherapy.