More than 25% of human infectious diseases are vector-borne diseases (VBDs). These diseases, caused by pathogens shared between animals and humans, are a growing threat to global health with more than 2.5 million annual deaths. Mosquitoes and ticks are the main vectors of arboviruses including flaviviruses, which greatly affect humans. However, all tick or mosquito species are not able to transmit all viruses, suggesting important molecular mechanisms regulating viral infection, dissemination, and transmission by vectors. Despite the large distribution of arthropods (mosquitoes and ticks) and arboviruses, only a few pairings of arthropods (family, genus, and population) and viruses (family, genus, and genotype) successfully transmit. Here, we review the factors that might limit pathogen transmission: internal (vector genetics, immune responses, microbiome including insect-specific viruses, and coinfections) and external, either biotic (adult and larvae nutrition) or abiotic (temperature, chemicals, and altitude). This review will demonstrate the dynamic nature and complexity of virus–vector interactions to help in designing appropriate practices in surveillance and prevention to reduce VBD threats.