Viruses intricately interact with and modulate cellular membranes at several stages of their replication, but much less is known about the role of viral lipids compared to proteins and nucleic acids. All animal viruses have to cross membranes for cell entry and exit, which occurs by membrane fusion (in enveloped viruses), by transient local disruption of membrane integrity, or by cell lysis. Furthermore, many viruses interact with cellular membrane compartments during their replication and often induce cytoplasmic membrane structures, in which genome replication and assembly occurs. Recent studies revealed details of membrane interaction, membrane bending, fission, and fusion for a number of viruses and unraveled the lipid composition of raft-dependent and -independent viruses. Alterations of membrane lipid composition can block viral release and entry, and certain lipids act as fusion inhibitors, suggesting a potential as antiviral drugs. Here, we review viral interactions with cellular membranes important for virus entry, cytoplasmic genome replication, and virus egress.