In this paper, we analyse mathematical models for the interaction between virus replication and immune responses. We show that the immune system can provide selection pressure for or against viral diversity. The paper provides new insights into the relationship between virus load (=the abundance of virus in an infected individual) and antigenic diversity. Antigenic variation can increase virus load during infections, but the correlation between load and diversity in comparisons among different infected individuals can be positive or negative, depending on whether individuals differ in their cross-reactive or strain-specific immune responses. We derive two models: our first model applies to any replicating parasite that can escape from immune responses; our second model includes immune function impairment, and specifically describes infections with the human immunodeficiency virus (HIV).