Abstract The age at which a father reproduces is predicted to affect not only his own fertility but also the fitness of his offspring. Specifically, offspring born to old fathers are assumed to be of a lower quality than those conceived by young fathers. However, when fathers have low mating rates, paternal age might be confounded with the duration for which mature sperm are stored in fathers prior to ejaculation. Studies that disentangle the confounding effects of paternal sperm storage duration from those of paternal age, on offspring, are lacking. We use Drosophila melanogaster to test the separate and interactive effects of paternal age and sperm storage duration (sexual rest) on the survival and lifetime reproduction of offspring. As expected, old fathers produce fewer offspring than young fathers. But surprisingly, paternal age does not influence the survival or lifetime reproductive success of either sons or daughters. Instead, sons conceived by fathers with long durations of sexual rest have lower reproductive success than sons conceived by fathers with short durations of sexual rest. We further discover that daughters of low reproductive quality selectively disappear with age, but sons do not, highlighting that demographic processes need to be accounted for when studying paternal age effects. Overall, our study suggests that paternal age effects might not be as pervasive as previously assumed. We emphasize that a more nuanced understanding of the mechanisms causing paternal age effects is required, so that studies do not misattribute effects of paternal sperm storage to age.