When rubber slides on a hard, rough substrate, the surface asperities of the substrate exert oscillating forces on the rubber surface leading to energy ``dissipation'' via the internal friction of the rubber. In this paper we extend an earlier published theory [B.N.J. Persson, J. Chem. Phys. $115,$ 3840 (2001)] to nonstationary sliding, and present a discussion of how the area of real contact and the friction force depend on the nature of the substrate surface roughness and on the history of the sliding motion. We consider in detail the case when the substrate surface has a self-affine fractal structure.