This study focuses on parental reading and mathematical difficulties, the home literacy environment, and the home numeracy environment as well as their predictive role in Finnish children’s reading and mathematical development through Grades 1–9. We examined if parental reading and mathematical difficulties directly predict children’s academic performance and/or if they are mediated by the home learning environment. Mothers (n = 1590) and fathers (n = 1507) reported on their reading and mathematical difficulties as well as on the home environment (shared reading, teaching literacy, and numeracy) when their children were in kindergarten. Tests for reading fluency, reading comprehension, and arithmetic fluency were administered to children in Grades 1, 2, 3, 4, 7, and 9. Parental reading difficulties predicted children’s reading fluency, whereas parental mathematical difficulties predicted their reading comprehension and arithmetic fluency. Familial risk was associated with neither formal nor informal home environment factors, whereas maternal education had a significant relationship with both, with higher levels of education among mothers predicting less time spent on teaching activities and more time spent on shared reading. In addition, shared reading was significantly associated with the development of reading comprehension up to Grades 3 and 4, whereas other components of the home learning environment were not associated with any assessed skills. Our study highlights that taken together, familial risk, parental education, and the home learning environment form a complex pattern of associations with children’s mathematical and reading skills.