The home learning environment (HLE) has been considered to contribute to children’s early math and reading development. Previous studies examined the HLE by examining the influence of parent-child math and reading activities on math and reading outcomes, however also parents’ own perceptions of math and reading and their math anxiety (MA) and reading anxiety (RA) contribute to the HLE but the latter factors have been scarcely explored. The aim of this study was to provide a more holistic view of the HLE and its relations with children’s cognitive and non-cognitive outcomes in math and reading at the start of primary school. This paper examined the relations within the HLE, and the relations between the HLE and children’s early math and reading outcomes. Participants were 301 first-grade children and their parents. The HLE was measured by the parent questionnaire. Children’s digit comparison, number line estimation, letter knowledge and phonological awareness skills were measured as well as their math and reading anxiety levels. The results demonstrated a significant association between parents’ perceptions and their anxiety towards math and reading. No significant associations were found between parents’ perceptions towards math and the frequency of home numeracy activities, whereas significant relations were found in the domain of reading. Socioeconomic status was found to provide a unique contribution in children’s digit comparison and math anxiety, while no significant relations were observed between other HLE factors and children’s outcomes. The current study suggests the importance of including parents’ perceptions and feelings to explore the dynamics of the HLE and its impact on children’s math and reading outcomes.