Rote counting skills have found to be a strong predictor of later arithmetic and reading fluency. However, knowledge of the underlying cognitive factors influencing counting skill is very limited. Present study examined to what extent language skills (phonology, vocabulary, and morphology), nonverbal reasoning skills, and memory at the age of five could explain counting skill at the beginning of first grade. Gender, parents’ education level and child’s persistence were included as control variables. The question was examined in a longitudinal sample (N = 101) with a structural equation model. Results showed that language skills together with memory, nonverbal reasoning skills and parent’s education explained only 22% of the variance in counting at the beginning of the first grade. Vocabulary, morphology, and verbal short-term memory were found to be interchangeable predictors, each explaining approximately 7%–9%, of counting skill. These findings challenge the interpretation of counting as a strongly language-based number skill. However, additional analysis among children with dyslexia revealed that memory and language skills, together with a child’s persistence and gender, had a rather strong predictive value, explaining 34%–46% of counting skill. Together these results suggest that verbal short-term memory and language skills at the age of five have not the same predictive value on counting skill at the beginning of school among a population-based sample as found in subjects with language impairment or learning difficulties, and thus, other cognitive factors should be taken into account in further research related to typical development of counting skill.