Wood can store carbon and help mitigate global climate change. Carbon density (CD), the basis for measuring and analyzing C storage, is the product of wood density (WD) and C concentration, which are dependent on wood structure, cellulose concentration (CC), hemicellulose concentration (HC), and lignin concentration (LC). However, little attention has been paid to the C concentration of cellulose, hemicellulose, and lignin, which are fundamental factors in C storage and affect the credibility of accurate CD estimates. In order to disentangle the CD drives, WD, C concentration, CC, HC, and LC of the branch, stem, and root were quantified for five Rosaceae species from temperate forests in Northeastern China. The species were Sorbus alnifolia (Sieb.et Zucc.) K. Koch, Pyrus ussuriensis Maxim., Malus baccata (L.) Borkh., Crataegus pinnatifida var. major N. E. Brown, and Padus racemosa (Linn.) Gilib. The WD, CC, HC, and LC differed among species and tree organs, with the highest variability for the HC. The structural carbon concentration (SCC) was lower than the organic carbon concentration (OCC) and even the Intergovernmental Panel on Climate Change (IPCC) default value of 45%, with a maximum deviation of 2.6%. CD differed dramatically among species and tree organs. Based on SCC calculations, the highest CD was found in Sorbus alnifolia root (0.27 × 106 g/m3), while the lowest was found in Padus racemosa branch (0.22 × 106 g/m3). The results suggest that when estimating CD accurately at species level, it is important to consider not only WD but also structural carbohydrates and lignin concentration, providing important information on C fluxes and long-term C sequestration for forests. The study findings provide valuable insights into CD variations among tree species and organs and are valuable for forest management and policy development to improve carbon sequestration.