Silicon nitride thin films were prepared by reactive sputtering from different sputtering targets and using a range of Ar/N2 sputtering gas mixtures. The hardness and the Young’s modulus of the samples were determined by nanoindentation measurements. Depending on the preparation parameters, the obtained values were in the ranges 8–23 and 100–210 GPa, respectively. Additionally, Fourier-transform infrared spectroscopy, Rutherford backscattering spectroscopy, and x-ray diffraction were used to characterize samples with respect to different types of bonding, atomic concentrations, and structure of the films to explain the variation of mechanical properties. The hardness and Young’s modulus were determined as a function of film composition and structure and conditions giving the hardest film were found. Additionally, a model that assumes a series coupling of the elastic components, corresponding to the Si–O and Si–N bonds present in the sample has been proposed to explain the observed variations of hardness and Young’s modulus.