Abstract In recent years, synthesizing inorganic nanostructures such as boron nitride nanotubes (BNNTs) has led to extensive studies on their exceptional properties. In this study, the torsional vibration behavior of boron-nitride nanotubes (BNNTs) is explored on the basis of molecular dynamics (MD) simulation. The results show that the torsional frequency is sensitive to geometrical parameters such as length and boundary conditions. The axial vibration is found to be induced by torsional vibration of nanotubes which can cause instability in the nanostructure. It is also observed that the torsional frequency of BNNTs is higher than that of their carbon counterpart. Moreover, the shear modulus is predicted by incorporating MD simulation numerical results into torsional vibration frequency obtained through continuum-based model of tubes. Finally, it is seen that the torsional frequency of double-walled boron-nitride nanotubes (DWBNNTs) is between the frequencies of their constituent inner and outer tubes.