Abstract This paper theoretically analyzes the nonlinear torsional vibration of a granular material specimen in the resonant column test (RCT). The RCT is a material testing method that has been well established in geotechnical engineering. The test provides some important dynamic properties of granular materials including the shear wave speed, effective dynamic shear modulus and damping capacity. Both the elastic modulus and damping capacity show a strong dependence on the amplitude of excitation, which is the essential behavior of materials with hysteretic nonlinearity. The analysis, starting with a constitutive relationship that describes the hysteretic nonlinear behavior of granular materials, is performed specifically on the forced torsional vibration of a circular bar with a fixed and inertia-loaded boundary condition as a model for the specimen in the RCT. Self-interaction of a single resonance mode is assumed in the perturbation analysis to obtain an approximate solution for near-resonance response. The theoretical results predict linear dependences of the resonance frequency and damping ratio on the shear strain; these are in agreement with the existing experimental observations. Notable is that the present model describes the hysteretic nonlinear behaviors with a single parameter, i.e., the hysteresis nonlinearity parameter in the constitutive relationship. The derived equations are applied to a jointed gneiss specimen where how the hysteresis nonlinearity parameter is determined from RCT data is demonstrated.