AbstractThis paper presents a novel modeling and compensation approach for the angular transmission error in harmonic drive gearings. In the modeling, physical phenomena of the transmission error due to nonlinear elastic deformations in micro‐displacement region are especially dealt with, as well as the synchronous component which has been discussed in a variety of conventional studies. On the basis of the analyses of the phenomena, the nonlinear elastic component is mathematically modeled by applying a modeling framework for the rolling friction with hysteresis attributes. The proposed transmission error model has been adopted to the positioning system as a model‐based feedforward compensation manner. Experimental results using a prototype show the effectiveness of the proposed modeling and compensation. Copyright © 2009 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.