In this paper, we present a novel approach to motion estimation (ME) of arbitrarily shaped video objects in wavelet domain. We explore the guiding role of binary alpha-plane in assisting ME of video objects and first devise a new block matching scheme of alpha-plane, by exploiting boundary expansion and boundary masks. To eliminate shift-variance, we modify low-band-shift (LBS) method via substituting variable-size block for wavelet block. Combining the modified LBS with a hierarchical structure, we further present a multiscale ME approach. Extensive experiments show that the proposed approach outperforms most of previous methods in terms of both subjective quality and objective quality. Moreover, significant reduction is achieved in computational complexity (89.05% at most) and memory requirement.