Secret image sharing has been widely applied in numerous areas, such as military imaging systems, remote sensing, and so on. One of the problems for image sharing schemes is to efficiently recover original images from their shares preserved by the shareholders. However, most of the existing schemes are based on the assumption that the shares are distortion-free. Moreover, the correspondence between secret images and their shares is definite. To overcome these shortcomings, we propose a novel secret sharing scheme using multiple share images based on the generalized Chinese remainder theorem (CRT) in this paper, where all of the shares are needed to recover the original images. Two categories of distortions are considered. In the first category, some pairs of shares with the same moduli are exchanged, while in the second category, some of pixels in the pairs of shares with the same moduli are exchanged. Based on these two sharing methods, we propose a generalized CRT based recovery method. Compared with the existing CRT based methods as well as combinatorial based methods, the proposed approach is much more efficient and secure. Furthermore, the conditions for successful recovery of two images from the given distorted shares are obtained. Simulations are also presented to show the efficiency of the proposed scheme.