The locally linear embedding (LLE) algorithm has been proven to have high output quality and applicability for voice conversion (VC) tasks. However, the major shortcoming of the LLE-based VC approach is the time complexity (especially in the matrix inversion process) during the conversion phase. In this paper, we propose a fast version of the LLE algorithm that significantly reduces the complexity. In the proposed method, each locally linear patch on the data manifold is described by a pre-computed cluster of exemplars, and thus the major part of on-line computation can be carried out beforehand in the off-line phase. Experimental results demonstrate that the VC performance of the proposed fast LLE algorithm is comparable to that of the original LLE algorithm and that a real-time VC system becomes possible because of the highly reduced time complexity.