Facial landmark detection is an important issue in many computer vision applications about faces. It is very challenging as human faces in wild conditions often present large variations in shape due to different poses, occlusions or expressions. Deep neural networks have been applied to learn the map from face images to face shapes. To the best of our knowledge, Recurrent Neural Network (RNN) has not been used in this issue yet. In this paper, we propose a method which utilizes RNN and Deep Neural Network (DNN) to learn the face shape. First, we build a global network using Long Short Term Memory (LSTM) architecture of RNN to get the initial landmark estimation of faces. Then, we use feed-forward neural networks for local search where a component-based searching method is explored. By using LSTM-RNN, the initial estimation is more reliable which makes the following component-based search feasible and accurate. Experiments show that the global network using LSTM-RNN gets better results than previous networks in both videos and single image. Our method outperforms the state-of-the-art algorithms especially in terms of fine estimation of landmarks.