Hashing is an effective and widely used technology for fast approximate nearest neighbor search in large-scale images. In recent years, it has been combined with a powerful feature learning model, convolutional neural network(CNN), to boost the efficiency of large-scale image retrieval. In this paper, we introduce a new Deep Double Center Hashing (DDCH) network to learn hash codes with higher discrimination between different people and compact hash codes between the same person for large-scale face image retrieval. Our method uses a deep neural network to learn image features as well as hash codes. We use a deep CNN to extract image features and a multi-layer neural network as the hash function. The whole model is trained end-to-end. In order to learn compact and discriminative hash codes, we impose a compact constraint on the codes to force lower intra-class variations of the codes. Our constraint is formulated as a center-loss over the learned codes, which encourages hash codes to be near the hash center of the same class. In addition, new discrete hashing modules and multi-scale fusion are designed to capture discriminative and multi-scale information. We conduct experiments on the most popular datasets, YouTubeFaces and FaceScrub, and demonstrates the efficient performance of DDCH over the state-of-the-art face image hashing methods.