As deep neural networks grow larger, they suffer from a huge number of weights, and thus reducing the overhead of handling those weights becomes one of key challenges nowadays. This paper presents a new approach to binarizing neural networks, where the weights are pruned and forced to take degenerate binary values. Experimental results show that the proposed approach achieves significant reductions in computation and power consumption at the cost of a slight accuracy loss.