Curcuma longa (turmeric) and Curcuma zanthorrhiza (temulawak) are members of the Zingiberaceae family that contain curcuminoids, essential oils, starch, protein, fat, cellulose, and minerals. The nutritional content proportion of turmeric is different from temulawak which implies differences in economic value. However, only a few people who understand herbal plants, can identify the difference between them. This study aims to build a model that can distinguish between the two species of Zingiberaceae based on the image captured from a mobile phone camera. A collection of images consisting of both types of rhizomes are used to build a model through a learning process using transfer learning, specifically pre-trained VGG-19 and Inception V3 with ImageNet weight. Experimental results show that the accuracy rates of the models to classify the rhizomes are 92.43% and 94.29%, consecutively. These achievements are quite promising to be used in various practical use.