Labeling objects in images plays a crucial role in many visual learning and recognition applications that need training data, such as image retrieval, object detection and recognition. Manually creating object labels in images is time consuming and, thus, becomes impossible for labeling a large image dataset. In this paper, we present a family of semi-automatic methods based on a graph-based semi-supervised learning algorithm for labeling objects in images. We first present SmartLabel that proposes to label images with reduced human input by iteratively computing the harmonic solutions to minimize a quadratic energy function on the Gaussian fields. SmartLabel tackles the problem of lacking negative data in the learning by embedding relevance feedback after the first iteration, which also leads to one limitation of SmartLabel-needing additional human supervision. To overcome the limitation and enhance SmartLabel, we propose SmartLabel-2 that utilizes a novel scheme to sample negative examples automatically, replace regular patch partitioning in SmartLabel by quadtree partitioning and applies image over-segmentation (superpixels) to extract smooth object contours. Evaluation on six diverse object categories have indicated that SmartLabel-2 can achieve promising results with a small amount of labeled data (e.g., 1%-5% of image size) and obtain close-to-fine extraction of object contours on different kinds of objects.