The conditional random fields (CRF) model, using patch-based classification bound with context information, has been widely adopted for image segmentation/ labeling. In this paper, we propose three components for improving the speed and accuracy, and illustrate them on a developed auto-context algorithm: (1) a new coding scheme for multiclass classification, named data-assisted output code (DAOC); (2) a scale-space approach to make it less sensitive to geometric scale change; and (3) a region-based voting scheme to make it faster and more accurate at object boundaries. The proposed multiclass classifier, DAOC, is general and particularly appealing when the number of class becomes large since it needs a minimal number of [log2 k] binary classifiers for k classes. We show advantages of the DAOC classifier over the existing algorithms on several Irvine repository datasets, as well as vision applications. Combining DAOC, the scale-space approach, and the region-based voting scheme for autocontext, the overall algorithm is significantly faster (5 ~ 10 times) than the original auto-context, with improved accuracy over many of the existing algorithms on theMSRC and VOC 2007 datasets.