Significant research has gone into engineering representations that can identify high-level semantic structure in images, such as objects, people, events and scenes. Recently there has been a shift towards learning representations of images either on top of dense features or directly from the pixel level. These features are often learned in hierarchies using large amounts of unlabeled data with the goal of removing the need for hand-crafted representations. In this thesis we consider the task of learning two specific types of image representations from standard size RGB images: a semi-supervised dense low-dimensional embedding and an unsupervised sparse binary code. We introduce a new algorithm called the deep matching pursuit network (DMP) that efficiently learns features layer-by-layer from the pixel level without the need for backpropagation fine tuning. The DMP network can be seen as a generalization of the single layer networks of Coates et. al. to multiple layers and larger images. We apply our features to several tasks including object detection, scene and event recognition, image auto-annotation and retrieval. For auto-annotation, we achieve competitive performance against methods that use 15 distinct hand-crafted features. We also apply our features for handwritten digit recognition on MNIST, achieving the best reported error when no distortions are used for training. When our features are combined with t-SNE, we obtain highly discriminative two dimensional image visualizations. Finally, we introduce the multi-scale DMP network for domain independent multimodal segmentation of medical images. We obtain the top performance on the MICCAI lung vessel segmentation (VESSEL12) competition and competitive performance on the MICCAI multimodal brain tumor segmentation (BRATS2012) challenge. We conclude by discussing how the deep matching pursuit network can be applied to other modalities such as RGB-D images and spectrograms.