3D scene understanding is crucial for robotics, augmented reality and autonomous vehicles. In those applications, the 3D structure can be computed by using stereo cameras or depth sensors. One can process these 3D measurements using deep learning techniques to achieve remarkable performance in various perception related tasks. However, different from images that have a dominant representation as 2D pixel arrays, 3D data has many representations, including voxels, meshes, depth images, point clouds, and etc. Among all of them, depth images and point clouds are closer to the direct output from 3D measurements, as depth images are computed by stereo cameras and point clouds are generated from LiDAR. The recent improved accessibility of those 3D measurements requires the need of algorithms to interpret them. Therefore, this dissertation develops algorithms for 3D scene understanding with deep learning techniques for depth images and point clouds. The first portion of this dissertation describes an algorithm to estimate accurate depth maps from stereo images. In particular, by solving the stereo matching problems, one can generate a disparity map and convert it into a dense depth image. During the processing, the semantic embedding learned from semantic segmentation further helps to guide the disparity estimation, especially for smooth, reflective and occluded regions. With the computed depth images and semantic segments, we can efficiently produce semantic 3D models. The second portion of the dissertation addresses the challenges of processing point cloud data which may be arbitrarily rotated. To solve perception tasks with random rotation in real-world point cloud data, traditional techniques employ data augmentation. However, this can increase training time and may require more complex deep learning models. To address rotations that may not exist in the training data, this dissertation proposes a 3D representation of point clouds that is designed to be rotationally invariant and introduces a novel neural network architecture to utilize this representation. The third portion of the dissertation devises methods to address the challenge of processing real-world point clouds due to partial observations. This dissertation applies a multivariate Gaussian distribution to model the output from each local point set and illustrates how to use each such local point set to infer the latent feature encoding information contained by a complete point cloud. This strategy ensures accurate prediction with a partial observed point set for different tasks, such as shape classification, part segmentation, and point cloud completion. The final portion of the dissertation focuses on point cloud completion. At processing the point clouds, existing approaches adopt encoder-decoder structures and output sparse distributed embeddings, which may lead to worse generalizability at testing. In addition, analysis of point cloud completion trained jointly with other tasks are lacking. To address those limitations, this dissertation proposes a novel module that includes a normalization layer to normalize embeddings into unit one, and the module can be integrated into existing approaches. Both the theory and empirical results are shown to demonstrate the effectiveness of the proposed method on improving point cloud completion performance.