We address the problem of unsupervised segmentation and grouping in 2D and 3D space, where samples are corrupted by noise, and in the presence of outliers. The problem has attracted attention in previous research work, but non-parametric outlier filtering and inlier denoising are still challenging. Tensor voting is a non-parametric algorithm that can infer local data geometric structure. Standard tensor voting considers outlier noise explicitly, but may suffer from serious problems if the inlier data is also noisy. In this paper, we propose probabilistic Tensor Voting, a Bayesian extension of standard tensor voting, taking into consideration both probabilistic and geometric meaning. Probabilistic tensor voting explicitly considers both outlier and inlier noise, and can handle them simultaneously. In the new framework, the representation consists of a 2nd order symmetric tensor, a polarity vector, and a new type 2 polarity vector orthogonal to the first one. We give a theoretical interpretation of our framework. Experimental results show that our approach outperforms other methods, including standard tensor voting.