In this paper, we propose a phase difference minimization algorithm to measure the specular surface shape in a displacement-free stereoscopic phase measuring deflectometry (PMD) system. The presented system is capable of solving the height-normal ambiguity appearing in a PMD system without moving any system component. Both the surface normal and the absolute height are simultaneously obtained by implementing phase difference minimization between the phase distributions in the LCD screen and the camera image plane. In particular, phase difference minimization is performed by using a second order polynomial fitting iteration method. Bi-cubic sub-pixel interpolation combined with 2D Fourier integration is used to reconstruct the surface. Finally, the performance of the proposed stereoscopic PMD system is verified by measuring the surface shapes of different mirrors and performing repeatability tests.