A three-dimensional polarimetric ptychography (3D-PP) technique is proposed to reconstruct the residual stress distribution in optical elements in three dimensions by combining polarization imaging and a 3D ptychographic iterative engine (3D-PIE). 3D-PP mainly uses two orthogonal linear polarizers placed along the light direction to capture the photoelastic information within an optical element. Based on phase shift theory, a three-step phase shift method is proposed to record three dark field images, allowing for calculating the isometric line of residual stress within the optical element. This method enables the layer-by-layer measurement of residual stress within thick samples and assesses the distribution of residual stress along the detection axis, enhancing the precision of stress measurements.