The neutral points are one of the most significant characteristics of the polarized skylight pattern in the whole sky. At present, detection of the neutral points mostly utilizes ellipse fitting of the degree of polarization. However, because the degree of polarization distribution characteristics of a polarized skylight pattern is easily affected by the environment, the robustness of the detection is unstable. Aiming at the problem, we analyzed the angle of polarization distribution characteristics of polarized skylight patterns in the region around the neutral point by measurement experiments. Based on this, we proposed an automatic detection method of neutral points using the angle of polarization of the polarized skylight pattern. The experimental results of different times in a continuous period of time show that compared with ellipse fitting of the degree of polarization, the detection accuracy of the proposed method is almost the same, but the robustness is better. It provides a novel method for the position detecting of the neutral point, which is in favor of the measurement applications of polarization technology.