Analog/digital hybrid precoder and combiner have been widely used in millimeter wave (mmWave) multiple-input multiple-output (MIMO) systems due to its energy-efficient and economic superiorities. Infinite resolution of phase shifters (PSs) for the analog beamformer can achieve very close performance compared to the full-digital scheme but will result in high complexity and intensive power consumption. Thus, more cost effective and energy efficient low resolution PSs are typically used in practical mmWave MIMO systems. In this paper, we consider the joint hybrid precoder and combiner design with one-bit quantized PSs in mmWave MIMO systems. We propose to firstly design the analog precoder and combiner pair for each data stream successively, aiming at conditionally maximizing the spectral efficiency. We present a novel binary analog precoder and combiner optimization algorithm under a Rank-1 approximation of the interference-included equivalent channel with lower than quadratic complexity. Then the digital precoder and combiner are computed based on the obtained baseband effective channel to further enhance the spectral efficiency. Simulation results demonstrate that the proposed algorithm outperforms the existing one-bit PSs based hybrid beamforming scheme.