Phase-resolved imaging of swept-source optical coherence tomography (SS-OCT) is subject to phase measurement instabilities involved with the sweep variation of a frequency-swept source. In general, optically generated timing references are utilized to track the variations imposed on OCT signals. But they might not be accurately synchronized due to relative time delays. In this research, we investigated the impact of the signal delays on the timing instabilities and the consequent deviations of the measured phases. We considered two types of timing signals utilized in a popular digitizer operation mode: a sweep trigger from a fiber Bragg grating (FBG) that initiates a series of signal sampling actions clocked by an auxiliary Mach-Zehnder interferometer (MZI) signal. We found that significant instabilities were brought by the relative delays through incoherent timing corrections and timing collisions between the timing references. The best-to-worst ratio of the measured phase errors was higher than 200 while only the signal delays varied. Noise-limited phase stability was achieved with a wide dynamic range of OCT signals above 50 dB in optimized delays. This demonstrated that delay optimization is very effective in phase stabilization of SS-OCT.