According to an in-depth analysis of the relationship among n-level polybinary transformation, the time-packing factor and the performance of the decoding algorithm, we find that the appropriate n-level polybinary transformation can improve the performance of the decoding algorithm within a certain range of the time-packing factor in the Faster than Nyquist (FTN) system. In this paper, we explain the reason that this phenomenon occurs. Based on the above analysis, we propose a modified blind phase search (BPS) algorithm to compensate for phase noise (PN) in the FTN system with an extremely small time-packing factor. As a result, the modified-BPS algorithm can cope with the PN with the linewidth × symbol rate at 1.07 × 10−5, 1.79 × 10−5, 2.86 × 10−5 and 3.57 × 10−5 under a time-packing factor of 0.55, 0.50 and 0.45, respectively. At the same time, the spectrum efficiency (SE) is improved to 3.27 bit/s/Hz, 4 bit/s/Hz and 4.88 bit/s/Hz.