Soliton perturbation theory is used to calculate the fluxon oscillator linewidth arising from fluxon interaction with background radiation. Detailed calculations in the case of an oscillator that is long compared with the Josephson length and for which the radiation field is thermal establish lower bounds for the linewidth of a real oscillator. These lower bounds are not in disagreement with recent, instrument-limited measurements of $X$-band linewidths less than 5 kHz.