The absolute measurement of infrared spectral radiance is very important for optical radiometry. In this paper, a system for absolute measurement of infrared spectral radiance is built up. The system consists of fixed-point blackbody sources, a variable temperature blackbody, a radiant source to be measured, Fourier Transform Infrared Radiometer (FTIR), relay optical system, non-contact infrared thermometer and so on. The emissivity of the variable temperature blackbody is 0.999; the temperature range is 50°C ~ 1050°C. The emissivity of the radiant source to be measured is larger than 0.995; the temperature range is 30°C ~ 550°C. The variable temperature blackbody source was calibrated and can be traced to the fixed-point blackbody source. In experiment, it was used as the standard radiant source. The spectral range of this system is 3 μm ~ 14 μm. A serial of experiments have been implemented to analyze the uncertainty of each component, including the repeatability, size-of-source effect, stability, uniformity and so on. To improve the system’s uncertainty, we have suppressed stray radiation and optimized optical system by installing a water-cooled aperture and a field stop at the entrance of the optical system and before the FTIR, respectively; optimizing the system based on optical simulation and replacing the reflective mirrors with one off-axis parabolic mirror. Next step, we will re-evaluate the uncertainty of the improved system.