Thermal radiation becomes a prominent feature in the continuum spectrum of Venus longwards of $\sim$3 $\mu$m. The emission is traceable to the upper cloud and haze layers in the planet's mesosphere. Venus' thermal radiation spectrum is punctuated by CO$_2$ bands of various strengths probing into different atmospheric depths. It is thus possible to invert measured spectra of thermal radiation to infer atmospheric temperature profiles and offer some insight into the cloud and haze structure. In practice, the retrieval becomes complicated by the fact that the outgoing radiation is multiply scattered by the ubiquitous aerosol particles before leaving the atmosphere. We numerically investigate the radiative transfer problem of thermal radiation from the Venus night side between 3 and 5 $\mu$m with a purpose-built model of Venus' mesosphere. Special emphasis is laid on the significance of scattering. The simulations explore the space of model parameters, which includes the atmospheric temperature, cloud opacity, and the aerosols' size and chemical composition. We confirm that aerosol scattering must be taken into account in a prospective temperature retrieval, which means an additional complication to the already ill-posed retrieval problem. We briefly touch upon the degeneracy in the spectrum's shape associated with parameterization of the Venus clouds. Reasonable perturbations in the chemical composition and size of aerosols do not significantly impact the model simulations. Although the experiments are specific to the technical characteristics of the Visual and Infrared Thermal Imaging Spectrometer on the Venus Express spacecraft, the conclusions are generally valid.