The neutron sensitivity of the C6D6 detector setup used at n_TOF facility for capture measurements has been studied by means of detailed GEANT4 simulations. A realistic software replica of the entire n_TOF experimental hall, including the neutron beam line, sample, detector supports and the walls of the experimental area has been implemented in the simulations. The simulations have been analyzed in the same manner as experimental data, in particular by applying the Pulse Height Weighting Technique. The simulations have been validated against a measurement of the neutron background performed with a natC sample, showing an excellent agreement above 1 keV. At lower energies, an additional component in the measured natC yield has been discovered, which prevents the use of natC data for neutron background estimates at neutron energies below a few hundred eV. The origin and time structure of the neutron background have been derived from the simulations. Examples of the neutron background for two different samples are demonstrating the important role of accurate simulations of the neutron background in capture cross-section measurements.