Band structure and spin can modify the simple two-mode far-infrared spectra of quantum dots. In our investigation of these effects we start from a k\ensuremath{\cdot}p Hamiltonian for the conduction, valence, and spin-orbit split-off bands, with potentials for subband formation and harmonic lateral confinement. Taking into account the nonparabolicity of the conduction band and spin-orbit coupling due to the confining potentials, we calculate the dipole excitation spectra of quantum dots on InSb with one and two electrons. Spin-orbit coupling leads to a level anticrossing in the ${\mathrm{\ensuremath{\omega}}}_{\mathrm{\ensuremath{-}}}$ mode and nonparabolicity modifies the ${\mathrm{\ensuremath{\omega}}}_{+}$ mode demonstrating violation of Kohn's theorem.