The permanent electrical dipole moment of colloidal quantum dots is important for their optoelectronic properties and can be determined by dielectric spectroscopy. Until now, however, colloidal interactions have not been taken into account in the interpretation of the spectra. Here, dielectric spectra of PbSe and CdSe colloidal quantum dots dispersed in an apolar liquid are measured from 1 Hz to 10 MHz. At frequencies of 10 kHz-1 MHz, Brownian rotation of nanoparticles with a permanent electric dipole moment is detected. At the lowest concentrations (∼0.1 vol %), the nanoparticles rotate independently of each other, and their dipole moment, for both PbSe and CdSe, is on the order of 40-50 D. At higher concentrations (≥0.3 vol %), the dipolar relaxation becomes slower, indicating the presence of nanoparticle structures. A simple model is used to estimate the interaction strength, which appears to be stronger than expected from the weak dipole moment, and has possibly also contributions from electrical moments of higher order. Our results indicate that nanoparticle interactions in liquid media lead to small equilibrium structures that affect dielectric measurements of the dipole moment already at concentrations of a few tenths of a volume percent.