Optical nonlinearities enable photon-photon interaction and lie at the heart of several proposals for quantum information processing, quantum nondemolition measurements of photons, and optical signal processing. To date, the largest nonlinearities have been realized with single atoms and atomic ensembles. We show that a single quantum dot coupled to a photonic crystal nanocavity can facilitate controlled phase and amplitude modulation between two modes of light at the single-photon level. At larger control powers, we observed phase shifts up to π/4 and amplitude modulation up to 50%. This was accomplished by varying the photon number in the control beam at a wavelength that was the same as that of the signal, or at a wavelength that was detuned by several quantum dot linewidths from the signal. Our results present a step toward quantum logic devices and quantum nondemolition measurements on a chip.