The circadian clock plays a pivotal role in the control of Arabidopsis hypocotyl elongation by regulating rhythmic expression of the bHLH factors PHYTOCHROME INTERACTING FACTOR 4 and 5 (PIF4 and 5). Coincidence of increased PIF4/PIF5 transcript levels with the dark period allows nuclear accumulation of these factors, and in short days it phases maximal hypocotyl growth at dawn. During early night, PIF4 and PIF5 transcription is repressed by the Evening Complex (EC) proteins EARLY FLOWERING3 (ELF3), EARLY FLOWERING4 (ELF4), and LUX ARRHYTHMO (LUX). While ELF3 has an essential role in EC complex assembly, several lines of evidence indicate that this protein controls plant growth via other mechanisms that are presently unknown. Here, we show that the ELF3 and PIF4 proteins interact in an EC-independent manner, and that this interaction prevents PIF4 from activating its transcriptional targets. We also show that PIF4 overexpression leads to ELF3 protein destabilization, and that this effect is mediated indirectly by negative feedback regulation of photoactive PHYTOCHROME B (phyB). Physical interaction of the phyB photoreceptor with ELF3 has been reported, but its functional relevance remains poorly understood. Our findings establish that phyB is needed for ELF3 accumulation in the light, most likely by competing for CONSTITUTIVELY PHOTOMORPHOGENIC1 (COP1)-mediated ubiquitination and the proteasomal degradation of ELF3. Our results explain the short hypocotyl phenotype of ELF3 overexpressors, despite their normal clock function, and provide a molecular framework for understanding how warm temperatures promote hypocotyl elongation and affect the endogenous clock.