Eukaryotes have evolved independent conserved pathways to regulate translation initiation in response to stress. One is the activation of the integrated stress response (ISR) via phosphorylation of the alpha subunit of the initiation factor eIF2 by kinases that sense biotic and abiotic stresses. This leads to reduction of initiator ternary complex and hence inhibition of global translation initiation. Paradoxically however, eIF2a phosphorylation promotes the translation of pro-survival effector genes, such as GCN4 in yeast. Complementing the ISR is the Target of Rapamycin (TOR) signaling pathway, whose inhibition leads to sequestration of the initiation factor eIF4E and subsequent inhibition of canonical initiation. Here we probe translational control in the absence of eIF4E by utilizing a temperature-sensitive allele of the factor in Saccharomyces cerevisiae. Intriguingly, we find that loss of eIF4E leads to translation of GCN4. In addition, we find that translation of GCN4 is neither accompanied by eIF2α phosphorylation nor reduction in initiator ternary complex. Our data suggest that when eIF4E levels are depleted, GCN4 translation is de-repressed via a unique mechanism that involves faster scanning by the small ribosome subunit due to increased local concentration of eIF4A. Overall, our findings suggest that the stoichiometry of eIF4F components is key to ribosome dynamics and may play important roles in translational control of gene expression.