Esophageal cancer (EC) is the sixth most deadly cancer, and its incidence is still increasing year by year. Although the researches on the molecular mechanisms of EC have been widely carried out and incremental progress has been made, its overall survival rate is still low. There is cumulative evidence showing that the esophageal microenvironment plays a vital role in the development of EC. In precancerous lesions of the esophagus, high-risk environmental factors can promote the development of precancerous lesions by inducing the production of inflammatory factors and the recruitment of immune cells. In the tumor microenvironment, tumor-promoting cells can inhibit anti-tumor immunity and promote tumor progression through a variety of pathways, such as bone marrow-derived suppressor cells (MDSCs), tumor-associated fibroblasts (CAFs), and regulatory T cells (Tregs). The formation of extracellular hypoxia and acidic microenvironment and the change of extracellular matrix stiffness are also important factors affecting tumor progression and metastasis. Simultaneously, primary tumor-derived cytokines and bone marrow-derived immune cells can also promote the formation of pre-metastasis niche of EC lymph nodes, which are beneficial to EC lymph node metastasis. Further research on the specific mechanism of these processes in the occurrence, development, and metastasis of each EC subtype will support us to grasp the overall pre-cancerous prevention, targeted treatment, and metastatic assessment of EC.