Tumor tissue comprises a highly complex network of diverse cell types. The tumor microenvironment (TME) can be mainly subdivided into cancer cells and stromal cell compartments, the latter include different types of immune cells, fibroblasts, endothelial cells, and pericytes. Tumor cells reprogram immune cells and other stromal cells in the TME to constrain their antitumor capacity by creating an immunosuppressive milieu and metabolism competition. Moreover, the reprogramming effect on immune cells is localized not only in the tumor but also at the systemic level. With wide application of single-cell sequencing technology, tumor-specific characteristics of immune cells and other stromal cells in the TME have been dissected. In this review, we mainly focus on how tumor cells reprogram immune cells both within the TME and peripheral blood. This information can further help us to improve the efficiency of current immunotherapy as well as bring up new ideas to combat cancer.