The tumor microenvironment (TME) is a very complex niche that consists of multiple cell types, supportive matrix and soluble factors. Cells in the TME consist of both host cells that are present at tumor site at the onset of tumor growth and cells that are recruited in either response to tumor- or host-derived factors. PCa (PCa) thrives on crosstalk between tumor cells and the TME. Crosstalk results in an orchestrated evolution of both the tumor and microenvironment as the tumor progresses. The TME reacts to PCa-produced soluble factors as well as direct interaction with PCa cells. In return, the TME produces soluble factors, structural support and direct contact interactions that influence the establishment and progression of PCa. In this review, we focus on the host side of the equation to provide a foundation for understanding how different aspects of the TME contribute to PCa progression. We discuss immune effector cells, specialized niches, such as the vascular and bone marrow, and several key protein factors that mediate host effects on PCa. This discussion highlights the concept that the TME offers a potentially very fertile target for PCa therapy.