The interplay between pancreatic epithelium and the surrounding microenvironment is pivotal for pancreas formation and differentiation as well as adult organ homeostasis. The mesenchyme is the main component of the embryonic pancreatic microenvironment, yet its cellular identity is broadly defined, and whether it comprises functionally distinct cell subsets is not known. Using genetic lineage tracing, transcriptome, and functional studies, we identified mesenchymal populations with different roles during pancreatic development. Moreover, we showed that Pbx transcription factors act within the mouse pancreatic mesenchyme to define a pro-endocrine specialized niche. Pbx directs differentiation of endocrine progenitors into insulin- and glucagon-positive cells through non-cell-autonomous regulation of ECM-integrin interactions and soluble molecules. Next, we measured functional conservation between mouse and human pancreatic mesenchyme by testing identified mesenchymal factors in an iPSC-based differentiation model. Our findings provide insights into how lineage-specific crosstalk between epithelium and neighboring mesenchymal cells underpin the generation of different pancreatic cell types.