Cell polarity in eukaryotes requires constant sorting, packaging and transport of membrane-bound cargo within the cell. These processes occur in two sorting hubs: the recycling endosome for incoming material and the trans-Golgi network for outgoing material. Phosphatidylinositol 3-phosphate and phosphatidylinositol 4-phosphate are enriched at the endocytic and exocytic sorting hubs, respectively, where they act together with small GTPases to recruit factors to segregate cargo and regulate carrier formation and transport. In this review, we summarize the current understanding of how these lipids and GTPases regulate membrane trafficking directly, emphasizing the recent discoveries of phosphatidylinositol 4-phosphate functions at the trans-Golgi network.