Caveolae are cell-surface membrane invaginations that play critical roles in cellular processes including signaling and membrane homeostasis. The cavin proteins, in cooperation with caveolins, are essential for caveola formation. Here we show that a minimal N-terminal domain of the cavins, termed HR1, is required and sufficient for their homo- and hetero-oligomerization. Crystal structures of the mouse cavin1 and zebrafish cavin4a HR1 domains reveal highly conserved trimeric coiled-coil architectures, with intersubunit interactions that determine the specificity of cavin-cavin interactions. The HR1 domain contains a basic surface patch that interacts with polyphosphoinositides and coordinates with additional membrane-binding sites within the cavin C terminus to facilitate membrane association and remodeling. Electron microscopy of purified cavins reveals the existence of large assemblies, composed of a repeating rod-like structural element, and we propose that these structures polymerize through membrane-coupled interactions to form the unique striations observed on the surface of caveolae in vivo.