Lowe syndrome and type 2 Dent disease are caused by defects in the inositol 5-phosphatase OCRL. Most missense mutations in the OCRL ASH-RhoGAP domain that are found in affected individuals abolish interactions with the endocytic adaptors APPL1 and Ses (both Ses1 and Ses2), which bind OCRL through a short phenylalanine and histidine (F&H) motif. Using X-ray crystallography, we have identified the F&H motif binding site on the RhoGAP domain of OCRL. Missense mutations associated with disease affected F&H binding indirectly by destabilizing the RhoGAP fold. By contrast, a disease-associated mutation that does not perturb F&H binding and ASH-RhoGAP stability disrupted the interaction of OCRL with Rab5. The F&H binding site of OCRL is conserved even in species that do not have an identified homolog for APPL or Ses. Our study predicts the existence of other OCRL binding partners and shows that the perturbation of OCRL interactions has a crucial role in disease.