The interferon-induced transmembrane proteins (IFITMs) broadly inhibit virus infections, particularly at the viral entry level. However, despite this shared ability to inhibit fusion, IFITMs differ in the potency and breadth of viruses restricted, an anomaly that is not fully understood. Here, we show that differences in the range of viruses restricted by IFITM1 are regulated by a C-terminal non-canonical dibasic sorting signal KRXX that suppresses restriction of some viruses by governing its intracellular distribution. Replacing the two basic residues with alanine (KR/AA) increased restriction of jaagsiekte sheep retrovirus and 10A1 amphotropic murine leukemia virus. Deconvolution microscopy revealed an altered subcellular distribution for KR/AA, with fewer molecules in LAMP1-positive lysosomes balanced by increased levels in CD63-positive multivesicular bodies, where jaagsiekte sheep retrovirus pseudovirions are colocalized. IFITM1 binds to cellular adaptor protein complex 3 (AP-3), an association that is lost when the dibasic motif is altered. Although knockdown of AP-3 itself decreases some virus entry, expression of parental IFITM1, but not its KR/AA mutant, potentiates inhibition of viral infections in AP-3 knockdown cells. By using the substituted cysteine accessibility method, we provide evidence that IFITM1 adopts more than one membrane topology co-existing in cellular membranes. Because the C-terminal dibasic sorting signal is unique to human IFITM1, our results provide novel insight into understanding the species- and virus-specific antiviral effect of IFITMs.