A search for charged leptons with large impact parameters using 139 fb$^{-1}$ of $\sqrt{s} = 13$ TeV $pp$ collision data from the ATLAS detector at the LHC is presented, addressing a long-standing gap in coverage of possible new physics signatures. Results are consistent with the background prediction. This search provides unique sensitivity to long-lived scalar supersymmetric lepton-partners (sleptons). For lifetimes of 0.1 ns, selectron, smuon and stau masses up to 720 GeV, 680 GeV, and 340 GeV are respectively excluded at 95% confidence level, drastically improving on the previous best limits from LEP. 30 pages in total, author list starting page 15, 2 figures, 2 tables, published in PRL. All figures including auxiliary figures are available at http://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PAPERS/SUSY-2018-14