A search for new phenomena is presented in final states with two leptons and one or no $b$-tagged jets. The event selection requires the two leptons to have opposite charge, the same flavor (electrons or muons), and a large invariant mass. The analysis is based on the full Run-2 proton-proton collision dataset recorded at a center-of-mass energy of $\sqrt{s} = 13$ TeV by the ATLAS experiment at the LHC, corresponding to an integrated luminosity of 139 fb$^{-1}$. No significant deviation from the expected background is observed in the data. A four-fermion contact interaction between two quarks ($b,s$) and two leptons ($ee$ or $����$), inspired by the $B$-meson decay anomalies, is used as a benchmark signal model. This model is characterized by the energy scale and coupling, $��$ and $g_*$ respectively. Contact interactions with $��/ g_*$ lower than 2.0 (2.4) TeV are excluded for electrons (muons) at the 95% confidence level, still far below the value which is favored by the $B$-meson decay anomalies. In addition, model-independent limits are set as a function of the selection on the dilepton invariant mass, which allows the results to be reinterpreted in other signal scenarios. 32 pages in total, author list starting page 12, 4 figures, 1 table, submitted to PRL. All figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PAPERS/EXOT-2018-16/