We investigate the trade-offs between price stability and the sustainability of sovereign debt, using a small open economy model where the government issues nominal defaultable debt and chooses fiscal and monetary policy under discretion. Inflation reduces the real value of outstanding debt, thus making it more sustainable; but it also raises nominal yields and entails direct welfare costs. We compare this scenario with a situation in which the government gives up the ability to deflate debt away, e.g. by issuing foreign currency debt or joining a monetary union with an anti-inflationary stance. We find that the benefits of giving up this adjustment margin outweigh the costs, both for our preferred calibration and for a wide range of parameter values.