A common scientific inverse problem is the placement of magnets that produce a desired magnetic field inside a prescribed volume. This is a key component of stellarator design, and recently permanent magnets have been proposed as a potentially useful tool for magnetic field shaping. Here, we take a closer look at possible objective functions for permanent magnet optimization, reformulate the problem as sparse regression, and propose an algorithm that can efficiently solve many convex and nonconvex variants. The algorithm generates sparse solutions that are independent of the initial guess, explicitly enforces maximum strengths for the permanent magnets, and accurately produces the desired magnetic field. The algorithm is flexible, and our implementation is open-source and computationally fast. We conclude with two new permanent magnet configurations for the NCSX and MUSE stellarators. Our methodology can be additionally applied for effectively solving permanent magnet optimizations in other scientific fields, as well as for solving quite general high-dimensional, constrained, sparse regression problems, even if a binary solution is required.