Androgens are not only essential for bone development but for the maintenance of bone mass. Therefore, conditions with androgen deficiency, such as male hypogonadism, androgen-insensitive syndromes, and prostate cancer with androgen deprivation therapy are strongly associated with bone loss and increased fracture risk. Here we summarize the skeletal effects of androgens—androgen receptors (AR) actions based on in vitro and in vivo studies from animals and humans, and discuss bone loss due to androgens/AR deficiency to clarify the molecular basis for the anabolic action of androgens and AR in bone homeostasis and unravel the functions of androgen/AR signaling in healthy and disease states. Moreover, we provide evidence for the skeletal benefits of androgen therapy and elucidate why androgens are more beneficial than male sexual hormones, highlighting their therapeutic potential as osteoanabolic steroids in improving bone fracture repair. Finally, the application of selective androgen receptor modulators may provide new approaches for the treatment of osteoporosis and fractures as well as building stronger bones in diseases dependent on androgens/AR status.