Astaxanthin is increasingly attracting commercial interest for its application in the nutraceutical and pharmaceutical industries. This study aimed to produce astaxanthin from molasses with our newly mutated strain of Rhodosporidium toruloides G17 and to evaluate biological activities of the produced astaxanthin. To maximize the astaxanthin yield, the response surface methodology was used so as to optimize the culture conditions. A maximum astaxanthin yield of 1262.08 ± 14.58 µg/L was achieved by growing R. toruloides G17 in a molasses-based medium containing 49.39 g/L reducing sugar, 1.00 g/L urea, 4.15 g/L MgSO4·7H2O, and 10.05% inoculum ratio. The produced astaxanthin was then purified and studied for its antioxidant and anticancer activities. This compound exhibited 123-fold higher antioxidant activity than α-tocopherol, with an IC50 value of 0.97 ± 0.01 µg/mL. The astaxanthin also showed a potent inhibitory ability against the following three cancer cell lines: HeLa, A549, and MCF7, with IC50 values of 69.07 ± 2.4 µg/mL, 55.60 ± 2.64 µg/mL, and 56.38 ± 4.1 µg/mL, respectively. This study indicates that astaxanthin derived from our newly mutated R. toruloides G17 is a promising anticancer and antioxidant agent for further pharmaceutical applications.