Molecular oxygen (O2) is key to all life on earth, as it is constantly cycled via photosynthesis and cellular respiration. Substantial scientific effort has been devoted to understanding every part of this cycle. Classical molecular dynamics (MD) simulations have been used to study some of the key processes involved in cellular respiration: O2 permeation through alveolar monolayers and cellular membranes, its binding to hemoglobin during transport in the bloodstream, as well as its transport along optimal pathways toward its reduction sites in proteins. Moreover, MD simulations can help interpret the results of several imaging techniques in which O2 is used because of its paramagnetic nature. However, despite the widespread use of computational models for the O2 molecule, their performances have never been systematically evaluated. In this paper, we assess the performances of 14 different models of O2 available in the literature by calculating four thermodynamic properties: density, heat of vaporization, free energy of hydration, and free energy of solvation in hexadecane. For each property, reliable experimental data are available. Most models perform reasonably well in predicting the correct trends, but they fail to reproduce the experimental data quantitatively. We then develop new models for O2, with and without a quadrupole moment, and compare their behavior with the behavior of previously published models. The new models show significant improvement in terms of density, heat of vaporization, and free energy of hydration. However, quantitative agreement with water-oil partitioning is not reached due to discrepancies between the calculated and measured free energies of solvation in hexadecane. We suggest that classical pairwise-additive models may be inadequate to properly describe the thermodynamics of solvation of apolar species, such as O2, in apolar solvents.