A quantitative description of optical refrigeration in ${\mathrm{Yb}}^{3+}$-doped ZBLAN glass in the presence of transition-metal and $\mathrm{OH}$ impurities is presented. The model includes the competition of radiative processes with energy migration, energy transfer to transition-metal ions, and multiphonon relaxation. Molecular dynamics calculations of pure ZBLAN and ZBLAN doped with transition-metal ions provide the structural information that, when combined with spectroscopic data, allows for the calculation of electric-dipole energy-transfer rates in the framework of the Dexter theory. The structural data is further used to extend the traditional energy-gap law to multiphonon relaxation via vibrational impurities. The cooling efficiency is sensitive to the presence of both $3d$ metal ions with absorption in the near infrared and high-frequency vibrational impurities such as $\mathrm{OH}$. The calculation establishes maximum impurity concentrations for different operating temperatures and finds ${\mathrm{Cu}}^{2+}$, ${\mathrm{Fe}}^{2+}$, ${\mathrm{Co}}^{2+}$, ${\mathrm{Ni}}^{2+}$, and $\mathrm{OH}$ to be the most problematic species. ${\mathrm{Cu}}^{2+}$ in particular has to be reduced to $l2\phantom{\rule{0.3em}{0ex}}\mathrm{ppb}$, and ${\mathrm{Fe}}^{2+}$, ${\mathrm{Co}}^{2+}$, ${\mathrm{Ni}}^{2+}$, and $\mathrm{OH}$ have to be reduced to $10--100\phantom{\rule{0.3em}{0ex}}\mathrm{ppb}$ for a practical $\mathrm{ZBLAN}:{\mathrm{Yb}}^{3+}$ optical cryocooler to operate at $100--150\phantom{\rule{0.3em}{0ex}}\mathrm{K}$.