Cu(In,Ga)Se2 based solar cells exceed power conversion efficiencies of 23 %. Yet, the fill factor of these solar cells, with best values around 80 %, is relatively low (Si reaches 84.9%) mostly due to diode factors greater than one. Recently, we proposed metastable defects, a general feature of the Cu(In,Ga)Se2 alloy, to be the origin of the increased diode factor. We measure the diode factor of the bare absorber layers by excitation-dependent photoluminescence. For high quality and thus high luminescent polycrystalline absorbers, we evaluate the diode factor excitation dependence over four orders of magnitude. Using simulations and the model of metastable defects, we can well describe the experimental findings on n- and p-type epitaxial films as well as the polycrystalline absorbers, providing additional evidence for this model. We find that the diode factors measured optically by photoluminescence impose a lower limit for the diode factor measured electrically on a finished solar cell. Interestingly, the lowest diode factor (optical and electrical) and consequently highest fill factor of 81.0 % is obtained by Ag alloying, i.e. an (Ag,Cu)(In,Ga)Se2 absorber. This finding hints to a pathway to increase fill factors and thus efficiencies for Cu(In,Ga)Se2-based solar cells.