We present a general method for optimizing the configuration of an experimental diagnostic to minimize uncertainty and bias in inferred quantities from experimental data. The method relies on Bayesian inference to sample the posterior using a physical model of the experiment and instrument. The mean squared error (MSE) of posterior samples relative to true values obtained from a high fidelity model (HFM) across multiple configurations is used as the optimization metric. The method is demonstrated on a common problem in dense plasma research, the use of radiation detectors to estimate physical properties of the plasma. We optimize a set of filtered photoconducting diamond detectors to minimize the MSE in the inferred X-ray spectrum, from which we can derive quantities like the electron temperature. In the optimization we self-consistently account for uncertainties in the instrument response with appropriate prior probabilities. We also develop a penalty term, acting as a soft constraint on the optimization, to produce results that avoid negative instrumental effects. We show results of the optimization and compare with two other reference instrument configurations to demonstrate the improvement. The MSE with respect to the total inferred X-ray spectrum is reduced by more than an order of magnitude using our optimized configuration compared with the two reference cases. We also extract multiple other quantities from the inference and compare with the HFM, showing an overall improvement in multiple inferred quantities like the electron temperature, the peak in the X-ray spectrum and the total radiated energy.