ABSTRACT Bayesian evidence ratios are widely used to quantify the statistical consistency between different experiments. However, since the evidence ratio is prior dependent, the precise translation between its value and the degree of concordance/discordance requires additional information. The most commonly adopted metric, the Jeffreys scale, can falsely suggest agreement between data sets when priors are chosen to be sufficiently wide. This work examines evidence ratios in a DES-Y1 simulated analysis, focusing on the internal consistency between weak lensing and galaxy clustering. We study two scenarios using simulated data in controlled experiments. First, we calibrate the expected evidence ratio distribution given noise realizations around the best-fitting DES-Y1 ΛCDM cosmology. Secondly, we show the behaviour of evidence ratios for noiseless fiducial data vectors simulated using a modified gravity model, which generates internal tension in the LCDM analysis. We find that the evidence ratio of noise realizations generated at all confidence levels was biased towards agreement and show, with a modified gravity model, that the choice of prior could conceal the discrepancies between weak lensing and galaxy clustering induced by prior effects in unlike cosmological models, concluding that the evidence ratio in a DES-Y1 study is, indeed, biased towards agreement. Boundary effects can also influence conclusions about the inconsistency induced by modified gravity, even in a noiseless data vector simulation.