The metabotropic glutamate 5 (mGlu5) receptor is a class C G protein-coupled receptor (GPCR) that is implicated in several CNS disorders making it a popular drug discovery target. Years of research have revealed allosteric mGlu5 ligands showing an unexpected complete switch in functional activity despite only small changes in their chemical structure, resulting in positive allosteric modulators (PAM) or negative allosteric modulators (NAM) for the same scaffold. Up to now, the origins of this effect are not understood, causing difficulties in a drug discovery context. In this work, experimental data was gathered and analyzed alongside docking and Molecular Dynamics (MD) calculations for three sets of PAM and NAM pairs. The results consistently show the role of specific interactions formed between ligand substituents and amino acid side chains that block or promote local movements associated with receptor activation. The work provides an explanation for how such small structural changes lead to remarkable differences in functional activity. While this work can greatly help drug discovery programs avoid these switches, it also provides valuable insight into the mechanisms of class C GPCR allosteric activation. Furthermore, the approach shows the value of applying MD to understand functional activity in drug design programs, even for such close structural analogues.