Despite increasing research into the use of syntax during statistical machine translation, the incorporation of syntax into language models has seen limited success. We present a study of the discriminative abilities of generative syntax-based language models, over and above standard n-gram models, with a focus on potential applications for Statistical Machine Translation (SMT). We show that in fact parsers are better able to discriminate between good and bad English, and that parsers, as well as n-gram language models, assign higher average log probabilities to references in comparison to SMT output.