As the science and technology of the brain and mind develop, so do the ways in which brains and minds may be surveilled and manipulated. Some cognitive libertarians worry that these developments undermine cognitive liberty or "freedom of thought." I argue that protecting an individual's cognitive liberty undermines others' ability to use their own cognitive liberty. Given that the threatening devices and processes are not relevantly different from ordinary and frequent intrusions upon one's brain and mind, strong protections of cognitive liberty may proscribe neurotechnological intrusions but also ordinary intrusions. Thus, the cognitive libertarian position "hog-ties" others' use of their own liberties. This problem for the cognitive libertarian is the same problem that ordinary libertarianism faces in protecting individual rights to property and person. But the libertarian strategies for resolving the problem don't work for the cognitive libertarian. I conclude that the right to mental privacy is weaker than what cognitive libertarians want it to be.