Abstract Public and scholarly debates surrounding free speech often assume free speech is a public good and/or should be approached as a problem of “drawing the line” between free and regulated or benign and harmful speech. In contrast, this article provides a genealogy of free speech in which liberal freedom of expression has, since its inception, been integral to white supremacist settler colonialism in the United Kingdom and its former settler colonies, the United States, Canada, Australia, and New Zealand. The article argues that, far from a noble struggle against regulation, liberal politics around free speech establish oppositions between white “civilized” speech and its Indigenized racially darkened “others” as well as controlling or silencing Indigenous, Black and/or otherwise racially othered speech across the Anglosphere. The article first traces free speech through two significant moments in its emergence: early European Enlightenment colonial expansion (embodied in John Locke's “toleration”) and 1800s British colonial industrialization (embodied in John Stuart Mill's “marketplace of ideas”). The article then examines how this genealogy informs the contemporary case study of contestation over free speech in universities, showing that engagements with free speech across the political spectrum extend its settler colonial rationality.