Humans are born free in the moral sense. This widely recognized principle of natural freedom implies that exercising freedom should not require moral justification; but rather; its restriction must be based on valid justifications. Nevertheless, the burden of justification appears to have been reversed in the context of free speech. A rich jurisprudence has been developed for justifying free speech. Why and how did such reversion happen? This paper is a historical analysis for answering that question through three arguments. (1) It argues that the primitive forms of mythologies, superstitions and religions can be considered the oldest sources of limitation on expressive activities. (2) It finds that the first restrictive laws of freedom of speech were religious in nature, and the first justificatory arguments for freedom of speech were reactions to such religious restrictions. (3) It concludes that justifications of freedom of expression have evolved in parallel with the manifestation of power in both religious and political institutions. These three critical arguments can be a strong base for asking whether exercising or restricting freedom of speech requires moral justifications.