What actions are too "political" for a Supreme Court justice? This question has been the subject of intense debate since the beginning of the republic. In this note, I reveal the misunderstood beginnings of this debate. In chronicling how it transformed the judicial role, I will show how critiquing the judiciary as "political" is a practice as old as the Constitution itself. I do so by examining the lengthy "charges" the Supreme Court Justices delivered to federal grand juries while they rode Circuit and the increasingly negative public responses to them. Though the lens of the charges, one sees that justices and their political opponents had a fundamental disagreement about what was appropriately "judicial" and what was too "political." This disagreement shaped our judiciary in its first years and is the origin of our current debates about the boundaries between politics and judging.