This paper summarises \textit{S. Kripke}'s account [in: Wittgenstein on rules and private language, Cambridge UP (1982)] of Wittgenstein's private language argument. Next it expounds Wittgenstein's later philosophy of mathematics, in which the validity of a proof is a sociological matter (acceptance by the community of mathematicians) and the proof itself is a means of concept change or concept formation. Then it is argued that Wittgenstein's anti-realist conclusions about mathematics do not follow (as the private language argument may) from rule-following and criteriological considerations alone, requiring also the premisses that mathematical objects are not public and that criteria and symptoms can be sharply distinguished: either of which may be denied by a mathematical realist.