Let $M_{n}$ denote a random symmetric $n\times n$ matrix, whose entries on and above the diagonal are i.i.d. Rademacher random variables (taking values $\pm 1$ with probability $1/2$ each). Resolving a conjecture of Vu, we prove that the permanent of $M_{n}$ has magnitude $n^{n/2+o(n)}$ with probability $1-o(1)$. Our result can also be extended to more general models of random matrices.