We consider II$_1$ factors of the form $M=\bar{\bigotimes}_{G}N\rtimes G$, where either i) $N$ is a non-hyperfinite II$_1$ factor and $G$ is an ICC amenable group or ii) $N$ is a weakly rigid II$_1$ factor and $G$ is ICC group and where $G$ acts on $\bar{\bigotimes}_{G}N$ by Bernoulli shifts. We prove that isomorphism of two such factors implies cocycle conjugacy of the corresponding Bernoulli shift actions. In particular, the groups acting are isomorphic. As a consequence, we can distinguish between certain classes of group von Neumann algebras associated to wreath product groups.