It is known that for graphs $A$ and $B$ with odd cycles, the direct product $A\times B$ is vertex-transitive if and only if both $A$ and $B$ are vertex-transitive. But this is not necessarily true if one of $A$ or $B$ is bipartite, and until now there has been no characterization of such vertex-transitive direct products. We prove that if $A$ and $B$ are both bipartite, or both non-bipartite, then $A\times B$ is vertex-transitive if and only if both $A$ and $B$ are vertex-transitive. Also, if $A$ has an odd cycle and $B$ is bipartite, then $A\times B$ is vertex-transitive if and only if both $A\times K_2$ and $B$ are vertex-transitive.