We develop a variational multiscale proper orthogonal decomposition (POD) reduced‐order model (ROM) for turbulent incompressible Navier‐Stokes equations. Under two assumptions on the underlying finite element approximation and the generation of the POD basis, the error analysis of the full discretization of the ROM is presented. All error contributions are considered: the spatial discretization error (due to the finite element discretization), the temporal discretization error (due to the backward Euler method), and the POD truncation error. Numerical tests for a three‐dimensional turbulent flow past a cylinder at Reynolds number show the improved physical accuracy of the new model over the standard Galerkin and mixing‐length POD ROMs. The high computational efficiency of the new model is also showcased. Finally, the theoretical error estimates are confirmed by numerical simulations of a two‐dimensional Navier‐Stokes problem. © 2013 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 30: 641–663, 2014