When the Fermi level lies in a gap, the Hall conductivity of three-dimensional (3D) electrons in a periodic potential can be expressed in a topologically invariant form with a set of three integers. These integers are explicitly found as a solution of a Diophantine equation, the structure of which relies on the flux of the magnetic field through three areas of the periodic lattice. In a simple geometry, we detail a tight-binding model which is found to be reduced to a generalized 1D Harper equation. The existence of a complex gap structure is explicitly shown. The spectrum depends on the field orientation.