We study compatible families of three-dimensional Galois representations constructed in the étale cohomology of a smooth projective variety. We prove a theorem asserting that the residual images will be generically large if certain easy-to-check conditions are satisfied. We only consider representations with coefficients in an imaginary quadratic field. For primes inert in this field, the residual representations (when irreducible) are unitary. We apply our result to an example constructed by van Geemen and Top, obtaining a family of special linear groups and one of special unitary groups as Galois groups over [inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="01i" /]. We also consider the case of cohomological modular forms for a congruence subgroup of SL [inline-graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="02i" /]. Assuming Clozel's conjecture stating that a geometric family of three-dimensional Galois representations can be attached to them, we verify for three examples the conditions guaranteeing generically large images and explicitly bound the finite set of primes with non-maximal image. We also discuss what intrinsic conditions a modular form should verify to guarantee that the images of the attached Galois representations will be generically large.