Topographic factor models separate overlapping signals into latent spatial functions to identify correlation structure across observations. These methods require the underlying structure to be held fixed and are not robust to deviations commonly found across images. We present autoencoding topographic factors, a novel variational inference scheme, to decompose irregular observations on a lattice into a superposition of low-rank sources. By exploiting recent developments in variational autoencoders, we replace fixed sources with a nonlinear mapping that parameterizes an unnormalized distribution on the lattice. In doing so, we permit sources to drift dynamically, filtering residual differences in location across comparable areas of interest. This gives an implicit mapping to a unique latent representation while simultaneously forcing the posterior to model group variability in spatial structure. Simulation results and applications to functional imaging demonstrate the effectiveness of our method and its ability to outperform existing spatial factor models.