Solving challenges of enhanced imaging (resolution or speed) is a continuously changing frontier of research. Within this sphere, ghost imaging (and the closely related single-pixel imaging) has evolved as an alternative to focal plane detector arrays owing to advances in detectors and/or modulation devices. The interest in these techniques is due to their robustness to varied sets of patterns and applicability to a broad range of wavelengths and compatibility with compressive sensing. To achieve a better control of illumination strategies, modulators of many kinds have long been available in the optical regime. However, analogous technology to control of phase and amplitude of electron beams does not exist. We approach this electron microscopy challenge from an optics perspective, with a novel approach to imaging with non-orthogonal pattern sets using ghost imaging. Assessed first in the optical regime and subsequently in electron microscopy, we present a methodology that is applicable at different spectral regions and robust to non-orthogonality. The distributed illumination pattern sets also result in a reduced peak intensity, thereby potentially reducing damage of samples during imaging. This imaging approach is potentially translatable beyond both regimes explored here, as a single-element detector system.