We study the moduli surface for pairs of elliptic curves together with an isomorphism between their N-torsion groups. The Weil pairing gives a "determinant" map from this moduli surface to (Z/NZ)*; its fibers are the components of the surface. We define spaces of modular forms on these components and Hecke correspondences between them, and study how those spaces of modular forms behave as modules for the Hecke algebra. We discover that the component with determinant -1 is somehow the "dominant" one; we characterize the difference between its spaces of modular forms and the spaces of modular forms on the other components using forms with complex multiplication. Finally, we show some simplifications that arise when N is prime, including a complete determination of such CM-forms, and give numerical examples.

30 pages