We study topological string theory on elliptically fibered Calabi-Yau threefolds using mirror symmetry. We compute higher genus topological string amplitudes and express these in terms of polynomials of functions constructed from the special geometry of the moduli space. The polynomials are fixed by the holomorphic anomaly equations supplemented by the expected behavior at the boundary in moduli space. We further expand the amplitudes in the base moduli of the elliptic fibration and find that the fiber moduli dependence is captured by a finer polynomial structure in terms of the modular forms of the modular group of the elliptic curve. We further find a recursive equation which governs this finer structure and which can be related to the anomaly equations for correlation functions.

56 pages, 1 figure. v2: typos corrected, eqns (3.14) - (3.16) removed, two examples added. v3: typos in (4.56) and (4.59) corrected