We apply crystal theory to affine Schubert calculus, Gromov-Witten invariants for the complete flag manifold, and the positroid stratification of the positive Grassmannian. We introduce operators on decompositions of elements in the type-$A$ affine Weyl group and produce a crystal reflecting the internal structure of the generalized Young modules whose Frobenius image is represented by stable Schubert polynomials. We apply the crystal framework to products of a Schur function with a $k$-Schur function, consequently proving that a subclass of 3-point Gromov-Witten invariants of complete flag varieties for $\mathbb C^n$ enumerate the highest weight elements under these operators. Included in this class are the Schubert structure constants in the (quantum) product of a Schubert polynomial with a Schur function $s_λ$ for all $|λ^\vee|< n$. Another by-product gives a highest weight formulation for various fusion coefficients of the Verlinde algebra and for the Schubert decomposition of certain positroid classes.

42 pages; version to appear in IMRN