We analyze the fractionalization of the Fourier transform (FT), starting from the minimal premise that repeated application of the fractional Fourier transform (FrFT) a sufficient number of times should give back the FT. There is a qualitative increase in the richness of the solution manifold, from U(1) (the circle S1) in the one-dimensional case to U(2) (the four-parameter group of 2 x 2 unitary matrices) in the two-dimensional case [rather than simply U(1) x U(1)]. Our treatment clarifies the situation in the N-dimensional case. The parameterization of this manifold (a fiber bundle) is accomplished through two powers running over the torus T2 = S1 x S1 and two parameters running over the Fourier sphere S2. We detail the spectral representation of the FrFT: The eigenvalues are shown to depend only on the T2 coordinates; the eigenfunctions, only on the S2 coordinates. FrFT's corresponding to special points on the Fourier sphere have for eigenfunctions the Hermite-Gaussian beams and the Laguerre-Gaussian beams, while those corresponding to generic points are SU(2)-coherent states of these beams. Thus the integral transform produced by every Sp(4, R) first-order system is essentially a FrFT.