The virial expansion characterizes the high-temperature approach to the quantum-classical crossover in any quantum many-body system. Here, we calculate the virial coefficients up to the fifth order of Fermi gases in one, two, and three dimensions, with attractive contact interactions, as relevant for a variety of applications in atomic and nuclear physics. To that end, we discretize the imaginary-time direction and calculate the relevant canonical partition functions. In coarse discretizations, we obtain analytic results featuring relationships between the interaction-induced changes Δb3, Δb4, and Δb5 as functions of Δb2, the latter being exactly known in many cases by virtue of the Beth-Uhlenbeck formula. Using automated-algebra methods, we push our calculations to progressively finer discretizations and extrapolate to the continuous-time limit. We find excellent agreement for Δb3 with previous calculations in all dimensions and we formulate predictions for Δb4 and Δb5 in one and two dimensions. We also provide, for a range of couplings, the subspace contributions Δb31, Δb22, Δb41, and Δb32, which determine the equation of state and static response of polarized systems at high temperature. As a performance check, we compare the density equation of state and Tan contact with quantum Monte Carlo calculations, diagrammatic approaches, and experimental data where available. Finally, we apply Padé and Padé-Borel resummation methods to extend the usefulness of the virial coefficients to approach and in some cases go beyond the unit-fugacity point.