For the investigation of processes happening on the time scale of the motion of bound electrons, well-controlled X-ray pulses with durations in the few-femtosecond and even sub-femtosecond range are a necessary prerequisite. Novel free-electron lasers sources provide these ultrashort, high-brightness X-ray pulses, but their unique aspects open up concomitant challenges for their characterization on a suitable time scale. In this review paper we describe progress and results of recent work on ultrafast pulse characterization at soft and hard X-ray free-electron lasers. We report on different approaches to laser-assisted time-domain measurements, with specific focus on single-shot characterization of ultrashort X-ray pulses from self-amplified spontaneous emission-based and seeded free-electron lasers. The method relying on the sideband measurement of X-ray electron ionization in the presence of a dressing optical laser field is described first. When the X-ray pulse duration is shorter than half the oscillation period of the streaking field, few-femtosecond characterization becomes feasible via linear streaking spectroscopy. Finally, using terahertz fields alleviates the issue of arrival time jitter between streaking laser and X-ray pulse, but compromises the achievable temporal resolution. Possible solutions to these remaining challenges for single-shot, full time–energy characterization of X-ray free-electron laser pulses are proposed in the outlook at the end of the review.