Accurate time-delay measurement is at the core of many modern technologies. Here, we present a temporal-mode demultiplexing scheme that achieves the ultimate quantum precision for the simultaneous estimation of the temporal centroid, the time offset, and the relative intensities of an incoherent mixture of ultrashort pulses at the single-photon level. We experimentally resolve temporal separations ten times smaller than the pulse duration, as well as imbalanced intensities differing by a factor of $10^{2}$. This represents an improvement of more than an order of magnitude over the best standard methods based on intensity detection.