Path delays in IP networks are important metrics, required by network operators for assessment, planning, and fault diagnosis. Monitoring delays of all source-destination pairs in a large network is however challenging and wasteful of resources. The present paper develops a spatio-temporal prediction approach to track and predict network-wide path delays using measurements on only a few paths. The proposed algorithm uses a space-time Kalman filter that exploits both topological as well as historical data. The resulting predictor is optimal in the class of linear predictors, and outperforms competing alternatives on real-world datasets.