The aim of this paper is to present the multi-factor swap market model with non-parametric local volatility functions and stochastic volatility scaling factors. We provide a Dupire-like formula with which calibration can be carried out with the particle algorithm in an efficient manner. We also discuss how the calibration method can be made applicable in the context of Libor Market Model. We show high accuracy of our calibration algorithm by numerical experiments.