Robust forecast comparison

Research, Article, Preprint English OPEN
Jin, Sainan ; Corradi, Valentina ; Swanson, Norman (2015)
  • Publisher: New Brunswick, NJ: Rutgers University, Department of Economics
  • Related identifiers: doi: 10.1017/S0266466616000426
  • Subject: convex loss function, empirical processes, forecast superiority, general loss function | C22 | C12 | forecast superiority | convex loss function | general loss function | empirical processes
    • jel: jel:C12 | jel:C22
      ddc: ddc:330

Forecast accuracy is typically measured in terms of a given loss function. However, as a consequence of the use of misspecified models in multiple model comparisons, relative forecast rankings are loss function dependent. This paper addresses this issue by using a novel criterion for forecast evaluation which is based on the entire distribution of forecast errors. We introduce the concepts of general-loss (GL) forecast superiority and convex-loss (CL) forecast superiority, and we establish a mapping between GL (CL) superiority and first (second) order stochastic dominance. This allows us to develop a forecast evaluation procedure based on an out-of-sample generalization of the tests introduced by Linton, Maasoumi and Whang (2005). The asymptotic null distributions of our test statistics are nonstandard, and resampling procedures are used to obtain the critical values. Additionally, the tests are consistent and have nontrivial local power under a sequence of local alternatives. In addition to the stationary case, we outline theory extending our tests to the case of heterogeneity induced by distributional change over time. Monte Carlo simulations suggest that the tests perform reasonably well in finite samples; and an application to exchange rate data indicates that our tests can help identify superior forecasting models, regardless of loss function.
  • References (59)
    59 references, page 1 of 6

    [1] Andrews, D.W.K. (2012), Similar-on-The-Boundary Tests for Moment Inequalities Exist, but Have Poor Power, Cowles Foundation Working Paper 1815R.

    [2] Andrews, D. W. K., and P. J. Barwick (2012), Inference for Parameters Defined by Moment Inequalities: A Recommended Moment Selection Procedure, Econometrica 80, 2805-2826.

    [3] Andrews, D. W. K. and D. Pollard (1994), An Introduction to Functional Central Limit Theorems for Dependent Stochastic Processes, International Statistical Review 62, 119-132.

    [4] Andrews, D. W. K., and X. Shi (2009), Inference Based on Conditional Moment Inequalities, Econometrica 81, 609-666.

    [5] Andrews, D. W. K., and X. Shi (2014), Nonparametric inference based on conditional moment inequalities, Journal of Econometrics 179, 31-45.

    [6] Andrews, D. W. K., and G. Soares (2010), Inference for Parameters Defined by Moment Inequalities Using Generalized Moment Selection, Econometrica 78, 119-157.

    [7] Bosq, D. (1996), Nonparametric Statistics for Stochastic Processes, Springer-Verlag, New York.

    [8] Chao, J. C., V. Corradi and N. R. Swanson (2001), An out of Sample Test for Granger Causality, Macroeconomic Dynamics 5, 598-620.

    [9] Chiang, T. (1986), Empirical Analysis of the Predictor of Future Spot Rates, Journal of Financial Research, 153-162.

    [10] Christoffersen, P. (1998), Evaluating Interval Forecasts, International Economic Review 39, 841-862.

  • Similar Research Results (1)
  • Metrics
    views in OpenAIRE
    views in local repository
    downloads in local repository

    The information is available from the following content providers:

    From Number Of Views Number Of Downloads
    Surrey Research Insight - IRUS-UK 0 7
Share - Bookmark