(Translated by https://www.hiragana.jp/)
Model Evaluation Using Stochastic Simulations: The Case of the Econometric Model KOSMOS
IDEAS home Printed from https://ideas.repec.org/p/hhs/nierwp/0061.html
   My bibliography  Save this paper

Model Evaluation Using Stochastic Simulations: The Case of the Econometric Model KOSMOS

Author

Listed:
  • Gajda, Jan B.

    (National Institute of Economic Research)

  • Markowski, Aleksander

    (National Institute of Economic Research)

Abstract

One aspect of model behaviour that is of interest to the model builder is sensitivity to different forms of errors. This can be investigated using stochastic simulations, as shown by Gajda [1995]. The method involves generating random numbers from a given (usually normal) distribution and introducing them as shocks to the model. In particular, stochastic simulations can be employed to make an empirical investigation into the following aspects of the model: the effects on the forecast of random disturbances, the effects on the forecast of random variation in equation parameters (sampling errors), error propagation and accumulation patterns in the model, the effects on the forecast of (random) errors in the exogenous variables. The results of stochastic simulations can provide information on - inter alia - the sampling distribution of the model forecast. In particular, the model builder may be interested in the shape of this distribution. If it is not symmetric, the mean (stochastic) forecast will be different from the median forecast (which under certain conditions is equal to the deterministic forecast). Furthermore, if the distribution is skewed, a typical stochastic forecast (represented by the mode) will systematically underestimate (or overestimate) both the mean and the median forecasts. The purpose of the present paper is to investigate KOSMOS, the econometric model of the National Institute of Economic Research in Stockholm, from the point of view of the first three aspects mentioned above. The main aim of this exercise is to look for ”weak links” in the model, i.e. to find out which equations introduce most uncertainty and at the same time are crucial for the forecast because of their strong influence on it. Thus, our interest is not only in assessing the forecast error variance as a descriptive statistic, but also - and primarily - in finding those equations that are important for error propagation and those coefficients whose values are crucial for the model. The outline of the paper is as follows. Section 2 discusses the analysis of expected forecast errors (for linear models) based on analytical formulae. In Section 3, model simulations and stochastic simulations are defined. The two subsequent sections discuss the purpose of our experiments and their design, respectively. Section 6 gives a brief description of the econometric model KOSMOS, whose equations are subject to our investigation. Section 7 describes the results of stochastic simulations with additive equation disturbances. Section 8 presents the results of stochastic simulations with both equation disturbances and disturbances to the estimated coefficients. Sampling distributions of forecasts are discussed in Section 9. Section 10 concludes.

Suggested Citation

  • Gajda, Jan B. & Markowski, Aleksander, 1998. "Model Evaluation Using Stochastic Simulations: The Case of the Econometric Model KOSMOS," Working Papers 61, National Institute of Economic Research.
    • Handle: RePEc:hhs:nierwp:0061
    as

    Download full text from publisher

    File URL: http://www.konj.se/download/18.4bf39736154c6660a10982ed/1463754704237/WP_61.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bianchi, Carlo & Calzolari, Giorgio & Corsi, Paolo, 1981. "Standard errors of multipliers and forecasts from structural coefficients with block-diagonal covariance matrix," MPRA Paper 22678, University Library of Munich, Germany, revised 1981.
    2. Fair, Ray C & Taylor, John B, 1983. "Solution and Maximum Likelihood Estimation of Dynamic Nonlinear Rational Expectations Models," Econometrica, Econometric Society, vol. 51(4), pages 1169-1185, July.
    3. Bianchi, Carlo & Calzolari, Giorgio & Corsi, Paolo, 1981. "Estimating asymptotic standard errors and inconsistencies of impact multipliers in nonlinear econometric models," Journal of Econometrics, Elsevier, vol. 16(3), pages 277-294, August.
    4. Bianchi, Carlo & Calzolari, Giorgio & Brillet, Jean-Louis, 1987. "Measuring forecast uncertainty : A review with evaluation based on a macro model of the French economy," International Journal of Forecasting, Elsevier, vol. 3(2), pages 211-227.
    5. Fair, Ray C, 1980. "Estimating the Uncertainty of Policy Effects in Nonlinear Models," Econometrica, Econometric Society, vol. 48(6), pages 1381-1391, September.
    6. Brown, Bryan W. & Mariano, Roberto S., 1989. "Predictors in Dynamic Nonlinear Models: Large-Sample Behavior," Econometric Theory, Cambridge University Press, vol. 5(3), pages 430-452, December.
    7. Calzolari, Giorgio, 1979. "Antithetic variates to estimate the simulation bias in non-linear models," Economics Letters, Elsevier, vol. 4(4), pages 323-328.
    8. Fair, Ray C, 1980. "Estimating the Expected Predictive Accuracy of Econometric Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 21(2), pages 355-378, June.
    9. Bianchi, Carlo & Calzolari, Giorgio, 1982. "Evaluating forecast uncertainty due to errors in estimated coefficients: empirical comparison of alternative methods," MPRA Paper 22559, University Library of Munich, Germany.
    10. Wallis, Kenneth F., 1982. "'Time-series' versus 'econometric' forecasts : A non-linear regression counterexample," Economics Letters, Elsevier, vol. 10(3-4), pages 309-315.
    11. Hall, S G, 1986. "The Application of Stochastic Simulation Techniques to the National Institute's Model 7," The Manchester School of Economic & Social Studies, University of Manchester, vol. 54(2), pages 180-201, June.
    12. Bianchi, Carlo & Calzolari, Giorgio & Corsi, Paolo, 1976. "Divergences in the results of stochastic and deterministic simulation of an Italian non linear econometric model," MPRA Paper 21287, University Library of Munich, Germany.
    13. Oliver, F. R., 1979. "Rebased and revised data -- their effects on econometric estimates," Economics Letters, Elsevier, vol. 4(4), pages 355-359.
    14. Bianchi, Carlo & Calzolari, Giorgio & Weihs, Claus, 1986. "Parametric and nonparametric Monte Carlo estimates of standard errors of forecasts in econometric models," MPRA Paper 29120, University Library of Munich, Germany.
    15. Sterbenz, Frederic P & Calzolari, Giorgio, 1990. "Alternative Specifications of the Error Process in the Stochastic Simulation of Econometric Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 5(2), pages 137-150, April-Jun.
    16. Feldstein, Martin S, 1971. "The Error of Forecast in Econometric Models when the Forecast-Period Exogenous Variables are Stochastic," Econometrica, Econometric Society, vol. 39(1), pages 55-60, January.
    17. Bianchi, Carlo & Calzolari, Giorgio, 1980. "The One-Period Forecast Errors in Nonlinear Econometric Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 21(1), pages 201-208, February.
    18. Mariano, Roberto S, 1982. "Analytical Small-Sample Distribution Theory in Econometrics: The Simultaneous-Equations Case," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 23(3), pages 503-533, October.
    19. Salmon, Mark & Wallis, Kenneth F, 1980. "Model Validation and Forecast Comparisons : Theoretical and Practical Considerations," The Warwick Economics Research Paper Series (TWERPS) 184, University of Warwick, Department of Economics.
    20. Brown, Bryan W & Mariano, Roberto S, 1984. "Residual-Based Procedures for Prediction and Estimation in a Nonlinear Simultaneous System," Econometrica, Econometric Society, vol. 52(2), pages 321-343, March.
    21. Hendry, David F., 1984. "Monte carlo experimentation in econometrics," Handbook of Econometrics, in: Z. Griliches† & M. D. Intriligator (ed.), Handbook of Econometrics, edition 1, volume 2, chapter 16, pages 937-976, Elsevier.
    22. Bianchi, Carlo & Calzolari, Giorgio & Corsi, Paolo, 1979. "A Monte Carlo approach to compute the asymptotic standard errors of dynamic multipliers," Economics Letters, Elsevier, vol. 2(2), pages 161-164.
    23. Fisher, Paul & Salmon, Mark, 1986. "On Evaluating the Importance of Nonlinearity in Large Macroeconometric Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 27(3), pages 625-646, October.
    24. Corker, R. J. & Holly, S. & Ellis, R. G., 1986. "Uncertainty and forecast precision," International Journal of Forecasting, Elsevier, vol. 2(1), pages 53-53.
    25. Brown, Bryan W & Mariano, Roberto S, 1989. "Measures of Deterministic Prediction Bias in Nonlinear Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 30(3), pages 667-684, August.
    26. Brillet, Jean-Louis & Calzolari, Giorgio & Panattoni, Lorenzo, 1986. "Coherent optimal prediction with large nonlinear systems: an example based on a French model," MPRA Paper 29057, University Library of Munich, Germany.
    27. Mariano, Roberto S & Brown, Bryan W, 1983. "Asymptotic Behavior of Predictors in a Nonlinear Simultaneous System," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 24(3), pages 523-536, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dobrescu, Emilian & Pauna, Bianca, 2007. "Stochastic simulations on the Romanian macroeconomic model," MPRA Paper 35723, University Library of Munich, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bianchi, Carlo & Calzolari, Giorgio & Brillet, Jean-Louis, 1987. "Measuring forecast uncertainty : A review with evaluation based on a macro model of the French economy," International Journal of Forecasting, Elsevier, vol. 3(2), pages 211-227.
    2. Calzolari, Giorgio & Panattoni, Lorenzo, 1990. "Mode predictors in nonlinear systems with identities," International Journal of Forecasting, Elsevier, vol. 6(3), pages 317-326, October.
    3. Calzolari, Giorgio, 1987. "La varianza delle previsioni nei modelli econometrici [Forecast variance in econometric models]," MPRA Paper 23866, University Library of Munich, Germany.
    4. Brillet, Jean-Louis & Calzolari, Giorgio & Panattoni, Lorenzo, 1986. "Coherent optimal prediction with large nonlinear systems: an example based on a French model," MPRA Paper 29057, University Library of Munich, Germany.
    5. Bianchi, Carlo & Calzolari, Giorgio, 1983. "Standard errors of forecasts in dynamic simulation of nonlinear econometric models: some empirical results," MPRA Paper 22657, University Library of Munich, Germany, revised 1983.
    6. Bianchi, Carlo & Calzolari, Giorgio & Weihs, Claus, 1986. "Parametric and nonparametric Monte Carlo estimates of standard errors of forecasts in econometric models," MPRA Paper 29120, University Library of Munich, Germany.
    7. Mariano, Roberto S, 1985. "Finite-Sample Properties in Stochastic Predictors in Nonlinear Systems : Some Initial Results," The Warwick Economics Research Paper Series (TWERPS) 266, University of Warwick, Department of Economics.
    8. Bianchi, Carlo & Calzolari, Giorgio, 1983. "Confidence intervals of forecasts from nonlinear econometric models," MPRA Paper 29025, University Library of Munich, Germany.
    9. Dag Kolsrud, 2008. "Stochastic Ceteris Paribus Simulations," Computational Economics, Springer;Society for Computational Economics, vol. 31(1), pages 21-43, February.
    10. Neil R. Ericsson & Jaime R. Marquez, 1990. "Evaluating the predictive performance of trade-account models," International Finance Discussion Papers 377, Board of Governors of the Federal Reserve System (U.S.).
    11. Bianchi, Carlo & Calzolari, Giorgio, 1982. "Evaluating forecast uncertainty due to errors in estimated coefficients: empirical comparison of alternative methods," MPRA Paper 22559, University Library of Munich, Germany.
    12. Calzolari, Giorgio & Bianchi, Carlo & Corsi, Paolo & Panattoni, Lorenzo, 1982. "Uncertainty of policy recommendations for nonlinear econometric models: some empirical results," MPRA Paper 28846, University Library of Munich, Germany.
    13. van Garderen, Kees Jan, 2001. "Optimal prediction in loglinear models," Journal of Econometrics, Elsevier, vol. 104(1), pages 119-140, August.
    14. Fair Ray C, 2003. "Bootstrapping Macroeconometric Models," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 7(4), pages 1-26, December.
    15. Bianchi, Carlo & Brillet, Jean-Louis & Calzolari, Giorgio, 1983. "Analysis and measurement of the uncertainty in Mini-Dms model for the French economy," MPRA Paper 29056, University Library of Munich, Germany.
    16. Bianchi, Carlo & Brillet, Jean-Louis & Calzolari, Giorgio, 1984. "Analyse et mesure de l'incertitude en prevision d'un modele econometrique. Application au modele mini-DMS [Analysis and measurement of forecast uncertainty in an econometric model. Application to m," MPRA Paper 22565, University Library of Munich, Germany, revised 1984.
    17. Skold, Karl Durwood, 1989. "The integration of alternative information systems: an application to the Hogs and Pigs report," ISU General Staff Papers 1989010108000010239, Iowa State University, Department of Economics.
    18. Neil R. Ericsson & Jaime Marquez, 1998. "A framework for economic forecasting," Econometrics Journal, Royal Economic Society, vol. 1(Conferenc), pages 228-266.
    19. Calzolari, Giorgio & Panattoni, Lorenzo, 1984. "Evaluating Forecast Uncertainty in Econometric Models: The Effect of Alternative Estimators of Maximum Likelihood Covariance Matrix," MPRA Paper 28806, University Library of Munich, Germany.
    20. Bianchi, Carlo & Calzolari, Giorgio & Corsi, Paolo & Panattoni, Lorenzo, 1985. "Asymptotic properties of dynamic multipliers in nonlinear econometric models," MPRA Paper 24401, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hhs:nierwp:0061. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sarah Hegardt Grant (email available below). General contact details of provider: https://edirc.repec.org/data/kongvse.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.