(Translated by https://www.hiragana.jp/)
A Method for Agent-Based Models Validation
IDEAS home Printed from https://ideas.repec.org/p/thk/wpaper/42.html
   My bibliography  Save this paper

A Method for Agent-Based Models Validation

Author

Listed:
  • Mattia Guerini

    (Institute of Economics, Scuola Superiore Sant'Anna, Pisa, Italy)

  • Alessio Moneta

    (Institute of Economics, Scuola Superiore Sant'Anna, Pisa, Italy)

Abstract

This paper proposes a new method to empirically validate simulation models that generate artificial time series data comparable with real-world data. The approach is based on comparing structures of vector autoregression models that are estimated from both artificial and real-world data by means of causal search algorithms. This relatively simple procedure is able to tackle both the problem of confronting theoretical simulation models with the data and the problem of comparing different models in terms of their empirical reliability. The paper also provides an application of the validation procedure to the Dosi et al. (2015) macro-model.

Suggested Citation

  • Mattia Guerini & Alessio Moneta, 2016. "A Method for Agent-Based Models Validation," Working Papers Series 42, Institute for New Economic Thinking.
    • Handle: RePEc:thk:wpaper:42
    DOI: 10.2139/ssrn.2772133
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.2139/ssrn.2772133
    File Function: First version, 2016
    Download Restriction: no
    File URL: https://libkey.io/10.2139/ssrn.2772133?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    2. S. Alfarano & T. Lux & F. Wagner, 2007. "Empirical validation of stochastic models of interacting agents," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 55(2), pages 183-187, January.
    3. Blanchard, Olivier Jean & Quah, Danny, 1989. "The Dynamic Effects of Aggregate Demand and Supply Disturbances," American Economic Review, American Economic Association, vol. 79(4), pages 655-673, September.
    4. Marco Del Negro & Frank Schorfheide, 2006. "How good is what you've got? DSGE-VAR as a toolkit for evaluating DSGE models," Economic Review, Federal Reserve Bank of Atlanta, vol. 91(Q 2), pages 21-37.
    5. repec:hal:spmain:info:hdl:2441/3qv4spsglp8tmorvev1h0duo4p is not listed on IDEAS
    6. Martin Fukac & Adrian Pagan, 2006. "Issues in Adopting DSGE Models for Use in the Policy Process," Working Papers 2006/6, Czech National Bank.
    7. Giorgio Fagiolo & Andrea Roventini, 2012. "Macroeconomic Policy in DSGE and Agent-Based Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 67-116.
    8. Del Negro, Marco & Schorfheide, Frank, 2008. "Forming priors for DSGE models (and how it affects the assessment of nominal rigidities)," Journal of Monetary Economics, Elsevier, vol. 55(7), pages 1191-1208, October.
    9. Jean-Luc Gaffard & Mauro Napoletano, 2012. "Improving the toolbox. New advances in Agent-based and Computational Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 7-13.
    10. repec:hal:spmain:info:hdl:2441/20d1ncsepb9ssq3b3v4s6nbc41 is not listed on IDEAS
    11. Johansen, Soren & Juselius, Katarina, 1990. "Maximum Likelihood Estimation and Inference on Cointegration--With Applications to the Demand for Money," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 52(2), pages 169-210, May.
    12. Matthias Lengnick & Hans-Werner Wohltmann, 2013. "Agent-based financial markets and New Keynesian macroeconomics: a synthesis," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 8(1), pages 1-32, April.
    13. Sylvain Barde, 2015. "Direct calibration and comparison of agent-based herding models of financial markets," Studies in Economics 1507, School of Economics, University of Kent.
    14. Andrew G. Haldane & Vasileios Madouros, 2012. "The dog and the frisbee," Proceedings - Economic Policy Symposium - Jackson Hole, Federal Reserve Bank of Kansas City, pages 109-159.
    15. Dosi, Giovanni & Fagiolo, Giorgio & Roventini, Andrea, 2010. "Schumpeter meeting Keynes: A policy-friendly model of endogenous growth and business cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1748-1767, September.
    16. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    17. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    18. Guerini, Mattia & Napoletano, Mauro & Roventini, Andrea, 2018. "No man is an Island: The impact of heterogeneity and local interactions on macroeconomic dynamics," Economic Modelling, Elsevier, vol. 68(C), pages 82-95.
    19. Kirman, Alan, 1989. "The Intrinsic Limits of Modern Economic Theory: The Emperor Has No Clothes," Economic Journal, Royal Economic Society, vol. 99(395), pages 126-139, Supplemen.
    20. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    21. Recchioni, Maria Cristina & Tedeschi, Gabriele & Gallegati, Mauro, 2015. "A calibration procedure for analyzing stock price dynamics in an agent-based framework," Journal of Economic Dynamics and Control, Elsevier, vol. 60(C), pages 1-25.
    22. Kirman, Alan, 2016. "Ants And Nonoptimal Self-Organization: Lessons For Macroeconomics," Macroeconomic Dynamics, Cambridge University Press, vol. 20(2), pages 601-621, March.
    23. Lux, Thomas, 2012. "Estimation of an agent-based model of investor sentiment formation in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1284-1302.
    24. Marco Del Negro & Frank Schorfheide, 2004. "Priors from General Equilibrium Models for VARS," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 45(2), pages 643-673, May.
    25. Bernanke, Ben S., 1986. "Alternative explanations of the money-income correlation," Carnegie-Rochester Conference Series on Public Policy, Elsevier, vol. 25(1), pages 49-99, January.
    26. Ireland, Peter N., 2004. "A method for taking models to the data," Journal of Economic Dynamics and Control, Elsevier, vol. 28(6), pages 1205-1226, March.
    27. Alfarano, Simone & Lux, Thomas & Wagner, Friedrich, 2006. "Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 38-42.
    28. Selva Demiralp & Kevin D. Hoover, 2003. "Searching for the Causal Structure of a Vector Autoregression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 745-767, December.
    29. Canova, Fabio & Sala, Luca, 2009. "Back to square one: Identification issues in DSGE models," Journal of Monetary Economics, Elsevier, vol. 56(4), pages 431-449, May.
    30. Katarina Juselius, 2011. "On the Role of Theory and Evidence in Macroeconomics," Chapters, in: John B. Davis & D. Wade Hands (ed.), The Elgar Companion to Recent Economic Methodology, chapter 17, Edward Elgar Publishing.
    31. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea, 2013. "Income distribution, credit and fiscal policies in an agent-based Keynesian model," Journal of Economic Dynamics and Control, Elsevier, vol. 37(8), pages 1598-1625.
    32. Granger, C. W. J., 1980. "Testing for causality : A personal viewpoint," Journal of Economic Dynamics and Control, Elsevier, vol. 2(1), pages 329-352, May.
    33. Finn E. Kydland & Edward C. Prescott, 1996. "The Computational Experiment: An Econometric Tool," Journal of Economic Perspectives, American Economic Association, vol. 10(1), pages 69-85, Winter.
    34. Jean-Luc Gaffard & Mauro Napoletano, 2012. "Improving the toolbox. New advances in Agent-based and Computational Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 7-13.
    35. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    36. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(1), pages 45-97.
    37. Matthew D. Shapiro & Mark W. Watson, 1988. "Sources of Business Cycle Fluctuations," NBER Chapters, in: NBER Macroeconomics Annual 1988, Volume 3, pages 111-156, National Bureau of Economic Research, Inc.
    38. Tommaso Ciarli, 2012. "Structural Interactions and Long Run Growth. An Application of Experimental Design to Agent Based Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 295-345.
    39. Blake LeBaron & Leigh Tesfatsion, 2008. "Modeling Macroeconomies as Open-Ended Dynamic Systems of Interacting Agents," American Economic Review, American Economic Association, vol. 98(2), pages 246-250, May.
    40. repec:hal:spmain:info:hdl:2441/53r60a8s3kup1vc9l564k4686 is not listed on IDEAS
    41. Alessio Moneta, 2008. "Graphical causal models and VARs: an empirical assessment of the real business cycles hypothesis," Empirical Economics, Springer, vol. 35(2), pages 275-300, September.
    42. repec:hal:spmain:info:hdl:2441/7kr9gv74ut9ngo58gia97t83i7 is not listed on IDEAS
    43. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    44. Kydland, Finn E & Prescott, Edward C, 1982. "Time to Build and Aggregate Fluctuations," Econometrica, Econometric Society, vol. 50(6), pages 1345-1370, November.
    45. Blanchard, Olivier Jean, 1989. "A Traditional Interpretation of Macroeconomic Fluctuations," American Economic Review, American Economic Association, vol. 79(5), pages 1146-1164, December.
    46. Sylvain Barde, 2017. "A Practical, Accurate, Information Criterion for Nth Order Markov Processes," Computational Economics, Springer;Society for Computational Economics, vol. 50(2), pages 281-324, August.
    47. David A. Bessler & Seongpyo Lee, 2002. "Money and prices: U.S. Data 1869-1914 (A study with directed graphs)," Empirical Economics, Springer, vol. 27(3), pages 427-446.
    48. Sims, Christopher A, 1980. "Macroeconomics and Reality," Econometrica, Econometric Society, vol. 48(1), pages 1-48, January.
    49. Mauro Napoletano & Jean-Luc Gaffard & Zakaria Babutsidze, 2012. "Agent Based Models," SciencePo Working papers Main hal-03461262, HAL.
    50. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    51. Simone Alfarano & Thomas Lux & Friedrich Wagner, 2005. "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model," Computational Economics, Springer;Society for Computational Economics, vol. 26(1), pages 19-49, August.
    52. Alessio Moneta & Doris Entner & Patrik O. Hoyer & Alex Coad, 2013. "Causal Inference by Independent Component Analysis: Theory and Applications," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 75(5), pages 705-730, October.
    53. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    54. Selva Demiralp & Kevin D. Hoover, 2003. "Searching for the Causal Structure of a Vector Autoregression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 745-767, December.
    55. Alan P. Kirman, 1992. "Whom or What Does the Representative Individual Represent?," Journal of Economic Perspectives, American Economic Association, vol. 6(2), pages 117-136, Spring.
    56. Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    57. Bianchi, Carlo & Cirillo, Pasquale & Gallegati, Mauro & Vagliasindi, Pietro A., 2008. "Validation in agent-based models: An investigation on the CATS model," Journal of Economic Behavior & Organization, Elsevier, vol. 67(3-4), pages 947-964, September.
    58. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    59. Lucas, Robert Jr, 1976. "Econometric policy evaluation: A critique," Carnegie-Rochester Conference Series on Public Policy, Elsevier, vol. 1(1), pages 19-46, January.
    60. Jakob Grazzini, 2012. "Analysis of the Emergent Properties: Stationarity and Ergodicity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 15(2), pages 1-7.
    61. Giorgio Fagiolo & Alessio Moneta & Paul Windrum, 2007. "A Critical Guide to Empirical Validation of Agent-Based Models in Economics: Methodologies, Procedures, and Open Problems," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 195-226, October.
    62. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    63. Hoover, Kevin D., 2004. "Lost Causes," Journal of the History of Economic Thought, Cambridge University Press, vol. 26(2), pages 149-164, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    2. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    3. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    4. repec:hal:spmain:info:hdl:2441/dcditnq6282sbu1u151qe5p7f is not listed on IDEAS
    5. repec:spo:wpmain:info:hdl:2441/dcditnq6282sbu1u151qe5p7f is not listed on IDEAS
    6. repec:hal:spmain:info:hdl:2441/20hflp7eqn97boh50no50tv67n is not listed on IDEAS
    7. repec:spo:wpmain:info:hdl:2441/20hflp7eqn97boh50no50tv67n is not listed on IDEAS
    8. Francesco Lamperti, 2018. "Empirical validation of simulated models through the GSL-div: an illustrative application," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 143-171, April.
    9. Giovanni Dosi & Andrea Roventini, 2019. "More is different ... and complex! the case for agent-based macroeconomics," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 1-37, March.
    10. Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
    11. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    12. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    13. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    14. repec:hal:spmain:info:hdl:2441/13thfd12aa8rmplfudlgvgahff is not listed on IDEAS
    15. Giorgio Fagiolo & Andrea Roventini, 2016. "Macroeconomic Policy in DGSE and Agent-Based Models Redux," Working Papers hal-03459348, HAL.
    16. Guerini, Mattia & Napoletano, Mauro & Roventini, Andrea, 2018. "No man is an Island: The impact of heterogeneity and local interactions on macroeconomic dynamics," Economic Modelling, Elsevier, vol. 68(C), pages 82-95.
    17. Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    18. Donovan Platt & Tim Gebbie, 2016. "The Problem of Calibrating an Agent-Based Model of High-Frequency Trading," Papers 1606.01495, arXiv.org, revised Mar 2017.
    19. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    20. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    21. Platt, Donovan, 2020. "A comparison of economic agent-based model calibration methods," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
    22. Mario Martinoli & Alessio Moneta & Gianluca Pallante, 2022. "Calibration and Validation of Macroeconomic Simulation Models by Statistical Causal Search," LEM Papers Series 2022/33, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    23. Barde, Sylvain, 2020. "Macroeconomic simulation comparison with a multivariate extension of the Markov information criterion," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    24. repec:spo:wpmain:info:hdl:2441/20d1ncsepb9ssq3b3v4s6nbc41 is not listed on IDEAS
    25. repec:hal:spmain:info:hdl:2441/20d1ncsepb9ssq3b3v4s6nbc41 is not listed on IDEAS
    26. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    27. Donovan Platt, 2022. "Bayesian Estimation of Economic Simulation Models Using Neural Networks," Computational Economics, Springer;Society for Computational Economics, vol. 59(2), pages 599-650, February.

    More about this item

    Keywords

    Models validation; Agent-Based models; Causality; Structural Vector Autoregressions;
    All these keywords.

    JEL classification:

    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • E37 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles - - - Forecasting and Simulation: Models and Applications

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:thk:wpaper:42. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Pia Malaney (email available below). General contact details of provider: https://edirc.repec.org/data/inetnus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.