(Translated by https://www.hiragana.jp/)
Can p-values be meaningfully interpreted without random sampling?
IDEAS home Printed from https://ideas.repec.org/p/osf/socarx/yazr8.html
   My bibliography  Save this paper

Can p-values be meaningfully interpreted without random sampling?

Author

Listed:
  • Hirschauer, Norbert
  • Gruener, Sven
  • Mußhoff, Oliver
  • Becker, Claudia
  • Jantsch, Antje

Abstract

Besides the inferential errors that abound in the interpretation of p-values, the probabilistic pre-conditions (i.e. random sampling or equivalent) for using them at all are not often met by observa-tional studies in the social sciences. This paper systematizes different sampling designs and discusses the restrictive requirements of data collection that are the sine-qua-non for using p-values.

Suggested Citation

  • Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2019. "Can p-values be meaningfully interpreted without random sampling?," SocArXiv yazr8, Center for Open Science.
    • Handle: RePEc:osf:socarx:yazr8
    DOI: 10.31219/osf.io/yazr8
    as

    Download full text from publisher

    File URL: https://osf.io/download/5d56ae446e9c70001cfe3dd1/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/yazr8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492, National Bureau of Economic Research, Inc.
    3. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    4. repec:bla:obuest:v:62:y:2000:i:2:p:305-22 is not listed on IDEAS
    5. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "Inference in economic experiments," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 14, pages 1-14.
    6. Joyce J Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 101(5), pages 1337-1352.
    7. Massenot, Baptiste & Pettinicchi, Yuri, 2018. "Can firms see into the future? Survey evidence from Germany," Journal of Economic Behavior & Organization, Elsevier, vol. 145(C), pages 66-79.
    8. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    9. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 318, University of California, Davis, Department of Economics.
    10. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 107, University of California, Davis, Department of Economics.
    11. Walter Krämer, 2011. "The Cult of Statistical Significance – What Economists Should and Should Not Do to Make their Data Talk," Schmollers Jahrbuch : Journal of Applied Social Science Studies / Zeitschrift für Wirtschafts- und Sozialwissenschaften, Duncker & Humblot, Berlin, vol. 131(3), pages 455-468.
    12. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    13. James G. MacKinnon, 2019. "How cluster-robust inference is changing applied econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 52(3), pages 851-881, August.
    14. Joyce J. Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, John Wiley & Sons, vol. 101(5), pages 1337-1352, October.
    15. J. B. Copas & H. G. Li, 1997. "Inference for Non‐random Samples," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 59(1), pages 55-95.
    16. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    17. Byung‐Joo Lee & L. C. Marsh, 2000. "Sample Selection Bias Correction for Missing Response Observations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 62(2), pages 305-322, May.
    18. David Trafimow, 2019. "Five Nonobvious Changes in Editorial Practice for Editors and Reviewers to Consider When Evaluating Submissions in a Post p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 340-345, March.
    19. Gary Solon & Steven J. Haider & Jeffrey M. Wooldridge, 2015. "What Are We Weighting For?," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 301-316.
    20. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    21. Andrew Gelman & John Carlin, 2017. "Some Natural Solutions to the -Value Communication Problem—and Why They Won’t Work," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 899-901, July.
    22. Donald Berry, 2017. "A -Value to Die For," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 895-897, July.
    23. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    24. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    25. Vella, Francis & Verbeek, Marno, 1999. "Two-step estimation of panel data models with censored endogenous variables and selection bias," Journal of Econometrics, Elsevier, vol. 90(2), pages 239-263, June.
    26. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    27. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    28. Greene, William H, 1981. "Sample Selection Bias as a Specification Error: Comment," Econometrica, Econometric Society, vol. 49(3), pages 795-798, May.
    29. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, October.
    30. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2019. "Economic experiments and inference," SocArXiv 67mws, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dufhues, Thomas & Möllers, Judith & Jantsch, Antje & Buchenrieder, Gertrud & Camfield, Laura, 2023. "Don’t look up! Individual income comparisons and subjective well-being of students in Thailand," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 24(2), pages 477-503.
    2. Filiptseva, Anna & Filler, Günther & Odening, Martin, 2023. "Compensation schemes for plant quarantine pest costs: A case study for Germany," Economic Analysis and Policy, Elsevier, vol. 80(C), pages 1381-1395.
    3. Hirschauer, Norbert & Grüner , Sven, 2021. "A Primer on p-Value Thresholds and αあるふぁ-Levels – Two Different Kettles of Fish," German Journal of Agricultural Economics, Humboldt-Universitaet zu Berlin, Department for Agricultural Economics, vol. 70(02), January.
    4. Gruener, Sven, 2019. "An empirical study on Internet-based false news stories: experiences, problem awareness, and responsibilities," SocArXiv xbez9, Center for Open Science.
    5. Schröter, Iris & Mergenthaler, Marcus, 2021. "Applying the HEXACO Model of Personality to German Livestock Farmers: Item Scale Validation, Personality Structure and Influence on Participation in Livestock Certification Schemes," International Journal on Food System Dynamics, International Center for Management, Communication, and Research, vol. 12(03), September.
    6. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    7. , Hirschauer, 2022. "Some Thoughts About Statistical Inference In The 21st Century," SocArXiv exdfg, Center for Open Science.
    8. Krzysztof Jajuga & Józef Pociecha & Mirosław Szreder, 2024. "Statistical inference and statistical learning in economic research – selected challenges," Ekonomista, Polskie Towarzystwo Ekonomiczne, issue 2, pages 138-154.
    9. Peter Backus & Thien Nguyen, 2021. "The Effect of the Sex Buyer Law on the Market for Sex, Sexual Health and Sexual Violence," Economics Discussion Paper Series 2106, Economics, The University of Manchester.
    10. J. M. Santos & H. Horta & H. Luna, 2022. "The relationship between academics’ strategic research agendas and their preferences for basic research, applied research, or experimental development," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4191-4225, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Miguel Santolino & Magnus Söderberg, 2014. "Modelling appellate courts’ responses in motor injury disputes," European Journal of Law and Economics, Springer, vol. 38(3), pages 393-407, December.
    2. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    3. Morrissey, Karyn & Kinderman, Peter & Pontin, Eleanor & Tai, Sara & Schwannauer, Mathias, 2016. "Web based health surveys: Using a Two Step Heckman model to examine their potential for population health analysis," Social Science & Medicine, Elsevier, vol. 163(C), pages 45-53.
    4. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    5. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    6. Embaye, Weldensie T. & Bergtold, Jason S. & Archer, David & Flora, Cornelia & Andrango, Graciela C. & Odening, Marting & Buysse, Jeroen, 2018. "Examining farmers' willingness to grow and allocate land for oilseed crops for biofuel production," Energy Economics, Elsevier, vol. 71(C), pages 311-320.
    7. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    8. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    9. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    10. P.W. Miller & S. Rummery, 1989. "Gender Wage Discrimination in Australia: A reassessment," Economics Discussion / Working Papers 89-21, The University of Western Australia, Department of Economics.
    11. Takashi Yamagata & Chris Orme, 2005. "On Testing Sample Selection Bias Under the Multicollinearity Problem," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 467-481.
    12. Benjamin L. Collier & Andrew F. Haughwout & Howard C. Kunreuther & Erwann O. Michel‐Kerjan, 2020. "Firms’ Management of Infrequent Shocks," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 52(6), pages 1329-1359, September.
    13. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    14. Turati, Riccardo, 2024. "Network Abroad and Culture: Global Individual-Level Evidence," GLO Discussion Paper Series 1488, Global Labor Organization (GLO).
    15. Ian Gazeley & Rose Holmes & Andrew Newell & Kevin Reynolds & Hector Gutierrez Rufrancos, 2023. "Escaping from hunger before WW1: the nutritional transition and living standards in Western Europe and USA in the late nineteenth century," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(3), pages 533-565, September.
    16. Gregory S. Crawford & Nicola Pavanini & Fabiano Schivardi, 2018. "Asymmetric Information and Imperfect Competition in Lending Markets," American Economic Review, American Economic Association, vol. 108(7), pages 1659-1701, July.
    17. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    18. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    19. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    20. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.

    More about this item

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:socarx:yazr8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://arabixiv.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.