(Translated by https://www.hiragana.jp/)
Placebo inference on treatment effects when the number of clusters is small
IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v213y2019i1p190-209.html
   My bibliography  Save this article

Placebo inference on treatment effects when the number of clusters is small

Author

Listed:
  • Hagemann, Andreas

Abstract

I introduce a general, Fisher-style randomization testing framework to conduct nearly exact inference about the lack of effect of a binary treatment in the presence of very few, large clusters when the treatment effect is identified across clusters. The proposed randomization test formalizes and extends the intuitive notion of generating null distributions by assigning placebo treatments to untreated clusters. I show that under simple and easily verifiable conditions, the placebo test leads to asymptotically valid inference in a very large class of empirically relevant models. Examples discussed explicitly are (i) least squares regression with cluster-level treatment, (ii) difference-in-differences estimation, and (iii) binary choice models with cluster-level treatment. A simulation study and an empirical example are provided. The proposed inference procedure is easy to implement and performs well with as few as three treated and three untreated clusters.

Suggested Citation

  • Hagemann, Andreas, 2019. "Placebo inference on treatment effects when the number of clusters is small," Journal of Econometrics, Elsevier, vol. 213(1), pages 190-209.
    • Handle: RePEc:eee:econom:v:213:y:2019:i:1:p:190-209
    DOI: 10.1016/j.jeconom.2019.04.011
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304407619300661
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jeconom.2019.04.011?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. repec:clg:wpaper:2013-20 is not listed on IDEAS
    2. James G. MacKinnon & Matthew D. Webb, 2018. "The wild bootstrap for few (treated) clusters," Econometrics Journal, Royal Economic Society, vol. 21(2), pages 114-135, June.
    3. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    4. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    5. Andrew V. Carter & Kevin T. Schnepel & Douglas G. Steigerwald, 2017. "Asymptotic Behavior of a t -Test Robust to Cluster Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 698-709, July.
    6. Ivan A. Canay & Joseph P. Romano & Azeem M. Shaikh, 2017. "Randomization Tests Under an Approximate Symmetry Assumption," Econometrica, Econometric Society, vol. 85, pages 1013-1030, May.
    7. Stephen G. Donald & Kevin Lang, 2007. "Inference with Difference-in-Differences and Other Panel Data," The Review of Economics and Statistics, MIT Press, vol. 89(2), pages 221-233, May.
    8. Hansen, Christian B., 2007. "Asymptotic properties of a robust variance matrix estimator for panel data when T is large," Journal of Econometrics, Elsevier, vol. 141(2), pages 597-620, December.
    9. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    10. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    11. Kloek, T, 1981. "OLS Estimation in a Model Where a Microvariable Is Explained by Aggregates and Contemporaneous Disturbances Are Equicorrelated," Econometrica, Econometric Society, vol. 49(1), pages 205-207, January.
    12. Xiong, Shifeng & Li, Guoying, 2008. "Some results on the convergence of conditional distributions," Statistics & Probability Letters, Elsevier, vol. 78(18), pages 3249-3253, December.
    13. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2005. "A New Asymptotic Theory For Heteroskedasticity-Autocorrelation Robust Tests," Econometric Theory, Cambridge University Press, vol. 21(6), pages 1130-1164, December.
    14. Rustam Ibragimov & Ulrich K. Müller, 2016. "Inference with Few Heterogeneous Clusters," The Review of Economics and Statistics, MIT Press, vol. 98(1), pages 83-96, March.
    15. El Machkouri, Mohamed & Volný, Dalibor & Wu, Wei Biao, 2013. "A central limit theorem for stationary random fields," Stochastic Processes and their Applications, Elsevier, vol. 123(1), pages 1-14.
    16. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2002. "Heteroskedasticity-Autocorrelation Robust Testing Using Bandwidth Equal To Sample Size," Econometric Theory, Cambridge University Press, vol. 18(6), pages 1350-1366, December.
    17. Bester, C. Alan & Conley, Timothy G. & Hansen, Christian B., 2011. "Inference with dependent data using cluster covariance estimators," Journal of Econometrics, Elsevier, vol. 165(2), pages 137-151.
    18. Arnold Janssen, 2005. "Resampling student'st-type statistics," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 57(3), pages 507-529, September.
    19. Horowitz, Joel L, 1992. "A Smoothed Maximum Score Estimator for the Binary Response Model," Econometrica, Econometric Society, vol. 60(3), pages 505-531, May.
    20. Janssen, Arnold, 1997. "Studentized permutation tests for non-i.i.d. hypotheses and the generalized Behrens-Fisher problem," Statistics & Probability Letters, Elsevier, vol. 36(1), pages 9-21, November.
    21. Ibragimov, Rustam & Müller, Ulrich K., 2010. "t-Statistic Based Correlation and Heterogeneity Robust Inference," Journal of Business & Economic Statistics, American Statistical Association, vol. 28(4), pages 453-468.
    22. Moulton, Brent R, 1990. "An Illustration of a Pitfall in Estimating the Effects of Aggregate Variables on Micro Unit," The Review of Economics and Statistics, MIT Press, vol. 72(2), pages 334-338, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    2. Kojevnikov, Denis & Song, Kyungchul, 2023. "Some impossibility results for inference with cluster dependence with large clusters," Other publications TiSEM 80b8e4ed-54bc-4a34-883f-f, Tilburg University, School of Economics and Management.
    3. Kevin Lang & Kaiwen Leong & Huailu Li & Haibo Xu, 2019. "Lending to the Unbanked: Relational Contracting with Loan Sharks," NBER Working Papers 26400, National Bureau of Economic Research, Inc.
    4. Rik Chakraborti & Gavin Roberts, 2023. "How price-gouging regulation undermined COVID-19 mitigation: county-level evidence of unintended consequences," Public Choice, Springer, vol. 196(1), pages 51-83, July.
    5. Wang, Wenjie & Zhang, Yichong, 2024. "Wild bootstrap inference for instrumental variables regressions with weak and few clusters," Journal of Econometrics, Elsevier, vol. 241(1).
    6. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    7. Kojevnikov, Denis & Song, Kyungchul, 2023. "Some impossibility results for inference with cluster dependence with large clusters," Journal of Econometrics, Elsevier, vol. 237(2).
    8. Hwang, Jungbin, 2021. "Simple and trustworthy cluster-robust GMM inference," Journal of Econometrics, Elsevier, vol. 222(2), pages 993-1023.
    9. Kevin Lang, 2020. "Effort and wages: Evidence from the payroll tax," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(1), pages 108-139, February.
    10. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    11. Felipe Lozano‐Rojas & Patrick Carlin, 2022. "The effect of soda taxes beyond beverages in Philadelphia," Health Economics, John Wiley & Sons, Ltd., vol. 31(11), pages 2381-2410, November.
    12. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.
    13. Yingfei Mu & Edward A. Rubin & Eric Zou, 2021. "What’s Missing in Environmental (Self-)Monitoring: Evidence from Strategic Shutdowns of Pollution Monitors," NBER Working Papers 28735, National Bureau of Economic Research, Inc.
    14. Andreas Hagemann, 2019. "Permutation inference with a finite number of heterogeneous clusters," Papers 1907.01049, arXiv.org, revised Feb 2023.
    15. Brantly Callaway, 2022. "Difference-in-Differences for Policy Evaluation," Papers 2203.15646, arXiv.org.
    16. Dao, Chi Danh & Fenig, Guidon & Sator, Georg & Yoon, Jin Young, 2024. "Assessing Robustness to Varying Clustering Methods and Samples in Ambuehl, Bernheim, and Lusardi (2022): Replication and Sensitivity Analysis," I4R Discussion Paper Series 110, The Institute for Replication (I4R).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andreas Hagemann, 2019. "Permutation inference with a finite number of heterogeneous clusters," Papers 1907.01049, arXiv.org, revised Feb 2023.
    2. Hansen, Bruce E. & Lee, Seojeong, 2019. "Asymptotic theory for clustered samples," Journal of Econometrics, Elsevier, vol. 210(2), pages 268-290.
    3. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    4. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    5. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    6. Hwang, Jungbin, 2021. "Simple and trustworthy cluster-robust GMM inference," Journal of Econometrics, Elsevier, vol. 222(2), pages 993-1023.
    7. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    8. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Testing for the appropriate level of clustering in linear regression models," Journal of Econometrics, Elsevier, vol. 235(2), pages 2027-2056.
    9. Ivan A. Canay & Andres Santos & Azeem M. Shaikh, 2018. "The wild bootstrap with a "small" number of "large" clusters," CeMMAP working papers CWP27/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    10. James G. MacKinnon, 2019. "How cluster‐robust inference is changing applied econometrics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 52(3), pages 851-881, August.
    11. Jungbin Hwang, 2017. "Simple and Trustworthy Cluster-Robust GMM Inference," Working papers 2017-19, University of Connecticut, Department of Economics, revised Aug 2020.
    12. James G. MacKinnon & Matthew D. Webb, 2017. "Pitfalls When Estimating Treatment Effects Using Clustered Data," Working Paper 1387, Economics Department, Queen's University.
    13. Timothy Conley & Silvia Gonçalves & Christian Hansen, 2018. "Inference with Dependent Data in Accounting and Finance Applications," Journal of Accounting Research, Wiley Blackwell, vol. 56(4), pages 1139-1203, September.
    14. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 106, University of California, Davis, Department of Economics.
    15. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2021. "Wild Bootstrap and Asymptotic Inference With Multiway Clustering," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 505-519, March.
    16. James G. MacKinnon & Morten {O}rregaard Nielsen & Matthew D. Webb, 2024. "Cluster-robust jackknife and bootstrap inference for binary response models," Papers 2406.00650, arXiv.org.
    17. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 318, University of California, Davis, Department of Economics.
    18. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    19. Andreas Hagemann, 2020. "Inference with a single treated cluster," Papers 2010.04076, arXiv.org.
    20. Andreas Hagemann, 2017. "Cluster-Robust Bootstrap Inference in Quantile Regression Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(517), pages 446-456, January.

    More about this item

    Keywords

    Cluster-robust inference; Randomization; Permutation;
    All these keywords.

    JEL classification:

    • C01 - Mathematical and Quantitative Methods - - General - - - Econometrics
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:213:y:2019:i:1:p:190-209. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.