Showing 1–11 of 11 results for author: Pashley, N E

Design-based Causal Inference for Balanced Incomplete Block Designs

Authors: Taehyeon Koo, Nicole E. Pashley

Abstract: Researchers often turn to block randomization to increase the precision of their inference or due to practical considerations, such as in multi-site trials. However, if the number of treatments under consideration is large it might not be practical or even feasible to assign all treatments within each block. We develop novel inference results under the finite-population design-based framework for… ▽ More Researchers often turn to block randomization to increase the precision of their inference or due to practical considerations, such as in multi-site trials. However, if the number of treatments under consideration is large it might not be practical or even feasible to assign all treatments within each block. We develop novel inference results under the finite-population design-based framework for a natural alternative to the complete block design that does not require reducing the number of treatment arms, the balanced incomplete block design (BIBD). This includes deriving the properties of two estimators for BIBDs and proposing conservative variance estimators. To assist practitioners in understanding the trade-offs of using BIBDs over other designs, the precisions of resulting estimators are compared to standard estimators for the complete block, cluster-randomized, and completely randomized designs. Simulations and a data illustration demonstrate the trade-offs of using BIBDs. This work highlights BIBDs as practical and currently underutilized designs. △ Less

Submitted 1 July, 2024; v1 submitted 29 May, 2024; originally announced May 2024.

Optimal allocation of sample size for randomization-based inference from $2^K$ factorial designs

Authors: Arun Ravichandran, Nicole E. Pashley, Brian Libgober, Tirthankar Dasgupta

Abstract: Optimizing the allocation of units into treatment groups can help researchers improve the precision of causal estimators and decrease costs when running factorial experiments. However, existing optimal allocation results typically assume a super-population model and that the outcome data comes from a known family of distributions. Instead, we focus on randomization-based causal inference for the f… ▽ More Optimizing the allocation of units into treatment groups can help researchers improve the precision of causal estimators and decrease costs when running factorial experiments. However, existing optimal allocation results typically assume a super-population model and that the outcome data comes from a known family of distributions. Instead, we focus on randomization-based causal inference for the finite-population setting, which does not require model specifications for the data or sampling assumptions. We propose exact theoretical solutions for optimal allocation in $2^K$ factorial experiments under complete randomization with A-, D- and E-optimality criteria. We then extend this work to factorial designs with block randomization. We also derive results for optimal allocations when using cost-based constraints. To connect our theory to practice, we provide convenient integer-constrained programming solutions using a greedy optimization approach to find integer optimal allocation solutions for both complete and block randomization. The proposed methods are demonstrated using two real-life factorial experiments conducted by social scientists. △ Less

Submitted 21 June, 2023; originally announced June 2023.

Comments: 27 pages

Journal ref: Journal of Causal Inference 12, no. 1 (2024)

Improving instrumental variable estimators with post-stratification

Authors: Nicole E. Pashley, Luke Keele, Luke W. Miratrix

Abstract: Experiments studying get-out-the-vote (GOTV) efforts estimate the causal effect of various mobilization efforts on voter turnout. However, there is often substantial noncompliance in these studies. A usual approach is to use an instrumental variable (IV) analysis to estimate impacts for compliers, here being those actually contacted by the investigators. Unfortunately, popular IV estimators can be… ▽ More Experiments studying get-out-the-vote (GOTV) efforts estimate the causal effect of various mobilization efforts on voter turnout. However, there is often substantial noncompliance in these studies. A usual approach is to use an instrumental variable (IV) analysis to estimate impacts for compliers, here being those actually contacted by the investigators. Unfortunately, popular IV estimators can be unstable in studies with a small fraction of compliers. We explore post-stratifying the data (e.g., taking a weighted average of IV estimates within each stratum) using variables that predict complier status (and, potentially, the outcome) to mitigate this. We present the benefits of post-stratification in terms of bias, variance, and improved standard error estimates, and provide a finite-sample asymptotic variance formula. We also compare the performance of different IV approaches and discuss the advantages of our design-based post-stratification approach over incorporating compliance-predictive covariates into the two-stage least squares estimator. In the end, we show that covariates predictive of compliance can increase precision, but only if one is willing to make a bias-variance trade-off by down-weighting or dropping strata with few compliers. By contrast, standard approaches such as two-stage least squares fail to use such information. We finally examine the benefits of our approach in two GOTV applications. △ Less

Submitted 28 June, 2024; v1 submitted 17 March, 2023; originally announced March 2023.

Causal inference from treatment-control studies having an additional factor with unknown assignment mechanism

Authors: Nicole E. Pashley, Kristen B. Hunter, Katy McKeough, Donald B. Rubin, Tirthankar Dasgupta

Abstract: Consider a situation with two treatments, the first of which is randomized but the second is not, and the multifactor version of this. Interest is in treatment effects, defined using standard factorial notation. We define estimators for the treatment effects and explore their properties when there is information about the nonrandomized treatment assignment and when there is no information on the a… ▽ More Consider a situation with two treatments, the first of which is randomized but the second is not, and the multifactor version of this. Interest is in treatment effects, defined using standard factorial notation. We define estimators for the treatment effects and explore their properties when there is information about the nonrandomized treatment assignment and when there is no information on the assignment of the nonrandomized treatment. We show when and how hidden treatments can bias estimators and inflate their sampling variances. △ Less

Submitted 7 February, 2022; originally announced February 2022.

Estimating Heterogeneous Causal Effects of High-Dimensional Treatments: Application to Conjoint Analysis

Authors: Max Goplerud, Kosuke Imai, Nicole E. Pashley

Abstract: Estimation of heterogeneous treatment effects is an active area of research. Most of the existing methods, however, focus on estimating the conditional average treatment effects of a single, binary treatment given a set of pre-treatment covariates. In this paper, we propose a method to estimate the heterogeneous causal effects of high-dimensional treatments, which poses unique challenges in terms… ▽ More Estimation of heterogeneous treatment effects is an active area of research. Most of the existing methods, however, focus on estimating the conditional average treatment effects of a single, binary treatment given a set of pre-treatment covariates. In this paper, we propose a method to estimate the heterogeneous causal effects of high-dimensional treatments, which poses unique challenges in terms of estimation and interpretation. The proposed approach finds maximally heterogeneous groups and uses a Bayesian mixture of regularized logistic regressions to identify groups of units who exhibit similar patterns of treatment effects. By directly modeling group membership with covariates, the proposed methodology allows one to explore the unit characteristics that are associated with different patterns of treatment effects. Our motivating application is conjoint analysis, which is a popular type of survey experiment in social science and marketing research and is based on a high-dimensional factorial design. We apply the proposed methodology to the conjoint data, where survey respondents are asked to select one of two immigrant profiles with randomly selected attributes. We find that a group of respondents with a relatively high degree of prejudice appears to discriminate against immigrants from non-European countries like Iraq. An open-source software package is available for implementing the proposed methodology. △ Less

Submitted 16 June, 2024; v1 submitted 4 January, 2022; originally announced January 2022.

Comments: Major revision; added Propositions 1 and 2. 27 pages (main text); 34 pages (supplementary information)

Block what you can, except when you shouldn't

Authors: Nicole E. Pashley, Luke W. Miratrix

Abstract: Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this paper, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a… ▽ More Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this paper, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a blocked design can cost, all in terms of precision. We discuss how the different findings are due to different sampling models and assumptions of how the blocks were formed. We also connect these ideas to common misconceptions, for instance showing that analyzing a blocked experiment as if it were completely randomized, a seemingly conservative method, can actually backfire in some cases. Overall, we find that blocking can have a price, but that this price is usually small and the potential for gain can be large. It is hard to go too far wrong with blocking. △ Less

Submitted 27 May, 2021; v1 submitted 26 October, 2020; originally announced October 2020.

Comments: arXiv admin note: text overlap with arXiv:1710.10342

Journal ref: Journal of Educational and Behavioral Statistics, 2022; 47(1):69-100

Conditional As-If Analyses in Randomized Experiments

Authors: Nicole E. Pashley, Guillaume W. Basse, Luke W. Miratrix

Abstract: The injunction to `analyze the way you randomize' is well-known to statisticians since Fisher advocated for randomization as the basis of inference. Yet even those convinced by the merits of randomization-based inference seldom follow this injunction to the letter. Bernoulli randomized experiments are often analyzed as completely randomized experiments, and completely randomized experiments are an… ▽ More The injunction to `analyze the way you randomize' is well-known to statisticians since Fisher advocated for randomization as the basis of inference. Yet even those convinced by the merits of randomization-based inference seldom follow this injunction to the letter. Bernoulli randomized experiments are often analyzed as completely randomized experiments, and completely randomized experiments are analyzed as if they had been stratified; more generally, it is not uncommon to analyze an experiment as if it had been randomized differently. This paper examines the theoretical foundation behind this practice within a randomization-based framework. Specifically, we ask when is it legitimate to analyze an experiment randomized according to one design as if it had been randomized according to some other design. We show that a sufficient condition for this type of analysis to be valid is that the design used for analysis be derived from the original design by an appropriate form of conditioning. We use our theory to justify certain existing methods, question others, and finally suggest new methodological insights such as conditioning on approximate covariate balance. △ Less

Submitted 19 August, 2021; v1 submitted 3 August, 2020; originally announced August 2020.

Journal ref: Journal of Causal Inference, vol. 9, no. 1, 2021, pp. 264-284

Design-Based Ratio Estimators and Central Limit Theorems for Clustered, Blocked RCTs

Authors: Peter Z. Schochet, Nicole E. Pashley, Luke W. Miratrix, Tim Kautz

Abstract: This article develops design-based ratio estimators for clustered, blocked randomized controlled trials (RCTs), with an application to a federally funded, school-based RCT testing the effects of behavioral health interventions. We consider finite population weighted least squares estimators for average treatment effects (ATEs), allowing for general weighting schemes and covariates. We consider mod… ▽ More This article develops design-based ratio estimators for clustered, blocked randomized controlled trials (RCTs), with an application to a federally funded, school-based RCT testing the effects of behavioral health interventions. We consider finite population weighted least squares estimators for average treatment effects (ATEs), allowing for general weighting schemes and covariates. We consider models with block-by-treatment status interactions as well as restricted models with block indicators only. We prove new finite population central limit theorems for each block specification. We also discuss simple variance estimators that share features with commonly used cluster-robust standard error estimators. Simulations show that the design-based ATE estimator yields nominal rejection rates with standard errors near true ones, even with few clusters. △ Less

Submitted 25 February, 2021; v1 submitted 4 February, 2020; originally announced February 2020.

Journal ref: Journal of the American Statistical Association 117, no. 540 (2022)

Note on the Delta Method for Finite Population Inference with Applications to Causal Inference

Authors: Nicole E. Pashley

Abstract: The delta method creates more general inference results when coupled with central limit theorem results for the finite population. This opens up a range of new estimators for which we can find finite population asymptotic properties. We focus on the use of this method to derive asymptotic distributional results and variance expressions for causal estimators. We illustrate the use of the method by… ▽ More The delta method creates more general inference results when coupled with central limit theorem results for the finite population. This opens up a range of new estimators for which we can find finite population asymptotic properties. We focus on the use of this method to derive asymptotic distributional results and variance expressions for causal estimators. We illustrate the use of the method by obtaining a finite population asymptotic distribution for a causal ratio estimator. △ Less

Submitted 31 May, 2022; v1 submitted 20 October, 2019; originally announced October 2019.

Journal ref: Statistics & Probability Letters 188 (2022)

Causal Inference for Multiple Treatments using Fractional Factorial Designs

Authors: Nicole E. Pashley, Marie-Abele C. Bind

Abstract: We consider the design and analysis of multi-factor experiments using fractional factorial and incomplete designs within the potential outcome framework. These designs are particularly useful when limited resources make running a full factorial design infeasible. We connect our design-based methods to standard regression methods. We further motivate the usefulness of these designs in multi-factor… ▽ More We consider the design and analysis of multi-factor experiments using fractional factorial and incomplete designs within the potential outcome framework. These designs are particularly useful when limited resources make running a full factorial design infeasible. We connect our design-based methods to standard regression methods. We further motivate the usefulness of these designs in multi-factor observational studies, where certain treatment combinations may be so rare that there are no measured outcomes in the observed data corresponding to them. Therefore, conceptualizing a hypothetical fractional factorial experiment instead of a full factorial experiment allows for appropriate analysis in those settings. We illustrate our approach using biomedical data from the 2003-2004 cycle of the National Health and Nutrition Examination Survey to examine the effects of four common pesticides on body mass index. △ Less

Submitted 28 January, 2022; v1 submitted 18 May, 2019; originally announced May 2019.

Insights on Variance Estimation for Blocked and Matched Pairs Designs

Authors: Nicole E. Pashley, Luke W. Miratrix

Abstract: Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated ave… ▽ More Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated average impact, but they are also built on different sets of assumptions. Neither literature handles cases with blocks of varying size that contain singleton treatment or control units, a case which can occur in a variety of contexts, such as with different forms of matching or post-stratification. In this paper, we reconcile the literatures by carefully examining the performance of variance estimators under several different frameworks. We then use these insights to derive novel variance estimators for experiments containing blocks of different sizes. △ Less

Submitted 29 June, 2020; v1 submitted 27 October, 2017; originally announced October 2017.

Journal ref: Journal of Educational and Behavioral Statistics, 46(3) (2021) p. 271-296