Essays on a study of statistical power in economics.

Abstract

Knowing the statistical power of an empirical analysis after it is completed can be very useful. Among other things, it can help one determine whether a finding of statistical insignificance is due to a small effect size or insufficient statistical power. This thesis consists of five studies linked together by my attempts to study how best to calculate ex post statistical power. Chapter One provides an introduction and the background for this thesis. In Chapter Two, I detail what is meant by statistical power (including ex ante and ex post power) and why it is important to researchers. I also identify the various factors that affect statistical power.

Chapter Three explains why ex post power has a “bad reputation.” A common practice for calculating ex post power employs an inappropriate method known as “observed power.” “Observed power” uses the estimated effect size as the assumed true effect size and then calculates the associated power. Though widely used, this method has been demonstrated to produce biased estimates of statistical power (Yuan & Maxwell, 2005). I present two approaches for calculating ex post power suggested by researchers to avoid the problems of “observed power”.

Chapter Four begins by replicating a recent paper by Brown, Lambert and Wojan (2019). BLW use a bootstrapping procedure to calculate ex post power and apply it to a benefit-cost analysis of a U.S. conservation program. I reproduce BLW’s results. I call their method for calculating ex post power BLW1. I then propose a variant of their method, which I call BLW2. I use Monte Carlo experiments to compare both methods in a simple data environment where there is no clustering.

In Chapter Five, I detail two more methods for calculating ex post power. The first procedure is taken from a blog post by David McKenzie and Owen Ozier (2019). I call this approach the SE-ES Method (for Standard Error – Effect Size). I then propose yet another variant of the BLW method, BLW3, which uses a wild-cluster bootstrap for handling clustered data. Chapter Five subjects all four methods to an extensive set of Monte Carlo experiments to assess their reliability in calculating ex post statistical power. I find that the SE-ES method is superior to BLW’s method (BLW1, BLW2 and BLW3) and has good overall performance.

Chapter Six applies the SE-ES method to a set of 23 development studies that were funded by the International Initiative for Impact Evaluation (3ie), a non-profit organization that supports research on ways to help the poor in low- and middle-income countries. I analyze the ex post power of these studies and explore factors that may be responsible for differences between ex ante and ex post statistical power. Chapter Seven concludes this thesis. It provides an overview of my chapters, as well as a summary of my main findings.