Java applets for power and sample size

Note: These require a web browser capable of running Java applets (version 1.1 or higher). If you do not see a selection list above, chances are that you either have disabled Java, your browser is not new enough., or you need to download a JRE plug-in from java.sun.com.  Due to a compatibility bug, many plug-ins size the applet window before allowing for an additional strip with a security warning.; drag the bottom of the window downward a bit to compensate.

Citing this software

Download to run locally

Questions?

Advice

• Retrospective power (a.k.a. observed power, post hoc power).  You've got the data, did the analysis, and did not achieve "significance."  So you compute power retrospectively to see if the test was powerful enough or not.  This is an empty question.  Of course it wasn't powerful enough -- that's why the result isn't significant.  Power calculations are useful for design, not analysis.
  (Note: These comments refer to power computed based on the observed effect size and sample size.  Considering a different sample size is obviously prospective in nature.  Considering a different effect size might make sense, but probably what you really need to do instead is an equivalence test; see Hoenig and Heisey, 2001.)
  
  
• Specify T-shirt effect sizes ("small", "medium", and "large").  This is an elaborate way to arrive at the same sample size that has been used in past social science studies of large, medium, and small size (respectively).  The method uses a standardized effect size as the goal.  Think about it: for a "medium" effect size, you'll choose the same n regardless of the accuracy or reliability of your instrument, or the narrowness or diversity of your subjects.  Clearly, important considerations are being ignored here.  "Medium" is definitely not the message!
  

• Use power prospectively for planning future studies.  Software such as is provided on this website is useful for determining an appropriate sample size, or for evaluating a planned study to see if it is likely to yield useful information.
  
  
• Put science before statistics.  It is easy to get caught up in statistical significance and such; but studies should be designed to meet scientific goals, and you need to keep those in sight at all times (in planning and analysis).  The appropriate inputs to power/sample-size calculations are effect sizes that are deemed clinically important, based on careful considerations of the underlying scientific (not statistical) goals of the study.  Statistical considerations are used to identify a plan that is effective in meeting scientific goals -- not the other way around.
  
  
• Do pilot studies.  Investigators tend to try to answer all the world's questions with one study.  However, you usually cannot do a definitive study in one step.  It is far better to work incrementally.  A pilot study helps you establish procedures, understand and protect against things that can go wrong, and obtain variance estimates needed in determining sample size.  A pilot study with 20-30 degrees of freedom for error is generally quite adequate for obtaining reasonably reliable sample-size estimates. 
  

• Lenth, R. V. (2001), ``Some Practical Guidelines for Effective Sample Size Determination,'' The American Statistician, 55, 187-193.
  
  
• Hoenig, John M. and Heisey, Dennis M. (2001), ``The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis,'' The American Statistician, 55, 19-24.
  
  
• An earlier draft of the Lenth reference above is _here_,  and a shorter summary of some comments I made in a panel discussion at the 2000 Joint Statistical Meetings in Indianapolis is _here_.
  
  
• Additional brief comments, prepared as a handout for my poster presentation at the 2001 Joint Statistical Meetings in Atlanta, are _here_.

---

Accuracy

• Formula accuracy
Most computations are ``exact'' in the sense that they are based on exact formulas for sample size, power, etc.  The exception is Satterthwaite approximations; see below.


• Machine accuracy
Even with exact formulas, computed values are inexact, as are all double-precision floating-point computations.   Many computations (especially noncentral distributions) require summing one or more series, and there is a serious tradeoff between speed and accuracy.  The error bound set for cdfs is 1E-8 or smaller, and for quantiles the bound is 1E-6.  Actual errors can be much larger due to accumulated errors or other reasons.  Quantiles, for example, are computed by numerically solving an equation involving the cdf; thus, in extreme cases, a small error in the cdf can create a large error in the quantile.
A warning (typically, ``too many iterations'') is generated when an error bound is not detected to have been achieved.  However, in the case of quantile computations, no warning message is generated for extreme quantiles.  If you want a power of .9999 at alpha=.0001, you can expect the computed sample size to not be accurate to the nearest integer!  If you specify reasonable criteria, the answers will be pretty reliable.


• Satterthwaite approximations
Some of the dialogs (two-sample t, mixed ANOVA) implement Satterthwaite approximations when certain combinations of inputs require an error term to be constructed.  These are of course not exact, even in their formulation.  Moreover, the Satterthwaite degrees of freedom is used as-is in computing power from a noncentral t or noncentral F distribution, and this introduces further errors that could be large in some cases.  In the two-sample t setting, I'd expect the worst errors to exist when there is a huge imbalance in sample sizes and/or variances.  In the dialogs for mixed ANOVA models (either F tests or multiple comparisons/contrasts), I expect these errors to get worse as more variance components are involved, especially when one or more of them is given negative weight.

---

Links to other sites

• A review of power software for PCs by Len Thomas and Charles Krebs
• Interactive page - Michael Friendly (ANOVA designs)
• Interactive page - David Schoenfeld (clinical trials designs; menu based on study type and measurement type)
• UnifyPow - A SAS module for sample-size analysis by Ralph O'Brien.
• SSize - ECHIP sample size calculator for Palm devices (freeware) by Bob Wheeler.