Comparing group differences for the evaluation of treatment effectiveness is a common practice. Parametric procedures such as t-tests and F-tests are widely used for this purpose. However, those procedures based upon centrality may mislead the researcher, especially in the case of heterogeneity of variance.
In recent years more and more researchers endorsed the use of confidence intervals (CI). By using CI, the researcher not only looks at the group differences by means, but also by variability. SAS/JMP provides a powerful tool named diamond plot to visualize variability, as demonstrated in the following figure. It condenses a lot of important information:
- Grand sample mean: represented by a horizontal dot line
- Group means: the horizontal line inside each diamond is the group means
- Confidence intervals: The diamond is the CI for each group
- Quantitle: In addition to CI, JMP also provides the option of overlaying a boxplot showing quantile information
The visualization of CI is straightforward--the flatter the diamond is, the tighter the CI is. In this example, it is obvious that in group two most observations cluster around the group mean. Moreover, the mean and the median are almost the same.
However, Payton, Greenstone and Schenker (2003) warned
researchers that inferring from non-overlapping CIs to significant mean
differences is a dangerous practice, because the error rate associated with this
comparison is quite large. The probability of overlap is a function of the
standard error. As the standard errors become less homogeneous, the probability
of overlap decreases. Simulations result showed that when the standard errors
are approximately equal, using 83% or 84% size for the intervals will give an
approximate alpha = 0.05 test, but using 95% confidence intervals, which is a
common practice, will give very conservative results. In short, CI overlaps
should be taken for reference only; it could not be used as a replacement of
hypothesis testing.
References
Payton, M. E., Greenstone, M. H., & Schenker, N. (2003).
Overlapping confidence intervals or standard error intervals: What do they mean
statistical significance? Journal of Insect Science, 3(34). Retrieved
April 21, 2008 from
http://insectscience.org/3.34
Return to Index