Analysis of Variance

While the t-test is limited to the comparison of two means, the ANOVA is an inferential statistic that can be used to compare two or more means. The ANOVA produces an F statistic that is a ratio of the variability of the means being compared to the variability of the observations within each set of data on which the means are based. The ANOVA can be used to compare two or more means (in theory the number of means is limitless) taken from different groups (resulting from a between-subjects design) or to compare means taken from the same group of subjects under varying conditions (resulting from a within-subjects or repeated-measures design) – the repeated measures ANOVA. The one-way ANOVA examines the effect of a single independent variable on a dependent variable. The two-way ANOVA examines the effects of two separate independent variables, as well as their interaction, on a dependent variable.

Consider an experimental investigation in which the effects of work environment on productivity are examined using a between-subjects design. A large corporation randomly assigns its workers to one of three different work environments; single closed office, single open cubby where the worker works alone but can see and hear other workers, or shared open cubby where the worker shares the work space with another worker and can see and hear other workers. They then measure each worker’s productivity on a standardized productivity schedule. The independent variable is the type of work environment and it has three levels rather than two as in the previous studying example. The dependent variable is productivity, and the null hypothesis is that work environment has no effect on productivity or that the productivity of all workers will be the same regardless of work environment. These data would be most appropriately analyzed with an ANOVA.

Identify the independent variable, dependent variable, and the null hypothesis from the following scenario:

A researcher would like to know if highlighting a textbook helps students to score better on exams. She randomly selects one-half of the students in an introductory class and instructs them to highlight their textbooks as they read. The other students are instructed to do NO highlighting as they read.

If the experimental manipulation has no effect, the experimental and control groups in the over-learning study would not differ significantly in their performance on the exam and the workers in the different work environments would all be equally productive. In those cases, we would fail to reject the null hypothesis. If, in the over-learning study, the experimental manipulation has an effect, the two groups would differ significantly in their performance on the exam. In that case, we would reject the null hypothesis. This would indirectly support the research hypothesis, which would predict that over-learning affects exam performance. But how large must a difference be between groups for it to be significant? How much more productive must one group of workers be than another for us to conclude that work environment affects productivity? To determine whether the difference between groups is large enough to minimize chance variation as an alternative explanation of the results, we must determine the statistical significance of the difference between them.

Critical Values

To determine whether a t statistic or F statistic has less than a .05 (or .01) probability of occurring by chance, the observed (i.e., calculated) statistic is compared to a critical value taken from a probability table. The exact critical value used for any one comparison depends on the level of significance chosen, the number of observations in each group, the number of groups being compared, and whether the researcher has a directional or non-directional hypothesis. Using information about the above factors a researcher obtains a critical value and compares the observed value to it. If the observed value fails to exceed the critical value, the null hypothesis is retained and the results of the study are said to be inconclusive. If the observed value is more extreme than the critical value, the null hypothesis is rejected, the results are said to be statistically significant, and the research hypothesis is said to have received support from the study.

Note that statistical significance is a statement of probability. We can never be certain that what is true of our samples is also true of the populations they represent. This is one of the reasons why all scientific findings are tentative. Moreover, statistical significance does not indicate practical significance. A statistically significant effect may be too small or be produced at too great a cost of time or money to be useful. What if those who practice over-learning must study two extra hours each day to improve their exam performance by a statistically significant, yet relatively small, 3 points? Knowing this, students might choose to spend their time in another way. As the American statesman Henry Clay (1777-1852) noted, in determining the importance of research findings, by themselves "statistics are no substitute for judgment."

Power

The difference between the means of groups will more likely be statistically significant under the following conditions:

1. When the samples are large.

2. When the difference between the means is large.

3. When the variability within the groups is small.

These are all factors involved in the power of a study. Power is the probability of your experiment allowing you to detect an effect that really exists in the world. The difference between the means of your groups is a measure of effect size – how big of an impact your independent variable has on your dependent variable. Larger samples are apt to be more representative of the population in question and, as sample size increases within groups variance typically decreases. Since one rarely has precise control over the difference between means, a good method for improving power is to increase the number of participants in a study.

contributed by Karen Burke, EdD

ANOVA: Significance and Eta Squared

When looking at the significance column when analyzing the results from an ANOVA in SPSS, look at the column labeled "Sig." This column indicates the level of significance (the likelihood the result is due to chance) the lower the significance, the less likely the differences between the groups are due to chance and the more likely they are due to the independent variable. For example, a significance level or probability of less than .01 means there's a fewer than 1 possibility in 100 that the results are due to chance.

The next column is labeled "Eta Squared," which is the measure of effect size. It is the percentage of the dependent variable explained by the independent variable. The higher the percentage (the closer to 1), the more important the effect of the independent variable. For example, an Eta Squared of .75 means that 75% of the independent variable is explained by the independent variable.

"contributed by Sheri Prendergast"