Suppose pre- and posttest data are available for the individuals in 2 intact groups, such as a classroom of kids in school X and a classroom of kids in school Y. If pretest performance is used as a covariate (i.e., “control”) variable in an analysis of covariance, are the 2 groups “equated” such that the comparison of the groups’ posttest means is fair? Many people think ANCOVA achieves this goal. It doesn’t. Even with several “control” variables, ANCOVA can’t truly equate the groups.
There are two reasons why the analysis of covariance cannot equate intact groups.
First, studies in theoretical statistics have shown that ANCOVA’s adjusted means turn out to be biased in the situation where the comparison groups differ with respect to their population means on the covariate variable. In other words, the sample-based adjusted means on the dependent variable do not turn out to be accurate estimates of the corresponding adjusted means in the population when the population means on the covariate variable are dissimilar.
Besides ANCOVA’s statistical inability to generate unbiased adjusted means when nonrandomly formed groups are compared, there is a second, logical reason why you should be on guard whenever you come across a research report in which ANCOVA was used in an effort to equate groups created without random assignment. Simply stated, the covariate variable(s) used by the researcher may not address one or more important differences between the comparison groups. Here, the problem is that a given covariate variable (or set of covariate variables) is limited in scope. For example, the covariate variable(s) used by the researcher might address knowledge but not motivation (or vice versa).
Consider, for example, the many studies conducted in schools or colleges in which one intact group of students receives one form of instruction whereas a different intact group receives an alternative form of instruction. In such studies, it is common practice to compare the two groups’ posttest means via an analysis of covariance, with the covariate being IQ, GPA, or score on a pretest. In the summaries of these studies, the researchers may say that they used ANCOVA “to control for initial differences between the groups.” However, it is debatable whether academic ability is reflected in any of the three covariates mentioned (or even in all three used jointly). In this and many other studies, people’s motivation plays no small part in how well they perform.