To Control or Not to Control: The question of whether experimental designs are appropriate for evaluating teaching innovations in higher education
Kember, D, To Control or Not to Control: The question of whether experimental designs are appropriate for evaluating teaching innovations in higher education, Assessment and Evaluation in Higher Education, 28, (1) pp. 89-101. ISSN 0260-2938 (2003) [Refereed Article]
It is becoming increasingly common for academics to engage in research
into their own teaching, usually by introducing some form of innovation into one of their
courses. When doing so, they are often inclined by disciplinary tradition, or urged by
senior colleagues, to use experimental designs in order that any observed outcomes can
be attributed to the innovation itself. This article points out the problems with experimental designs for naturalistic studies of innovative teaching in higher education. A
genuine control is impossible. Practical difficulties in separating groups often result in
contamination of designs. Educational issues are complex with many variables involved.
Therefore, experimental designs with limited numbers of cells result in oversimplification because they deal with only a few of the relevant factors. Particular types
of innovation are not precisely reproducible so generalisation can be misleading. As an
alternative, triangulation across multi-method evaluations from several sources is
recommended, with the aim of establishing evidence beyond reasonable doubt. Comparison and synthesis across related projects appears to be a promising way to derive
recommendations for ways to formulate important aspects of innovations.