Date: Nov 17, 2012 7:14 PM
Author: Richard Hake
Subject: Randomized Control Trials: The Strange Case of the Contradictory<br> Graphs (was In Defense of the NRC's 'Scientific Research in Education')
Some subscribers to MathEdCC might be interested in a recent post
"Randomized Control Trials: The Strange Case of the Contradictory
Graphs (was In Defense of the NRC's 'Scientific Research in
Education')" [Hake (2012)]. The abstract reads:
ABSTRACT: Susan Skidmore at <http://bit.ly/Uov4sU> alerted the
Math-Teach list to her valuable articles with Bruce Thompson in the
June/July 2012 issue of the "Educational Researcher": (a)
"Propagation of Misinformation About Frequencies of RFTs/RCTs in
Education: A Cautionary Tale" [Skidmore & Thomson (2012a)] at
<http://bit.ly/SPN361>, and (b) "Things (We Now Believe) We Know"
[S&T (2012b)] at <http://bit.ly/ZSQ5v5>.
S&T (2012a) discuss the CONTRADICTORY GRAPHS of cumulative numbers of
Randomized Control Trials (RCT's) vs time for Criminology, Education,
Psychology, and Social fields (showing education first, tied for
second, and last) presented by influential scholars in prominent
settings [that, along with the attendant sequence of events] "may
have gratuitously damaged the already fragile reputation of education
research as a field."
After reviewing the history, S&T (2012b) conclude: "We believe that
the errors were unintentional . . . . . . "But the history as
recounted by S&T (2012a) [and in the same "Educational Researcher"
issue by Robinson (2012) at <http://bit.ly/WHhdiU> and Petrosino
(2012) at <http://bit.ly/SU3K3O>] seems to contradict S&T's
Thomas Cook submitted an article with the title "A critical appraisal
of the case against using experiments to assess school (or community)
effects" [Cook (2001a)] at <http://bit.ly/Uyd3CY> with NO GRAPH to
the Hoover Institution's "Education Next"
<http://educationnext.org/>. Evidently without Cook's knowledge, his
academic article was heavily edited and published as "Sciencephobia:
Why education researchers reject randomized experiments" [Cook
(2001b)] at <http://bit.ly/SQox50> WITH A GRAPH of cumulative numbers
of Randomized Control Trials (RCT's) vs time for Criminology,
Education, Psychology, and Social fields showing education LAST,
consistent with the provocative new title. The graph was erroneously
attributed to Boruch, De Moya, & Snyder (2001) - the data should have
been 2002) - at <http://bit.ly/UoX3sA>, despite the fact that the
Boruch et al. graph showed education tied for second, not last. Are
we to believe that Education Next's degradation of the accurate
academic Cook (2001a) to the inaccurate hooverized Cook (2001b) was
A side issue: to those who regard RCT's as the "gold standard" of
education research, the higher the curve of cumulative numbers of
Randomized Control Trials (RCT's) vs time for a field, the higher the
merit of research in that field. But not everyone would agree - see
e.g., "A Summative Evaluation of RCT Methodology: & An Alternative
Approach to Causal Research" [Scriven (2008] at
<http://bit.ly/93VcWD>, "Seventeen Statements by Gold-Standard
Skeptics #2" [Hake (2010)] at <http://bit.ly/TNpTR9>, and the present
signature quote of Thomas Cook and Monique Payne.
To access the complete 46 kB post please click on <http://bit.ly/TQdfhX>.
Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: <http://bit.ly/a6M5y0>
Links to Socratic Dialogue Inducing (SDI) Labs: <http://bit.ly/9nGd3M>
"In science education, there is almost nothing of proven efficacy."
- Grover Whitehurst, former director, Institute of Education
as quoted by Sharon Begley (2004)
"In some quarters, particularly medical ones, the randomized
experiment is considered the causal 'gold standard.' It is clearly
not that in educational contexts, given the difficulties with
implementing and maintaining randomly created groups, with the
sometimes incomplete implementation of treatment particulars, with
the borrowing of some treatment particulars by control group units,
and with the limitations to external validity that often follow from
how the random assignment is achieved."
- Thomas Cook and Monique Payne in "Evidence Matters"
[Mosteller & Boruch (2002)]
"Physics educators have led the way in developing and using objective
tests to compare student learning gains in different types of
courses, and chemists, biologists, and others are now developing
similar instruments. These tests provide convincing evidence that
students assimilate new knowledge more effectively in courses
including active, inquiry-based, and collaborative learning, assisted
by information technology, than in traditional courses."
- Wood & Gentile (2003)
REFERENCES [URL's shortened by <http://bit.ly/> and accessed on 17 Nov 2012.]
Begley, S. 2004. "To Improve Education, We Need Clinical Trials To
Show What Works," Wall Street Journal, 17 December, page B1; online
as a 41 kB pdf at <http://bit.ly/SSmaym>, thanks to David Klahr.
Hake, R.R. 2012. "Randomized Control Trials: The Strange Case of the
Contradictory Graphs (was In Defense of the NRC's 'Scientific
Research in Education')" online on the OPEN! AERA-L archives at
<http://bit.ly/TQdfhX>. Post of 17 Nov 2012 10:45:11-0800 to AERA-L
and Net-Gold. The abstract and link to the complete post are being
transmitted to several discussion lists and are also on my blog
"Hake'sEdStuff" at <http://bit.ly/Uy5ufa> with a provision for
Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized
Trials in Education Research." Brookings Institution, publisher's
information at <http://bit.ly/UoX3sA>. Amazon.com information at
<http://amzn.to/n6T0Uo>. An expurgated Google book preview is online
Wood, W.B., & J.M. Gentile. 2003. "Teaching in a research context,"
Science 302: 1510; 28 November; online as a 213 kB pdf at
<http://bit.ly/SyhOvL> thanks to Portland State's "Ecoplexity" site.