Date: Dec 5, 2012 2:46 PM
Author: Richard Hake
Subject: [math-learn] Randomized Control Trials - The Tarnished Gold Standard

ome subscribers to Math-Learn might be interested in a recent post 
"Randomized Control Trials - The Tarnished Gold Standard" [Hake
(2012a)]. The abstract reads:

************************************************
ABSTRACT: In response to "The Randomistas' War On Global Poverty -
Erratum & Addendum" at <http://bit.ly/YfMESg>, Art Burke of the
EvalTalk list pointed to an NYT piece "Applying Evidence to Social
Programs" by Jon Baron at <http://nyti.ms/Um9vVI>.

Baron wrote (slightly edited): "Scientifically rigorous studies -
particularly, the 'gold standard' of Randomized Controlled Trials
(RCT''s) - are a mainstay of medicine, providing conclusive evidence
of effectiveness for most major medical advances in recent history.
In social spending, by contrast, such studies have only a toehold.
Where they have been used, however, they have demonstrated the same
ability to produce important, credible evidence about what works -
and illuminated a path to major progress."

In this post I cite arguments that the "gold standard" RCT studies
may not be as lustrous as claimed by Baron:

(1) Ever since the pioneering work of Halloun & Hestenes (1985a) at
<http://bit.ly/fDdJHm>, physicists have been engaged in social
science of Physics Education Research (PER) that has made useful,
reliable, and nonobvious predictions without resort to RCT's - e.g.
"Why Not Try a Scientific Approach to Science Education?" [Wieman
(2007)] at <http://bit.ly/anTMfF>.

(2) In "A Response to 'It's Not All About Class Size' " [Hake
(2009)], I pointed out that according to the California Class Size
Reduction Research Consortium [CCSRRC (2002)] at
<http://bit.ly/V923Ms>, California's attempt to duplicate the vaunted
Tennessee RCT study of reduced class size benefits yielded *no
conclusive evidence of increased student achievement*.

(3) In "A Summative Evaluation of RCT Methodology: & An Alternative
Approach to Causal Research" [Scriven (2008] at
<http://bit.ly/93VcWD> wrote: "In standard scientific usage,
experiments are just carefully constrained explorations, and the RCT
is simply a special case of these. To call the RCT the only 'true
experiment' is part of an attempt at redefinition that distorts the
original and continuing usage, and excludes experiments designed to
test many simple hypotheses about - or simple efforts to find out -
what happens if we do *this*."

(4) In "Seventeen Statements by Gold-Standard Skeptics #2" [Hake
(2010)] at <http://bit.ly/TNpTR9> I cite, among others, the RCT
skepticism of the American Education Research Association; the
American Evaluation Association; the National Education Association;
the European Evaluation Society; Thomas Cook and Monique Payne, Hugh
Burkhardt & Alan Schoenfeld; Margaret Eisenhart & Lisa Towne; Burke
Johnson; Annette Lareau & Pamela Barnhouse; Joseph Maxwell; Dennis
Phillips; Barbara Schneider, Martin Carnoy, Jeremy Kilpatrick,
William Schmidt, & Richard Shavelson; Mack Shelley, Larry Yore, &
Brian Hand; Deborah Stipek; and Carol Weiss.

(5) In "Why Most Published Research Findings Are False," John
Ioannidis' (2005) at <http://1.usa.gov/YxUxkL> states ". . . .
there is strong evidence that selective outcome reporting, with
manipulation of the outcomes and analyses reported, is a common
problem even for randomized trails [Chan et al. (2004)] at
<http://1.usa.gov/X8SB1T>.

(6) the present signature quote of Thomas Cook and Monique Payne.
************************************************

To access the complete 18 kB post please click on <http://bit.ly/VzVc0K>.

Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: <http://bit.ly/a6M5y0>
Links to Socratic Dialogue Inducing (SDI) Labs: <http://bit.ly/9nGd3M>
Academia: <http://bit.ly/a8ixxm>
Blog: <http://bit.ly/9yGsXh>
GooglePlus: <http://bit.ly/KwZ6mE>
Twitter: <http://bit.ly/juvd52>

"In some quarters, particularly medical ones, the randomized
experiment is considered the causal 'gold standard.' It is clearly
not that in educational contexts, given the difficulties with
implementing and maintaining randomly created groups, with the
sometimes incomplete implementation of treatment particulars, with
the borrowing of some treatment particulars by control group units,
and with the limitations to external validity that often follow from
how the random assignment is achieved."
- Thomas Cook and Monique Payne in "Evidence Matters"
[Mosteller & Boruch (2002)]


REFERENCES [URL shortened by http://bit.ly/ and accessed on 04 Dec 2012.]
Hake, R.R. 2012a. "Randomized Control Trials - The Tarnished Gold
Standard," online on the OPEN! AERA-L archives at
<http://bit.ly/VzVc0K>. Post of 4 Dec 2012 19:26:48-0800 to AERA-L
and Net-Gold. The abstract and link to the complete post are being
transmitted to several discussion lists and are also on my blog
"Hake'sEdStuff" at <http://bit.ly/TEIwY5> with a provision for
comments.

Hake, R.R. 2012b. "The Randomistas' War On Global Poverty - ERRATUM &
ADDENDUM," online on the OPEN! AERA-L archives at
<http://bit.ly/YfMESg>. Post of 30 Nov 2012 12:15:33-0800 to AERA-L
and Net-Gold. The abstract and link to the complete post are being
transmitted to several discussion lists and are also on my blog
"Hake'sEdStuff" at <http://bit.ly/11c5w3e> with a provision for
comments.

Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized
Trials in Education Research." Brookings Institution, publisher's
information at <http://bit.ly/UoX3sA>. Amazon.com information at
<http://amzn.to/n6T0Uo>. An expurgated Google book preview is online
at <http://bit.ly/RX1k3u>.

[Non-text portions of this message have been removed]