The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » Education » mathedcc

Topic: In Defense of the NRC's "Scientific Research in Education'"
Replies: 4   Last Post: Oct 31, 2012 1:41 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Wayne Mackey

Posts: 312
Registered: 12/4/04
RE: In Defense of the NRC's "Scientific Research in Education'"
Posted: Oct 30, 2012 8:53 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply
att1.html (34.5 K)
Emoticon1.gif (0.3 K)
us.gif (2.2 K)

The IES requires randomized cluster trials now. Pretty scientific.


From: [] on behalf of Clyde Greeno @ MALEI []
Sent: Tuesday, October 30, 2012 4:55 PM
To: Richard Hake;;;; Alain Schremmer
Subject: In Defense of the NRC's "Scientific Research in Education'"


Thanks for devoting so much time and attention to my comments about the National Academies'
document, "SCIENTIFIC RESEARCH IN EDUCATION". Yes, it was published by the NRC's Center for Education . You have blessed me (tongue in cheek, I presume) with a label (in Title caps no less) which I presume is as amusing to you as to me and others ... since there are no such "MEG" animals.[Smile emoticon]

Be assured that I respect your contributions toward educational progress at least as much as you seem to (usually) honor my own. In that vein, I regard your response NOT as opening a debate, but as an invitation to further pursue our mutual interests in scientific instructology. As Schremmer noted, you and I are not into dispute, but into viewing the cited report from differing angles.

For sure, your prompting question, "Is 'Education Research' 'Scientific Research' ? " ... which must be answered "NO!" ... because it appears to speak of ALL educational research ... has some closely related questions: (1) CAN 'Education Research' be 'Scientific Research'?" ... whose answer is assuredly, "YES!" (2) "Has SOME 'Education Research' also been 'Scientific Research' ?" ... whose answer is assuredly, "YES!" (3) "How MUCH of 'Education Research' has been also 'Scientific Research' ?" ... whose answer is assuredly, "relatively meager!" (4) Can whatever 'Education Research' is also 'Scientific Research' become important and significant enough to warrant distinguishing it from non-scientific 'Education Research' ? ... whose optimistic answer is "yes", but in mathematics, the demonstrations are not readily apparent. Nonetheless, I am convinced that scientific research in instructology is the only reliable guide that can navigate the evolution of curricular education so as to achieve genuine progress.

The issue that catalyzed your initial question ... about the scientific credibility of Boaler's instructological research ... is quite peripheral to our subsequent discourse about the SRE document's "six principles", but not fully irrelevant.

I do fear that you have sorely misinterpreted my comments and their basis. Before attempting to clarify, I must confess to being a myopic specialist who focuses primarily on mathematics instructology (scientific and otherwise). Your repeated reference to scientific progress in physics instructology is heart warming ... even though my own plate presently is too full to seek enlightenment in that field. Nonetheless, I do greatly relish whatever you disclose about scientific instructology per mathematics.

You seem to have presumed that I have "criticized" the NRC report. Actually, I have no quarrel with it ... other than recognizing the potentials for its gross misuse by hoards of education researchers. Necessarily, the NRC relies on whatever "experts" it can identify ... for this panel: . Of its seventeen non-staff members, only three seem visibly associated with science ... six others are from university schools of education and some others are in the same arena. Of course, such a constituency is far more disposed toward viewing scientific research through the eyes of educational research, than toward viewing educational research through the eyes of scientific research.

Actually, it is very rare for a committee to author anything ... even though authors might attempt to amalgamate contributions from committee members, and later achieve their approval. In this case, Shavelson (formerly Stanford U's Dean of its School of Education) and Towne (an educational consultant and formerly an NRC staff member) have composed the document and are responsible for its contents. Any maladies in the document, as such, rest on their shoulders.

The document reveals that the committee was saddled with one truly impossible task, so undertook another task which it hoped would serve similar purposes. My "caveat" apparently misled you into thinking that I had not scanned the report. Actually, I had ... and quickly (if rashly) concluded that the bulk of it was attempting to describe "scientific research" as it might happen across all arenas of educational research. So, I am still amazed that the NRC would allow its Center for Education to publish such a global and (necessarily) nebulous document ... to be commonly believed to be a "definition" of scientific educational research.

I was/remain as skeptical about the scientific viability/credibility/value of any such global undertaking as I am about the scientific value of most of what has been published as "educational research." So, I did/will not have the time or interest needed for actually studying the body of the document. However, I would encourage all who are interested in such stuff to examine it.

Due to your initial prompts, I did focus on the report's statements of six principles which it claims as "underlying" all of scientific research. Those are of special interest because you had earlier judged that Boaler's study met all of those conditions. That prompted my own concerns that other educators might use those *underlying principles* as if they were necessary and sufficient criteria for judging the science-ness of educational research activities and/or findings.

My earlier comments about those six principles were primarily about their non-sufficiency ... their inadequacy for collectively serving as criteria for scientific research. [For sure, I might have better clarified that perspective.] Unfortunately, much of the educational research which conforms with those principles badly fails to be scientific. The report does NOT proffer those principles as being N&S criteria ... but it is certain that educational researchers will use them as if the report had so declared them.

As a case in point (as Schremmer noted), the report's [Box 5-5] citation of (committee member) Confrey's own study as being illustrative of "scientific education research" is quite illustrative ... of how easily the report can be used for misconstruing good, "bricks and boards", instructological research as being "scientific."

I did/could not *dispute* that each of those six principles *underlies* much of scientific research. And we can acknowledge that the committee's accepted goal required that it refer to scientific principles which are far too broad to be contested (other than through how they are worded). Nonetheless, since the report will (rightly or wrongly) be used as a criterion for the science-ness of educational research, its six principles of scientific research likewise will (rightly or wrongly) be used as necessary and sufficient conditions for educational research to qualify as being scientific.

To reiterate, my own narrow focus on mathematics instructology leads me to so view the potentials, dangers, and projected mis-orientations of educational research that arise from allowing things like the Confrey study to be called "scientific" ... which would badly undermine the potentials of science for guiding future progress. As yet, the NRC's Center for Education is a long way from distinguishing scientific studies in instructology [e.g. of concept inventories] from non-scientific studies [like Comfrey's].

So my comments about the six principles should be construed not as *disputing* them, but as a call for *refining* them toward better serving as scientific criteria for research ... and for instructological research, in particular. After your response, I carefully re-read the Executive Summary, and found it to say nothing different from what it said when I earlier studied it with care. Perhaps your presumptions that I had not studied it resulted from your better abilities to read, between the lines, things that I was unable or unwilling to speculate. I also have re-examined my earlier comments about each of those six. True, the rhetoric could better clarify the difference between "underlying principles" and "scientific criteria" for research. But I do not personally see my comments as disputing any of those six ... only as calling for something more adequate (at least as instructological research).

However, my skepticism ... about any attempt to formulate scientific criteria that could be universally applied to *all* kinds of educational research .... does lead toward a more constructive and viable venue, and to this corresponding invitation to you and others.

At least for now, let us set about to propose actual *criteria* for identifying educational research *in specific areas* as being genuinely scientific, or not. Let us begin by focusing specifically on instructological research ... on human learning, and the instructional guidance, thereof. Indeed, let us do that within each of the various arenas of human learning, because not all scientific criteria that apply to learning/instruction in one area necessarily apply in all other areas of learning.

Thus, you and yours may proceed with developing viable/appropriate criteria for scientific physics-instructology ... and counterpart specialists in other fields can follow suit. You might wish to regard your contributions as being "a contextual particularization" appendix to the NRC document.

Should you choose to pursue such a goal, you might get some ideas from the scientific (STAGML) theory of the State-Transition Analysis and Guidance of Mathematical Learning ... and how it is empirically grounded to clinical case-studies that focus on personal educational health (rather than on scholastic measures).

I continue to look forward to our parallel pursuits and mutual stimulation.



From: Richard Hake<>
Sent: Friday, October 26, 2012 10:05 PM
Subject: [math-learn] In Defense of the NRC's "Scientific Research in Education'"

Some subscribers to Math-Learn might be interested in a recent post
"In Defense of the NRC's 'Scientific Research in Education' " [Hake
(2012)]. The abstract reads:

ABSTRACT: In my post "Is 'Education Research' 'Scientific Research'
? YES!" [Hake (2012)] at <> I listed the
*one-line headings* of the "six guiding principles that underlie all
scientific inquiry (including education research)" as set down in the
Executive Summary of the NRC's "Scientific Research in Education"
[Shavelson & Towne (2002)] at <>.

In response, Math Education Guru <> Clyde Greeno
(2012) at <> denounced the six principles as
"glibly superficial and badly unknowing about the nature of science,"
on the basis of their one-line headings, evidently not bothering to
scan the book "Scientific Research in Education" or even the
paragraph-long elaborations of the six principles in the Executive
Summary. In this post I juxtapose for each of NRC's six one-line
headings (a) Greeno's criticism, and (b) NRC's one-paragraph
elaboration, and invite readers to judge the validity of Greeno'
criticisms. Greeno ended his critique with: (a) "I am amazed that the
NRC would allow such a publication," and (b) "[The World is] more
than anxious to learn of whatever educational research efforts
qualify as being genuinely 'scientific.' "

Regarding (a) above, I am amazed that Greeno would denounce the NRC's
six guiding principles on the basis of what he (often mistakenly)
perceives them to mean from their one-line headings. Regarding (b)
above, he and other skeptics might consider scanning: (1)"The future
of physics education research: Intellectual challenges and practical
concerns" [Heron & Meltzer (2005)] at <>; (2) "A
Developmental History of Physics Education Research" [Cummings (2011)
at <>; (3) "The Impact of Concept Inventories On
Physics Education and It's Relevance For Engineering Education" [Hake
(2011a)] at <> (8.7 MB); and (4) and "Resource
Letter ALIP-1: Active-Learning Instruction in Physics" [Meltzer &
Thornton (2012)] at <>.

To access the complete 46 kB post please click on <>.

Richard Hake, Emeritus Professor of Physics, Indiana University
Links to Articles: <>
Links to Socratic Dialogue Inducing (SDI) Labs: <>
Academia: <>
Blog: <>
GooglePlus: <>
Twitter: <>

"Education is not rocket science, it's much harder."
- George Nelson, astronaut, astrophysicist, and former
director of the AAAS
Project 2061, as quoted by Redish (1999)

"Physics educators have led the way in developing and using objective
tests to compare student learning gains in different types of
courses, and chemists, biologists, and others are now developing
similar instruments. These tests provide convincing evidence that
students assimilate new knowledge more effectively in courses
including active, inquiry-based, and collaborative learning, assisted
by information technology, than in traditional courses."
- Wood & Gentile (2003)

REFERENCES [All URL's shortened by <> and accessed on
26 Oct 2012.]
Hake, R.R. 2012. "In Defense of the NRC's 'Scientific Research in
Education' " online on the OPEN! AERA-L archives at
<>. Post of 26 Oct 2012 17:04:49-0700 to AERA-L
and Net-Gold. The abstract and link to the complete post are being
transmitted to several discussion lists and are also on my blog
"Hake'sEdStuff" at <> with a provision for

Redish, E.F. 1999. "Millikan lecture 1998: building a science of
teaching physics," Am. J. Phys. 67(7): 562-573; online as a 258 kB
pdf at <>.

Wood, W.B. & J.M. Gentile. 2003. "Teaching in a research context,"
Science 302: 1510; 28 November; online to subscribers at
<>. A summary is online to all at

[Non-text portions of this message have been removed]

Reply via web post<;_ylc=X3oDMTJxOWFjM2JqBF9TAzk3MzU5NzE0BGdycElkAzI3Mjk3MzkEZ3Jwc3BJZAMxNzA1MDgyMjQyBG1zZ0lkAzEyODM3BHNlYwNmdHIEc2xrA3JwbHkEc3RpbWUDMTM1MTMwNzExOQ--?act=reply&messageNum=12837> Reply to sender <> Reply to group <> Start a New Topic<;_ylc=X3oDMTJlazltNmVhBF9TAzk3MzU5NzE0BGdycElkAzI3Mjk3MzkEZ3Jwc3BJZAMxNzA1MDgyMjQyBHNlYwNmdHIEc2xrA250cGMEc3RpbWUDMTM1MTMwNzExOQ--> Messages in this topic<;_ylc=X3oDMTM2dWZvNjA1BF9TAzk3MzU5NzE0BGdycElkAzI3Mjk3MzkEZ3Jwc3BJZAMxNzA1MDgyMjQyBG1zZ0lkAzEyODM3BHNlYwNmdHIEc2xrA3Z0cGMEc3RpbWUDMTM1MTMwNzExOQR0cGNJZAMxMjgzNw--> (1)
Recent Activity:

Visit Your Group<;_ylc=X3oDMTJlbGw0MzNhBF9TAzk3MzU5NzE0BGdycElkAzI3Mjk3MzkEZ3Jwc3BJZAMxNzA1MDgyMjQyBHNlYwN2dGwEc2xrA3ZnaHAEc3RpbWUDMTM1MTMwNzExOQ-->
[Yahoo! Groups]<;_ylc=X3oDMTJkYTllNHRmBF9TAzk3MzU5NzE0BGdycElkAzI3Mjk3MzkEZ3Jwc3BJZAMxNzA1MDgyMjQyBHNlYwNmdHIEc2xrA2dmcARzdGltZQMxMzUxMzA3MTE5>
Switch to: Text-Only<>, Daily Digest<> ? Unsubscribe<> ? Terms of Use<> ? Send us Feedback <>

I do choose to regard your critique of my critique as being NOT a challenge to debate the veracity of some authors' perceptions/descriptions of "scientific educational research", but as an invitation to collaborate on the continued furtherance of scientific instructology.

Within that context, I confess to being a myopic specialist who focuses primarily on mathematics instructology. Your repeated reference to scientific progress in physics instructology is heart warming ... even though my own plate presently is a bit too full to seek enlightenment in that field. Nonetheless, I do greatly relish whatever you disclose about scientific instructology per mathematics.

Thanks for your copious and time consuming critique of my own critique of the National Academies' Education Center's document about scientific educational research. Yes, the Center exists and is responsible for the document in question. The National Academies consists of four organizations, one of which is the NRC. The NAS has its own Center for Education ... which it lists as authoring the document of reference. The NRC's Center for Education is cited at But all of that is sheerly peripheral to our dialog.

You have blessed me with a label (in Title caps no less) which I presume is as amusing to you as to me(presumably tongue in cheek). Be assured that I respect your contributions toward educational progress at least as much as you seem to honor my own.

For sure, there have been many recent advances in mankind's scientific knowledge of instructology [the methods of "concept inventories" being a case in point]. But for the benefit of future generations, I deem it essential that we clearly distinguish between genuinely scientific advances, and the kind of question-answering findings and data-suggesting hypotheses that characterize classical educational research.

That brings us back to your critique of my critique. Our dialog has reached far beyond the issues/disputes about whether or not Jo Boaler's findings are scientifically reliable ... or even about the National Academies' document. We now are talking about the scientific credibility of the diverse kinds of instructological research. In particular, we presently are talking about how well the Shavelson and Town criteria suffice for discerning which educational research is genuinely *scientific* ... and which is not. I do not disagree with what they say ... only with what they allow as being "scientific educational research."

Yes, I did earlier read their "executive summary" elaborations of their six criteria. That is precisely how I concluded that their espoused concepts of "science" were biased to accommodate non-scientific educational research. The raising/pursuit of "questions" characterizes classical educational research ... which has badly failed to produce any scientific theory of instructology. The bottom line is that the NAS Education Center's six criteria accept as "scientific findings" too many findings that have no scientific credibility.

You cite the "scholars" behind that document. I have personally talked with, and corresponded with, some NRC "experts" in mathematics education ... and found them to be sorely lacking in professional expertise. So your list of "scholars" is of no scientific interest. Science is all about empirical realities ... not about opinions of committees or "experts" (or of you or myself).

Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2017. All Rights Reserved.