Even without reference to Boaler's papers, the NRC's Education Center's criteria for "scientific educational research" (as published by the National Academies Press) might be worth examining in some detail ... before even attempting to use them in support or denial of the scientific credibility of research by Boaler or anyone else.
Of course, the over-riding question is of how well the Education Center's criteria characterize "scientific research in education." For, those criteria appear to be glibly superficial and badly unknowing about the nature of science.
(1) Pose Significant Questions That Can Be Investigated Empirically: Scientific knowledge does not advance only through merely posing "questions." All substantial scientific advancements have come through the development and verification of scientific *theories* which describe whatever phenomena they respectively attend. True, "questions" have their place in motivating particular pursuits within the context of specific scientific theories. But to assert that all scientific theories are developed through pursuing specific "questions" is a bit far fetched and much too scholastic for serious scientists to swallow. Rather, that assertion appears to be the Center's defense of educators' claims that their "boards and bricks" kinds of fragmentary investigations can in some ways be "scientific."
In the macrocosm, what is the scientific theory which Boaler's works serve to advance? In the microcosm, what are the "questions" that she has pursued? Are they genuinely *scientific* questions about specific phenomena that are sufficiently well-defined so that other researchers can readily focus on exactly the same questions? ... or are they too nebulous to determine what her "answers" actually say, if anything.
And when do the Education Center's scientific questions become "significant"? When the answers open the way for progress ... if not for mankind, then at least for further development of the scientific theory. No theory? ... no significance. No leads toward important future developments? ... no significance.
The scientific importance, significance, and credibility of Boaler's works (or anyone else's) can be assessed only within the context of how well-defined her scientific "questions" are, within the broader context of whatever scientific theories she is helping to develop.
Traditionally, educational research has badly failed to be "scientific", because it typically has been about collecting "bricks and boards data" that merely hint at possible truths about nebulous populations or phenomena ... rather than about developing scientific theories about well defined kinds of entities. [My personal skepticism about the science-ness of traditional "educational research" precludes my own serious investigations into her works.]
(2) Link Research to Relevant Theory Not just to "relevant theory", but specifically to the *encompassing scientific* theory. To speak of "scientific research" within the context of badly non-scientific "theories" ... that are no more than personal/committee/community opinions ... is a gross misuse of the label.
Only if/whenever Boaler's works/questions/answers can be regarded within the context of an emerging, but definitive scientific theory can they be regarded as being *scientific findings*. [True, some persons might feel that they perceive a scientific theory into which her works might fit ... but unless/until she provides such a scientific context, her works do not qualify as being "scientific." [Again, my skepticism about traditional "educational research" precludes my investigations into her works.]
(3) Use Methods That Permit Direct Investigation of the Question, Methods which *others* can use for achieving *the same (well-defined) answers* to the same, well-defined "questions" ... or, in the broader context, methods which invariably yield exactly the same findings, about phenomena of essentially the same kind. Boaler's "conclusions" can be scientifically credible only if her "experiments" can be replicated by others ... and invariably produce the same results. Only if the nature and proceedings of her investigations are so clear that others can do as she did, can she hope for her works to be scientifically credible. [Again, my skepticism about traditional "educational research" precludes my investigations into her works.]
(4) Provide a Coherent and Explicit Chain of Reasoning Of *scientific* reasoning ... done within the context of an evolving scientific theory. Such a theory must clearly describe the kind of things it is talking about ... clearly define its crucial concepts ... clearly specify its established facts and conclusions ... and combine all of those with some newly acquired concepts/facts to logically derive the "answers" to the newly posed "questions." If/when Boaler's conclusions have that kind of theoretical basis, they thereby become admissible as candidates for scientific credibility. [Again, my skepticism about traditional "educational research" precludes my investigations into her specific works.]
(5) Replicate and Generalize Across Studies The scientific *studies* must be replicable ... and their *results* must generalize across all such studies. But true reliability and generality can happen only to the extent that the "studies" and results are well-defined within the context of an evolving scientific theory. Nebulous "studies" are of no scientific value ... and their "findings" enjoy no scientific credibility. Likewise for educators' "bricks and boards" data-collections, in general. But the attacks on her "studies" and the "findings" of her experiments are scientifically naïve (or scientifically irresponsible) straw men. Identifying weaknesses in her methodologies cannot (as the challenges should know) disprove her claims of generality. The only responsible mode of refuting her "findings" lies in subsequent studies "of the same" kind failing to produce findings of her kind ... or in producing even findings that directly contradict hers. Otherwise, their attacks on her experimental methods can be regarded only as being non-scientific efforts to uphold some contrary beliefs. On the other hand, her only academic defense lies in subsequent studies "of the same" kind producing findings "of the same kind." Endeavors of either kind necessarily rely on her studies being so well defined that they can be replicated by others. [Again, my skepticism about traditional "educational research" precludes my investigations into her specific works.]
(6) Disclose Research to Encourage Professional Scrutiny and Critique. "Disclose research"??? ... whatever might the Education Center mean by that!? My best guess is that they should mean "disclose the theoretical framework, the motivations for the particular investigations, the procedures used for conducting the investigations, the conclusions, the rationale for reaching the conclusions, and any empirical verifications of those conclusions" ... all of which are to be done within the context of the evolving scientific theory. For, these six criteria are supposed to be "... six guiding principles that underlie all scientific inquiry, including education research. [Again, my skepticism about traditional "educational research" precludes my investigations into her specific works.]
I am even more skeptical about trying to use the Education Center's description of "scientific educational research" as a basis for supporting or denying the scientific-ness of any educational research effort ... or of anything else, for that matter. I am amazed that the NRC would allow such a publication.
So what does the above analysis say about Boaler's works? Absolutely nothing ... except that they cannot be scientifically credible without being scientifically definitive, within the context of a scientific theory! What about the challengers? Absolutely nothing ... except that they cannot be scientifically credible without being scientifically definitive within the context of a scientific theory! Their sociological disputes over her methods of educational research have no scientific meanings.
About Hake's, "Yes!": If Richard knows of some evolving scientific theory to which Boaler has contributed to to its development, I'm all ears ... not so much because of her own contributions, as because of the earthshaking importance and significance of any such scientific theory. Methinks that, unfortunately, Richard's "Yes!" stems not from science, as such, but from his other instructological information which has yet to congeal as a bona fide science. If so, Boaler's "findings" (whatever her methodology) might somehow help nurture the development of such a science. But unless such a science already hides within an inner sanctum of mathematics instructology, it is worse than rash to claim that her findings are scientifically reliable.
Yes, there is growing, scientifically reliable evidence that some specific kinds of mathematics-instruction "reforms" will result in easier, more productive ways of achieving even traditional goals of curricular mathematics education, plus others. Some of those reforms are clearly in harmony with some of Hake's contentions. But that is quite a different matter from proclaiming specific educational research efforts to be in some way "scientific." We (the world) are more than anxious to learn of whatever educational research efforts qualify as being genuinely "scientific." They are not at all easy to locate.
From: Richard Hake Sent: Friday, October 19, 2012 5:00 PM To: firstname.lastname@example.org Subject: [math-learn] Is "Education Research" "Scientific Research" ? YES!"
Some subscribers to Math-Learn might be interested in a recent post "Is 'Education Research' 'Scientific Research' ? YES!" [Hake (2012)]. The abstract reads:
********************************************* ABSTRACT: GS Chandy at <http://bit.ly/PfqJqt> wrote (liberally paraphrasing and drastically condensing); "Judging from his posts on the over 40-post Math-Teach thread 'Jo Boaler reveals attacks by Milgram and Bishop' at <http://bit.ly/Ty9tbf>, Robert Hansen is wrong in his claim that 'Education Research' is not 'Scientific Research.' "
To which Hansen at <http://bit.ly/RHESbF> responded with the following statement (paraphrasing) of which the 2nd and 3rd sentences are blatantly false (as I indicate in this post): "In scientific research, conclusions are tested repeatedly by many different researchers. In educational research, they are not. One does a study, claims success, publishes it and that is the end of it."
The National Academies' "Scientific Research in Education" at <http://bit.ly/VjrQaV> suggests six guiding principles that underlie all scientific inquiry, including education research: (1) Pose Significant Questions That Can Be Investigated Empirically, (2) Link Research to Relevant Theory, (3) Use Methods That Permit Direct Investigation of the Question, (4) Provide a Coherent and Explicit Chain of Reasoning, (5) Replicate and Generalize Across Studies, and (6) Disclose Research to Encourage Professional Scrutiny and Critique.
REFERENCES [All URL's shortened by <http://bit.ly/> and accessed on 19 Oct 2012.]
Hake, R.R. 2012. "Is 'Education Research' 'Scientific Research' ? YES!" online on the OPEN! AERA-L archives at <http://bit.ly/Vdj88z>. Post of 19 Oct 2012 13:11:11-0700 to AERA-L and Net-Gold. The abstract and link to the complete post are being transmitted to several discussion lists and are also on my blog "Hake'sEdStuff" at <http://bit.ly/OPmCjt> with a provision for comments.
[Non-text portions of this message have been removed]