The normalized gain does not confuse the issue, but rather it is just a different way to look at the results. It doesn't change the results. The normalized gain is used to remove the bias of student innate tendency to understand. It is more sensative to what is done in class because it is one way of factoring out what students already know. Of course it is not totally insenstive to prior knowledge because the ability to think increases both prior knowledge and gain. If you use effect size you also see dramatic gain. Some researchers have just shown the graphs. But each method of presentation has its own bias. The papers on Interactive Lecture Demonstrations show just the graphs for individual questions and the results are extremely striking with very large gains from either Real Time Labs, or ILDs as compared to lectures. Looking at the evidence shown at the beginning of the ILD book, pre instruction is between 5 and 17%, after traditional instruction the scores rise to getween 10 and 23%, but after the ILDs they rise to between 70 and 90%. This word was done by Sokoloff, Thornton, and Laws with one paper being in The Physics Teacher 36: 6, 340 (1977), but there are others in American Journal of Physics.
As to a full blown exam, Eric Mazur showed that students did better on a "full blown physics final" when he had gotten improved understanding as measured by the FCI. So of course the reaction was that Mazur could not create a standard physics exam, in other words he was a fool. But he is a respected Harvard professor who holds a number of chairs. So why not contact him and ask him to release to you his "standard" physics exam and also get one from a colleague and compare them. In research one usually has to take the claim made by the other researchers at face value, and then do an experiment to refute their results. Saying they are incompetent may work in politics, but it only works temporarily in science. Goethe's theories of color vision trumped other scientists because Goethe was extremely nasty and had a high reputation. But after his death, the competitors were finally able to score their points.
As to whether the students who get a 5 on an AP test really understand calculus, one has to do that experiment to know the result. You can not assume the result without obtaining it. I am sure that there is a fraction (maybe even large) of high AP scorers who do not understand calculus, but I am willing to say I am wrong if the experiment shows something else. We have found that some graduate students in physics have difficulty with the FCI, and one would assume that graduate students would have perfect results. The CCI is not measuring the same things that the AP measures. I suspect that the AP exam would not be able to measure the ability to transfer as well as the CCI, but one would need more testing for that hypothesis. The essence of research is when someone claims that the theory is so good that a particular result is absolutely true, but the experiment disproves it. This sort of thing has happened time and again in medicine. Physicians denied that bacteria were the prime cause of ulcers for many years after some definitive experiments, and they continued to cut out parts of stomachs.
As to the questions on the FCI or FMCE testing purely rote memorization, the answers to those questions have been told to students, and they still get them wrong. Part of the psychology behind what is going on can be found in the web reference. http://www.sciencenewsline.com/psychology/2012012017430043.html Students when given a question couched in everyday language then give an answer based on their gut feeling and do not analyze. They particularize the way they answer questions, so they only turn on an analysis appropriate to the type of question. So when they see the heading proportions, they create ratios, but when there is no explicit clue they often can't make a ratio because they don't understand proportional reasoning. So it our judgement based on what we have seen and the testing we have done that the concept inventories are not answerable usually by rote memorization. The reason why they convinced us is that researchers were in doubt until they gave the inventories. Then they saw that students who had been told the results, still got the questions wrong. Joe (Edward) Redis, and Eric Mazur are prime examples of this. They even tried better lectures, but still didn't get better results.
Let us remember that all exams are just measuring something, and even when we think the measurement is good, it may not be. So extreme faith in the AP is not any more reasonable than faith in our tests. There are other issues, and with time they are being researched. At one time the IQ test was considered to be an immutable test of intelligence. In other words it wasn't subject to change for an individual and it was a true measure of intelligence. But researchers like Feuerstein blew this out of the water by producing a long term large change in IQ. The Flynn effect also blows this out of the water.
As to how people think, even the most educated and "intelligent" people use gut reactions instead of analysis. Everyone does it and neither you nor I am immune to it. We often draw non rational conclusions because of this tendency. Lawson found that understanding of concepts in science was much more difficult for things that could not be seen, and showed that there seems to be a level of thinking above the top level proposed by Piaget. A lot of the argumentation here is actually gut feeling, and people tend to manufacture evidence based on the feeling without looking at evidence. As part of this process people tend to see correlations that do not exist, and they unconciously ignore evidence that is contrary to their notions. Global warming and evolution deniers do exactly that, and ignore the fact that the experts are convinced. Logical arguments do not change people's minds. They just harden positions and bring out more opposition. This of course can be seen in the development of things like the helicentric univers vs the geocentric universe, or medical advances like antiseptics.
One of the things that IE seems to do is break down the student gut feelings which tie them to positions which are at odds with evidence. The ILDs do this by memory reconsolidation. One would assume that a student who has taken calculus would be able to understand that acceleration does not go to zero just because velocity goes to zero, yet we have found that this misconception exists even in students who have had 2 years of calculus (I have seen it in class). Our position is that to be able to analyze well the students have to line up their gut feelings with their ability to analyze. IE uses specialized techniques along with consistent coherent instruction.
So there are basically 2 views here. One can assume that because IE works better for average or below average students that it also works for the superior student. But of course that has not been proven. The assumption that standard instruction works best for the superior student is the other point of view. It has been acknowledged that it doesn't work well for the lower 80%. But the superiorty for the top 1% hasn't been proven either. The only evidence that I can point to are some early papers in JRST cited by Lawson in his book "Science Teaching and the Development of Reasoning" where they found that the learning cycle works for all students. The evidence was the exploration had to come first for the students below the formal operational level, but concept development (lecture) could come first for the others. But they found that the formal operational students still needed an exploration phase.
If someone could actually show that the top 1% benefits more by conventional instruction I would be happy to consider it. But I would like to make sure that the ability to transfer the learning was tested. I think the PER community would agree that the ability to transfer is one of the most important abilities a student can have. And of course the experiment MUST have two equivalent groups given very different treatments. I suspect that IE will do a better job, but I am extremely doubtful that conventional teaching does a better job. My "belief" here is based on the fact that education up until now has been done by gut feelings rather than by an understanding of how students learn and "accommodate" new ideas. In other words what is brain incompatible for lower students is likely to be incompatible for all students. Until this experiment has been done, it is quite obvious that arguments using informal observation will not change my mind. Can anyone come up with some experimental evidence here? I always ask for evidence and never get any citations to it!
John M. Clement Houston, TX
>Jerry said that most students that pass calculus don't even know what calculus is. I agree with him. I wasn't talking >about that group and the Jerry wasn't talking about students scoring 5's on AP exams. I put the bar high enough so that >any more quibbling would be silly. > > > I have read and studied a great deal of the data John (and > Hake) have pointed me to. And I have looked, as much as > possible, at the scant "live" examples of IE that they have > pointed me to. > > My simple request for something real, like showing me top > students practicing I.E. is born of all that review. The > "research" John keeps pointing me to is ruined by a > fundamental (and absurd) flaw in the design of the > experiment. Essentially, an I.E. class is set up next to a > traditional class and an inventory test is taken at the > beginning and end (by each class). And then the results are > compared. Actually, rather than comparing the results > directly and absolutely, they are obfuscated by converting > them to "normalized gain" prior to the comparison. But that > isn't the flaw I was talking about. The flaw is that an > inventory test is no substitute for a full blown calculus or > physics exam. Who would form an hypothesis that IE classes > are better than traditional classes and then not set out to > prove it with full blown traditional exams? You can't make a > claim and then redefine how you test the claim as well. To > make the claim that IE is better than traditional you have to > face the music and test it using full blown exams, not > fragments of an exam. >