I am not sure what population we checked this with, or in fact whether someone else did this, but we found that the normalized gain was independent of the pre-test score, a worthwhile thing to keep in mind. his is a major part of why this seems to be such a worthwhile measure.
On 1/27/2012 1:33 PM, John Clement wrote: > > The normalized gain does not confuse the issue, but rather it is just a > different way to look at the results. It doesn't change the results. The > normalized gain is used to remove the bias of student innate tendency to > understand. It is more sensative to what is done in class because it > is one > way of factoring out what students already know. Of course it is not > totally insenstive to prior knowledge because the ability to think > increases > both prior knowledge and gain. If you use effect size you also see > dramatic > gain. Some researchers have just shown the graphs. But each method of > presentation has its own bias. The papers on Interactive Lecture > Demonstrations show just the graphs for individual questions and the > results > are extremely striking with very large gains from either Real Time > Labs, or > ILDs as compared to lectures. Looking at the evidence shown at the > beginning of the ILD book, pre instruction is between 5 and 17%, after > traditional instruction the scores rise to getween 10 and 23%, but > after the > ILDs they rise to between 70 and 90%. This word was done by Sokoloff, > Thornton, and Laws with one paper being in The Physics Teacher 36: 6, 340 > (1977), but there are others in American Journal of Physics. > > As to a full blown exam, Eric Mazur showed that students did better on a > "full blown physics final" when he had gotten improved understanding as > measured by the FCI. So of course the reaction was that Mazur could not > create a standard physics exam, in other words he was a fool. But he is a > respected Harvard professor who holds a number of chairs. So why not > contact him and ask him to release to you his "standard" physics exam and > also get one from a colleague and compare them. In research one > usually has > to take the claim made by the other researchers at face value, and then do > an experiment to refute their results. Saying they are incompetent may > work > in politics, but it only works temporarily in science. Goethe's > theories of > color vision trumped other scientists because Goethe was extremely > nasty and > had a high reputation. But after his death, the competitors were finally > able to score their points. > > As to whether the students who get a 5 on an AP test really understand > calculus, one has to do that experiment to know the result. You can not > assume the result without obtaining it. I am sure that there is a fraction > (maybe even large) of high AP scorers who do not understand calculus, > but I > am willing to say I am wrong if the experiment shows something else. We > have found that some graduate students in physics have difficulty with the > FCI, and one would assume that graduate students would have perfect > results. > The CCI is not measuring the same things that the AP measures. I suspect > that the AP exam would not be able to measure the ability to transfer as > well as the CCI, but one would need more testing for that hypothesis. The > essence of research is when someone claims that the theory is so good > that a > particular result is absolutely true, but the experiment disproves it. > This > sort of thing has happened time and again in medicine. Physicians denied > that bacteria were the prime cause of ulcers for many years after some > definitive experiments, and they continued to cut out parts of stomachs. > > As to the questions on the FCI or FMCE testing purely rote > memorization, the > answers to those questions have been told to students, and they still get > them wrong. Part of the psychology behind what is going on can be found in > the web reference. > http://www.sciencenewsline.com/psychology/2012012017430043.html > Students when given a question couched in everyday language then give an > answer based on their gut feeling and do not analyze. They particularize > the way they answer questions, so they only turn on an analysis > appropriate > to the type of question. So when they see the heading proportions, they > create ratios, but when there is no explicit clue they often can't make a > ratio because they don't understand proportional reasoning. So it our > judgement based on what we have seen and the testing we have done that the > concept inventories are not answerable usually by rote memorization. The > reason why they convinced us is that researchers were in doubt until they > gave the inventories. Then they saw that students who had been told the > results, still got the questions wrong. Joe (Edward) Redis, and Eric Mazur > are prime examples of this. They even tried better lectures, but still > didn't get better results. > > Let us remember that all exams are just measuring something, and even when > we think the measurement is good, it may not be. So extreme faith in > the AP > is not any more reasonable than faith in our tests. There are other > issues, > and with time they are being researched. At one time the IQ test was > considered to be an immutable test of intelligence. In other words it > wasn't subject to change for an individual and it was a true measure of > intelligence. But researchers like Feuerstein blew this out of the > water by > producing a long term large change in IQ. The Flynn effect also blows this > out of the water. > > As to how people think, even the most educated and "intelligent" > people use > gut reactions instead of analysis. Everyone does it and neither you nor I > am immune to it. We often draw non rational conclusions because of this > tendency. Lawson found that understanding of concepts in science was much > more difficult for things that could not be seen, and showed that there > seems to be a level of thinking above the top level proposed by Piaget. A > lot of the argumentation here is actually gut feeling, and people tend to > manufacture evidence based on the feeling without looking at evidence. As > part of this process people tend to see correlations that do not > exist, and > they unconciously ignore evidence that is contrary to their notions. > Global > warming and evolution deniers do exactly that, and ignore the fact > that the > experts are convinced. Logical arguments do not change people's minds. > They just harden positions and bring out more opposition. This of course > can be seen in the development of things like the helicentric univers > vs the > geocentric universe, or medical advances like antiseptics. > > One of the things that IE seems to do is break down the student gut > feelings > which tie them to positions which are at odds with evidence. The ILDs do > this by memory reconsolidation. One would assume that a student who has > taken calculus would be able to understand that acceleration does not > go to > zero just because velocity goes to zero, yet we have found that this > misconception exists even in students who have had 2 years of calculus (I > have seen it in class). Our position is that to be able to analyze > well the > students have to line up their gut feelings with their ability to analyze. > IE uses specialized techniques along with consistent coherent instruction. > > So there are basically 2 views here. One can assume that because IE works > better for average or below average students that it also works for the > superior student. But of course that has not been proven. The assumption > that standard instruction works best for the superior student is the other > point of view. It has been acknowledged that it doesn't work well for the > lower 80%. But the superiorty for the top 1% hasn't been proven either. > The only evidence that I can point to are some early papers in JRST > cited by > Lawson in his book "Science Teaching and the Development of Reasoning" > where > they found that the learning cycle works for all students. The > evidence was > the exploration had to come first for the students below the formal > operational level, but concept development (lecture) could come first for > the others. But they found that the formal operational students still > needed an exploration phase. > > If someone could actually show that the top 1% benefits more by > conventional > instruction I would be happy to consider it. But I would like to make sure > that the ability to transfer the learning was tested. I think the PER > community would agree that the ability to transfer is one of the most > important abilities a student can have. And of course the experiment MUST > have two equivalent groups given very different treatments. I suspect that > IE will do a better job, but I am extremely doubtful that conventional > teaching does a better job. My "belief" here is based on the fact that > education up until now has been done by gut feelings rather than by an > understanding of how students learn and "accommodate" new ideas. In other > words what is brain incompatible for lower students is likely to be > incompatible for all students. Until this experiment has been done, it is > quite obvious that arguments using informal observation will not change my > mind. Can anyone come up with some experimental evidence here? I always > ask for evidence and never get any citations to it! > > John M. Clement > Houston, TX > > >Jerry said that most students that pass calculus don't even know what > calculus is. I agree with him. I wasn't talking > >about that group and the Jerry wasn't talking about students scoring > 5's on > AP exams. I put the bar high enough so that > >any more quibbling would be silly. > > > > > > I have read and studied a great deal of the data John (and > > Hake) have pointed me to. And I have looked, as much as > > possible, at the scant "live" examples of IE that they have > > pointed me to. > > > > My simple request for something real, like showing me top > > students practicing I.E. is born of all that review. The > > "research" John keeps pointing me to is ruined by a > > fundamental (and absurd) flaw in the design of the > > experiment. Essentially, an I.E. class is set up next to a > > traditional class and an inventory test is taken at the > > beginning and end (by each class). And then the results are > > compared. Actually, rather than comparing the results > > directly and absolutely, they are obfuscated by converting > > them to "normalized gain" prior to the comparison. But that > > isn't the flaw I was talking about. The flaw is that an > > inventory test is no substitute for a full blown calculus or > > physics exam. Who would form an hypothesis that IE classes > > are better than traditional classes and then not set out to > > prove it with full blown traditional exams? You can't make a > > claim and then redefine how you test the claim as well. To > > make the claim that IE is better than traditional you have to > > face the music and test it using full blown exams, not > > fragments of an exam. > > > >
[Non-text portions of this message have been removed]