The comparison to a mythical piano playing inventory is interesting, but it does not apply. Concept inventories are usually validated by free response and also by interview protocols. So they have been validated by seeing the student in action. And various researchers have correlated them with other factors. The Heller's have a program that is designed to improve performance on rich context problems, and the FCI gain is very good. Mazur emphasizes improving FCI gain, but has also as a result seen improvement in problem solving skills on a conventional exam.
Since science and math skills are usually tested by examinations, a concept inventory is just a specialized exam. But musicianship is generally tested by authentic evaluation of performance on projects such as writing or performing music.
But remember I pointed out that getting more data such as AP data is much more difficult. As Jerry pointed out mechanical skills are not generally tested by concept inventories, and mechanical skills are not as highly correlated with performance as people would like to think. AP has a lot of testing of purely mechanical skills. My example here is that arithmetic prowess is not highly correlated with the ability to do higher level thinking because they reside in different portions of the brain. Both I and my daughter are examples of this. Dyslexia blocks prowess in simple mechanical recall of "math facts", but algebra is easy if you have high level thinking. Neither of us would be good bookkeepers, but a PhD in science is possible. There is even a chemist who is totally disnumeric and can not do arithmetic without a calculator. The myth the arithmetic prowess is necessary for advanced studies probably comes from the fact that students who had trouble with math facts were held back. Then after a couple of failures they dropped out. So dyscalculia became a self fulfilling prophesy. Once it was recognized, individuals could be helped and achieved high goals.
So let us make it a math proposition. Inability to do arithmetic well on timed tests is an indicator that students are not capable of higher math. FALSE because I just cited two examples where this was not true. It is a simple proof by contradiction.
So although there may be a correlation between AP and the other tests, the other tests are actually testing things that are not necessarily totally relevant to AP performance. But they are probably relevant to other types of performance. One would think that a straight A medical student would be better than one who had lower grades, but in truth, there is no strong correlation. Indeed the correlation between the MCAT and physician performance is pretty low. As I recall, some students who passed AP still scored low on other evaluations such as the FCI. So even such long evaluations such as AP are only measuring a fraction of what is needed to eventually do well.
Even IQ is not as good a predictor as is often thought. It can and does change. For dramatic evidence here see "Instrumental Enrichment" by Reuven Feuerstein. It has been estimated that it only measures about 1/3 of the human potential. The ability to be innovative and transfer is not well measured by AP or most other exams. As a result the Oriental school systems are looking at the US system for the solution to how to make students more innovative. I fear that the current emphasis on just state test scores is killing our student's ability to innovate.
John M. Clement Houston, TX
> > Actually, they do here (in the U.S.) and one can easily match > that performance to the TIMSS performance here and in > Shanghai. Without meaningful and whole comparisons, like > comparing performance on AP exams, it will be very hard for > IE to gain any traction at all. But let's move on. The > University of Michigan actually uses the AP exam format for > their Calculus 1 and Calculus 2 classes (115 & 116). Have you > tried to compare exam results that involve everything > (concept, functional and problem solving skills)? Such a > comparison though will not be very helpful because the curve > used at UM is very pronounced. For example, in Math-116 > (Calculus 2), 32% on the Final exam will garner a "C". A > comparison in that environment is more like saying who is > failing the least. You compared IE class performance to the > stellar (and complete) performance of Shanghai and I think to > be credible, you (or John) must supply us with verifiable > examples of such, beyond pre and post gains on a concept > inventory that really does not provide the whole picture. As > an analogy, imagine if I said I created a music concept > inventory for piano playing. I seriously doubt that you would > accept that as an indication of success in learning to play > the piano without actually seeing if the kid can play the piano. >