On 2013-02-20, Dave <email@example.com> wrote: > Wow, I hadn't expected my post to generate such a fire storm. I guess neutrality IS a big deal, LOL.
> Okay, so let me carefully address "significance." Overall, I agree with you. Indeed, as memory serves me, Fisher preferred the reporting of p values instead of the rejection/acceptance region idea. But that isn't really the issue.
> It isn't always about an action. In business or engineering applications there is usually an action to be taken and significance testing neither includes cost functions nor gives you automatic direction. Rejecting an hypothesized value for the precision of a machine may be informative, but failing to reject isn't informative. That is clearly a problem. On the other hand, in science it can be valuable to perform significance testing.
> Fisher, again this is as I understand it, started down the Frequentist path in order to place some regularity on problems where no prior data existed and I do believe he stated that the Bayesian method should be used where prior information existed.
At the time Fisher started, "Bayesian" had become a dirty word, in no small part because the philosophers of science had erroneously believed that the data forced the model. This was never true; the data only allowed choice among the models envisaged.
After Neyman-Pearson, Fisher attempted to come up with similar "objective" methods, leading to "fiducial inference". It has all the markings of the same confusion wihich existed earlier.
> Science is a combination of deductive and inductive reasoning. Inductive reasoning is incomplete by its nature, even if in practice it is powerful. Frequentism can be powerful if it is properly used as an extension of modus tollens. Indeed, many areas of error within science have come from using inductive reasoning rather than deductive reasoning. The fact that the point null hypothesis is always false isn't a problem for two reasons that I will show below.
Inductive logic leads to Bayesian.
> Let us imagine that you believed that someone was a con man and for some reason you wanted to invest your money to prove it. You gamble in either a coin tossing game or a game similar to three card monte. It doesn't matter really.
Even here one should be using likelihood ratios. There are infinitely many statistical tests, and it was not until Neyman-Pearson that the choice of test between two point hypotheses was reduced to the likelihood ratio. Bayesian procedures are what extend this to more general cases.
> You decide to be Frequentist and use the null hypothesis that p(coin=heads)=1/2 or if you are doing three card monte that each choice has a 1 chance in 3 of happening.
> If you flip a US quarter enough times you can show, some grad students at Harvard did this for some reason, that it is not a fair coin. I think they had a robot flip the coin 50 or 100,000 times. Likewise, psychological research on the ability of humans to replicate random behavior shows that humans will not choose all three piles evenly.
Statistics does not concern itself about how ignorant humans operate. It is concerned with doing the best for the user. `
> If you flip the coin enough, you are guaranteed to get evidence the other party is not using a fair coin even if it is a fair coin in the sense they are not engaging in policy to cheat the other party. This is true as a Frequentist or a Bayesian.
> Except in the real world we rarely flip the coin enough. Statistics is an approximation. If you really have enough data, you don't really need statistics. If you stand Michael Jordan against a five year old and you are asked who is taller, getting out a measuring tape would only be required by journal editors.
> The Frequentist hypothesis may be false, but that is okay, as the goal is to prove that it is really really false and not just really false. A frequentist is simply saying, "I believe the world works thus and if it is true then certain things should rarely happen." They could happen, but it would be surprising and you should probably consider alternate models if they do.
> That said, you would still be better off with the Bayesian model as there is information in the problem that is ignored. For a con man to get away with such a thing, the coin needs to look fair and so an ignorance prior is inappropriate.
> Frequentism is helpful when two criteria are met. One, you don't have enough data to decide without statistical tools. Two, you can define clearly what "I am wrong," means. The broad error in statistical education is no one teaches that.
One can include an action which has one wait; in principle, one should consider all possible actions. In pracice, one has to approximate. The current methods are only tradition, based on poor and incorrect reasoning.
> No one sits down and says, this is a Bayesian problem or this is a Frequentist problem. Why we ram t-tests down the throats of sophomores is beyond me. Why no one has constructed a book on reasoning makes no sense at all.
Why we teach the standard procedures without the necessary backgruound to understand them is what makes no sense. They are turned off from understanding it later, just as students are turned off from understanding the much simpler idea of the integral after they have learned how to antidifferentiate. Learning how to calculate anything seems to always make it harder to understand what is being calculated.
> One more note, I try and avoid self-consistent behavior. I realized when I was 26 years old and Santa didn't bring me a present that year that the universe might not be self consistent. I do admit it wasn't my best year on the "naughty/nice" spectrum, but you don't expect the rug to be pulled out from you. Since then his track record has been spotty. Figuring that the universe can fail at being self-consistent, I don't mind a little inconsistency either.
Self-consistency here concerns only a single moment of time. If the universe is not self-consistent, we could not do science; think about the problem of deducing the rules behind observations if they were changing, not according to some rules. This allows changes with time, again orderly.
-- This address is for information only. I do not claim that these views are those of the Statistics Department or of Purdue University. Herman Rubin, Department of Statistics, Purdue University firstname.lastname@example.org Phone: (765)494-6054 FAX: (765)494-0558