T-Tests, P-Values, and Statistical Significance

Date: 03/18/2004 at 22:15:49
From: Matt
Subject: What does the t test tell you?

I am doing a lab report comparing two different samples of fish.  For 
the results the teacher wants a t-test.  I have calculated it and lots
of other information.  Could you tell me what this information means
and how I interpret it?	
       
What does the t-value and two-tailed P-value tell me and how do they 
compare to each other?  Is this information "significant" enough to 
say that variable 2 came from the same family as variable 1?

Here is my data:

                Variable 1     Variable 2
Mean	        0.562	       0.09152
Variance	0.00097	       0.003081962
Observations	    5	           5
Pooled Variance	       0.002025981	
Hypothesized Mean Difference  0	
df	                      8	
t Stat	                16.52697943	
P(T<=t) one-tail	9.06766E-08	
t Critical one-tail	1.85954832	
P(T<=t) two-tail	1.81353E-07	
t Critical two-tail	2.306005626

Date: 03/19/2004 at 09:24:34
From: Doctor Achilles
Subject: Re: What does the t test tell you?

Hi Matt,

Thanks for writing to Dr. Math.

A t-test tells you the probability that two sets of values come from 
different groups.  Using a one-tailed P-value assumes you already know 
before you even see the values which group should be larger and which 
should be smaller.  Since this is usually not true, you should almost 
always use a two-tailed test.

Let's say, for example, that I have the following hypothesis: "The 
average age of trees in Yellowstone National Park is significantly 
different than the average age of trees in Yosemite National Park."  
How would I test that (let's assume that I have a way to accurately 
determine the age of a tree without cutting it down)?  I don't have 
the resources to check the ages of all the trees in each park, so I 
will take a small random sample from each park and then use a t-test 
to compare them.

A two-tailed P-value of 0.6, for example, would mean that there is a 
0.6 (or 60%) chance that the two sets of values come from the same 
group.  In other words, there is a 60% chance that the average age of 
the trees in each park is the same, and that whatever difference I 
may have seen in my random sample can be explained by the fact that I 
only sampled a small portion of the trees.  If I got a P-value of 
0.6, I would say that there is no significant difference between the 
ages of the two populations.

A two-tailed P-value of 0.1 would mean that there is a 0.1 (or 10% 
chance) that the two sets come from the same group.  In this case, 
there is a pretty good chance that the ages of the two populations is 
different.  However, in order to be on the safe side, it is 
traditional in science to say that a P-value of 0.1 is NOT 
significant.  Why?  Because if 0.1 were considered significant, then 
10% of all scientific findings would be false.  So even if I got a P-
value of 0.1, I couldn't say anything for sure, the most I could say 
is that more study is probably required.

The traditionally accepted P-value for something to be significant is 
P < 0.05.  So if there is less than a 5% chance that two sets came 
from the same group, then it is considered a significant difference 
between the two sets.

A t-test computes a "t-value".  There is a complicated mathematical 
relationship that I don't know off the top of my head between a t-
value and a P-value that depends on the size of the samples (and one 
or two other variables).  Larger t-values translate into smaller P-
values.  So the larger the t-value is the more likely the difference 
is significant.  A "critical t-value" is the minimum t-value you need 
in order to have P < 0.05.  If your t-value is greater than or equal 
to the critical t-value, then you will have a significant difference.

In your problem, your critical t-value is 2.306005626, your t-value 
is 16.52697943.  So, your t-value is greater than the critical t-
value, therefore the difference between the two sets is significant.

This is confirmed by the fact that your two-tailed P-value is 
1.81353E-07 or 0.000000181353; this is extremely small (much less 
than 0.05).

I would call the difference between these values "highly significant".

(If I were you, I'd go back and recheck that you entered all the 
values correctly, since this difference is much more significant than 
one usually gets with only 5 samples in each set; but if you did, then 
congratulations: you have found a big effect of whatever you were 
testing!)

I hope this explanation is helpful.  If anything is unclear, or you'd 
like to talk about some of this more, please write back.

- Doctor Achilles, The Math Forum
  http://mathforum.org/dr.math/