On p-value

Albert Schweitzer said: “Example is not the main thing in influencing others. It is the only thing.”, so I start with it.

Generate two random samples from Normal distribution.
Test the hypothesis number one: \mu_1 = \mu_2 = 0, do not reject, then test the more general hypothesis number two: \mu_1 = \mu_2.
Logic states that since there is at least one value we do not reject for (in this case, 0), the more general statement would also not be rejected.

This is not always the case.

?View Code RSPLUS
PVgen = function(mu1,mu2){
n = 100
x1 = rnorm(n,mu1,1)
x2 = rnorm(n,mu2,1)
X = cbind(x1,x2)
mx = apply(X,2,mean)
### LR test for NULL mu1 = mu2 = 0
lambda1 = exp((-n/2)*( t(mx)%*%mx ))
Tstat1 = -2*log(lambda1)
### LR test for NULL mu1 = mu2
l = c(1,1)
II = diag(2)
lambda2 = exp((-n/2)*( t(mx)%*% (II - .5*(l%*%t(l))) %*% mx ) )
Tstat2 = -2*log(lambda2)
LT1 = ifelse(Tstat1>qchisq(.5,2),T,F) ; LT2 = ifelse(Tstat2>qchisq(.5,1),T,F)
pv1  <-  2*(1 - pchisq(Tstat1,2, lower.tail = LT1))
pv2  <-  2*(1 - pchisq(Tstat2,1, lower.tail = LT2))
s0 <- s1  <- s2 <- pvalues <-  NULL
for (i in 1:500){
	pvalues  <- rbind(pvalues, PVgen(.16,-.16)	)
	s1[i]  <-  sign(pvalues[i,1] - .05)
	s2[i]  <-  sign(pvalues[i,2] - .05) 
# which was rejected when the tests do not agree?
	s0[i]  <- ifelse(s1[i] != s2[i],which.min(pvalues[i,]),0 ) 
sum(s0==2) # when the second was rejected but the first did not..

Since the code involves simulations, results may differ between users, despite the repetition procedure applied.

So in about 14% = 70/500 cases, we can not reject hypothesis number one, but we reject the more general hypothesis number two. Note, using the same data.

What is the point to take from this?
Well, that there is no logical contradiction here, we just need to be careful with the interpretation of p-values:
“one agent that does not believe in a proposition does NOT imply the she believes in its negation” (Dubois and Prade, 2001).

The idea for the post came from reading the paper: A classical measure of evidence for general null hypotheses. The author suggest an alternative measure to p-value.


2 thoughts on “On p-value

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>