On Saturday, August 16, 2014 10:43:01 PM UTC-4, Rich Ulrich wrote: > On Sat, 16 Aug 2014 10:50:30 -0700 (PDT), firstname.lastname@example.org > > wrote: > > > > >Hi, > > > > > >We are testing some attribute of pairs of objects that satisfy a > > >given crierion vs. pairs that do not. However, in the sample we have > > >all the pairs satisfy the criterion. > > > > You want to test A vs. B; and you have no B. > > Therefore: You do not have any test. That seems straightforward. > > > > Can you make some argument about how that test would result, > > if data existed for it? ... conceivably. I would think of depending > > on observably high (near perfect?) correlations. - That, however, > > would (it seems to me, though I have no concrete example) rule > > out the strategy that you go on to propose. Assuming that I > > understand your proposal at all. > > > > > > > I proposed randomly paring the objects repeatedly and check the > > >attribute. The permutation would be w/o restriction. It would allow > > >pairs that satisfy as well as pairs that do not satisfy the criterion. > > >Some members of our group claim that we should exclude from the random > > >permutations the pairs that satisfy the criterion, keep randomizing > > >untill none of the observed pairs appears in the permutation. Is this > > >an accespted approach? My recollection is that in a permutation test > > >the observed data must be a possible outcome. Could you plese point me > > >to any reference that explaints which way is correct? > > > > > > > Nope, I don't follow; but I'm not trying very hard. Neither the data > > description nor the strategy rings any bells of familiarity. > > > > Maybe this all makes sense when you plug in the concrete terms, but > > I'm not intrigued enough to fiddle with my own made-up examples (which > > are unlikely to be anything like your data). > > > > Try stating the specifics? > > > > -- > > Rich Ulrich
Thank you for your response. I'll try to be more clear with a very similar example. We want to test whether the heights of pairs of brothers are more similar than the heights of random pairs of boys of similar age but with no relation to each other. Problem is that the sample we have has only pairs of brothers but we think that the boys in the sample are representative of the population. Thus, to have an idea of how similar the heights of random pairs of brothers we randomly make pairs with the boys in the sample and calculate our similarity metric. This metric we compare with the metric we obtained from the brothers' pairs. We repeat the random pairing a large number of times and count the number of random pairings whose similarity metric is as extreme as or more extreme than that of the paired brothers. The ratio of this number to the total number of random pairings is our p-value.
Our argument is whether or not to allow two brothers in the same random pair. Now, from the little I know about permutation test, the observed data must be one of the possible random permutations, otherwise we're comparing a possible outcome against an impossible one.
Thank you again for your response and sorry for my unclear question.