Random Variables and Order Statistics

Date: 08/16/2001 at 00:42:11
From: Sam
Subject: Random variables
             
Can you give me some hints concerning the following problem?
             
Let X1, X2, ....., Xn be n independent and continuous random variables
with the same distribution.
             
Let X denote the maximum and Y the minimum of these random variables.
             
Let U = X/Y.
             
Find the distribution of X, Y, and U.

Date: 08/16/2001 at 11:57:35
From: Doctor Jordi
Subject: Re: Random variables
             
Hello, Sam - thanks for writing to Ask Dr. Math.
             
The problem you propose is an exercise in order statistics and 
transformations. Let us first look at the distribution of your maximum 
order statistic and of your minimum.  I will denote the minimum 
observation by X_min and the maximum by X_max.
             
Suppose you have ordered your sample of n Xi observations. That is, 
say we have a streak such that

     W1 < W2 < ... < Wn
             
where every Wi corresponds to some Xi and X_min = W1 = min(X1, X2, 
..., Xn) and X_max = Wn = max(X1, X2, ..., Xn).  Let F_X(x) denote the 
cumulative distribution function of the Xi. Given that the Xi are 
i.i.d., the distribution of each may be found according to the 
following argument.
             
We will proceed by first finding the distribution function of X_max.  
That is, we seek F_max(x) = P(X_max < x). Now, think about this 
problem this way: the probability that the maximum is less than any 
given number is not an event independent of the other observations.  
In fact, X_max will be less than any given x if and only if all the 
other observations are also less than this given x.
             
What is the probability that W1 is less than x (hint: what is the 
meaning of the cumulative distribution function F_X(x) of W1)?  What 
is the probability that W2 is also less than x? What is the 
probability that W1 AND W2 are less than x, given that W1 and W2 are 
independent (how do you find the probability of two _independent_ 
events)?  Therefore, what is the probability that W1 AND W2 AND W3 AND 
... AND Wn = X_max are all less than x?
             
I hope the above hints are enough for you. Once you have found the 
cumulative distribution function of X_max, remember to differentiate 
to obtain its density function.
             
The argument for finding the distribution of the minimum proceeds 
similarly. That is, we seek

 F_min(x) = P(X_min < x) = 1 - P(X > x)
             
and we know that the minimum will be greater than any given x if and 
only if all the other observations are greater than this given x.  
Remember that

 P(Xi > x) = 1 - P(Xi < x) = 1 - F_X(x)
             
The last step is to find the distribution function of the quotient of 
these two random variables. This can be done with similar reasoning.
             
We seek

 F_U(u) = P(U < u) = P(X_max / X_min < u)
             
That is, the probability of the set (x_max, x_min) such that 
x_max / x_min < u. If x_min is greater than 0, then this will be the 
set x_max < u*x_min. If x_min is less than 0, the set in question will 
then be x_max > u*x_min. This allows us to set up the following double 
integrals to evaluate the probability in question:

 F_U(u) = Int[-oo, 0]Int[u*x_min, oo] f(x_max, x_min) dx_max dx_min  +
            Int[0, oo]Int[-oo, u*x_min] f(x_max, x_min) dx_max dx_min
             
Where the symbol Int[a, b] f(x)dx is used to denote the definite 
integral of f(x) evaluated from a to b. (I've omitted the appropriate 
subscripts of the density functions for better readability). To 
evaluate these integrals, it is helpful to make the change-of-variable 
x_max = x_min*v in the inner integrals.
             
Does this help? Please write back if you feel I need to explain myself 
better or if you would like to talk about anything else.
             
- - Doctor Jordi, The Math Forum
    http://mathforum.org/dr.math/   

Date: 08/16/2001 at 18:39:16
From: Sam
Subject: Re: Random variables
             
I am extremely thankful for the answer you provided, and I think I got 
the idea concerning the calculation of the first part. However, 
concerning the part that deals with the ratio ( max / min ) , I am 
still wondering how I can calculate the joint p.d.f (f (x_max, x_min))  
you used in the double integrals.
             
Is there any general formula that allows me to calculate it for the 
ordered set of random variables you talked about at the beginning?
             
If yes I would be glad to receive your answer.
             
Thank you again.
Sincerely, Sam

Date: 08/17/2001 at 21:02:36
From: Doctor Jordi
Subject: Re: Random variables
             
Hello again.
             
I will treat in more detail the general case for the transformation of 
the quotient of two continuous random variables (without assuming 
independence).  I hope you have no difficulty applying it to the 
problem at hand.
             
Suppose we have two random variables, X and Y, with joint density 
f(x,y), and we seek the density function of Z = Y/X.  We proceed first 
by finding the cdf of Z.  F_Z(z) = P(Z < z) = P(Y/X < z).
             
Now, this last probability can be expressed as the sum of two disjoint 
events: [1]: P(Y < xz) if x > 0 and [2]: P(Y > xz) if x < 0.  (Since I 
am assuming continuous random variables, it is okay to use strict 
inequalities). By the additive axiom of disjoint events of the 
probability function,

P(Z < z) = P(Y < xz|x > 0) + P(Y > xz|x < 0)
             
It is now simple to setup the appropriate integrals to evaluate this 
probability.  In [1] Y ranges from -oo to xz, while x ranges from 
0 to oo.  In the other summand above, Y ranges from xz to oo and x 
from -oo to 0. The resulting double integrals look as follows:

F_Z(z) = Int[o,oo]Int[-oo,xz]f(x,y)dydx +
                Int[-oo,0]Int[xz,oo]f(x,y)dydx
             
In order to remove the dependence of x in the inner limits of 
integration, 
we make the change-of-variable v = y/x; dv = dy/x, which gives

Int[0,oo]Int[-oo,z] xf(x,xv) dvdx + Int[-oo,0]Int[z,-oo] xf(x,xv) dvdx

(take careful note of the changes in the limits of integration and
make sure you understand why they changed as they did).
             
Flipping the inner limits of integration on the right integral and 
using the absolute value function allows us to express this more 
neatly as follows:

Int[0,oo]Int[-oo,z] xf(x,xv) dvdx + Int[-oo,0]Int[-oo,z] - 
   xf(x,xv) dvdx

Int[0,oo]Int[-oo,z] |x|f(x,xv) dvdx
             
Now, if we can use the assumption that f(x,xv) is continuous, we can 
exchange the order of integration and differentiate with respect to z 
to obtain the appropriate density function

f_z(z) = Int[0,oo] |x|f(x,xz) dx
             
as required.
             
I will now address your question about the distribution of the kth
order statistic.
             
The setup, recall, is an i.i.d. sample of n observations. We take 
these observations and order them from least to greatest.  Now we want 
their distribution.
             
Let X1, X2, ..., Xn be independent, identically distributed random 
variables with common density f_x(x) and let Y1, Y2, ..., Yn be such 
that every Yi corresponds to some Xi and Y1 < Y2 < ... < Yn.  (note: 
since I am restricting myself to the continuous case, the previous 
inequalities are strict, although to be perfectly rigorous and 
general, they should be "less than or equal to").  We have already 
considered previously the distribution of Y1 and Yn (the min and the 
max). Let us now concentrate on the case n=2 and turn to the joint 
distribution of Y1 and Y2.
             
It is apparent that no two-order statistics will be independent. So 
whatever the joint distribution of Y1 and  Y2 is, it will not be the 
product of their marginal distributions.
             
The compound event (Y1 < y1, Y2 < y2) means that either 
(X1 < y1, X2 < y2) or (X2 < y1, X1 < y2). Notice that Y1 can be X1 or 
X2, whichever one is smaller. This means that the event (Y1 < y1 and 
Y2 < y2) is the union of two events. Recall the additive law of 
probability that states that for two events A and B we have

P(A union B) = P(A) + P(B) - P(A intersection B).
             
So, since y1 < y2, for F_Y1,Y2(y1, y2)  = P(Y1 < y1, Y2 < y2) we have 
the following:

P(Y1 < y1, Y2 < y2) = P(X1 < y1, X2 < y2) + P(X2 < y1, X1 < y2) -
   P(X1 < y1, X2 < y2)
             
Since X1 and X2 are independent with common distribution F_x(x), we 
can calculate

F_Y1,Y2(y1, y2) = F(y1)F(y2) + F(y2)F(y1) - F(y1)F(y1)

                = 2F(y1)F(y2) - [F(y1)]^2
             
To obtain the joint density f_Y1,Y2(y1, y2) we differentiate first 
with respect to y1 and then with respect to y2 to obtain simply

f_Y1,Y2(y1, y2) = 2f(y1)f(y2) for y1 < y2 and 0 elsewhere
             
This method can be applied to any number of n order statistics. In 
general, we obtain the following joint density of all the order 
statistics:

f_Y1,Y2,...Yn(y1, y2, ..., yn) = n!f(y1)f(y2)...f(yn)
                                             for y1 < y2 < ... < yn
                                             and 0 elsewhere
             
From this joint density, the marginal of the kth order statistic can 
be found by integration.
             
As an example, say n = 3 and we seek the marginal density function of 
Y2. We set up the following double integral and "integrate out" y1 
and y3. The limits of integration are set by the requirement that 
y1 < y2 < y3.

f_Y2(y2) = Int[-oo,y2]Int[y2,oo] 3! f(y1)f(y2)f(y3) dy1dy3

         = 3!f_X(y2)F_X(y2)[1 - F_X(y2))
             
Although this procedure can be generalized to obtain the density of 
the kth statistic, it is easier to proceed by a heuristic differential 
argument, which can be made rigorous if desired.
             
Think of the density function of a continuous random variable as an 
approximate measure of the probability that the random variable 
realizes a value in that neighbourhood.  P( x < X < x + dx) is 
approximately equal to f(x)dx for small dx. These approximations 
become exact in the limit as dx -> 0.
             
Say now we seek the marginal density function f_Yk(yk) of the kth 
order statistic. This can be done if we consider the problem as an 
exercise in applying the multinomial distribution. We have n 
observations. We want to know the probability that k-1 of those 
observations are less than yk, that exactly one of them is close to 
yk, and that the remaining n-k are greater than yk. The following 
table summarizes a bit of this information.

Class                  How many of them       Associated probability
--------------------------------------------------------------------
Yi's less than yk           k-1                 [F_X(yk)]^(k-1)

Yi's close to yk             1                   f_X(yk)

Yi's greater than yk        n-k                 [1 - F_X(yk)]^(n-k)
             
From the multinomial distribution, we now compute the probability of 
the event f_X(yk) = P("Yk is close to yk").

                 n!
f_Yk(yk) = --------------- [F_X(yk)]^(k-1)*f_X(yk)*[1 - F_X(yk)]^(n-k)
           (k-1)! 1! (n-k!)
             
An entirely analogous argument allows us to find the joint density of 
any two-order statistics. You will have to use this when you find the 
joint density of the maximum and the minimum. HINT: If two-order 
statistics are under consideration then there will be 5 classes for 
the multinomial distribution instead of 3. What are these classes?
             
This should be enough to help you answer your question.  If you still 
need anything else, please let us know.
             
- Doctor Jordi, The Math Forum
  http://mathforum.org/dr.math/