Virtual University of Pakistan Lecture No. 34 of the course on Statistics and Probability by Miss Saleha Naghmi Habibullah
IN THE LAST LECTURE, YOU LEARNT •Sampling Distribution of X1 − X 2 (continued) Sampling Distribution of pˆ1 − pˆ 2 •Point Estimation •Desirable Qualities of a Good Point Estimator –Unbiasedness –Consistency
TOPICS FOR TODAY •Desirable Qualities of a Good Point Estimator: •Efficiency •Methods of Point Estimation: •The Method of Moments •The Method of Least Squares •The Method of Maximum Likelihood •Interval Estimation: •Confidence Interval for µ
The students will recall that, in the last lecture, we presented the basic definition of a point estimator, and considered some desirable qualities of a good point estimator.
Three main qualities of a good point estimator are:
DESIRABLE QUALITIES OF A GOOD POINT ESTIMATOR
•unbiasedness •consistency •efficiency
In the last lecture, we discussed in some detail the concepts of unbiasedness and consistency.
Why considered quality?
is
consistency a
desirable
In order to obtain an idea regarding the answer to this question, consider the following:
As a sample is only a part of the population, it is obvious that the larger the sample size, the more representative we expect it to be of the population from which it has been drawn.
In agreement with the above argument, we will expect our estimator to be close to the corresponding parameter if the sample size is large. Hence, we will naturally be happy if the probability of our estimator being close to the parameter increases with an increase in the sample size. As such, consistency is a desirable property.
Another important desirable quality of a good point estimator is EFFICIENCY:
EFFICIENCY An unbiased estimator is defined to be efficient if the variance of its sampling distribution is smaller than that of the sampling distribution of any other unbiased estimator of the same parameter.
In other words, suppose that there are two unbiased estimators T1 and T2 of the same parameter θ. Then, the estimator T1 will be said to be more efficient than T2 if Var (T1) < Var (T2).
In the following diagram, since Var (T1) < Var (T2), hence T1 is more efficient than T2 : Sampling Distribution of T1
Sampling Distribution of T2
The relative efficiency of T1 compared to T2 (where both T1 and T2 are unbiased estimators) is given by the ratio
Var (T2 ) Ef = . Var (T1 )
And, if we multiply the above expression by 100, we obtain the relative efficiency in percentage form.
It thus provides a criterion for comparing different unbiased estimators of a parameter.
Both the sample mean and the sample median for a population that has a normal distribution, are unbiased and consistent estimators of Âľ
but
the variance of the sampling distribution of sample means is smaller than the variance of the sampling distribution of sample medians.
Hence, the sample mean is more efficient than the sample median as an estimator of Âľ. The sample mean may therefore be preferred as an estimator.
Next, we consider various methods of point estimation. A point estimator of a parameter can be obtained by several methods. We shall be presenting a brief account of the following three methods:
METHODS OF POINT ESTIMATION •The Method of Moments •The Method of Least Squares •The Method of Maximum Likelihood
These methods give estimates which may differ as the methods are based on different theories of estimation. We begin with the Method of Moments:
THE METHOD OF MOMENTS
The method of moments which is due to Karl Pearson (1857-1936), consists of calculating a few moments of the sample values and equating them to the corresponding moments of a population, thus getting as many equations as are needed to solve for the unknown parameters.
The procedure is described below:
Let X1, X2, â&#x20AC;Ś, Xn be a random sample of size n from a population. Then the rth sample moment about zero is
m 'r =
r â&#x2C6;&#x2018; Xi
n
, r = 1,2,...
and the corresponding population moment is Âľ'r .
rth
We then match these moments and get as many equations as we need to solve for the unknown parameters.
The following examples illustrate the method:
EXAMPLE-1 Let X be uniformly distributed on the interval (0, θ). Find an estimator of θ by the method of moments.
SOLUTION The probability density function of the given uniform distribution is 1 f ( x) = , 0 ≤ x ≤ θ θ
Since the uniform distribution has only one parameter, (i.e. θ), therefore, in order to find the maximum likelihood estimator of θ by the method of moments, we need to consider only one equation.
The first sample moment about zero is ∑ Xi m'1 = . n And, the first population moment about zero is θ 2 1 1 x θ µ'1 = ∫ x.f ( x ) dx = ∫ x. dx = = θ 2 2 0 0 θ 0 θ
θ
Matching these moments, we obtain: ∑ Xi θ = or θ = 2 X. n 2
Hence, the moment estimator of θ is equal to 2 X i.e. θˆ = 2 X. In other words, the moment estimator of θ is just twice the sample mean.
An Interesting Observation: It should be noted that, for the above uniform distribution, the mean is given by θ µ= . 2 (This is so due to the absolute symmetry of the uniform distribution around the value θ . ) 2
Now, µ = θ
implies that
2
θ = 2µ .
In other words, if we wish to have the exact value of θ, all we need to do is to multiply the population mean µ by 2.
Generally, it is not possible to determine µ, and all we can do is to draw a sample from the probability distribution, and compute the sample mean X. Hence, naturally, the equation θ = 2µ will be replaced by the equation θˆ = 2 x . (As 2 x provides an estimate of θ, hence a ‘hat’ is placed on top of θ.)
It is interesting to note that 2 x is exactly the same quantity as what we obtained as an estimate of θ by the method of moments ! (The result obtained by the method of moments coincides with what we obtain through simple logic.)
EXAMPLE-2 Let X1, X2, …, Xn be a random sample of size n from a normal population with parameters µ and σ 2. Find these parameters by the method of moments.
SOLUTION Here we need two equations as there are two unknown parameters, 2 µ and σ .
The first two sample moments about zero are 1 1 2 m'1 = â&#x2C6;&#x2018; X i = X and m'2 = â&#x2C6;&#x2018; X i . n n
The corresponding two moments of a normal distribution are µ′ 1 = µ and µ′ 2 = σ + 2
µ. 2 2 ( σ = µ′ 2 – µ′ 1 2 = µ′ 2 – µ ) 2
To get the desired estimators by the method of moments, we match them. Thus, we have : 1 1 2 2 2 µ = ∑ Xi and σ + µ = ∑ Xi n n
Solving the above equations simultaneously, we obtain:
1 ˆ = ∑X i =X , and µ n 2 Xi 1 2 ∑ 2 2 2 ˆ = σ −X = ∑(X i −X ) =S . n n 2
as the moment estimators for µand σ .
A shortcoming of this method is that the moment estimators are, in general, inefficient.
Next, we consider the Method of Least Squares:
The method of Least Squares, which is due to Gauss (1777-1855) and Markov (1856-1922), is based on the theory of linear estimation. It is regarded as one of the important methods of point estimation.
THE METHOD OF LEAST SQUARES
An
estimator found by minimizing the sum of squared deviations of the sample values from some function that has been hypothesized as a fit for the data, is called the least squares estimator.
The method of least-squares has already been discussed in connection with regression analysis that was presented in Lecture No. 15. The students will recall that, when fitting a straight line y = a+bx to real data, ‘a’ and ‘b’ were determined by minimizing the sum of squared deviations between the fitted line and the data-points.
The y-intercept and the slope of the fitted line i.e. ‘a’ and ‘b’ are least-square estimates (respectively) of the y-intercept and the slope of the TRUE line that would have been obtained by considering the entire population of datapoints, and not just a sample.
Next, we consider the
Method of Maximum Likelihood:
The method of maximum likelihood is regarded as the MOST important method of estimation, and is the most widely used method. This method was introduced in 1922 by Sir Ronald A. Fisher (1890-1962).
The mathematical technique of finding Maximum Likelihood Estimators is a bit advanced, and involves the concept of the Likelihood Function. In this course, we consider only the method:
overall
logic of this
RATIONALE
OF THE METHOD OF MAXIMUM LIKELIHOOD (ML)
â&#x20AC;&#x153;To consider every possible value that the parameter might have, and for each value, compute the probability that the given sample would have occurred if that were the true value of the parameter. That value of the parameter for which the probability of a given sample is greatest, is chosen as an estimate.â&#x20AC;?
An estimate obtained by this method is called the maximum likelihood estimate (MLE).
It should be noted that the method of maximum likelihood is applicable to both discrete and continuous random variables.
Let us consider a few examples:
EXAMPLES OF MLE’s IN CASE OF DISCRETE DISTRIBUTIONS
Example-1: For the Poisson distribution given by -µ x e µ P(X = x) = , x = 0,1,2, ......, x! the MLE of µ is X (the sample
mean).
Example-2: For the geometric distribution given by P(X = x) = pq
the MLE of p is
x â&#x2C6;&#x2019;1
, x = 1,2,3......,
1 . X
Hence, the MLE of p is equal to the reciprocal of the mean.
Example-3: For the Bernoulli distribution given by x 1â&#x2C6;&#x2019; x P(X = x) = p q , x = 0,1 , the MLE of p is X
(the sample mean).
EXAMPLES OF MLE’s IN CASE OF CONTINUOUS DISTRIBUTIONS
Example-1: For the exponential distribution given by f ( x ) = θe
− θx
the MLE of θ is
, x > 0, θ > 0 , 1 . X
(the reciprocal of the sample mean).
Example-2: For the normal distribution with parameters µ and σ 2, the joint ML estimators of µ and
σ are the sample mean X , 2
and the sample variance S (which is not an unbiased 2 estimator of σ ). 2
As indicated many times earlier, the normal distribution is encountered frequently in practice, and, in this regard, it is both interesting and important to note that, in the case of this frequently encountered distribution, the simplest formulae (i.e. the sample mean and the sample variance) fulfil the criteria of the relatively advanced method of maximum likelihood estimation !
The last example among the five presented above (the one on the normal distribution) points to another important fact --and that is :
The Maximum Likelihood Estimators are consistent and efficient but not necessarily unbiased.
(As we know, S is not an unbiased estimator of 2 Ď&#x192; .) 2
Let us apply this concept to a real-life example:
EXAMPLE It is well-known that human weight is an approximately normally distributed variable. Suppose that we are interested in estimating the mean and the variance of the weights of adult males in one particular province of a country.
A random sample of 15 adult males from this particular population yields the following weights (in pounds): 131.5 135.2 131.6
136.9 129.6 136.7
133.8 134.4 135.8
130.1 130.5 134.5
133.9 134.2 132.7
Find the maximum likelihood 2 estimates for θ 1 = µ and θ 2 = σ .
SOLUTION The above data is that of a random sample of size 15 from N(µ, σ 2). It has been mathematically proved that the joint maximum likelihood estimators of µ and σ 2 areX and S2.
We compute these quantities for this particular sample, and obtain 鵃出 = 133.43, and S = 5.10 . 2
These are the Maximum Likelihood Estimates of the mean and variance of the population of weights in this particular example.
Having discussed the concept of point estimation in some detail, we now begin the discussion of the concept of
interval estimation:
As stated earlier, whenever a single quantity computed from the sample acts as an estimate of a population parameter, we call that quantity a point estimate e.g. the sample mean x is a point estimate of the population mean Âľ.
The limitation of point estimation is that we have no way of ascertaining how close our point estimate is to the true value (the parameter).
For example, we know that x is an unbiased estimator of µ i.e. if we had taken all possible samples of a particular size from the population and calculated the mean ( x ) of each sample, then the mean of the sample means ( µ x ) would have been equal to the population mean (µ), but in an actual survey we will be selecting only one sample from the population and will calculate its mean x . We will have no way of ascertaining how close this particular x is to µ.
Whereas a point estimate is a single value that acts as an estimate of the population parameter,
interval
estimation
is a procedure of estimating the unknown parameter which specifies a range of values within which the parameter is expected to lie.
A confidence interval is an interval computed from the sample observations x1, x2â&#x20AC;Ś.xn, with a statement of how confident we are that the interval does contain population parameter.
the
We develop the concept of interval estimation with the help of the example of the Ministry of Transport test to which all cars, irrespective of age, have to be submitted :
EXAMPLE Let us examine the case of an annual Ministry of Transport test to which all cars, irrespective of age, have to be submitted. The test looks for faulty breaks, steering, lights and suspension, and it is discovered after the first year that approximately the same number of cars have 0, 1, 2, 3, or 4 faults.
You will recall that when we drew all possible samples of size 2 from this uniformly distributed population, the sampling distribution of 鵃出 was triangular:
Sampling Distribution of鵃出 for n = 2 P( x ) 5/25 4/25 3/25 2/25 1/25 0
0.0 0.5
1.0
1.5
2.0
2.5 3.0
3.5 4.0
X
But when we considered what happened to the shape of the sampling distribution with if the sample size is increased, we found that it was somewhat like a normal distribution:
Sampling Distribution of鵃出 for n = 3 P( x ) 20/125 16/125 12/125 8/125
3.00 3.33 3.67 4.00
0
0.00 0.33 0.67 1.00 1.33 1.67 2.00 2.33 2.67
4/125
X
And, when we increased the sample size to 4, the sampling distribution resembled a normal distribution even more closely :
Sampling Distribution of鵃出 for n = 4 P( x ) 100/625 80/625 60/625 40/625
0
0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00
20/625
X
It is clear from the above discussion that as larger samples are taken, the shape of the sampling distribution of 鵃出 undergoes discernible changes. In all three cases the line charts are symmetrical, but as the sample size increases, the overall configuration changed from a triangular distribution to a bellshaped distribution.
In other words, for large samples, we are dealing with a normal sampling distribution of x . In other words:
When sampling from an infinite population such that the sample size n is large,X is normally distributed with mean µ and variance σ 2 σ N i.e. X is µ , n
2
.
n
Hence, the standardized X −µ version of X i.e. Z=
σ n
is normally distributed with mean 0 and variance 1 i.e. Z is N(0, 1).
Now, for the standard normal distribution, we have:
For the standard normal distribution, we have:
0.0250 -1.96
0.4750
0.4750
0
0.0250 1.96
The above is equivalent to P(-1.96 < Z < 1.96) = 0.4750 + 0.4750 = 0.95
Z
0.025 -1.96
0.95
0
0.025 1.96
Z
In other words: X −µ P − 1.96 ≤ ≤ 1.96 = 0.95 σ n
The above can be re-written as: σ σ = 0.95 P − 1.96 ≤ X − µ ≤ 1.96 n n
or σ σ = 0.95 P − X − 1.96 ≤ −µ ≤ − X + 1.96 n n
or σ σ = 0.95 P X + 1.96 ≥ µ ≥ X − 1.96 n n
or
σ σ = 0.95 P X − 1.96 ≤ µ ≤ X + 1.96 n n
The above equation yields the 95% confidence interval for Âľ :
The 95% interval for µ is
confidence
σ σ X − 1.96 . , X + 1.96 n n
In other words, the 95% C.I. for µ is given by
σ X ± 1.96 n
In a real-life situation, the population standard deviation is usually not known and hence it has to be estimated.
It can be mathematically proved that the quantity
( X − X) ∑ s = 2
2
n −1
is an unbiased estimator of σ 2 (the population variance).
(just as the sample mean X is an unbiased estimator of Âľ).
In this situation, the 95% Confidence Interval for µ is given by: s s P X − 1.96 < µ < X + 1.96 = 95% n n
The points
s s X â&#x2C6;&#x2019; 1.96 and X + 1.96 n n are called the
limits interval.
lower and upper
of the 95% confidence
IN TODAY’S LECTURE, YOU LEARNT •Desirable Qualities of a Good Point Estimator: •Efficiency •Methods of Point Estimation: •The Method of Moments •The Method of Least Squares •The Method of Maximum Likelihood •Interval Estimation: •Confidence Interval for µ
IN THE NEXT LECTURE, YOU WILL LEARN
•Confidence Interval for µ (continued) •Confidence Interval for µ 1-µ 2 •Large Sample Confidence Intervals for p and p1-p2 •Determination of Sample Size