Page 86 - 4660

Estimation

Consistency. An estimator ˆa is consistent if its value tends to the true value a in the large-sample
limit, i.e.
lim ˆa = a.
N→∞
Consistency is usually a minimum requirement for a useful estimator. An equivalent statement of
consistency is that in the limit of large N the sampling distribution P(ˆa|a) of the estimator must
satisfy
lim P(ˆa|a) → δ(ˆa − a).
N→∞

Bias. The expectation value of an estimator ˆa is given by

∫ ∫
N
E(ˆa) = ˆ aP(ˆa|a)dˆa = ˆ a(x)P(x|a)d x, (2.1)

where the second integral extends over all possible values that can be taken by the sample
elements x 1 , x 2 , . . . , x N . This expression gives the expected mean value of ˆa from an infinite
number of samples, each of size N. The bias of an estimator ˆa is then defined as

b(a) = E(ˆa) − a. (2.2)

We note that the bias b does not depend on the measured sample values x 1 , x 2 , . . . , x N . In
general, though, it will depend on the sample size N, the functional form of the estimator ˆa and,
as indicated, on the true properties a of the population, including the true value of a itself. If
b = 0 then ˆa is called an unbiased estimator of a.

Example 2.1. An estimator ˆa is biased in such a way that E(ˆa) = a + b(a), where
the bias b(a) is given by (b 1 − 1)a + b 2 and b 1 and b 2 are known constants. Construct
an unbiased estimator of a. ,

Solution. Let us first write E(ˆa) is the clearer form

E(ˆa) = a + (b 1 − 1)a + b 2 = b 1 a + b 2 .

The task of constructing an unbiased estimator is now trivial, and an appropriate choice is
′
ˆ a = (ˆa − b 2 )/b1, which (as required) has the expectation value

E(ˆa) − b 2
E(ˆa ) = = a.
′
b 1

Efficiency. The variance of an estimator is given by

∫ ∫
2
N
2
Var(ˆa) = (ˆa − E(ˆa)) P(ˆa|a)dˆa = (ˆa(x) − E(ˆa)) P(x|a)d x (2.3)
and describes the spread of values ˆa about E(ˆa) that would result from a large number of samples,
each of size N. An estimator with a smaller variance is said to be more efficient than one with a
larger variance. As we show in the next section, for any given quantity a of the population there
exists atheoretical lower limit on the variance of any estimator ˆa. This result is known as Fisher’s
inequality (or the Cramér-Rao inequality) and reads

( ) 2 [ 2 ]
∂b ∂ ln P
Var(ˆa) ≥ 1 + /E − , (2.4)
∂a ∂a 2

86