Properties of estimators

Prof. Maria Tackett

Oct 01, 2024

Announcements

  • Project Proposal due Thursday, October 3 at 11:59pm

  • Lab 03 due Thursday, October 3 at 11:59pm

  • HW 02 due Thursday, October 3 at 11:59pm (released after class)

  • Exam 01: Tuesday, October 8 (in class + take-home)

    • Lecture recordings available until the start of the in-class exam (Link on side bar of webpage)

    • Exam review on Thursday

    • Monday’s lab: Exam office hours

    • No office hours while take-home exam is out

Topics

  • Properties of the least squares estimator

Note

This is not a mathematical statistics class. There are semester-long courses that will go into these topics in much more detail; we will barely scratch the surface in this course.

Our goals are to understand

  • Estimators have properties

  • A few properties of the least squares estimator and why they are useful

Properties of β^

Motivation

  • We have discussed how to use least squares to find an estimator of β^

  • How do we know whether our least-squares estimator is a “good” estimator?

  • When we consider what makes an estimator “good”, we’ll look at three criteria:

    • Bias
    • Variance
    • Mean squared error

  • We’ll take a look at these and motivate why we might prefer using least squares to compute β^ versus other methods

Bias and variance

Suppose you are throwing darts at a target

Image source: Analytics Vidhya

  • Ideal scenario: Darts are clustered around the target (unbiased and low variance)

  • Worst case scenario: Darts are widely spread out and systematically far from the target (high bias and high variance)

  • Acceptable scenario: There’s some trade-off between the bias and variance.

Properties of β^

Finite sample ( n ) properties

  • Unbiased estimator

  • Best Linear Unbiased Estimator (BLUE)


Infinite sample ( n→∞ ) properties

  • Consistent estimator

  • Efficient estimator

Finite sample properties

Unbiased estimator

The bias of an estimator is the difference between the estimator’s expected value and the true value of the parameter

Let θ^ be an estimator of the parameter θ. Then

Bias(θ^)=E(θ^)−θ

An estimator is unbiased if the bias is 0 and thus E(θ^)=θ

Unbiased estimator

E(β^)=E[(XTX)−1XTy]=E[(XTX)−1XT(Xβ+ϵ)]=E[(XTX)−1XTXβ]+E[(XTX)−1XTϵ]=β+(XTX)−1XTE(ϵ)=β

The least-squares estimator β^ is an unbiased estimator of β

Variance of β^

Var(β^)=Var((XTX)−1XTy)=[(XTX)−1XT]Var(y)[(XTX)−1XT]T=[(XTX)−1XT]σϵ2I[X(XTX)−1]=σϵ2[(XTX)−1XTX(XTX)−1]=σϵ2(XTX)−1

“Linear” regression model

What does it mean for a model to be a “linear” regression model?

  • Linear regression models are linear in the parameters, i.e. given an observation yi

    yi=β0+β1f1(xi1)+⋯+βpfp(xip)+ϵi

  • The functions f1,…,fp can be non-linear as long as β0,β1,…,βp are linear in Y




Gauss-Markov Theorem

The least-squares estimator of β in the model y=Xβ+ϵ is given by β^. Given the errors have mean 0 and variance σϵ2I , then β^ is BLUE (best linear unbiased estimator).

“Best” means β^ has the smallest variance among all linear unbiased estimators for β .

Gauss-Markov Theorem Proof

Suppose β~ is another linear unbiased estimator of β that can be expressed as β~=Cy , such that y^=Xβ~=XCy


Let C=(XTX)−1XT+B for a non-zero matrix B.


What is the dimension of B?

Gauss-Markov Theorem Proof

β~=Cy=((XTX)−1XT+B)y

We need to show

  • β~ is unbiased

  • Var(β~)>Var(β^)

Gauss-Markov Theorem Proof

E(β~)=E[((XTX)−1XT+B)y]=E[((XTX)−1XT+B)(Xβ+ϵ)]=E[((XTX)−1XT+B)(Xβ)]=((XTX)−1XT+B)(Xβ)=(I+BX)β

  • What assumption(s) of the Gauss-Markov Theorem did we use?

  • What must be true for β~ to be unbiased?

Gauss-Markov Theorem Proof

  • BX must be the 0 matrix (dimension = (p+1)×(p+1)) in order for β~ to be unbiased

  • Now we need to find Var(β~) and see how it compares to Var(β^)

Gauss-Markov Theorem Proof

Var(β~)=Var[((XTX)−1XT+B)y]=((XTX)−1XT+B)Var(y)((XTX)−1XT+B)T=σϵ2[(XTX)−1XTX(XTX)−1+(XTX)−1XTBT+BX(XTX)−1+BBT]=σϵ2(XTX)−1+σϵ2BBT

What assumption(s) of the Gauss-Markov Theorem did we use?

Gauss-Markov Theorem Proof

We have

Var(β~)=σϵ2(XTX)−1+σϵ2BBT

We know that σϵ2BBT≥0.


When is σϵ2BBT=0?

Therefore, we have shown that Var(β~)>Var(β^) and have completed the proof.




Gauss-Markov Theorem

The least-squares estimator of β in the model y=Xβ+ϵ is given by β^. Given the errors have mean 0 and variance σϵ2I , then β^ is BLUE (best linear unbiased estimator).

“Best” means β^ has the smallest variance among all linear unbiased estimators for β .

Properties of β^

Finite sample ( n ) properties

  • Unbiased estimator ✅

  • Best Linear Unbiased Estimator (BLUE) ✅


Infinite sample ( n→∞ ) properties

  • Consistent estimator

  • Efficient estimator

Infinite sample properties

Mean squared error

The mean squared error (MSE) is the squared difference between the estimator and parameter.

Let θ^ be an estimator of the parameter θ. Then

MSE(θ^)=E[(θ^−θ)2]=E(θ^2−2θ^θ+θ2)=E(θ^2)−2θE(θ^)+θ2=E(θ^2)−E(θ^)2⏟Var(θ^)+E(θ^)2−2θE(θ^)+θ2⏟Bias(θ)2

Mean squared error

MSE(θ^)=Var(θ^)+Bias(θ^)2


The least-squares estimator β^ is unbiased, so MSE(β^)=Var(β^)

Consistency

An estimator θ^ is a consistent estimator of a parameter θ if it converges in probability to θ. Given a sequence of estimators θ^1,θ^2,..., then for every ϵ>0,

limn→∞P(|θ^n−θ|≥ϵ)=0

This means that as the sample size goes to ∞ (and thus the sample information gets better and better), the estimator will be arbitrarily close to the parameter with high probability.


Why is this a useful property of an estimator?

Consistency



Important

Theorem

An estimator θ^ is a consistent estimator of the parameter θ if the sequence of estimators θ^1,θ^2,… satisfies

  • limn→∞Var(θ^)=0

  • limn→∞Bias(θ^)=0

Consistency of β^

Bias(β^)=0, so limn→∞Bias(β^)=0


Now we need to show that limn→∞Var(β^)=0

  • What is Var(β^)?

  • Does Var(β^)→0 as n→∞?

Efficiency

  • The efficiency of an estimator is concerned with the asymptotic variance of an estimator.

  • The estimator with the smallest variance is considered the most efficient.

  • By the Gauss-Markov Theorem, we have shown that the least-squares estimator is the most efficient among linear unbiased estimators.

Recap

Finite sample ( n ) properties

  • Unbiased estimator ✅

  • Best Linear Unbiased Estimator (BLUE) ✅


Infinite sample ( n→∞ ) properties

  • Consistent estimator ✅

  • Efficient estimator ✅

🔗 STA 221 - Fall 2024

1 / 29
Properties of estimators Prof. Maria Tackett Oct 01, 2024

  1. Slides

  2. Tools

  3. Close
  • Properties of estimators
  • Announcements
  • Topics
  • Properties of β^
  • Motivation
  • Bias and variance
  • Properties of β^
  • Finite sample properties
  • Unbiased estimator
  • Unbiased estimator
  • Variance of β^
  • “Linear” regression model
  • Gauss-Markov Theorem...
  • Gauss-Markov Theorem Proof
  • Gauss-Markov Theorem Proof
  • Gauss-Markov Theorem Proof
  • Gauss-Markov Theorem Proof
  • Gauss-Markov Theorem Proof
  • Gauss-Markov Theorem Proof
  • Gauss-Markov Theorem...
  • Properties of β^
  • Infinite sample properties
  • Mean squared error
  • Mean squared error
  • Consistency
  • Consistency
  • Consistency of β^
  • Efficiency
  • Recap
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • r Scroll View Mode
  • b Toggle Chalkboard
  • c Toggle Notes Canvas
  • d Download Drawings
  • ? Keyboard Help