SLR: Matrix representation

Prof. Maria Tackett

Sep 05, 2024

Topics

  • Matrix representation for simple linear regression
    • Model form
    • Least square estimate
    • Predicted (fitted) values
    • Residuals
  • Matrix representation in R

Matrix representation of simple linear regression

SLR: Statistical model (population)

When we have a quantitative response, Y, and a single quantitative predictor, X, we can use a simple linear regression model to describe the relationship between Y and X. Y=β0+β1X+ϵ,ϵ∼N(0,σϵ2)


  • β1: Population (true) slope of the relationship between X and Y
  • β0: Population (true) intercept of the relationship between X and Y
  • ϵ: Error

SLR in matrix form

Suppose we have n observations.

[y1⋮yn]⏟y=[1x1⋮⋮1xn]⏟X[β0β1]⏟β+[ϵ1⋮ϵn]⏟ϵ


What are the dimensions of y, X, β, and ϵ?

Sum of squared residuals

We use the sum of squared residuals (also called “sum of squared error”) to find the least squares line:

SSR=∑i=1nei2=eTe=(y−y^)T(y−y^)


  • What is the dimension of SSR?

  • What is y^ in terms of y, X, and/or β ?

Minimize sum of squared residuals

We want to find values of β=[β0β1] that minimize the sum of squared residuals eTe=(y−Xβ)T(y−Xβ)

Minimize sum of squared residuals

We want to find values of β=[β0β1] that minimize the sum of squared residuals eTe=(y−Xβ)T(y−Xβ)=(yT−βTXT)(y−Xβ)

Minimize sum of squared residuals

We want to find values of β=[β0β1] that minimize the sum of squared residuals eTe=(y−Xβ)T(y−Xβ)=(yT−βTXT)(y−Xβ)=yTy−yTXβ−βTXTy+βTXTXβ

Minimize sum of squared residuals

We want to find values of β=[β0β1] that minimize the sum of squared residuals eTe=(y−Xβ)T(y−Xβ)=(yT−βTXT)(y−Xβ)=yTy−yTXβ−βTXTy+βTXTXβ=yTy−2βTXTy+βTXTXβ

Least squares estimators

SSR=eTe=yTy−2βTXTy+βTXTXβ


The least squares estimators must satisfy

∇βSSR=−2XTy+2XTXβ=0


β^=(XTX)−1XTy

Did we find a minimum?

∇β2SSR∝2XTX=0


  • X is full rank ⇒ XTX is positive definite

  • Therefore we have found the minimizing point

Matrix representation in R

Obtain y vector

Let’s go back to the Duke Forest data. We want to use the matrix representation to fit a model of the form:

price=β0+β1 area+ϵ,ϵ∼N(0,σϵ2)

Get y, the vector of responses

y <- duke_forest$price


Let’s look at the first 10 observations of y

y[1:10]
 [1] 1520000 1030000  420000  680000  428500  456000 1270000  557450  697500
[10]  650000

Obtain X matrix

Use the model.matrix() function to get X

X <- model.matrix(price ~ area, data = duke_forest)


Let’s look at the first 10 rows of X

X[1:10,]
   (Intercept) area
1            1 6040
2            1 4475
3            1 1745
4            1 2091
5            1 1772
6            1 1950
7            1 3909
8            1 2841
9            1 3924
10           1 2173

Calculate β^

Matrix functions in R. Let A and B be matrices

  • t(A): transpose A
  • solve(A): inverse of A
  • A %*% B: multiply A and B

Now let’s calculate β^

beta_hat <- solve(t(X)%*%X)%*%t(X)%*%y
beta_hat
                   [,1]
(Intercept) 116652.3251
area           159.4833

Compare to result from lm

duke_forest_model <- lm(price ~ area, data = duke_forest)
tidy(duke_forest_model) |> kable(digits = 3)
term estimate std.error statistic p.value
(Intercept) 116652.325 53302.463 2.188 0.031
area 159.483 18.171 8.777 0.000 


beta_hat 
                   [,1]
(Intercept) 116652.3251
area           159.4833

Predicted values and residuals

Predicted (fitted) values

Now that we have β^, let’s predict values of y using the model

y^=Xβ^=X(XTX)−1XT⏟Hy=Hy

Hat matrix: H=X(XTX)−1XT

  • H is an n×n matrix
  • Maps vector of observed values y to a vector of fitted values y^

Residuals

Recall that the residuals are the difference between the observed and predicted values

e=y−y^=y−Xβ^=y−Hy=(I−H)y

e=(I−H)y

Recap

  • Introduced matrix representation for simple linear regression
    • Model from
    • Least square estimate
    • Predicted (fitted) values
    • Residuals
  • Used R for matrix calculations

Next class

  • Multiple linear regression

  • See Sep 10 prepare

🔗 STA 221 - Fall 2024

1 / 22
SLR: Matrix representation Prof. Maria Tackett Sep 05, 2024

  1. Slides

  2. Tools

  3. Close
  • SLR: Matrix representation
  • Topics
  • Matrix representation of simple linear regression
  • SLR: Statistical model (population)
  • SLR in matrix form
  • Sum of squared residuals
  • Minimize sum of squared residuals
  • Minimize sum of squared residuals
  • Minimize sum of squared residuals
  • Minimize sum of squared residuals
  • Least squares estimators
  • Did we find a minimum?
  • Matrix representation in R
  • Obtain y vector
  • Obtain X matrix
  • Calculate β^
  • Compare to result from lm
  • Predicted values and residuals
  • Predicted (fitted) values
  • Residuals
  • Recap
  • Next class
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • r Scroll View Mode
  • b Toggle Chalkboard
  • c Toggle Notes Canvas
  • d Download Drawings
  • ? Keyboard Help