Gauss-Markov Theorem

Large N & Leeuwenhoek (70700173)

Yue Hu

Overview

\[Y_i = \beta_0 + \beta_1X + \epsilon.\]

1. BLUE (Best Linear Unbiased Estimator)
2. OLS vs. association
3. OLS in linear algebra

1 BLUE

1.1 How is OLS superior to other simulations

1.2 Best linear simulation

1.3 Conventional ways to find the best

Wasting computing power, slow ending with large data

1.4 Smarter solution based on Statistics

Is the statistical solution (given by last lecture) unbiased and consistent?

Unbiasedness

\[E(\hat\beta_1|X) = \beta_1\]

Proof:

\[\begin{align} E(\hat\beta_1|X) =& E[\frac{\sum(X - \bar X)(Y - \bar Y)}{\sum(X - \bar X)^2}|X]= E[\frac{\sum(X - \bar X)Y}{\sum(X - \bar X)^2}|X],\\ =& \frac{1}{\sum(X - \bar X)^2}E[\sum(X - \bar X)Y|X] = \frac{\sum(X - \bar X)}{\sum(X - \bar X)^2}E(Y|X),\\ =& \frac{\sum(X - \bar X)}{\sum(X - \bar X)^2}(\beta_0 + \beta_1X + \color{red}{\epsilon}) = \frac{\sum(X - \bar X)}{\sum(X - \bar X)^2}(\beta_0 + \beta_1X),\\ =& \frac{1}{\sum(X - \bar X)^2}[\beta_0\sum(X - \bar X) + \beta_1X\sum(X - \bar X)],\\ =& \frac{\beta_1\sum(X - \bar X)X}{\sum(X - \bar X)^2}, \text{given the "transformation gadget"} \sum X_i(X_i - \bar X) = \sum (X_i - \bar X)(X_i - \bar X),\\ =& \frac{\beta_1\sum(X - \bar X)(X - \bar X)}{\sum(X - \bar X)^2} =\beta_1.\blacksquare \end{align}\]

\(\epsilon\) went away by assuming

1.5 Consistency

1. \(var(\hat{\beta_1}|X) = \frac{\sigma^2}{\sum (X_i - \bar X)^2}\).
   • So when N increases, \(\sum (X_i - \bar X)^2\) increases → var(β₁) decreasing.
     • \(\displaystyle{\lim_{n\to\infty}} var(\beta_1) = 0.\)
2. \(var(\hat{\beta_1}|X) = \sigma^2 \frac{\sum X_i^2}{n \sum (X_i - \bar X)^2}\), the denominator increases faster than the nominator. (The proof is omitted.)

The variance of ( _0 ) is:

[ (_0 | X) = ^2 ]

where:

1. The numerator is ( X_i^2 ),
2. The denominator is ( n (X_i - X)^2 ).

1.5.1 Step 1: Impact of Adding ( X_{n+1} ) on ( X_i^2 ) (Numerator)

When adding ( X_{n+1} ), the new numerator becomes:

[ {i=1}^{n+1} X_i^2 = {i=1}^n X_i^2 + X_{n+1}^2 ]

This increases by ( X_{n+1}^2 ), so the numerator grows linearly with each added term ( X_i^2 ).

1.5.2 Step 2: Impact of Adding ( X_{n+1} ) on ( (X_i - X)^2 ) (Denominator)

The denominator involves ( (X_i - X)^2 ), which represents the total variability of ( X_i ) around its mean. When we add ( X_{n+1} ), both ( {X} ) and ( (X_i - X)^2 ) change:

1. The new mean ( X_{} ) with ( n+1 ) observations is:

   [ X_{} = ]

2. The new sum of squares around the mean becomes:

   [ {i=1}^{n+1} (X_i - X{})^2 = {i=1}^n (X_i - X{})^2 + (X_{n+1} - X_{})^2 ]

This increases by more than a linear term because the spread around the new mean incorporates variability from all ( X_i ) observations. Therefore, ( (X_i - X)^2 ) tends to grow faster as it captures both the variability of previous points and the new data point’s deviation from the updated mean.

1.5.3 Step 3: Comparing Growth Rates

Thus, while the numerator ( X_i^2 ) increases by ( X_{n+1}^2 ) (a linear increase), the denominator ( n (X_i - X)^2 ) increases more than linearly due to the compounded effect on the sum of squares. Hence, the denominator grows faster than the numerator when ( X_{n+1} ) is added.

1.6 “Ten Commandments”

1. Nonstochastic X (“X is fixed”), and X has positive noninfinite variance (σ_X > 0);
2. Correct specification;
3. Linearity in the parameter;
4. Full rank/identification
   • N > K; K = 2 for a simple OLS);
5. Mean zero errors
   • E(ε_i|X_i) = 0;
6. Exogeneity: No covariance between X_i and ε_i
   • E(X_iε_i) = cov(x_i, ε_i) =0;

7. No autocorrelation/serial correlation: E(ε_i, ε_J|X_i, X_j) = cov(ε_i, ε_J|X_i, X_j) = 0, ∀ i, j);
8. Normality: \(U|X \sim N(0, \sigma^2I).\)
9. Homoskedasticity/Spherical disturbances: constant variance of ε_i, var(ε_i|X) = σ²;
10. No perfect collinearity: there are more than one X, ∄ X_i s.t., X_i = a + b∑_{j = 1}b_jX_j

→ Classic Linear Regression Model (CLRM, esp. 4~9)

• Spherical disturbances: the assumption of homoscedasticity,
  • Disturbance: the variance of error term
• s.t. such that
• homoskedasticity: residual randomly distributed in the same variance across the values of x

1.7 Gauss-Markov Theorem

In a linear regression model in which the errors are uncorrelated, have equal variances, and expectation value of zero, the best linear unbiased estimator of the coefficients is given by the ordinary least squares (OLS) estimator, provided it exists.

https://www.youtube.com/watch?v=nXeTsWGPT0w

In Ordinary Least Squares (OLS), the term “ordinary” refers to the standard approach of minimizing the sum of squared residuals (errors) in a linear regression model.

To break it down:

1. Ordinary: This implies a straightforward, basic approach without special weighting or transformations. The method treats all observations equally, giving no special preference to any particular data points.

2. Least Squares: This part means that OLS estimates the parameters by minimizing the sum of the squared differences between the observed values and the values predicted by the linear model. Specifically, it finds the line (or hyperplane in multiple dimensions) that results in the smallest total squared error across all data points.

In contrast, Generalized Least Squares (GLS) or Weighted Least Squares (WLS) introduce adjustments, such as weighting observations differently, often to account for issues like heteroskedasticity or autocorrelation. In OLS, however, we assume constant variance (homoscedasticity) and independence across observations.

1.8 Distribution of OLS Paramenters

\[\begin{align} u_i|X \sim& \text{i.i.d.} N(0, \sigma^2)\\ \hat\beta_1|X\sim& N(\beta_1, \frac{\hat\sigma^2}{\sum (X_i - \bar X)^2})\\ \hat\beta_0|X\sim& N(\beta_0, \frac{\hat\sigma^2\sum X_i^2}{n\sum (X_i - \bar X)^2}) \end{align}\]

\[\begin{align} \frac{\hat\beta_1 - \beta_1}{\sqrt{\frac{\hat\sigma^2}{\sum (X_i - \bar X)^2}}}\sim& N(0, 1^2)\\ \frac{\hat\beta_0 - \beta_0}{\sqrt{\frac{\hat\sigma^2\sum X_i^2}{n\sum (X_i - \bar X)^2}}}\sim& N(0, 1^2)\\ \frac{\hat\sigma^2}{\frac{\sigma^2}{n - 2}}\sim& \chi^2_{n - 2} \end{align}\]

2 OLS vs. Association

2.1 \(\rho\), Χ², and F

χ² is adding-up of n square normals representing variances; F is the ratio of two χ²s.

In other words, they are consistent with t-test and OLS.

2.2 ANOVA vis-á-vis OLS

Source	Sum Square	d.f.	Mean Square
Treat	\(SST = \sum n_i (\bar X_i - \bar{\bar{X}})^2\)	K - 1	MST = SST/(K - 1)
Error	\(SSE = \sum \sum (X_{ik} - \bar{X_i})^2\)	N - K	MSE = SSE/(N - K)
Total	\(SS = SST + SSE\)	N - 1	\(F_{\alpha, K-1, N-1} = MST/MSE\)

	\(\sum(Y_i - \bar Y)^2\)	\(= \hat\beta_1^2(X_i - \bar X)^2\)	\(+ \sum\hat \epsilon_i^2\)
	SST	SSE	SSR
d.f.	n - 1	1	n - 2
MSS	\(\frac{\sum(Y_i - \bar Y)^2}{n - 1}\)	\(\frac{\hat\beta_1^2\sum(X_i - \bar X)}{1}\)	\(\frac{\sum\hat u_i^2}{n - 2}\)

n-1: Used out 1 to calculate \(\bar Y\); 1: The only thing varies is \(\hat\beta_1\); n-2: Used out to calculate \(\hat\beta_0, \hat\beta_1\).

Meaningless content:

\[\begin{align} E[\hat\beta_1^2\sum(X_i - \bar X)|X] =& E\{[\frac{\sum(X_i - \bar X)(Y_i - \bar Y)}{\sum(X_i - \bar X)^2}]\sum(X_i - \bar X)^2\}\\ =& \frac{\sum(X_i - \bar X)^2}{[\sum(X_i - \bar X)^2]^2}E[(\sum(X_i - \bar X)(Y_i - \bar Y))^2|X]\\ =& \frac{E\{[\sum(X_i - \bar X)Y_i]^2|X\}}{\sum(X_i - \bar X)^2}\\ =& \frac{E\{[\sum(X_i - \bar X)(\beta_0 + \beta_1X_i + u_i)]^2|X\}}{\sum(X_i - \bar X)^2}\\ =& \frac{E\{[\sum(X_i - \bar X)(\beta_1X_i + u_i)]^2|X\}}{\sum(X_i - \bar X)^2}\\ =& \frac{E\{[\beta_1\sum(X_i - \bar X)X_i + \sum(X_i - \bar X)u_i)]^2|X\}}{\sum(X_i - \bar X)^2}\\ =& \frac{E\{[\beta_1\sum(X_i - \bar X)(X_i - \bar X) + \sum(X_i - \bar X)u_i)]^2|X\}}{\sum(X_i - \bar X)^2}\\ =& \frac{E\{[\beta_1\sum(X_i - \bar X)^2 + \sum(X_i - \bar X)u_i)]^2|X\}}{\sum(X_i - \bar X)^2}\\ =& \frac{E\{[\beta_1\sum(X_i - \bar X)^2 + \sum(X_i - \bar X)u_i)]^2|X\}}{\sum(X_i - \bar X)^2} \end{align}\]

\[\begin{align} =& \frac{1}{\sum(X_i - \bar X)^2}\{E\{[\beta_1\sum(X_i - \bar X)^2]^2|X\} + E\{[\sum(X_i - \bar X)u_i)]^2|X\}\}\\ &+ E\{2\beta_1\sum(X_i - \bar X)^2\sum(X_i - \bar X)u_i)|X\}\\ =& \frac{1}{\sum(X_i - \bar X)^2}\{[\beta_1\sum(X_i - \bar X)^2]^2 + E\{[\sum(X_i - \bar X)u_i)]^2|X\}\\ &+ 2\beta_1\sum(X_i - \bar X)^2\sum(X_i - \bar X)E(u_i|X)\}\\ =& \frac{[\beta_1\sum(X_i - \bar X)^2]^2 + \sum(X_i - \bar X)^2\sigma^2 + 0}{\sum(X_i - \bar X)^2}\\ =& \beta_1^2\sum(X_i - \bar X)^2 + \sigma^2 \end{align}\]

nominator transformation

Ignore \(\beta_0\) for now

Then \(F_{1, n - 2}\sim\frac{\hat\beta_1^2(X_i - \bar X)^2}{\sigma^2} = \frac{\beta_1^2\sum(X_i - \bar X)^2 + \sigma^2}{\sigma^2} = \frac{\beta_1^2\sum(X_i - \bar X)^2}{\sigma^2} + 1\)

\([\beta_1\sum(X_i - \bar X)]^2\), constant

\(\frac{MSE}{MSR} = \frac{\hat\beta_1^2(X_i - \bar X)^2\sim\chi^2}{\sigma^2\sim\chi^2}\sim F_{1, n - 2}\)

\[\begin{align} F_{1, n - 2}\sim& \frac{\beta_1^2\sum(X_i - \bar X)^2}{\sigma^2}\\ =& \frac{\beta_1^2}{\frac{\sigma^2}{\sum(X_i - \bar X)^2}} = (\frac{\bar X - \mu}{\hat\sigma_X})^2. \end{align}\]

F and t

We know \(\frac{\bar X - \mu}{\hat\sigma_X}\sim t\). Therefore, F provides identical information as t.

• ANOVA and OLS should yield identical inferences.

2.3 Take-home point

Appendix: OLS in Linear Algebra

2.4 Elementary to Linear Algebra

\[\begin{align} Y_i =& \beta_0 + \beta_iX_i + \epsilon_i\\ \boldsymbol{Y} =& \boldsymbol{X\beta} + \boldsymbol{\epsilon}\\ \left(\begin{array}{c} Y_1\\ Y_2\\ \vdots\\ Y_n\end{array}\right)=& \left(\begin{array}{cc} 1 & X_1\\ 1 & X_2\\ \vdots & \vdots\\ 1 & X_n\end{array}\right) \left(\begin{array}{c} \beta_1\\ \beta_2\\ \vdots\\ \beta_n\end{array}\right) + \left(\begin{array}{cc} \epsilon_1\\ \epsilon_2\\ \vdots\\ \epsilon_n\end{array}\right) \end{align}\]

• Y: Response vector;
• X: Design matrix;
• β: Parameter vector;
• ε: Error vector;

the X is a diagonal matrix, writing in such a format showing each line has one single element

2.5 Estimator & Covariance Matrix of Error

Goal: Finding the β minimizing the squared residuals

\[\sum\epsilon^2 = \boldsymbol{\epsilon'\epsilon} = (\boldsymbol{Y} - \boldsymbol{X}\beta)'(\boldsymbol{Y} - \boldsymbol{X}\beta)\]

Then, seek for the value of β that lets the derivative of the above equation respected of β to be 0.

\[\begin{align} \hat\beta =& (\boldsymbol{X'X})^{-1}\boldsymbol{X'Y}.\\ var(\beta) =& \sigma^2(\boldsymbol{X'X})^{-1}, \text{where}\ \sigma^2 = \frac{\boldsymbol{\epsilon'\epsilon}}{n - k}. \end{align}\]

’ means transpose (exchanging the rows and columns)

According to the homoscedasiticity,

\[\sigma^2\{\epsilon\}_{n\times n} = \sigma^2\boldsymbol{I}_{n\times n} = \sigma^2\{\boldsymbol{Y}\}_{n\times n}.\]

In other words, \(\epsilon\sim N(\boldsymbol{0}, \sigma^2\boldsymbol{I})\).

2.6 Differential Rules for Linear Algebra

How to conduct derivatives for matrix:

\[\begin{align} \frac{\boldsymbol{a'b}}{\boldsymbol{b}} =& \frac{\boldsymbol{b'a}}{\boldsymbol{b}} = \boldsymbol{a}\\ \frac{\boldsymbol{b'Ab}}{\boldsymbol{b}} =& 2\boldsymbol{Ab} = 2\boldsymbol{b'A} \end{align}\]

A is an arbitrary symmetric matrix.

According to the above rules,

\[\begin{align} \frac{d2\boldsymbol{\beta'X'Y}}{\boldsymbol{\beta}}=& \frac{d2\boldsymbol{\beta'(X'Y)}}{\boldsymbol{\beta}} = 2\boldsymbol{X'Y}\\ \frac{d2\boldsymbol{\beta'X'X\beta}}{\boldsymbol{\beta}} =& \frac{d2\boldsymbol{\beta'(X'X\beta)}}{\boldsymbol{\beta}} = 2\boldsymbol{X'X\beta} \end{align}\]

2.7 \(\hat\beta = \beta\): A Proof

\[\begin{align} \boldsymbol{\epsilon\epsilon'}= \frac{d(\boldsymbol{Y} - \boldsymbol{X}\beta)'(\boldsymbol{Y} - \boldsymbol{X}\hat\beta)}{d\hat\beta} =& 0,\\ -2\boldsymbol{X'(\boldsymbol{Y} - \boldsymbol{X}\hat\beta)} =& 0,\\ \text{Given } \boldsymbol{X'Y} = \boldsymbol{X'X}\hat\beta, \hat\beta =& (\boldsymbol{X'X})^{-1}\boldsymbol{X'Y}.\\ \text{Within this}, \boldsymbol{X'X} =& \left(\begin{array}{cc} n & \sum X_i\\ \sum X_i & \sum X_i^2 \end{array}\right) \Rightarrow (\boldsymbol{X'X})^{-1} = \frac{\left(\begin{array}{cc} \sum X_i^2 & -\sum X_i\\ -\sum X_i & n \end{array}\right)}{nS_X},\\ \boldsymbol{X'Y} =& \left(\begin{array}{c} \sum Y_i\\ -\sum X_iY_i \end{array}\right).\\ \end{align}\]

\[\begin{align} \text{Then, } E(\hat\beta) =& [\boldsymbol{(X'X)^{-1}X'}](\boldsymbol{X}\beta + \epsilon),\\ =& [\boldsymbol{(X'X)^{-1}X'X}\beta] + [\boldsymbol{(X'X)^{-1}X'\epsilon}],\\ =& \beta.\\ var(\beta) =& \sigma^2(X'X)^{-1}, \text{where}\ \sigma^2 = \frac{\epsilon'\epsilon}{n - k}.\blacksquare \end{align}\]

X^-1: Inverse matrix; X’: Transposition.

2.8 C.f. Elementary Algebra

\[\begin{align} E(\hat\beta_1|X) =& E[\frac{\sum(X - \bar X)(Y - \bar Y)}{\sum(X - \bar X)^2}|X]= E[\frac{\sum(X - \bar X)Y}{\sum(X - \bar X)^2}|X],\\ =& \frac{1}{\sum(X - \bar X)^2}E[\sum(X - \bar X)Y|X] = \frac{\sum(X - \bar X)}{\sum(X - \bar X)^2}E(Y|X),\\ =& \frac{\sum(X - \bar X)}{\sum(X - \bar X)^2}(\beta_0 + \beta_1X + \epsilon) = \frac{\sum(X - \bar X)}{\sum(X - \bar X)^2}(\beta_0 + \beta_1X),\\ =& \frac{1}{\sum(X - \bar X)^2}[\beta_0\sum(X - \bar X) + \beta_1X\sum(X - \bar X)],\\ =& \frac{\beta_1\sum(X - \bar X)X}{\sum(X - \bar X)^2}, \text{given} \sum(X - \bar X) = \sum X - \sum\bar X = 0,\\ =& \frac{\beta_1\sum(X - \bar X)(X - \bar X)}{\sum(X - \bar X)^2} =\beta_1.\blacksquare \end{align}\]

2.9 About the Error

2.9.1 \(\boldsymbol{X'\epsilon} = 0\)

\[\begin{align} \boldsymbol{X'Y} =& \boldsymbol{X'X}\hat\beta,\\ \boldsymbol{X'(X\hat\beta + \epsilon)} =& \boldsymbol{X'X}\hat\beta,\\ \boldsymbol{X'\epsilon} =& 0.\blacksquare \end{align}\]

2.9.2 Hat matrix

For the predicted Y, \[\hat{\boldsymbol{Y}} = \boldsymbol{X}\beta = \boldsymbol{X(X'X)^{-1}X'Y} = \boldsymbol{[X(X'X)^{-1}X']Y},\]

\(H = [X(X'X)^{-1}X']\) is called the hat matrix.

Then, \[\epsilon = \boldsymbol{Y} - \hat{\boldsymbol{Y}} = \boldsymbol{Y} - \boldsymbol{HY} = \boldsymbol{(I - H)Y}.\]

Two properties of H:

1. Symmetric: \(\boldsymbol{H = H'};\boldsymbol{(I - H) = (I - H)'}.\)
2. Idempotent: \(\boldsymbol{H^2 = H; (I - H)(I - H) = (I - H)}.\)

Idempotent: 幂等

Stretch

https://www.bilibili.com/video/BV1E54y1r76o/?buvid=Y045340D2FD9FA3A46419EEFE4578279ECBD&from_spmid=main.space-search.0.0&is_story_h5=false&mid=7RnjBONLRMus4FQZFBWD2g%3D%3D&p=2&plat_id=114&share_from=ugc&share_medium=iphone&share_plat=ios&share_session_id=79FE08C9-F2B1-4C18-A16C-B31179817B42&share_source=WEIXIN&share_tag=s_i&timestamp=1728732278&unique_k=skSWw7j&up_id=390316092&vd_source=f38aeefd0d38cecba9017eeee43e71c8

Meditation

松茸的世界：5分钟正念冥想-自信之心

https://www.xiaoyuzhoufm.com/episode/65b752ed0bef6c207457f51b?s=eyJ1IjogIjYwNDA1OGJiZTBmNWU3MjNiYmY3MjZiZSJ9