Simple OLS and Properties of Estimators

Large N & Leeuwenhoek (70700173)

Yue Hu

Overview

1. Road to OLS
2. OLS Components

1 Road to OLS

1.1 Where are We

Goal: Understanding data patterns in terms of random variable

Path through

1. Probability theory
   • PDF/CDF
   • Confidence intervals/p-values
2. Description
   • Single: Moments
   • Relation: Association
3. Hypothesis testing
   • Hypothesis setup
   • Statistical inference

Path forward

Linear (OLS)

• Simple linear regression
• Multiple regression

↓

Generalized linear

• MLE(Binary/Ordered/Nominal)

↓

Nonlinear

• Moderation (& mediation)

1.2 Review: Expectation

\[\begin{align} E(X)_{disc.} =& \sum^n_{i = 1} X_if(X_i).\\ E(X)_{cont.} =& \int^{+\infty}_{-\infty}X_if(X_i)dX,\\ =& \mu.\\ \sigma^2 =& E[X - E(X)]^2, \\ =& E[(X - \mu)^2],\\ =& E(X^2 - 2X\mu + \mu^2),\\ =& E(X^2) - 2\mu E(X) + \mu^2,\\ =& E(X^2) - \mu^2. \end{align}\]

\[\begin{align} cov(X, Y) =& \sum(X - \mu_X)(Y - \mu_Y)p(X, Y),\\ =& E[X - E(X)][Y - E(Y)],\\ =& E[XY - X\cdot E(Y) -\\ &Y\cdot E(X) + E(X)E(Y)],\\ =& E(XY) - E(Y)E(X) -\\ &E(X)E(Y) + E(X)E(Y),\\ =& E(XY) - E(X)E(Y). \end{align}\]

1.3 Review (Continued): Variance

\[\begin{align} var(aX + b) =& E[(aX + b) - E(aX + b)]^2 = E[aX - aE(X)]^2, \\ =& a^2E[X - E(X)]^2 = a^2var(X). \end{align}\]

\[var(aX_1 + bX_2+ c) = a^2var(X_1) + b^2var(X_2) + 2ab\cdot cov(X_1, X_2).\]

When X_i is i.i.d., \[var(a_1X_1 + a_2X_2 + ... + a_nXn) = var(\sum a_iX_i) = \sum a_i^2var(X_i).\]

\[var(\bar X) = var(\frac{\sum X_i}{n}) = \sum^n_{i = 1}\frac{var(X_i)}{n^2} = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}\]

Comparison of deviation measurements

Pop: \(\sigma^2 = \frac{\sum (X_i - \mu)^2}{N}\); Sample: \(s^2 = \frac{\sum (X_i - \bar{X})^2}{\color{darkred}{n - 1}}\); Sample means: \(\text{Var}(\bar{X}) = \frac{\sigma^2}{n}\); \(\text{SE}(\bar{X}) = \frac{\sigma}{\sqrt{n}}.\)

1. When calculating the population mean, μ, we have the full information. The mean is merely the mean; so does it when calculating the sample mean.
2. When calculating the sample variance, however, our goal is to estimate the population, with the sample mean as a reference.
3. However, the sample mean introduces bias because \(\bar{X}\) tends to be closer to the individual data points than the true population mean (\(\mu\)).
4. We use \(n - 1\) to reduce the influence of this bias. The method is called Bessel’s correction, which compensates for the underestimation of variability due to using \(\bar{X}\). This adjustment gives a more accurate estimate of the population variance.
5. When calculating the variances of means of a linear combination of random variables (\(var(a_1X_1 + a_2X_2 + ... + a_nXn)\)), we do not estimate the population but being interested in how the mean of a sample varies, which is inherently different from measuring the variance within the sample data itself.
6. In SE calculation, we include the correciton in σ as the nominator, the \(\sqrt{n}\) reflects merely the number of independent observations in the sample. It uses \(n\) in the denominator because it reflects the precision of the sample mean estimate, considering the combined information from all \(n\) observations.

1.4 Review (Continued): Property of a good estimator \(\hat \theta\)

• Unbiased: On average, the estimator gives the right answer, formally, \(E(\hat\theta) = \theta.\)
• Consistent: As the sample size increases, the variance decreases.
• Efficiency: Smallest variance among unbiased estimators¹

Sometimes, one may want to trade off a little bias against variance.

Root Mean Square Error (RMSE) is widely used across various fields, especially in statistics, machine learning, and predictive modeling, where accuracy of estimates or predictions is critical. Here are some key scenarios where RMSE is commonly used or referred to:

1.4.1 1. Model Evaluation in Machine Learning and Regression Analysis

• Predictive Accuracy: RMSE is frequently used to assess the accuracy of models by measuring the average difference between predicted and observed values. For example, in linear regression, it shows how well the regression line fits the data.
• Comparing Models: RMSE is used to compare the performance of multiple models on the same data. Lower RMSE values indicate better predictive accuracy, making it a convenient metric for model selection.
• Cross-Validation and Hyperparameter Tuning: During cross-validation or tuning of model hyperparameters, RMSE is used as a scoring metric to optimize the model.

1.4.2 2. Forecasting and Time Series Analysis

• Evaluating Forecast Accuracy: In forecasting, such as demand prediction or stock price forecasting, RMSE measures how close predicted values are to actual outcomes. It is one of the primary metrics to evaluate models like ARIMA, exponential smoothing, or other time series models.
• Measuring Seasonal Adjustments: RMSE can be used to check the accuracy of seasonal adjustments and trend components in time series data.

1.4.3 3. Statistical Estimation

• Assessing Estimator Quality: RMSE is valuable for evaluating estimators in cases where both bias and variance are of concern. It provides a comprehensive measure of estimation error, especially when working with biased estimators.
• Choosing Between Biased and Unbiased Estimators: RMSE allows for comparing biased and unbiased estimators on an equal footing by quantifying the total error (bias + variance) of each estimator.

1.4.4 4. Quality Control and Engineering

• Process Control: In manufacturing or engineering, RMSE can measure deviations of observed measurements from target values, providing insight into the precision and accuracy of processes.
• Signal Processing: RMSE is often used in signal processing to quantify the error between processed signals and the original signal, particularly when reducing noise.

1.4.5 5. Environmental and Climate Science

• Model Validation: Climate models, weather forecasts, and environmental simulations often rely on RMSE to assess the accuracy of predicted temperatures, precipitation levels, or pollution concentrations.
• Spatial Accuracy: In geographic and environmental studies, RMSE can evaluate the spatial accuracy of geospatial models and remote sensing measurements.

1.4.6 6. Econometrics and Financial Modeling

• Evaluating Financial Models: RMSE is used to assess the accuracy of econometric models in forecasting economic indicators, like GDP growth or inflation rates, by comparing predictions to actual outcomes.
• Risk Assessment: In finance, RMSE helps quantify the accuracy of risk models by showing the average deviation between predicted and actual returns or risk measures.

1.4.7 7. Deep Learning and Neural Networks

• Loss Function: RMSE (or mean squared error, MSE) is often used as a loss function in regression-based neural networks to minimize the difference between predicted and target values.
• Evaluation Metric: For neural network models performing regression tasks, RMSE is used as an evaluation metric to interpret prediction accuracy on test data.

1.4.8 Why RMSE is Preferred in Many Applications

• Sensitivity to Large Errors: RMSE penalizes larger errors more than smaller errors (due to squaring), making it particularly useful when large errors are more detrimental in the application.
• Interpretability: Since RMSE is in the same units as the data, it’s easy to interpret and communicate results.

In summary, RMSE is a versatile metric used in any situation where quantifying the total error (combining bias and variance) in predictions or estimates is essential, providing a clear, interpretable measure of model accuracy.

1. RMES: Root mean square error, \(\sqrt{bias^2 + var}\).

1.5 Ordinary Least Squares (OLS)

One type of simulation of the reality among many others (unnecessarily the best one).

Important

Interpretation: How does variable Y change on average when some explanatory variable X changes (while the others don’t)?

2 OLS Component

2.1 Linearity

Which X has the linear relationship with Y?

\[\begin{align} Y_i =& \beta_0 + \beta_1X_i + \epsilon_i;\\ Y_i =& \frac{1}{\beta_0} + \frac{X_i}{\beta_1} + \epsilon_i;\\ Y_i =& \beta_0 + \beta_1ln(X_i) + \epsilon_i;\\ Y_i =& \beta_0 + \frac{\beta_1}{X_i} + \epsilon_i;\\ Y_i =& \beta_0 + X_i^{\beta_1} + \epsilon_i.\\ \end{align}\]

• First two are linear.

2.2 OLS Formally¹

\[\begin{align} Y_i =& \beta_0 + \beta_1X_1 + \epsilon_i.\\ E(Y_i|X_i) =& E(\beta_0 + \beta_1X_1 + \epsilon_i|X_i),\\ =& E(\beta_0|X_i) + E(\beta_1X_1|X_i) + E(\epsilon_i|X_i),\\ =& \beta_0 + \beta_1X_i + E(\epsilon_i|X_i). \end{align}\]

• Estimand: Parameter in the population which is to be estimated in a statistical analysis
• Estimator: A rule for calculating an estimate of a given quantity based on observed data. Function of the observations, i.e., how observations are put together
• Estimation: The process of finding an estimate, or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable (value derived from the best information available)

&epsilon_i: Things we can’t explain but hope to be zero.

\(\beta_1X_i\) consistent

E(_i|X_i) assumed 0

Sample Regression Function (SRF)

\[\begin{align} Y_i =& (\hat\beta_0 + \hat\beta_1X_i) + \hat \epsilon_i,\\ =& \hat Y_i + \hat \epsilon_i.\\ =& E(Y_i|X_i) + \epsilon_i,\\ \Leftrightarrow \epsilon_i =& Y_i - E(Y_i|X_i). \end{align}\]

1. β: Estimand/parameter, function: estimator, result: estimate

2.3 Minimizing Sum of Squares

2.4 A Good Estimator → Minimizing Expected \(\epsilon\)

β₀ and β₁ that make \(\sum[Y_i - (\hat\beta_0 + \hat\beta_1X_i)]^2 = 0\)

\[\begin{align} \frac{\partial\sum[Y_i - (\hat\beta_0 + \hat\beta_1X_i)]^2}{\partial\hat\beta_1} = -\sum 2X_i(Y_i - \hat\beta_0 - \hat\beta_1X_i) =& 0;\\ \Leftrightarrow \sum X_iY_i - \sum X_i\hat\beta_0 - \sum\hat\beta_1 X_1^2 =& 0;\\ (\sum X_iY_i =& \sum X_i\hat\beta_0 + \sum\hat\beta_1 X_1^2);\\ \sum X_iY_i - \sum X_i(\bar Y - \hat\beta_1\bar X) - \sum\hat\beta_1 X_1^2 =& 0;\\ \sum X_iY_i - \sum X_i\bar Y + \sum X_i\hat\beta_1\bar X - \sum\hat\beta_1 X_1^2 =& 0;\\ \sum X_i(Y_i - \bar Y) + \hat\beta_1\sum X_i(\bar X - X_i) =& 0;\\ \sum X_i(Y_i - \bar Y) =& \hat\beta_1\sum X_i(X_i - \bar X);\\ \hat\beta_1 =& \frac{\sum X_i(Y_i - \bar Y)}{\sum X_i(X_i - \bar X)}. \end{align}\]

A transformation gadget:

\[\begin{align} \sum X_i(Y_i - \bar Y) =& \sum X_i(Y_i - \bar Y) - \bar X(n\bar Y - n\bar Y)\\ =& \sum X_i(Y_i - \bar Y) - \bar X(\sum Y_i - \sum\bar Y)\\ =& \sum X_i(Y_i - \bar Y) - \bar X\sum (Y_i - \bar Y)\\ =& \sum (X_i - \bar X)(Y_i - \bar Y) \blacksquare \end{align}\]

“Quod Erat Demonstrandum” (QED) which loosely translated means “that which was to be demonstrated”.

\[\begin{align} \hat\beta_1 =& \frac{\sum X_i(Y_i - \bar Y)}{\sum X_i(X_i - \bar X)}, \text{using the above}\\ \hat\beta_1 =& \frac{\sum (X_i - \bar X)(Y_i - \bar Y)}{\sum (X_i - \bar X)^2}. \end{align}\]

2.5 \(\beta_1\)

\[\begin{align} \hat\beta_1 =& \frac{\sum (X_i - \bar X)(Y_i - \bar Y)}{\sum (X_i - \bar X)^2},\\ =& \frac{\sum (X_i - \bar X)(Y_i - \bar Y)}{\sqrt{\sum (X_i - \bar X)^2}\color{darkred}{\sqrt{\sum (Y_i - \bar Y)^2}}}\cdot\frac{\color{darkred}{\sqrt{\sum (Y_i - \bar Y)^2}}}{\sqrt{\sum (X_i - \bar X)^2}},\\ =& r_{X, Y}\frac{s_Y}{s_X}. \end{align}\]

Tip

So, when the variance of Y(s_Y) increases, β₁ increases.

A special case: Standardized X and Y, i.e., \(s_Y, s_X\) are 1s, then,

\[\beta_1 = r_{X, Y}\frac{s_Y}{s_X} = r_{X,Y}.\]

β₀ is efficient (minimizng the u) depending on β₁

2.6 Linear nature of the OLS coefficient¹

\[\begin{align} \hat\beta_1 =& \frac{\sum (X_i - \bar X)(Y_i - \bar Y)}{\sum (X_i - \bar X)^2},\\ =& \frac{1}{\sum (X_i - \bar X)^2}\sum (X_i - \bar X)Y_i,\\ =& \frac{1}{\sum (X_i - \bar X)^2}\sum (X_i - \bar X)(\beta_0 + \beta_1X_i + \epsilon_i),\\ =& \frac{1}{\sum (X_i - \bar X)^2}[\sum (X_i - \bar X)(\beta_0 + \beta_1X_i) + \sum (X_i - \bar X)\epsilon_i],\\ =& \frac{\sum (X_i - \bar X)(\beta_0 + \beta_1X_i)}{\sum (X_i - \bar X)^2} + \frac{\sum (X_i - \bar X)\epsilon_i}{\sum (X_i - \bar X)^2}. \end{align}\]

Let \(k_i=\frac{X_i - \bar X}{\sum (X_i - \bar X)^2},\) then \(\hat\beta_1 = \sum k_i\beta_0 + \sum k_i X_i\beta_1 + \sum k_i\epsilon_i.\)

• A linear combination with errors
• Min/max(X) influences a lot.

\(\sum(X_i - \bar X) = 0.\)

β1=constant+β1constant+variation due to error

A special case: \(\hat\beta_1 = \beta_1 + \sum k_i\epsilon_i\).

1. \(\sum X_i(Y_i - \bar Y) = \sum(X_i - \bar X)(Y_i - \bar Y).\)

2.7 Var(\(\beta_1\))

\(\hat\beta_1 = \sum k_i\beta_0 + \sum k_i X_i\beta_1 + \sum k_i\epsilon_i.\)

\[\begin{align} \sigma^2 =& var(\epsilon_i|X),\\ =& var(Y_i - \hat\beta_0 - \hat\beta_1X|X),\\ =& \frac{\sum(\hat \epsilon_i^2)}{n - 2},\\ =& \hat\sigma^2. \end{align}\]

β1=constant+β1constant+variation due to error

n - 2: \(\hat\beta_0, \hat\beta_1\), also the extension of Bessel’s correction in regression

\[\begin{align} var(\hat \beta_1|X) =& var(\frac{\sum(X_i - \bar X)(Y_i - \bar Y)}{\sum(X_i - \bar X)^2}|X)\\ =& var(\beta_1 + \sum k_i\epsilon_i|X)\\ =& var(\sum k_i\epsilon_i|X), \text{given}\ \beta_1\sum k_iX_i \text{constant}\\ =& \sum var(k_i\epsilon_i|X), \text{assuming}\ \epsilon_i\ \text{independent}\\ =& \sum k_i^2 var(\epsilon_i|X)\\ =& \sum[\frac{X_i - \bar X}{\sum (X_i - \bar X)^2}]^2\sigma^2\\ =& \frac{\sum(X_i - \bar X)^2}{[\sum (X_i - \bar X)^2]^2}\sigma^2 = \frac{\sigma^2}{\sum (X_i - \bar X)^2} \end{align}\]

Assumption is IMPORTANT!!!

If ε_i is not independent, then cov(k, ε) > 0, and this estimator is underestimated.

2.8 \(\beta_0\)

\[\begin{align} \frac{\partial\sum[Y_i - (\hat\beta_0 + \hat\beta_1X_i)]^2}{\partial\hat\beta_0} =& -\sum 2(Y_i - \hat\beta_0 - \hat\beta_1X_i) = 0\\ \Leftrightarrow\sum Y_i - \sum\hat\beta_0 - \sum\hat\beta_1X_i =& 0\\ \sum Y_i =& \sum\hat\beta_0 + \sum\hat\beta_1X_i\\ =& n\hat\beta_0 + \hat\beta_1\sum X_i\\ \hat\beta_0 =& \frac{\sum Y_i}{n} - \hat\beta_1\frac{\sum X_i}{n} = \bar Y - \hat\beta_1\bar X \end{align}\]

Normal equations

\[\begin{align} \sum Y_i =& n\hat\beta_0 + \hat\beta_1\sum X_i\\ \sum X_iY_i =& \sum X_i\hat\beta_0 + \sum\hat\beta_1 X_1^2 \end{align}\]

\(\hat\beta_0, \hat\beta_1\) are called coefficients.

2.9 Var(\(\beta_0\))

\[\begin{align} var(\hat \beta_0|X) =& var(\bar Y - \hat\beta_1\bar X|X),\\ =& var[\frac{\sum(\beta_0 + \beta_1X_i + \epsilon_i)}{n} - \hat\beta_1\bar X|X],\\ \because \beta_0 +& \beta_1X_i\text{ is constant & independent},\\ =& var(\frac{\sum \epsilon_i}{n}|X) + var(\hat\beta_1\bar X|X), \\ =& var(\frac{\sum \epsilon_i}{n}|X) + var(\hat\beta_1\bar X|X) = \frac{var(\sum \epsilon_i|X)}{n^2} + \bar X^2var(\hat\beta_1|X),\\ =& \frac{n\sigma^2}{n^2} + \frac{\bar X^2\sigma^2}{\sum (X_i - \bar X)^2} = \sigma^2[\frac{1}{n} + \frac{\bar X^2}{\sum (X_i - \bar X)^2}],\\ =& \sigma^2[\frac{\sum (X_i - \bar X)^2 + n\bar X^2}{n\sum (X_i - \bar X)^2}] = \sigma^2[\frac{\sum X_i^2 - \sum\bar X^2 + n\bar X^2}{n\sum (X_i - \bar X)^2}],\\ =& \sigma^2[\frac{\sum X_i^2 - n\bar X^2 + n\bar X^2}{n\sum (X_i - \bar X)^2}]= \sigma^2\frac{\sum X_i^2}{n\sum (X_i - \bar X)^2}. \end{align}\]

2.10 Characteristics of OLS Coefficients

1. Calculated using observed data
2. Unique point estimates
3. SRF passes through \((\bar X, \bar Y)\)
4. \(\bar{\hat Y} (\text{predicted}) = \hat Y (\text{observed})\), \(\frac{\sum\hat Y_i}{n} = \frac{\sum Y_i}{n}\)
5. \(\bar{\hat \epsilon_i} = \frac{\sum{\hat \epsilon_i}}{n} = 0\)
6. \(\sum X_i\hat \epsilon_i = 0\)

\[\begin{align} cov(X_i, \epsilon_i) =& 0,\\ \frac{\sum(X_i - \bar X)(\hat \epsilon_i - \bar{\hat \epsilon_i})}{n-1} =& 0,\\ \frac{\sum X_i\hat \epsilon_i}{n-1} =& 0. \blacksquare \end{align}\]

\[\begin{align} \frac{\partial\sum[Y_i - (\hat\beta_0 + \hat\beta_1X_i)]^2}{\partial\hat\beta_0} =& 0\\ -2\sum[Y_i - (\hat{\beta_0} + \hat{\beta_1}X_i)] =& 0\\ \sum (Y_i-\hat Y_i) = \sum{\hat \epsilon_i} =& 0\\ \Rightarrow \frac{\sum{\hat \epsilon_i}}{n} = \bar{\hat \epsilon_i} =& 0 \blacksquare \end{align}\]

2.11 Take-home point

Appendix

Stretch

https://www.bilibili.com/video/BV1E54y1r76o/?buvid=Y045340D2FD9FA3A46419EEFE4578279ECBD&from_spmid=main.space-search.0.0&is_story_h5=false&mid=7RnjBONLRMus4FQZFBWD2g%3D%3D&p=2&plat_id=114&share_from=ugc&share_medium=iphone&share_plat=ios&share_session_id=79FE08C9-F2B1-4C18-A16C-B31179817B42&share_source=WEIXIN&share_tag=s_i&timestamp=1728732278&unique_k=skSWw7j&up_id=390316092&vd_source=f38aeefd0d38cecba9017eeee43e71c8

Meditation

松茸的世界：5分钟正念冥想-自信之心

https://www.xiaoyuzhoufm.com/episode/65b752ed0bef6c207457f51b?s=eyJ1IjogIjYwNDA1OGJiZTBmNWU3MjNiYmY3MjZiZSJ9