Multicollinearity & Heteroskedasticity

Large N & Leeuwenhoek (70700173)

Yue Hu

Overview

  1. Collinearity
    • Violation
    • Diagnosis
    • Adjustment
  2. Heteroscedasticity
    • Violation
    • Diagnosis
    • Adjustment

1 Multicollinearity

1.1 Before going into details

Why should we use multiple regressions, and why controls?

One observes a set of covariates that is, after statistical adjustment, sufficient to make treatment status as-if random.—Keele et al. 2020

1.2 Statistical Adjustment

  1. Linearity in the parameter; ← Specification, Moderation & GLM
  2. Nonstochastic X (“given X,” a.k.a., “X is fixed”); ← Data collection
  3. X has positive noninfinite variance (var(X)); ← Data collection
  4. Correct specification; ← Specification
  5. Identification (N > K; K = 2 for a simple OLS); ← Data collection & perfect collinearity (Today)
  6. Mean zero errors (E(εi|Xi) = 0); ← Specification (Omited variables, etc.) & Missing
  7. No covariance between Xi and εi; ← Endogeneity issue
  8. No autocorrelation; ← Autocorrelation issue
  9. No perfect collinearity ← (Today)
  10. Homoskedasticity; ← (Today)

1.3 Perfect Collinearity

\[X_{2i} = \hat\delta_0 + \hat\delta_1X_{1i} + \hat r_{1i}.\]

When cov(X1, X2) = 1, r1i = 0, then \(\hat\beta_1 = \frac{\sum\hat r_{1i}y_i}{\hat r_{1i}^2}\) cannot be estimated.

E.g., \(X_{2i} = 1 + 2X_{1i}.\)

\[\begin{align} Y_i =& \hat\beta_0 + \hat\beta_1X_{1} + \hat\beta_2X_{2} + \hat \epsilon, \text{(PRF)}\\ =& \hat\beta_0 + \hat\beta_1X_{1} + \hat\beta_2(1 + 2X_1) + \hat \epsilon, \\ =& (\hat\beta_0 + \hat\beta_2) + (\hat\beta_1 + 2\hat\beta_2)X_1 + \hat \epsilon,\\ \Rightarrow Y_i =& \tilde\beta_0 + \tilde\beta_1X_{1} + \hat \epsilon. \end{align}\]

  • Consequence
    • \(\hat\epsilon\) is unbiased
    • PRF (β2) is nonidentifiable

1.4 Multicollinearity

  • Coefficient, \(\hat\beta = (\boldsymbol{X'X})^{-1}\boldsymbol{X'Y}\)
    • Small changes in X can lead to dramatic changes in \((\boldsymbol{X'X})^{-1}\), and thus \(\hat\beta\).

Example

\[X = \begin{bmatrix} 1 & 0.99 \\ 0.99 & 1 \end{bmatrix}, \tilde X = \begin{bmatrix} 1 & 0.99 \\ 0.99 & 1.01 \end{bmatrix} \Rightarrow X^{-1} \approx \begin{bmatrix} 50.25 & -49.75 \\ -49.75 & 50.25 \end{bmatrix}, \tilde X^{-1} \approx \begin{bmatrix} 33.44 & 0.3311 \\ 0.3311 & 33.78 \end{bmatrix}\]

  • Variance
    • \(var(\hat\beta_1|X) = \sigma^2(\boldsymbol{X'X})^{-1}= \frac{\sigma^2}{\sum(X_{1i} - \bar X_1)^2(1 - \beta_{12}^2)},\) where \(\beta_{12} = cov(X_1, X_2)\).
    • As β12 increases, the variance also increases (Means?)

1.5 Illustration with simulated data

E.g., \(X'_2 = 0.5X_1 + 0.5X_2 + \nu.\)

n <- 10000

df_sim <- tibble(
  x1 = rnorm(n, mean = 3.13), x2 = rnorm(n, sd = 0.4),
  y = 2 + .2 * x1 + .4 * x2 + rnorm(n)
)

df_fake <- df_sim |> 
  mutate(x2 = 0.5 * x1 + 0.5 * x2 + rnorm(n, sd = 0.1))  

cor(df_fake$x1, df_fake$x2)
[1] 0.9118169

1.6 Consequence of multicollinearity: Illustration

ls_result <- map(list(df_sim, df_fake), \(dataset){
  lm(y ~ x1 + x2, data = dataset)
}) |>
  set_names(c("True", "W. multicol"))

small_multiple(ls_result)

1.7 Diagnosis

Variance Inflation Factors (VIF, [1, +∞]): A measure of how much the variance of the estimated coefficient βx is “inflated” by the correlation among the predictor variables.

\[VIF = \frac{1}{1 - {R}^{2}_{X_i}} = \frac{1}{Tolerance}\]

  • \({R}^{2}_{X_i}\): The coefficient of determination for a regression model where Xi is the dependent variable, and all other predictor variables are independent variables.
    • 1: no correlation between X1 and the rest
    • Rule of thumb: 4, 10
    • E.g., VIF of the true model is 1.0000473, 1.0000473 in the previous slide, and for the one with multicollinearity is 5.9315546, 5.9315546
  • Adjustment
    1. Removed the highly collinear variables (Risk?)
    2. Remeasure the collinear variables (e.g., EFA, CFA, IRT)

2 Heteroscedasticity

2.1 What’s Heteroscedasticity

  • Homo/heteroscedasticity ≈ Homo/heterogeneity (of variance)
    • var(εi|X) = σ2
    • Violation: \(var(\epsilon_i|X) = \sigma_i^2, \sigma_i^2 \neq \sigma_j^2, \forall i, j.\)

2.2 Consequence

\[\begin{align} var(\hat\beta|X) =& \frac{\sum(X_i - \bar X)^2}{[\sum(X_i - \bar X)^2]^2}\sigma_i^2.\\ H_0: var(\hat\beta|X) =& \sum(X_i - \bar X)^2;\\ \Leftrightarrow\frac{\sum(X_i - \bar X)^2}{[\sum(X_i - \bar X)^2]^2}\sigma_i^2 =& \frac{\sigma^2}{\sum(X_i - \bar X)^2},\\ \sum(X_i - \bar X)^2\frac{\sigma_i^2}{\sigma^2} =& \sum(X_i - \bar X)^2 \end{align}\]

  • Consequence
    • Unbiased, expected consistent, but inefficient
    • Only when \(\frac{\sigma_i^2}{\sigma^2} = 1\), the estimate is efficient.
  • Diagnosis: Breusck-Pagan, Goldfeld-Quandt, Park, White, etc.

2.3 Diagnosis: Breusck-Pagan(-Godfreg) test

  1. Assuming \(\boldsymbol{\sigma_i^2} = \boldsymbol{X\delta} + \boldsymbol{v_i}\)
  2. Regress \(\hat \epsilon_i^2\) on \(X_1, X_2, \cdots, X_{k - 1}\),
    • \(H_0: \sigma_1 = \sigma_2 =\cdots=\sigma_{k - 1} = 0\)
    • \(H_1\): At least one of the σ≠0
    • Statistics \(F_{k - 1, n - k}\)

2.4 Diagnosis: Goldfeld-Quandt test

  1. Divide the domain of X to three parts, \(\frac{n -c}{2}, c, \frac{n -c}{2}\).1
  2. Regress Y on X as usual for each part.
  3. Calculate \(\sigma_L^2\) from part 1 and \(\sigma_H^2 = \frac{\sum^n_{i = \frac{n + c}{2} + 1}}{\frac{n - c}{2} - k}\hat \epsilon_i^2\) from part 3.
  4. Then test \(H_0: \sigma_L^2 = \sigma_H^2\)
    • Statistics: \(\frac{\sigma_L^2}{\sigma_H^2}\sim F_{\frac{n - c - 2k}{2}, \frac{n - c - 2k}{2}}\)

2.5 Diagnosis: Park test

Ocular-inspection test: Use the scalar points of \(\hat \epsilon_i^2\) against \(X_i\)1

Regress \(ln(\hat \epsilon_i^2)\) on some \(X_{ki}\): \[ln(\hat \epsilon_i) = \hat\delta_0 + \hat\delta_1ln(X_{ki}) + \hat\gamma_i.\]

Do the t-test of coefficient in \(ln(X_i)\): \(H_0: \hat\delta_1 = 0.\)

2.6 Diagnosis: White Test

  1. Regress Y on X to get \(\hat \epsilon_i^2\)
  2. Regress \(\hat \epsilon_i^2\) on \(\sum X_i + \sum X_i^2 + \sum X_iX_j\) or \(\hat Y_i\).
  3. Test this model against a model with no variables (F test).

2.7 When to use which

  • Choose based on data conditions
    • E.g., BP test examines Xi affects the squared residuals, GQ tests differences between subsamples ⇒ Not yield the same result

2.8 Statistics of X1 and X2 from different tests


    Goldfeld-Quandt test

data:  m_fit
GQ = 9.0616, df1 = 4997, df2 = 4997, p-value < 2.2e-16
alternative hypothesis: variance increases from segment 1 to 2

    Goldfeld-Quandt test

data:  m_fit
GQ = 0.94963, df1 = 4997, df2 = 4997, p-value = 0.9661
alternative hypothesis: variance increases from segment 1 to 2

    studentized Breusch-Pagan test

data:  m_fit
BP = 1599.8, df = 1, p-value < 2.2e-16

    studentized Breusch-Pagan test

data:  m_fit
BP = 1.9111, df = 1, p-value = 0.1668
  • If you have a specific hypothesis about the variance changes (e.g., increasing with the level of an explanatory variable), the Goldfeld-Quandt or Park test might be appropriate.
  • If you’re unsure of the form of heteroscedasticity or have multiple Xs, the Breusch-Pagan or White test would be more suitable.

2.9 Fixed Effect

Least Square with Dummy Variables (LSDV)

  • Type I: \(Y_{it} = \beta_1X_{it} + \alpha_i + \epsilon_{it},\) in which α is unit-specific mean differences (unit fixed effect).
  • Type II: \(Y_it = \sum^T_{t = 1}\delta_tD_{t_i} + \beta_1X_{it} + \epsilon_{it},\) in which \(\delta_tD_{t_i}\) is the fixed effect for time.
  • Issues for using LSDV
    1. Adding a lot of variables (risk of using out of the d.f.);
    2. Additional high multicollinearity;
    3. Losing time invariant variables;
    4. Inefficient estimates of FE on binary and dependent variables.

2.10 Alternatives corrections

  1. Demeaning
  2. Fixed effect vector decomposition (FEVD)

Cons:

  1. Can’t offer sufficient information (heterogeneous to what extent?).
  2. Difficult to calculate substantive effects, e.g., first differences.

Multilevel modeling (MLM)

2.11 MLM

  • Target: Heteroscedasticity can be theoretically meaningful, esp., when researchers focus on how the variance changes.
  • Also called as hierarchical linear models (HLM), mixed models, random coefficient models, and variance component models

Modeling variance:

Random Intercept (Two-Level)

\[\begin{align} Y_{ij} = \beta_{0j}& + \beta_{1j}X_{ij} + \epsilon_{ij}, \epsilon_{ij}\sim N(0, \sigma^2)\\ \beta_{0j}& = \gamma_{00} + \gamma_{01}Z_j + u_{0j}, u_{0j}\sim N(0, \tau^2) \end{align}\]

Z is the group indicator.

Intraclass correlation: \(\rho = \frac{\tau^2}{\sigma^2 + \tau^2}.\)

2.12 Random-Slop MLM

\[\begin{align} Y_{ij} = &\beta_{0j} + \beta_{1j}X_{ij} + \epsilon_{ij}\\ &\beta_{0j} = \gamma_{00} + \gamma_{01}Z_j + u_{0j}\\ &\beta_{1j} = \gamma_{10} + u_{1j}.\\ \text{Assume}\left( \begin{array}{c} u_{0j}\\ u_{1j}\end{array}\right)&\sim BVN\left[\left(\begin{array}{c} 0\\0\end{array}\right), \left(\begin{array}{cc} \tau_0^2 & \tau_0\tau_1\\ \tau_0\tau_1 & \tau_1^2 \end{array}\right)\right] \end{align}\]

  1. Identity: τ02 = τ12 = 1;
  2. Exchangeable: τ02= τ12;
  3. Independent: τ02τ2 = 0;
  4. Unstructured: no specific relationships is assumed for the τs.

2.13 Alternative Methods

  • White covariance matrix
  • Weighted least square
  • Feasible Generalized Linear Squares (FGLS)
  • “Sandwich” Matrix
  • Robust Standard Error

2.14 Take-home point

Appendix

2.15 White covariance matrix

(a.k.a., Heteroscedasitic-consistent covariance matrix).

\[E(\epsilon_i|X) = E\{\sqrt{[\epsilon_i - E(\epsilon_i)]^2}|X\}\]

Since E(εi|X) = 0 by assumption, we can estimate \(\epsilon_i\) with \(\hat \epsilon_i^2\) and estimate \(var(\hat\beta_1|X)\) with \(\frac{\sum(X_i - \bar X)^2}{[\sum(X_i - \bar X)^2]^2}\hat \epsilon_i^2.\)

The estimates turn out to be biased, but converge asymptotically in n to the true distribution. (Means?)

2.16 Weighted least square

Reduce the substantive effect of the change in X by \(\frac{1}{\sigma_i}\), in order to squeeze the value towards the middle: If σi is known,

\[\begin{align} \frac{Y_i}{\sigma_i} =& \frac{\beta_0}{\sigma_i} + \frac{\beta_1}{\sigma_i}X_i + \frac{\epsilon_i}{\sigma_i};\\ \text{Then, } var(\frac{\epsilon_i}{\sigma_i}) =& \frac{1}{\sigma_i^2}var(\epsilon_i) = \frac{\sigma_i^2}{\sigma_i^2} = 1;\\ \Rightarrow Y_i^* =& \beta_0X^*_{0i} + \beta_1X^*_{1i} + \epsilon_i^*. \end{align}\]

The last equation is homoscedastistic. However, in most cases, we don’t know σi ⇒ var(εi)∼ X1i, i.e., var(εi) = σi2 = σ2Xi = hiσ2.

2.17 Weighted least square (continued)

Given the goal var(ε*i) = var(ε*j), ∀ i,j.

\[\begin{align} \epsilon^*_i =& \frac{\epsilon_i}{\sqrt{h_i}},\\ var(\frac{\epsilon_i}{\sqrt{h_i}}) =& \frac{var(\epsilon_i)}{h_i} = \frac{h_i\sigma^2}{h_i} = \sigma^2,\\ \Rightarrow Y_i^* =& \frac{Y_i}{\sqrt{h_i}}; X_{0i}^* = \frac{1}{\sqrt{h_i}}; X_{1i}^* = \frac{X_{1i}}{\sqrt{h_i}}, \text{ assuming } X_i\in R^+. \end{align}\]

Tip

In practice, there are different ways to estimate the weight, one need to carefully choose the proper one.

2.18 Feasible Generalized Linear Squares (FGLS)

\[\begin{align} \boldsymbol{Y} =& \boldsymbol{X\beta} + \boldsymbol{\epsilon}; \\ var(\boldsymbol{\epsilon}) =& \Omega_{n\times n} = \left( \begin{array}{cccc} \sigma_1^2 & 0 & \cdots & 0\\ 0 & \sigma_2^2 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & \cdots & \sigma_n^2\\ \end{array}\right),\\ \text{Then, } \boldsymbol{\hat\beta_{GLS}} =& (\boldsymbol{X'\Omega X})^{-1}(\boldsymbol{X'\Omega Y}).\\ \text{Let }\boldsymbol{H}: \boldsymbol{\Omega} = \boldsymbol{HH^{-1}},\text{then, } \boldsymbol{H^{-1}Y} =& \boldsymbol{H^{-1}X\beta} + \boldsymbol{H^{-1}\epsilon}, \boldsymbol{H^{-1}\epsilon} = \boldsymbol{H^{-1}(H^{-1})'};\\ var(\boldsymbol{\epsilon})=& (\boldsymbol{HH'})^{-1}\boldsymbol{\Omega} = \boldsymbol{\Omega}^{-1}\boldsymbol{\Omega} = \boldsymbol{I},\\ var(\boldsymbol{\hat\beta_{GLS}}) =& (\boldsymbol{X'X})^{-1}(\boldsymbol{X'\Omega X})(\boldsymbol{X'X})^{-1}. \end{align}\]

  • If there’s no heteroscedasticity, \(\boldsymbol{\Omega} = \sigma\boldsymbol{I}.\)
  • If there is heteroscedasticity and \(\boldsymbol{\Omega}\) is known, then WLS (a special type of GLS).
  • If \(\boldsymbol{\Omega}\) is unknown, run \(\boldsymbol{Y} = \boldsymbol{X\beta} + \boldsymbol{\epsilon}\) to estimate \(\boldsymbol{\Omega}\) with \(\boldsymbol{\hat\Omega} = \boldsymbol{\hat\epsilon\hat\epsilon'}\)

Warning

NB: This method does not get SE, and also biased for small N.

2.19 “Sandwich” Matrix

FGLS is a type of “sandwich” estimator.

In a more general view, let \(\boldsymbol{Q} = \boldsymbol{X'X}\) and

\[\begin{align} \boldsymbol{Q} = \left( \begin{array}{cccc} \sigma_1^2 & 0 & \cdots & 0\\ 0 & \sigma_2^2 & \cdots & 0\\ \vdots & \vdots & \vdots & \vdots \\ 0 & 0 & \cdots & \sigma_n^2\\ \end{array}\right) \end{align}\]

Then for regular OLS, \(var(\beta) = \sigma^2(\boldsymbol{X'X})^{-1} = \sigma^2\boldsymbol{Q}^{-1}\).

But when heteroscedasticity occurs, \(var(\boldsymbol{\beta}|X)\neq \sigma^2\boldsymbol{Q}^{-1}\).

Instead, let \(\boldsymbol{G} = \boldsymbol{X'GX}\), then \(var(\boldsymbol{\beta}|X) = \boldsymbol{Q^{-1}GQ}^{-1}.\)

2.20 Heteroscedasticity in TSCS Data

  • TSCS: Time-series cross-sectional data.
    • Sometimes, they are called “pooled data”, “panel data”(WRONG!)
  • Reason of heterosceasticity
    • Spatical heterogeneity
    • Temporal heterogeneity
  • Deal with heteroscedasticity in TSCS data:
    1. Robust standard error
    2. Fixed effect
    3. Multilevel modeling

2.21 Robust Standard Error

  • Sandwich SE
    • In the FGLS version, the “meat” can be identified only when T > N, and before this, the autocorrelation (\(cov(\epsilon_i, u_j|X_i, X_j) = 0, \forall i, j\)) has to be eliminated.
  • Panel-corrected SE
    • An alternative version of the sandwich SE. Ω is clustered by time periods: \(\hat\Omega_{ij} = \frac{\sum^T_{t = 1}\epsilon_{it}\epsilon_{jt}}{T}\)
  • Cluster SE
    • Adjust SE to account for correlations within clusters.
  • Within-between model
    1. Run separate models in each group
    2. Run an aggregate regression

Let \(\bar Y_i = \frac{\sum^n_{i=1}Y_{it}}{n_i}, \bar X_i = \frac{\sum^n_{i=1}X_{it}}{n_i}, \bar \epsilon_i = \frac{\sum^n_{i=1}\epsilon_{it}}{n_i}\). Then,

\[\begin{align} \bar{Y_i} =& \beta_0 + \beta_1\bar{X_i} + \epsilon_i + \epsilon_i\\ Y_{it} - \bar{Y_i} =& (\beta_0 - \beta_0) + \beta_1(X_{it} - \bar{X_i}) + (\epsilon_{it} - \epsilon_i)\\ \hat{Y_i} =& \beta_1\hat{X_{it}} + \hat{\epsilon_{it}}, \text{(a.k.a., the within model)}\\ Y_{it} =& \beta_0 + \beta_1(X_{it} - \bar{X_i}) + \beta_2\bar{X_i} + \epsilon_{it}, \text{(a.k.a., the between model)} \end{align}\]

Stretch

Meditation

松茸的世界:5分钟正念冥想-自信之心

松茸的世界:5分钟正念冥想-自信之心