Hypothesis Testing: Concept and Operation

Large N & Leeuwenhoek (70700173)

Yue Hu

Overview

• Understand hypothesis testing
• Test hypotheses

1 Understand hypothesis testing

1.1 Function of Hypothesis

Theory (logic) → real world (conceptually)

↓

Ideal world (?) vs. real world (theoretically)

↓

Random distribution vs. empirical distribution (statistically)

↓

• Identification ⇒ Statement (hypothetically)

↓

Inferences (coefficients + confidence)

↓

Implications

1.2 How to get the coefficients

Probability: The chance of occurrence of X_i given the PDF of X, P(O|θ).

Likelihood : How probable a given set of observations is for certain values of the parameters of a distribution, ℒ(θ|O) = \(\prod_{i=1}^n y_i\)¹

ℒ Lagrangian or Laplace transform

Formally, let O be the set of observed outcomes and θ be the set of parameters that describe the stochastic process.

Probability: The integral, the area, the results.

Likelihood: The parameter, the product, the hypothesis.

ℒ can above 1.

• Goal: Model (parameter) selection
  • What’s the most appropriate estimation?
    • Maximizing the probability that ℒ(θ|O) = P(O|θ)
  • Point estimation: Unbiasdness
  • Interval estimation: Range of plausible values
  • Goodness of fit: Explained variances
  • Diagnostic estimation: What if the assumptions are violated

E.g., Given binomial distribution \(L(\pi) = {n \choose s}\pi^s(1 - \pi)^{n - s}\), what π reaches the maximum likelihood?

\[log[L(\pi)] = log{n\choose s} + s\cdot log(\pi) + (n - s)\cdot log(1 - \pi).\]

To get the peak value, we use the derivative:

\[\frac{dlog[L(\pi)]}{d\pi} = \frac{s}{\pi} - \frac{n - s}{1 - \pi}.\]

To minimize this, let it equal 0

\[\begin{align} \frac{s}{\pi} - \frac{n - s}{1 - \pi} = 0 \Rightarrow \frac{s}{\pi} &= \frac{n - s}{1 - \pi},\\ s(1 - \pi) &= \pi(n - s),\\ s - s\pi &= n\pi - s\pi \Rightarrow\pi = \frac{s}{n}. \end{align}\]

1. Maximum Likelihood Estimation: Maximizing the probability that ℒ(θ|O) = P(O|θ).

1.3 How to get confidence

Parametric

Nonparametric

1.4 Format

• Null: ℒ(null): Y = β₀ + ε (Random distribution)
• Theoretical: ℒ(theory): Y = β₀ + β₁X + ε (Your thought)

Hypothesis framing ABC

• Assertion normative statement
• Observable
• Clear direction

2 Test hypotheses

2.1 Statistical Foundations

Law of Large Number(LLN): For a sample with size n of a random variable X,

\[{\displaystyle \lim _{n\to \infty }\sum _{i=1}^{n}{\frac {X_{i}}{n}} = \lim _{n\to \infty }\bar{X} = \mu.} \]

When n gets larger, the ratio of outcomes, X̄, will converge to the theoretical (population) average μ

Converge = concentrated around

Convergence in probability

Proof: https://blog.csdn.net/a19990412/article/details/85283729

Central limit theorem (CLT): For a random sample of n, X̄ fluctuate around μ with an uncertainty,

\[When\ n \rightarrow \infty, Pr(\bar X) \sim \mathcal{N(\mu, \sigma)}. \]

When independent random variables are summed up, their properly normalized sum tends toward a normal distribution, even if the original variables themselves are not normally distributed.

Proof: https://zhuanlan.zhihu.com/p/93738110

When n gets larger…

‍LLN: X̄ will approach to μ
‍CLT: X̄ will distribute normally.

Contradictory?

Becoming a value or becoming a distribution?

CLT shows the the shape, LLN shows the location of the center.

2.2 Steps (parametric test)

1. An observed sample;
2. Making the IID assumption;
3. Set up the null (and alternative) hypotheses
4. Set the confidence level;
5. Compare the X̄ with μ.

2.3 IID

1. An observed sample;
2. Making the IID assumption;¹
   • Identical: X and Y are from the same distribution, \({\displaystyle F_{X}(x)=F_{Y}(x)\,\forall x\in I}\)
   • Independent: \({\displaystyle F_{X,Y}(x,y)=F_{X}(x)\cdot F_{Y}(y)\,\forall x,y\in I}\)
3. Set up the null (and alternative) hypotheses
4. Set the confidence level;
5. Compare the X̄ with &mu.

Unbiased point estimate:

\[\begin{align} Given \bar X = \frac{\sum X}{n}, E(\bar X) =& \frac{1}{n}[E(X_1) + E(X_2) + \dots + E(X_n)],\\ =& \frac{1}{n}[\mu + \mu + \dots + \mu] = \mu. \end{align}\]

Unbiased variance (standard error):

\[\begin{align} s(\bar X)^2 =& \frac{[E(s(X_1)^2) + E(s(X_2)^2) + \dots + E(s(X_n)^2)]}{n^2}\\ =& \frac{1}{n}[\sigma^2 + \sigma^2 + \dots + \sigma^2] = \frac{\sigma^2}{n}\\ \therefore\ SE =& \frac{\sigma}{\sqrt{n}}. \end{align}\]

Lindeberg-Feller CLT extend CLT to non-iid random variables, which does not require identical σ but as long as there is not big outliers, CLT should be fine.

1. No need under Lindeberg-Feller CLT.

2.4 Hypothesis Setting

1. An observed sample;
2. Making the IID assumption;
   • Identical: X and Y are from the same distribution, \({\displaystyle F_{X}(x)=F_{Y}(x)\,\forall x\in I}\)
   • Independent: \({\displaystyle F_{X,Y}(x,y)=F_{X}(x)\cdot F_{Y}(y)\,\forall x,y\in I}\)
3. Set up the null (and alternative) hypotheses
   • H₀: Specifying values for one or more population parameters in a random distribution (μ, π rather than X̄, P);
   • H₁: the population parameter is something other than the value in the stochastic status;
4. Set the confidence level;
5. Compare the X̄ with &mu.

2.5 Confidence Interval

1. An observed sample;
2. Making the IID assumption;
   • Identical: X and Y are from the same distribution, \({\displaystyle F_{X}(x)=F_{Y}(x)\,\forall x\in I}\)
   • Independent: \({\displaystyle F_{X,Y}(x,y)=F_{X}(x)\cdot F_{Y}(y)\,\forall x,y\in I}\)
3. Set up the null (and alternative) hypotheses
   • H₀: Specifying values for one or more population parameters in a random distribution (μ, π rather than X̄, P);
   • H₁: the population parameter is something other than the value in the stochastic status;
4. Set the confidence level;
   • 1 - α
5. Compare the X̄ with &mu.

2.6 About alpha

Decision	H0 T	H0 F
Reject	Type I error (Pr = α)	Pr = 1 - β
Fail to Reject	Pr = 1 - α	Type II error (Pr = β)

Type I: 无中生有 Type II: 闪

β is the power of the test.

Avoiding Type I is more emergent.

2.7 Why 0.05

It’s not arbitrary, α = 0.05 ⇒ 1 - α = 0.95 (one-tailed) or 0.975 (two tailed)

Well…it’s actually arbitrary annnnnnnnnnd qualitative.

Not calculate the p-value and set the significance but set the confidence level and conduct the test. This is scientific

2.8 How to get p < 0.05

You don’t pursue p

习近平在中央党校（国家行政学院）中青年干部培训班开班式上的讲话(2022年3月1日)

守住拒腐防变防线，最紧要的是守住内心，从小事小节上守起，正心明道、怀德自重，勤掸“思想尘”、多思“贪欲害”、常破“心中贼”，以内无妄思保证外无妄动。

2.9 Deploy Hypothesis testing

1. An observed sample;
2. Making the IID assumption;
3. Set up the null (and alternative) hypotheses
4. Set the confidence level;
5. Compare the X̄ with μ, two steps:
   • SE
   • Critical value / Confidence intervals

2.10 Standard Error (Review)

\[SE(\bar X) = \frac{s}{\sqrt{n}}.\]

E.g. Given the population mean as 69 and standard deviation as 3.2, how would the mean of a random sample of four observations fluctuate?

\[\begin{align} E(\bar X) =& \mu = 69; \sigma = 3.2, \\ \therefore SE(\bar X) =& \frac{3.2}{\sqrt 4} = 1.6. \end{align}\]

2.11 Correct SE

Finite population correction

For small population, not small sample.

• FPC: Without replacement, SE of X̄ ↑ ⇒ uncertainty overestimated ⇒ correction: \(\sqrt{\frac{N - n}{N - 1}}\)
  • n = 1 (sampling only one sample), Then \(SE = \frac{\sigma}{\sqrt{n}}\sqrt{\frac{N - n}{N - 1}} = \frac{\sigma}{\sqrt{n}}\);
  • n = N (sampling all), SE = 0.
  • n = 1000, N = 100,000,000 (large sample of a large population), \(FPC = \sqrt{\frac{100000000 - 1000}{100000000 - 1}}\approx .999\), little changes.
  • n = 100, N = 108 (large sample of a small population), \(FPC = \sqrt{\frac{108 - 100}{108 - 1}}\approx 0.075\), some changes.

2.12 What does the FPC correct

2.13 Statistical Test: Individual Level

• SE → Z(t)-score →
  1. Critical values (z/t), or
  2. Confidence intervals

\[Z = \frac{\bar X - \mu}{SE}= \frac{\bar X - \mu}{\sigma/\sqrt n}.\]

In a sample of N, the sample proportion P fluctuates around the population proportion π with SE, \(\sqrt{\frac{\pi(1 - \pi)}{n}}\). (Why?)

Proof: \(\sigma^2 = E(X^2) - \mu^2 = \pi - \pi^2 = \pi(1 - \pi)\). ∴ \(\sigma = \sqrt{\pi(1 - \pi)}, SE = \sigma/\sqrt{n} = \sqrt{\frac{\pi(1 - \pi)}{n}}.\)

Then, \(Z = \frac{P - \pi}{\sqrt{\frac{\pi(1 - \pi)}{n}}}\), the confidence interval: \(\pi = P \pm Z\sqrt{\frac{P(1 - P)}{n}}.\)

E.g. Given \(\mu\) = 72 and \(\sigma\) = 9, and a random sample of 10. Calculate the probabilities of P(X > 80) and P(X̄ > 80)

\(Z = \frac{80 - 72}{9} = .89\Rightarrow P(Z > .89) =\) 0.1867;

\(Z = \frac{80 - 72}{9/\sqrt{10}} = 2.81\Rightarrow P(Z > 2.81) =\) 0.0025.

2.14 Statistical Test: Aggregated Level

• SE → Z(t)-score →
  1. Critical values (z/t), or
  2. Confidence intervals

\[Z = \frac{P - \pi}{\sqrt{\frac{\pi(1 - \pi)}{n}}}.\]

E.g., Given the Republican are 60% of the U.S. population, what’s the probability that the Republican are the minority in a random sample of 100 people from the national population?

Minority means \(P(\pi < 0.5)\).

Then, \(Z = \frac{0.5 - 0.6}{\sqrt{\frac{0.6(1 - 0.6)}{100}}} =\) -2.0412, therefore, P(Z < 2.0412) = 0.0206.

2.15 Full Process Illustration

Given a virus can influence 10% of the population. Now there’s a sample of senior people, n = 527, within which there are 14% infected. Are senior people more likely to be victimized?

• Assuming the sample is IID sampled and \(H_0: \pi \leq 10; H_1: \pi > 10.\)
• α = 0.05
• Confidence-interval method:
  • \(\pi = 0.14 \pm 1.96 * \sqrt{\frac{0.14 * (1 - 0.14)}{527}} = 0.14 \pm 0.03,\) that is [0.11, 0.17] > 0.1. \(H_0.\) rejected.
• Critical value method
  • \(Z_{obs} = \frac{P - \pi}{\sqrt{\frac{\pi(1 - \pi)}{n}}} = \frac{14 - 10}{\sqrt{\frac{0.1 * 0.9}{527}}} = 3.06.\) Given the level of \(\alpha = 0.05\), \(Z_{critical}\) = 1.96 < \(Z_{obs}\),¹ therefore reject the \(H_0.\)

• Most applications are one-tail tests, while most software gives two-tail results.
• One-tailed test more often apply the critical value than confidence interval method.

1. In r, qnorm(0.975). 0.975, huh?

2.16 Take-home point

Appendix

Stretch

https://www.bilibili.com/video/BV1E54y1r76o/?buvid=Y045340D2FD9FA3A46419EEFE4578279ECBD&from_spmid=main.space-search.0.0&is_story_h5=false&mid=7RnjBONLRMus4FQZFBWD2g%3D%3D&p=2&plat_id=114&share_from=ugc&share_medium=iphone&share_plat=ios&share_session_id=79FE08C9-F2B1-4C18-A16C-B31179817B42&share_source=WEIXIN&share_tag=s_i&timestamp=1728732278&unique_k=skSWw7j&up_id=390316092&vd_source=f38aeefd0d38cecba9017eeee43e71c8

Meditation

松茸的世界：5分钟正念冥想-自信之心

https://www.xiaoyuzhoufm.com/episode/65b752ed0bef6c207457f51b?s=eyJ1IjogIjYwNDA1OGJiZTBmNWU3MjNiYmY3MjZiZSJ9