Case Illustration: Econometric Analysis

Understanding Policies (10700193-90)

Yue Hu

Tsinghua University

Overview

• WHAT & WHY
  • An analysis on how many data points?
  • Why not experiments?
  • What do we look for with LNA?
• HOW

What & Why

What’s LNA

In lieu of “quantitative”…

Large-N: Number of observations—not participants, but trials/units

What’s a 定量研究?

• 定结果的量
• 定数据的量
  • 都是量吗？binary，categorical， ordinary

Did you really 定量？

How large? 10, 100, 1000, 10000?

• We don’t know……

• It depends……

• The larger, the better!

To know how large, you have to know why using this method…

Why Not Experiments?

Experiment is powerful!

• Excluding confounders
• Clear outcomes
• Clear causal chain

But

• Difficult to conduct
• Difficult to represent
• No “cool” methods

Alternative: Quasi-experiment¹

A.k.a., pseudo-random assignment

1. Collins Thesaurus: Quasi- = pseudo-, apparent, seeming, semi-, so-called, would-be.

Why Must N Be Large?

• Sufficient variance to fake the control-treatment structure
• Sufficient amount to represent the population

Statistical foundation

• Law of Large Numbers
• Central Limit Theorem

Representation

Law of Large Numbers(LLN)

As the number of experiments (sample) increases, the ratio of outcomes will converge to the theoretical (population) average.

• → A rule of thumb: 100

Central Limit Theorem(CLT)

The sampling distribution of the sample means approaches a normal distribution as the sample size gets larger.

• → A rule of thumb: 1000 (Where does this number come from? We’ll talk about that soon)

Many natural phenomena and measurements tend to follow a normal distribution (a bell-shaped curve) because of how randomness and averages work. Here’s why:

Why Things Often Look “Normal”

1. The “Averaging Out” Effect
   Imagine rolling a single die: each number (1–6) is equally likely. But if you roll 10 dice and take the average, the results will cluster around 3.5 (the middle). This happens because extreme highs and lows cancel out when you average many tries. This is called the Central Limit Theorem—when you collect enough data, averages tend to form a bell curve, even if the original data isn’t normal.

2. Many Small Factors
   Things like human height or test scores are influenced by lots of tiny, random factors (genes, diet, study habits). No single factor dominates, so their combined effect creates a balanced, symmetric pattern around an average.

Why Some Things Start Non-Normal But Become Normal

Some patterns start off skewed or uneven but turn normal as you gather more data:

Example 1: Counting Rare Events (Poisson Distribution)

• Small numbers: If you count something rare, like “how many shooting stars you see in 10 minutes,” most counts will be 0 or 1, with a few higher numbers. This creates a lopsided (right-skewed) distribution.
  
• Large numbers: If you count shooting stars over 100 nights, the counts will spread out and average out, forming a bell curve. The randomness balances out with more data.

Example 2: Coin Flips (Binomial Distribution)

• Few flips: Flip a coin 10 times. The number of heads might be 3, 5, or 7—unpredictable and uneven.
  
• Many flips: Flip it 1,000 times. The number of heads will almost always be close to 500, forming a normal curve. More trials = more predictability.

---

Key Takeaway

• Normal distribution is common because randomness often “smooths out” when you average many small, independent effects.
  
• Non-normal at first? With enough data (or larger sample sizes), even skewed patterns tend to look normal due to the Central Limit Theorem.

Think of it like stirring a lumpy cake batter: at first, you see chunks (unevenness), but after enough mixing (more data), it becomes smooth!

You gonna feel it

My highly-educated fellows, let’s play a happy-happy kid game!

And we are going outside! 😈

1. Toss a fair, single die
   • Odds: One step to the ←
   • Evens: One step to the →
2. Toss six times
3. Going forward

Replicate the “magic”

Why Does CLT Require Over 1,000?

The larger the better?

\[\text{Margin of Error (M.E.)} = Z \times \frac{S}{\sqrt n}.\]

Sample Size (n)	Margin of Error (M.E.)
200	7.1%
1000	3.2%
2000	2.2%
4000	1.6%

To cut the margin of error in half, like from 3.2% down to 1.6%, you need four times as big of a sample, like going from 1000 to 4000 respondents. To cut the margin of error by a factor of five, you need 25 times as big of a sample

Now you know how large is large! Good for you 👍🎉

LNA for Public Policy

任何政策都建立在对事物差异性的分析和把握之上，没有差异性就没有政策。我国社会差异性特征明显，反映在地域、城乡、民族、人群等多个层面。我们的决策部署有综合性的，也有专项性的……这些都需要充分熟悉情况、深入分析论证、科学把握尺度。要坚持科学决策、民主决策、依法决策，对情况进行深入分析 (习近平 2019, 118–19)。

《中央政治局的同志必须有很强的看齐意识》(2015年12月28日、29 日),习近平《论坚持党对一切工作的领导》,中央文献出版社2019年版，第118-119页

How

Univariate Analysis

Let’s say we’ll test a school policy of course management

Student_ID	Courses
2025097963	9
2025091648	9
2025097782	12
2025093288	10
2025093709	8
2025091864	10
2025094798	15

How to Describe a Variable

Given the courses students took: (9, 9, 12, 10, 8, 10, 15)

• Mean: \(\frac{9 + 9 + 12 + 10 + 8 + 10 + 15}{7} \approx 10.\)
• Median: 8, 9, 9, 10, 10, 12, 15
• Mode: 8, 9, 9, 10, 10, 12, 15

Which one is the best?

How about this

Levels of Descriptive Statistics

• Number
• Statistics
• Plot

	Unique	Missing Pct.	Mean	SD	Min	Median	Max	Histogram
mpg	25	0	20.1	6.0	10.4	19.2	33.9
disp	27	0	230.7	123.9	71.1	196.3	472.0
hp	22	0	146.7	68.6	52.0	123.0	335.0
drat	22	0	3.6	0.5	2.8	3.7	4.9
wt	29	0	3.2	1.0	1.5	3.3	5.4
qsec	30	0	17.8	1.8	14.5	17.7	22.9
		N	%
cyl	4	11	34.4
	6	7	21.9
	8	14	43.8
vs	FALSE	18	56.2
	TRUE	14	43.8
gear	3	15	46.9
	4	12	37.5
	5	5	15.6

Bivariate Analysis

Weight of a car and how long it can run

Not Always Reliable

How about this data then?

Multivariable Analysis

Multivariable Analysis: Not a Panacea

Present Your Model Efficiently

Take-Home Points

• What: “Quantitative” analysis
  • How large is large?
    • LLN: 100
    • CLT: 1000
• Why
  • Why Not experiment?
  • What do we look for from the LNA?
    • Average and distribution
• How
  • Univariate
  • Bivariate
  • Multivariate

Bonus: If you wanna know more

“治理技术专题：政治数据分析” (70700173) 😋

Reference

Pizzi, Elise, and Yue Hu. 2022. “Does Governmental Policy Shape Migration Decisions? The Case of China’s Hukou System.” Modern China (Thousand Oaks) 48 (5): 1050–79. https://doi.org/10.1177/00977004221087426.

习近平. 2019. 论坚持党对一切工作的领导. 中央文献出版社. https://books.google.com?id=yvt0zQEACAAJ.

胡悦, and 朱萌. 2022. “以语塑心与国民治理：外语习得对政治认知能力的塑造机制研究.” 治理研究 38 (04): 51–65+125. https://doi.org/10.15944/j.cnki.33-1010/d.2022.04.007.

Bonus: Policy Analysis with LNA

Example 1: Hukou Policy and Migration Decisions? (Pizzi and Hu 2022)

Research Question

Does governmental hukou (户口) policy shape migrants’ destination choices in China?

• Cities set their own rules: how hard to acquire local hukou, and what benefits it confers
• Millions migrate despite strict policies — so what actually matters?

Why this is a large-N problem

• Hukou policy varies across hundreds of cities
• Individual migration choices are noisy — only patterns across thousands of cases reveal policy effects
• Need enough variation to separate hukou effects from economic ones

Theories & hypotheses

H₁: Barriers

Strict hukou acquisition rules deter migrants from choosing that city

• Harder to get hukou → city less appealing
• Migrants actively avoid locked cities

H₂: Benefits

People prefer destinations where they can access public services and welfare

• H₂.₁: Cities where one can get local hukou are more attractive
• H₂.₂: Cities where services are accessible regardless of hukou status are even more attractive

Note

These two hypotheses make opposite predictions about what drives migration — only data can tell us which is right

Data: Two Sources, One Theory

LESS 2018 (survey experiment)

• N = 1,100; 50% migrants, 50% non-migrants
• Online, quota-sampled (gender × education)
• Covers 31 provinces
• Key feature: embedded experiment — randomly shows respondents different hukou scenarios

CLDS 2016 (national representative survey)

• N = 21,086 respondents
• Multi-stage stratified probability sample
• 145 cities across 29 provinces
• Checks generalizability of experimental results

Plus: objective hukou strictness scores for 297 cities → Estimated via random forest using seed data from 45 cities (Zhang & Tao, 2012) → Scores range 0–4; Shanghai, Shenzhen, Beijing, Guangzhou rank strictest

Variables and Analytical Strategy

Key Variables

Variable	Measurement
OV (experiment)	Destination preference (1–5 Likert)
OV (survey)	Intent to stay long-term (1–5)
OV (survey)	Desire to acquire local hukou (1–5)
Treatment	easier hukou vs. education w/o hukou
EV (observed)	City hukou strictness (0–4)

Controls: Age, gender, education, job type, income satisfaction

Methods — from bivariate to multivariate

• Ordered logit — appropriate for ordinal DV (e.g., “how likely to stay: 1–5”)
• Kolmogorov-Smirnov test — compare full distributions between experiment groups
• Average marginal effects — interpret conditional effects across city tiers

Findings: Hukou as a Barrier?

Known

Careness

H₁ Rejected: Strictness Is NOT a Deterrent

• Respondents correctly perceive hukou strictness differences across cities
• Yet they rank hukou strictness last among six factors in destination choice (income and job security ranked first)
• No demographic group — including current migrants — weights strictness heavily

Findings: Hukou as a Benefit?

Experiment results (all groups faced the same economic trade-off):

Group	Prefer City B (lower wage, but…)
Control (no hukou info)	13.4%
Hukou treatment (easier to get hukou in B)	28.1%
Education treatment (no hukou needed for schools in B)	41.7%

→ In the national survey: stricter cities reduce migrants’ intention to stay long-term (ordered logit, p < 0.05), robust across all city tiers

Policy takeaway: Extending services to non-hukou-holders is more effective than relaxing hukou acquisition rules

Example 2: English-Learning Policy and Political Minds? (胡悦 and 朱萌 2022)

Why this is a large-N problem

• Language effects on political cognition are subtle and cumulative — only detectable across thousands of respondents
• Need enough variation in English proficiency to separate language effects from education, income, and other confounders

Research Question

Does English proficiency shape Chinese citizens’ political cognitive capabilities?

• China has 400+ million English learners — a product of 40 years of mandatory education
• But does this language learning change how people think about politics?

The Language Policy Field Model

Core idea

Language effects are policy-conditioned: the same language produces different political effects in different policy environments

Four hypothesized mechanisms

Mechanism	Path	Prediction
H₁ Information	English → internet use	↑ internal & external efficacy
H₂ Values	English → Western values	↑ internal, ↓ external efficacy
H₃ Self-evaluation	English → self-confidence	↑ internal efficacy
H₄ Competition	Relative English advantage	↑ internal & external efficacy

Data

CGSS 2010

• Chinese General Social Survey (中国综合社会调查)
• Multi-stage stratified probability sample
• 31 provinces, 140 cities, 2,762 communities
• N ≈ 11,700 usable respondents
• Contains rare items on both English proficiency and political efficacy

Operation

Variable	Measurement
OV	Internal political efficacy (1–5)
OV	External political efficacy (1–5)
EV	English proficiency (self-rated listening + speaking, 1–5)
Mediator 1	Internet use frequency (1–5)
Mediator 2	Attitude toward free speech (1–5)
Mediator 3	Self-rated social class (1–10)
Mediator 4	Relative English advantage (Δ from city mean)
Controls	Education, Mandarin, income (log), age, gender, party membership, rural status, migrant status, ethnicity; provincial fixed effects

Findings: What Drives Political Capability?

1. Effect Confirmed: English Proficiency Boosts Both Types of Efficacy

• OLS results (controlling for education, Mandarin, income, and demographics): English proficiency has a significant positive effect on both internal and external political efficacy — robust across all specifications 1.Internal efficacy — “Can I understand and participate in politics?”
• Information acquisition: 70%+ of mediated effect (dominant)
• Self-evaluation (self-confidence): ~24% of mediated effect
• Value westernization: NOT significant

1. External efficacy — “Does the government respond to people like me?”

• Relative English advantage: 90%+ of mediated effect (dominant)
• Self-evaluation also contributes positively (consistent with internal efficacy linkage)

Policy takeaway: English learning shapes political capability through literacy, not values — no evidence of westernization; instead, it raises citizens’ confidence in understanding government and their sense of competitive standing in society