Case Illustration: Experimental Analysis

Understanding Policies (10700193-90)

Yue Hu

Tsinghua University

You have been different

• Perspectives
  • Rational choice/Incremental model (micro) ← strategic conditions
    • System theory/Structural functionalism (macro) ← institutional conditions
      • Elitism (micro) ← power conditions
        • Group theory (meso) ← organizational conditions
• Process: How to make a decision → how to process a decision → whose call → call for what

Offclass Quiz

Cesium Fallout

焚城中刘德华代表技术官僚，莫文蔚代表行政官僚，他们共同构成精英阶层，而他们又重视民众生死

Overview

How do we know the perspectives explains the reality?

↓

How do we scientifically study public policies?

Today’s mission

• Starting point
  • What’s a (social scientific) experiment
  • What’s a good experiment

(Social scientific) experimental method

Experimental Method: a watershed

Experimental method is the watershed between traditional politics and political science, and also between humanity method and scientific method

A Political Mission

我们必须坚持解放思想、实事求是、与时俱进、求真务实，一切从实际出发，着眼解决新时代改革开放和社会主义现代化建设的实际问题，不断回答中国之问、世界之问、人民之问、时代之问，作出符合中国实际和时代要求的正确回答，得出符合客观规律的科学认识，形成与时俱进的理论成果，更好指导中国实践。 (习近平 2022)

科学研究是政治任务

Challenges from the limitation of statistics

What’s the problem？

• Causation is the phenomenon (why); causality is the mechanism (how)
  • So correlation can imply causation, but it’s not causation without evidence of causality

The target of expeirment

Discussion

Discuss with your team members and answer the following questions:

1. Find two examples of correlation but not causation
2. Find two examples of correlation which implies causation
   • And explain how you know

Theorizing the problem

How?

Ideal design for the problem

Everything everywhere all at once

我要想看这个人的职业导致不同的命运，那我就改变职业，看会不会有不同的结果

我记得成龙给我发短信说，‘祝贺你!你知道你的人是先来找我谈的’，我说，‘谢谢你，哥们儿，你可是帮了我一个大忙。’

https://www.bcbay.com/ent/2023/02/08/842770.html

Practical Solution

从个体的转换变成群体的转换

Procedure

Test	Assignment (Group T)	Assignment (Group C)	Effect
Pre	Group T	Group C	Diff(pre)
Post	Group T	Group C	Diff(treated)

Steps

1. Assignment
2. Pre-test
3. Treatment
4. Post-test
5. Comparing

Which step does the trick?

Randomization

Outcome: Balanced data in multiple dimensions

A Good Experiment

Simplifying experimental designs

Common practice (like all the time)

Type 1

1. Assignment
2. Pre-test
3. Stimulating
4. Post-test
5. Comparing

Assumption? Previously the same on the investigated variable

Type 2

1. Assignment
2. Pre-test
3. Stimulating
4. Post-test
5. Comparing

Assumption? The treatment is the only intervention.

Improvements could result from retest, regression to the mean effects, effect of intervening events (history), motivation and expectations (placebo), social contact, etc.

Validity

The approximate truth of the inference or knowledge claim (Morton and Williams 2010, 254).

Validation for experiment

Internal Validity

The approximate truth of the inference or knowledge claim within a target population studies.

• Construct validity
• Causal validity
• Statistical validity

• Construct validity: Whether the inferences from the data are valid for the theory the researcher is evaluating in a theory testing experiment
• Causal validity: Whether the relationship the researcher finds within the target population analyzed are causal
• Statistical validity: Whether there is a statistically significant convariance between the variables the researcher is interested in and whether the relationship is sizable.

External Validity

The approximate truth of the inference or knowledge claim beyond a target population studies.

Ecological Validity

Whether the methods, materials, and settings of the research are similar to a given target environment.

Source: Dall-E 3

Discussion

Evaluate the following experiment

• Question: Do people view others differently because of the language they use?
  
• Subjects: 500 students from Japan and Korea
• Design:
  1. Watch the video on the right
  2. Answer the survey about the characteristics of Nezha (masculinity, intimacy, powerfulness, literacy, etc.)

https://www.bilibili.com/video/BV1euPkerEnp/?vd_source=f38aeefd0d38cecba9017eeee43e71c8

Take-Home Points

• What’s an Experiment
  • Problem: Correlation is not causation
  • Practical solution: Group-level observation
  • Key trick: randomization
• What’s a good experiment
  • Internal validity
  • External validity
  • Ecological validity

Reference

Banerjee, Abhijit V., Shawn Cole, Esther Duflo, and Leigh Linden. 2007. “Remedying Education: Evidence from Two Randomized Experiments in India.” The Quarterly Journal of Economics 122 (3): 1235–64. https://doi.org/10.1162/qjec.122.3.1235.

Hu, Yue. 2020. “Culture Marker Versus Authority Marker: How Do Language Attitudes Affect Political Trust?” Political Psychology 41 (4): 699–716. https://doi.org/10.1111/pops.12646.

Morton, Rebecca B., and Kenneth C. Williams. 2010. Experimental Political Science and the Study of Causality: From Nature to the Lab. Cambridge University Press.

Pizzi, Elise, and Yue Hu. 2022. “Does Governmental Policy Shape Migration Decisions? The Case of China’s Hukou System.” Modern China (Thousand Oaks) 48 (5): 1050–79. https://doi.org/10.1177/00977004221087426.

习近平. 2022. “高举中国特色社会主义伟大旗帜 为全面建设社会主义现代化国家而团结奋斗.” 人民日报 (北京), October.

Bonus: Design Experiments

Design Typology

Treatment

1. Scenario ~
2. Behavior ~

Environment

1. Lab ~
2. Survey ~
3. Field ~
4. Natural ~

Lab Experiment

Hu (2020)

Subject: students or convenience sample

• Pro: Internal validity
• Con: External and ecological validity

Survey Experiment

Subject: Sample from the target population

• Pro: Ecological and external validity
• Con: (Other aspects of) internal validity

Pizzi and Hu (2022)

Field Experiment

Subject: Sample from the target population

Banerjee et al. (2007)

2019 NObel price of economics

Random experiments in schools in urban India in 2007, hiring young women, test score increase .28 sd.

• Pro: Ecological validity
• Con: (Other aspects of) internal and external validity

Natural “Experiment”

“Treatment”: Something just happened.

ShortCutsTV: https://www.youtube.com/watch?v=jC8_88OqXa4

Rural north carolina, mental health of children, 1996, casino opened, profit distributed to some family. Turns out to be effect for improving mental health

Why is natural experiments not an experimental design?

Facts You Should Know

1. Not every research in the lab is an experiment.
2. Not every experiment has to be conducted in the lab.
3. Not every method called “experiment” is an experiment.
4. Not every research question can be tested by an experiment in practice.

• what you did in chemistry in middle school.
• field, survey
• natural experiment