Experimental Methods

# Experimental Methods
## Public Policy Analysis: Perspectives and Methods (30700953)
### Yue HU

---

## Overview

1. What  .red[can] social-scientific experiments do?
1. What's a .red[good] experiment?
1. .red[How] to design a political experiment?

---

## What can experiments do?
# Causal Inference

---

## A Direction of Social Science' Future

- Causal effect
- Causal mechanism
]

???

Three paradigms to study causality: Rubin's model, iv, sem

Discussed with Per Johansson

- Experiment
- Quasi-experiment
]

---

## Causal Effect and Counterfactual Logic

.bg-black.golden.ba.shadow-5.ph4.mt3[
.center[Neyman-Rubin Causality Model (RCM)]
]

???

Captian carter

---

## "Empiricalize" Counter-Facts

Unable to observe individual counter-facts simultaneously   
&dArr;

Group comparison    
&dArr;

???
  
.red[Average] treatment effect on the treated (ATT): `\(ATT = \mathbb{E}(\delta_i | T_i = 1).\)`

`\(ATE(W) = \mathbb{E}(\delta_i|W_i); \boldsymbol Y \sim \boldsymbol W\boldsymbol\beta.\)`     
`\(ACTE(W) = \mathbb{E}(\delta_i | W_i, T_i = 1)\)`

ATE estimates the causal effect of treatment on a randomly drawn individual in the population
ATT estimates the causal effect of treatment on a randomly drawn untreated individual.

### Stable Unit Treatment Value Assumption (SUTVA)

SUTVA is .red[THE assumption]；No SUVA &rArr; no causal inference

???

Sut-va

+ `\(T_i\)` .red[only] affects `\(Y_i\)`.
+ In estimating ATE or ATT, the treatment is .red[homogeneous] across individuals.
+ `\(T_i\)` is .red[invariant] with respect to the mechanism by which the treatment is provided.
+ All possible states of the world are .red[observed].
+ The only causality question of interest is a .red[historical] one for the dataset analyzed.
+ The only causality question of interest is one in which treatment is received .red[before] a choice is made; there are no simultaneous relations between treatment and choice.

---

## Advantages of RCM

1. Causality `\(\leadsto\)` causal effects of single variables
1. Defining causality
1. Making causality observable and detectable

???

Neyman-Rubin causality theory

This is what experiments .normal[can] do.

---

## What's a .red[good] experiment?
# Evaluation of Experimental Design

---

## Components

1. Subjects + Stimulus `\(\leadsto\)` Intervention

???

1. 研究者必须首先明确实验的目标人群以及核心干预因素，然后据此决定被试的来源和对其进行的干预模式。

1. Reducing observed and unobserved confounders

???

1. 研究设计必须充分考虑对可观察的干扰变量的控制，并通过随机分配控制组和实验组等方法对不可见干扰变量和难以控制的可见干扰变量进行进一步的剔除。

1. Control vs. treatments
    + Existence, magnitude, direction
    + Pre - post

???

1. 研究者需要对控制组和实验组在接受干预因素影响前后的态度和行为表现进行比对，并根据统计原则进行跨组比较，据此判断干预因素与行为态度之间的因果关系是否存在。

---

## Three Common Strategies of Experimental Design

.center[
<img src="../../guestLecture/images/lab_lab.png" width = 33% />
<img src="../../guestLecture/images/lab_field.jpg" width = 33% />
<img src="../../guestLecture/images/lab_survey.jpg" width = 32% />
]

---

## Goodness of Experimental Design

???

Do the respondents' answer you want to know

---

The approximation between the inference and truth

Inference &harr;

- Subjects
- People beyond the subjects
- Real world
]

---

## Dimensions of Validity

- Constructive
- Causal
- Statistical
]
    
???

* 内部效度: 目标人群
    + 建构性：是否能合理且有针对性地评鉴目标理论
    + 因果性：Identification
    + 统计性: 核心因、果因素间具有显著且稳定的统计关系

(Subjects, Population&rbrack;

]

???

* 外部效度: 超越目标人群 (Hint:目标人群可不一定就是总体人群)

???

* 生态效度: 与目标人群在非实验环境中行为决策的一致性
    + *并不关注实证推断与真实规律的联系*，而考察的是研究环境与实际环境的相似程度

???

两个大小不同的正方型 vs. 一个正方形，一个三角形

效度是个整体概念，不能孤立看待!

---

## Misunderstandings of Validity

### "Experiments are valid"
]

### Crisis of Replication

+ Top three in Psychology（40%/100, Open Science Collaboration 2015）
+ Top thirteen in Economics（50%/67, Chang & Li 2015）
+ Top three in Poli Sci（Solt, Hu, et al. 2016 & 2017; Tai, Hu, Solt 2021; Xu 2021）

]

???

Xu Yiqing 对65篇政治学论文的复刻

---

## Research Opportunity I

Experiment Replication

- Original research project
- Less challenge for junior scholars
- Publication opportunity (Chinese)
- Research experience and certificate

]

- An 8-month project
- Some basic skills of quantitative analysis
- Public administration
]

- 1 RA
- Graduates/Senior undergrads
- No immediate deadlines

]

---

## Research Opportunity II

Theory and practice of validity

- Student friendly
- Publication opportunity (Chinese)
- Research experience and acknowledgement
- DCPO project

]

- A 3-month project
- English and Chinese literature
]

- 1 RA
- Undergrads with interest in social science
- No immediate deadlines

]

---

## Misunderstandings of Validity

### "Experiments are valid"
### "Randomization provides validity"
]

In a model with .red[ten] variables, let &alpha; = 0.05, the probability of randomization failure:

Data imbalance &rArr; low causal & statistical validity!
]

]

???

Morgan, Kari Lock, and Donald B. Rubin. 2012. “Rerandomization to Improve Covariate Balance in Experiments.” *The Annals of Statistics* 40(2): 1263–82.

---

## Misunderstandings of Validity

.left-column[
### "Experiments are valid"
### "Randomization provides validity"
### "External" validity
]

1. **Subject bias**

> "Subjects are always students. It has zero meaning to use results on them to estimate the population"

The key judgement should be who's the population for research:

- If students, it's generally ok
- If a broader strategy, this is an .red[internal] instead of external problem

]

.footnote[
.blue[Trivia]: Empirical research found that there are not that much difference between students and social members at least on certain issues.
]

???

EGAS M, RIEDL A：《The Economics of Altruistic Punishment and the Maintenance of Cooperation》：载《Proceedings of the Royal Society of London B: Biological Sciences》2008年第1637期第871–878页。

KAM C D, WILKING J R, ZECHMEISTER E J：《Beyond the ’Narrow Data Base’: Another Convenience Sample for Experimental Research》：载《Political Behavior》2007年第4期第415–440页

---

## Misunderstandings of Validity

### "Experiments are valid"
### "Randomization provides validity"
### "External" validity

]

1. **Subject bias**
1. **Experimental effects/experimenter effects**

> In most cases, the subjects know when the experiments start and end, which may affect their choice and judgements, and thus make different decisions from what they do in real life.

- *Ecological* rather than "external" validity
- It may becomes an advantage, e.g., Bassi (2011)

]

???

e.g., Anna Bassi （2011）等对于选民身份与投票意愿地研究中，被试者的选民身份就没有参照现实或进行随机分配，而是根据形式模型的条件进行了人为划分，从而实现了对不同特征群体内选民行为规律的全面考察。

---

## .red[How] to design a political experiment?
# Designing "High-High" Experiments

???

High internal and high external validity

---

## Improving Internal Validity

- Replicating experiments

- Randomization + covariates

- Removing confounders
    - Observable: EEG, fMRI
    - Unobservable: Compensation, time and environment setting, IAT, eye-tracking

- Improving constructive validity: Formal theory approach (FTA)

???

functional 脑部核磁共振(fMRI)，脑电图(EEG)， implicit association test (IAT)

---

## Improving External Validity

- Reducing subject bias
    - Examining on diverse subjects
    - Delicate research strategy, e.g., the least-likely case design

- Reducing experimental effect
    - Deception
    - Stimuli from the real life
    - Mimic real environment
    - Compensation

---

## Rethink of the Three Common Strategies

What are the fundamental differences?

Time, space, subjects...?

---

## Sensitive Question Technique

Common strategies:

Endorsement experiment    
.small[(Exploiting evaluation bias)]

List experiment   
.small[(Aggregation)]

Randomized response   
.small[Adding random noise]
]

???

Endorsement: In the control group, respondents are asked to evaluate some relatively uncontroversial issue or object (e.g., rate a policy on a Likert scale). In the treatment group, that issue or object is associated with the sensitive item before being evaluated (e.g., the same policy is said to be endorsed by a controversial political group).

Randomized response: Using a coin to determine whether to answer the question or a given answer

Rosenfeld et al. 2014: The endorsement experiment and the randomized response yield least bias, but the estimates based on the endorsement experiment (with a single item) are noisier than other indirect questioning methods.

---

## Application

Core Topic: Who watch state-media broadcast and why?

+ Objective: State media (Xinwen Lianbo)

+ Problem: sociopolitical desirability &rarr; sensitive questions

.footnote[
Hu, Yue, and Zijie Shao. 2022. “What Drives Chinese Internet Users to Watch State-Media Broadcasts? An Audience Analysis.” *Journal of Chinese Political Science* 27(1): 19–40.
]

+ Solution: List experiment + A general sense of political desirability
    
???

A: 王刚、张强、刘爽、赵芳
B: 王刚、张强、刘爽、赵芳、买买提·阿卜力孜

对于官媒总体禁忌，对所有相关问题的影响

---

## Experiment Setting

1. Random assignment
1. Answer the question:

> Which of the following statements you agree with? Please answer the total number instead of exactly which one:

---

A. The radio audience is very small in nowadays society.

B. I doubt about the audience rate of the Spring Gala. There shouldn't be that many people still watching it.

C. The real name registration on Weibo does not affect me much.

D. All the newspapers published in the mainland China are consistent with *People's Daily*
]

A. The radio audience is very small in nowadays society.

B. I doubt about the audience rate of the Spring Gala. There shouldn't be that many people still watching it.

C. The real name registration on Weibo does not affect me much.

D. All the newspapers published in the mainland China are consistent with *People's Daily*

E. The reports in Xinwen Lianbo is credible.
]

---

Further question: How to apply this result to correct survey answers?

Several ways......

---

## Mapping Experimental Method in Social Science
In terms of the goal...

---

In terms of the approach...

- **Experimental methods**

- Quasi-experimental methods
    - Large N
    - Bayesian
    - Big data
]

- Interviews
- Process tracing
- Critical joint
- QCA
]

---

background-image: url("images/exp_mindmap.png")
background-position: center
background-size: contain

## Take-Home Points

---

## Procedure

The presenters talk.

### Challenging (10min)

Drawing the Luck Dog

]

*Process*

1. Presenters query.
    - A question about the paper
1. Challengers respond.
1. Challengers query.
    - A question about the presentation
1. Presenters respond.
]

---

background-image: url("images/cha_challenge.gif")
background-position: center
background-size: contain