Scientific Approach to Understanding Policies

Understanding Policies (10700193-90)

Yue Hu

Tsinghua University

Overview

Goal: Understand the scientific approach to analyzing policies

• Dissect: What does being “scientific” mean
• Process: How to conduct a scientific inquiry
• Apply: When (not why) understanding policies scientifically

Dissect Science

Paradigms For Understanding Policies

Paradigm: a bag of theories, research methods, postulations, and standards for what constitutes contributions to a field.

• Positivist:
  • Evidence-based
  • Problem-solving through systematic queries
• Post-positivist
  • Culture-based (norms, morality, values)
  • Normative, ethical process

Dominant, 实证主义

Re-imported from the French word positivisme(/positee vish men/), derived from positif in its sense of ‘imposed on the mind by experience’.

Being positivist = being scientific?

A Story

A TCM¹ 🧑‍⚕️told a story about drinking water:²

每天早晨喝一杯凉水，是一个错误的养生做法……[某大历害老医师]是偶然发现这个问题的，当时他在给一位女士调理身体，本来调理得很好……突然有一天，再诊脉的时候，发现她的舌质又淡白了，舌体胖大，病情反复，忙问这是怎么回事儿？她回答，半个月前，看到电视里面有养生专家让晨起喝凉水，于是坚持了这些天，已经腹泻了很多次了。这么样，损失了阳气，导致水湿重。有位中医也被这一说法骗了……最后竟然收集了几百例的病例，都是晨起喝凉水，导致的各种复杂古怪的病症的。

1. Traditional Chinese Medicine, TCM

2. 《为什么外国人喝凉水没事》之“每天早晨喝一杯凉水，是一个错误的养生做法”

Question

• Will the seniors in your family be convinced by this piece?
• Is it a scientific inquiry?

Let’s Decompose It

Problem

“给一位女士调理身体，本来调理得很好……突然有一天，再诊脉的时候，发现她的舌质又淡白了，舌体胖大，病情反复。”

Theory

“损失了阳气，导致水湿重。”

Evidence

“晨起喝凉水，于是坚持了这些天，已经腹泻了很多次了。”

Conclusion

“每天早晨喝一杯凉水，是一个错误的养生做法”

Why Convincing

It’s a human inquiry: a question/puzzle -> point + evidence

Example

Q: Why do you Sichuanese/Jiangxiness/Huanese like to eat spicy food?

A: Our place is humid. Spices can “祛湿.”

后面有解释

Another Example

“你递啊，你伸啊！”

The video shows that the toll collector and the driver were in a standoff for 1 minute.

Question

Why aren’t above inquires not scientific inquiries?

\[ \text{Human inquiry} = Point + Evidence. \]

\[ \text{Scientific inquiry} = \color{red}{Procedure} (Point + Evidence). \]

Process Science

Why not stay with human inquiries

湃客(2020)

王静爱、左伟(2010)

• 湃客. 2020. 《如果吃辣有段位，你的家乡排第几？》. 澎湃号. https://www.thepaper.cn/newsDetail_forward_8315589 (2025年2月23日).
• 王静爱和左伟. 2010. 中国地理图集. 中国地图出版社. https://book.douban.com/subject/4733762/ (2025年2月23日).

https://www.bilibili.com/video/BV17RAgeoEvs/?spm_id_from=333.1387.favlist.content.click&vd_source=f38aeefd0d38cecba9017eeee43e71c8

Risks of human inquiry

• Incomplete information
• Illogical reasoning
  
• Observation selection
  
• Inaccurate observation
  
• Overgeneralization
  
• ……

Human → Scientific Inquiries

Guess/Hypothesis

Data

• Be Valid
  • Being representative
• Be Sufficient
  • Being diverse

Analysis

• Be Proper & consistent

• Point
  → Data + Method → Analysis
  ⇒ Evidence/conclusions

• Guess: Charle’s Angle, 2019, Columbia Pictures, Elizabeth Banks’ film
• Data: Solving the incomplete, selective, or inaccurate observation
• Analysis: Solving the overgeneralization and illogical reasoning

Scientific Procedure

Three (or Four) Steps¹

1. Lay out (assumptions & hypotheses)
2. Description
3. Explanation
4. (Prediction)

Two Approaches

Inductive vis-a-vis Deductive

1. No need to complete in one piece.

Group discussion

Scientific inquiry:

• Being valid
• Being sufficient
• Being proper & consistent

Why can the scientific procedure reduce human-inquiry risks?

Scientific inquiry addresses each risk of human inquiry through three requirements:

1. Being valid (representative data) → counters selective observation and inaccurate observation: by requiring data to be representative of the population rather than cherry-picked, it prevents researchers from drawing conclusions from a biased or unrepresentative sample.

2. Being sufficient (diverse data) → counters incomplete information: by requiring diverse data points, the scientific procedure ensures coverage across different cases and conditions, preventing conclusions drawn from too narrow a base.

3. Being proper & consistent (rigorous analysis) → counters illogical reasoning and overgeneralization: by requiring that methods be appropriate to the research question and applied uniformly, the procedure enforces logical discipline—conclusions must follow from evidence through explicit, replicable steps rather than intuition or informal leaps.

Together, these requirements transform subjective human impressions into intersubjectively verifiable knowledge: the hypothesis-data-analysis chain forces transparency at every step, allowing others to check and challenge the reasoning.

Apply Science

What can science do

• Give an answer? No.¹
• Win an argument? [Not exactly, but]
  • Provide acceptable evidence
  • Win an argument among scientists

1. A true story in the public policy area.

Social science

1. Can human behavior being scientifically researched?
2. Can social scientific results scientifically reliable?
3. Is social science different from natural science?

When do policymakers need science

Depends on what kind of conclusion s/he wants to make:

Credible conclusions vs. Strong conclusions

The Law of Decreasing Credibility

Strength of assumptions ↑ ⇒ credibility ↓

• 只要有疫苗了，我们就安全了 vs. 疫苗是战胜新冠病毒之制胜关键

Power and science

Ranges are for cattle. Give me a number.

Lyndon Johnson, the 36th POTUS

Jerry Hausman, MIT economist, Durbin-Wu-Hausman test

You can’t give the client a bound. The client needs a point.

Johnson use the anecdote to reply an economist’s description of the uncertainty of his forecast

Both being considered and being solid can be important

Question

What’s your choice?

Take-Home Points

• Dissect Science: What does being “scientific” mean
  • \(\text{Scientific inquiry} = \color{Procedure}{red}(Point + Evidence).\)
• Process Science: How to conduct a scientific inquiry
  • Why human inquiries can be risky?
  • Steps of processing science
• Apply Science: When (not why) understanding policies scientifically
  • Social science
  • Policymakers’ view

Four steps:

1. Lay out (assumptions & hypotheses)
2. Description
3. Explanation
4. (Prediction)

Two approaches: deductive and inductive

Challenge is on the way

Who are the lucky dogs today??

• Your missions
  • Hansen(2019)
• Presenter:
  • Content
  • Connection
  • Inspiring
  • & A question: Examining if the challengers do read the article
• Challengers:
  • A question: Testing whether the presenters understand the article thoroughly

Appendix: Public policy in the real world

A story about problems brought by science

U.S. Congress, 2010: Congress debated the biggest healthcare reform in decades

• ~50 million Americans had no health insurance
• Democrats proposed the Affordable Care Act (ACA), popularly known as “Obamacare”
• Central political battleground: Would it cost the government money or save it?

The Patient Protection and Affordable Care Act was signed by President Obama on March 23, 2010, followed by the Health Care and Education Reconciliation Act days later. Together they expanded insurance coverage, introduced insurance market regulations, and created subsidies for low-income Americans. The cost question was pivotal: a bill projected to blow up the deficit would face far greater political resistance.

Who does the math?

Before Congress votes on major legislation, it must be officially “scored” — i.e., its fiscal impact estimated

CBO (Congressional Budget Office)

• Non-partisan agency of Congress
• Produces official cost estimates for all major bills
• Both parties are supposed to trust its numbers

JCT (Joint Committee on Taxation)

• Non-partisan; handles revenue/tax side of scoring
• Works alongside CBO on healthcare legislation

The CBO was created in 1974 precisely to give Congress an independent, non-partisan source of economic data — a counterweight to the executive branch’s Office of Management and Budget (OMB), which serves the President. Its director at the time was Douglas Elmendorf, an economist who later became dean of Harvard Kennedy School.

Two experts, two conclusions

Enacting both pieces of legislation…would produce a net reduction of changes in federal deficits of $138 billion over the 2010–2019 period as a result of changes in direct spending and revenue.

Douglas Elmendorf, CBO Director (2009–2015)

The health care reform legislation would raise, not lower, federal deficits, by $562 billion.

Douglas Holtz-Eakin, former CBO Director (2003–2005)

A gap of 138 + 562 = $700 billion 😱

Douglas Holtz-Eakin directed the CBO from 2003–2005 and was chief economic policy adviser to Senator John McCain’s 2008 presidential campaign — putting him firmly in the Republican camp. His estimate was published by the American Action Forum, a conservative think tank he founded. Both men are credentialed economists who formerly led or directed the same office, yet arrived at opposite conclusions.

Same data, opposite answers

• Both analyses used publicly available legislative text and economic data
• Key differences in assumptions:
  • Whether certain spending cuts in the bill would actually be implemented
  • How to treat revenues from Medicare provisions
  • Baseline projections for healthcare cost growth
• Neither side fabricated numbers—they made different modeling choices

Holtz-Eakin’s critique: CBO took the bill’s stated provisions at face value — even politically unrealistic ones (e.g., sustained Medicare payment cuts). He “corrected” for what he called budget gimmicks.

Why CBO’s convention is still legitimate (if students push back):

• Role: CBO scores policy choices, not political will — if it second-guessed Congress, it would be making policy
• Discipline: “What changes if this law passes vs. baseline?” — injecting political predictions contaminates the comparison
• Anti-gaming: if CBO discounted provisions, legislators could write aggressive savings they never intend to keep
• Humility: predicting which provisions survive future politics is outside CBO’s expertise — being wrong destroys credibility

The real problem with Holtz-Eakin: his “realism correction” was itself a political judgment (assuming Democrats wouldn’t enforce their own bill) dressed as economic analysis.

The Essence of the problem

All such estimates are uncertain and that actual future impacts could differ significantly from the estimates of any given organization.