1 / 263

Introduction to Empirical Social Research Thomas Plümper Professor of Quantitative Social Research 2020

2 / 263

Learning Outcomes - the logic of research - theoretical foundations of empirical research - empirical foundations of theories knowledge and understanding of: - inferential strategies -> randomization of cases -> selection of cases -> randomization of treatment -> control of confounders - research designs -> observational data vs. experimental data -> inferential techniques for observational data -> regression techniques for observational data

3 / 263

Table of Contents 1. The Scientific Method 2. The Nature of Data 3. Empirical Research and the Logic of Inference 4. The (Wannabe) Gold Standard of Social Science Methodology: Experiments 5. From Survey Research to Survey Experiments 6. Observational Data Analysis 7. Causal Models 1: Matching 8. Causal Models 2: Regression Discontinuity Note: We may not have enough time to cover all aspects…

4 / 263

The Course… The course provides an introduction into the logic of empirical social science. “After all, science is a mode of inquiry rather than a belief structure.” In short: Science is a communicative process that aims at generating valid causal inferences. The scientific method has been invented to improve the validity of causal inferences. The scientific method seeks to eliminate the influence of priors (ideology, religion, etc. ) on the choice of scientific research designs and methods and on the interpretation of results. The course introduces into the logic of the scientific method and the three main principles of scientific inferences:

- random selection of observations into a sample - randomization of treatment - stratification of samples and selection of observations

5 / 263

Chapter 1. The Scientific Method

6 / 263

What are these things called ‘Research’ and ‘Science’?

7 / 263

What are these things called ‘Research’ and ‘Science’? Research is a structured process that ultimately aims at the development of valid theories – theories that help us understand the world we are living in, guide our behaviour, and allow us to manipulate outcomes.

8 / 263

What is this thing called ‘Research’? Research is a structured process that ultimately aims at the development of valid and useful theories – theories that help us understand the world we are living in, guide our behaviour, and allow us to manipulate outcomes. What does “structured” mean? The term structured process refers to the scientific method: a set of rules and ideas that (ought to) guide researchers in their work. And what is validity? Validity refers to a set of criteria that allow us to evaluate the relevance of scientific research findings. Understanding? Understanding of social phenomena (usually) requires simplification and an idea of a causal mechanism that links causes to consequences.

9 / 263

A Closer Look: The Scientific Method Wikipedia: The scientific method is an empirical method of acquiring knowledge that has characterized the development of science since at least the 17th century. It involves careful observation, applying rigorous scepticism about what is observed, given that cognitive assumptions can distort how one interprets the observation. It involves formulating hypotheses, via induction, based on such observations; experimental and measurement-based testing of deductions drawn from the hypotheses; and refinement (or elimination) of the hypotheses based on the experimental findings. These are principles of the scientific method, as distinguished from a definitive series of steps applicable to all scientific enterprises. The scientific method is a set of rules that relates theories to empirical observation (and vice versa) with the intention to learn about the validity, usefulness, and explanatory power of scientific theories. The scientific method links theories to reality (and yes, reality is a big word).

10 / 263

The Logic of Science “Science is a public process. It uses systems of concepts called theories to help interpret and unify obser-vation statements called data; in turn the data are used to check or ‘test’ the theories. Theory creation may be inductive, but demonstration and testing are deductive, although, in inexact subjects, testing will involve statistical inference. Theories that are at once simple, general and coherent are valued as they aid productive and precise scientific practice.” David F. Hendry 1980 Watch Eugenie Scott: Scientific Theories (4’42”) https://www.youtube.com/watch?v=-M1hxGj5bMg Matt Antigone: What is the difference between scientific law and theory? (5’11”) https://www.youtube.com/watch?v=GyN2RhbhiEU

11 / 263

The Scientific Method as Process

12 / 263

1. Formulation of Research Interest

Based on observation: Why do crude birth rates decline as per capita income increases? Why do poor people die earlier than rich people and men earlier than women? Why are natural disasters more costly in rich countries and more deadly in poor? Based on theory: Why don’t party platforms fail to converge in majoritarian two-party systems? Why do corporate taxes continue to exist? Why do people behave altruistically?

13 / 263

2. Theory Development A (positive) theory is the justification of the answer to a ‘why’ question. Example: birth rates decline as per capita income increases for basically two reasons: - child mortality decreases - the dependence of older people on transfers and support from their children declines

14 / 263

Theory A theory is a theory if and only if it can be wrong. Necessary Conditions for a Theory − generalizations over a category of phenomena − predictions over outcomes given a state or a change of state − consistency. In other words, if something cannot be expressed by an arrow diagram, it is not a theory. More importantly, theories must be falsifiable. A set of assumptions and their derivations is formulated in a way that cannot be falsified, is useless.

15 / 263

Theories can be Deterministic or Probabilistic A deterministic theory is universally valid for a set of cases: For all a, if x then y. A probabilistic theory is not universally valid: For all a, x increases the probability of y. Claim: All social science theories are probabilistic. Counterexample anyone?

16 / 263

Are probabilistic theories causal and does the probability matter? Yes and no. Yes: probabilistic theories are causal. Bullet in the head causes death. Probability: roughly 90 percent. Famous counterexamples: Rudy Dutschke any many more https://www.grunge.com/32897/people-amazingly-survived-gunshot-head/

17 / 263

Bullet to the Head, Bullet to the Leg Most people being shot to the head die. Most, but not all. It is also possible to die from a bullet shot to the leg: “A 25-year-old man died Tuesday morning (Feb. 19), nearly five days after he was shot in the leg in Covington, the city’s police department reported. The gunfire was reported about 11:45 p.m. Thursday at a Covington home. However, it’s unclear when or how responding officers were informed that the man had been struck; the release only states that investigators “later” learned a 25-year-old man had been shot in the leg.” A high probability does not imply a deterministic relation, but it is causal like a bullet in the head. A low probability does not preclude causality…

18 / 263

Constitutive Elements of Theories - assumptions - causal mechanism(s) - prediction(s) (hypotheses) In the social sciences, assumptions and mechanisms evolve around the average behaviour of individuals.

19 / 263

1. Assumptions In social science theories, assumptions are made about the nature of the world and the nature of human beings. Assumptions serve the purpose of simplification. Assumptions are not right or wrong. They are useful to a degree. Simplification is always costly. But not simplifying is the most costly strategy as it leads to nothing.

20 / 263

Example: Rational Choice Theories assume that human beings maximize their own utility. That should be an uncontroversial tautology. Whatever human beings do, they maximize their own utility.

21 / 263

However, rational choice theories stir a lot of debate. Many social scientists believe that rational choice assumptions - are wrong, or - simplifying, or - oversimplifying. Wrong. Depends on how we write the utility function. We can write it in ways so that the maximizing assumption is consistent with any behaviour. -> RCT is an empty shell. Fill it with content. -> Simplifying. Yes, but that is a good thing. -> Oversimplifying. Maybe. But then we need to show that a more realistic, complex assumptions explains the same phenomena better or more phenomena equally well.

22 / 263

The Principle of Parsimony or, Okkham’s Razor https://www.youtube.com/watch?v=M5WDdvkFaDg

23 / 263

2. Causal Mechanisms A causal mechanism is the answer researchers give to a ‘why’ question: Why does a bullet to the head cause death? https://www.scienceabc.com/humans/gunshot-to-the-head-does-this-mean-instant-death.html in short, the answer is: shock, blood loss, swelling of the brain and consecutive brain damage

24 / 263

Some more Questions in Search of a Causal Answer Why do crude birth rates decline as societies become wealthier? Why are divorce rates higher in urban areas? Why do poorer countries have (ceteris paribus) higher growth rates? Why are wars between democracies extremely rare? Why do extremist parties increase their vote share? All answers require a causal mechanism. However, importantly, suggesting a causal mechanism does not make our theory right. We need to test it – and we need to test it independently of the data that we may have used to develop the theory in the first place.

25 / 263

3. Predictions Predictions are the logical (consistent) consequences of derivations from assumptions and causal mechanism. If the theory is correct, then we expect the following (observable?) outcomes. Example: Einstein predicts …

26 / 263

Einstein 1: Time Dilution of Light Light sent down into a gravity well is blueshifted, whereas light sent in the opposite direction (i.e., climbing out of the gravity well) is redshifted; collectively, these two effects are known as the gravitational frequency shift. More generally, processes close to a massive body run more slowly when compared with processes taking place farther away; this effect is known as gravitational time dilation. Gravitational redshift has been measured in the laboratory and using astronomical observations. Gravitational time dilation in the Earth's gravitational field has been measured numerous times using atomic clocks, while ongoing validation is provided as a side effect of the operation of the Global Positioning System (GPS).

27 / 263

Einstein 2: Light Deflection General relativity predicts that the path of light will follow the curvature of spacetime as it passes near a star. This effect was initially confirmed by observing the light of stars or distant quasars being deflected as it passes the Sun. This and related predictions follow from the fact that light follows what is called a light-like or null geodesic—a generalization of the straight lines along which light travels in classical physics. Such geodesics are the generalization of the invariance of lightspeed in special relativity.

28 / 263

Einstein 3: Gravitational Waves Predicted in 1916 by Albert Einstein, there are gravitational waves: ripples in the metric of spacetime that propagate at the speed of light. These are one of several analogies between weak-field gravity and electromagnetism in that, they are analogous to electromagnetic waves. On February 11, 2016, the Advanced LIGO team announced that they had directly detected gravitational waves from a pair of black holes merging.

29 / 263

Einstein 4: Precession of Apsides In general relativity, the apsides of any orbit (the point of the orbiting body's closest approach to the system's center of mass) will precess; the orbit is not an ellipse, but akin to an ellipse that rotates on its focus, resulting in a rose curve-like shape (see image). Einstein first derived this result by using an approximate metric representing the Newtonian limit and treating the orbiting body as a test particle.

30 / 263

And many more… Why do we think Einstein is a great scientist? Because he made predictions based on very little observation and a great but also very general theory. Relativity theory does not make just one prediction, it made many predictions. And then the predictions were supported by evidence. It is not equally scientifically important to formulate a theory that predicts the known. (Sometimes we call it postdict…)

31 / 263

Note - theories are more useful if they make more predictions - theories can be tested in a more convincing fashion if they make more predictions In the social sciences, many theories make one prediction…

32 / 263

4. Testing Theories, testing Causal Mechanisms, testing assumptions, or testing Predictions? In principle, a test of a theory should provide evidence for or against the existence and relevance of the causal mechanism. However, we are often only able to test the predictions theories make based on their assumptions. Underdetermination (Quine and Duhem): Every empirical observation is consistent with more than one theory. Empirical support for one theory does not falsify another, potentially competing theory.

33 / 263

How do Scientists test Theories? A theory test is a systematic (that is method-based) investigation of whether the real world behaves as predicted by the hypothesis. Methods: - experiments field lab survey - analysis of observational data regression analysis descriptive statistics causal analysis

34 / 263

5. Interpretation of Empirical Results - criteria for ‘predictions are correct’ significance identification replicability robustness - criteria for the choice of analysis power unbiasedness efficiency criteria for the interpretation of findings generalization causal homogeneity or heterogeneity plausibility?

35 / 263

The Purpose of the Scientific Method The scientific method has been invented to eliminate the influence of - religion, - believes, - preferences, - ideology, - values, and - other subjective priors on scientific results and scientific knowledge.

36 / 263

A Very Short Philosophy of Science Introduction into the ‘Falsification versus Verification’ Issue Induction is not deductively valid. -> No empirical observation that confirms a theory proves the validity of the theory for other cases. -> Generalization to all other cases requires a deterministic theory and knowledge of the population boundaries. -> Generalization of probabilistic causal relations remains, well, probabilistic. Therefore, predictions and forecast on single cases cannot be made with certainty if theories are probabilistic. Define Population: A population is the set of observations of the same phenomena for which a theory claims to be valid.

37 / 263

An Illustrative Example: Kepler’s Moons show that Induction is not deductively valid

In 1726, Swift Johannes Kepler predicted the existence of two Mars moons. In 1877 these two moons were actually detected. One of these moons is now called Swift, the other one Voltaire (scientifically, they are called Phobos and Deimos), because Voltaire made an identical prediction in 1750. Causal Mechanism and Prediction: Well, Kepler predicted two moons because in 1726 it was known that the Earth has one moon and Jupiter four moons. Since Kepler believed in the symmetry of a god-given universe, he predicted the existence of two Mars moons. Indeed, according to Kepler, the next planet, Saturn, should have 8 moons – it has five, Uranus should have 64 moons – and has five as well. Hence, no symmetry. Correct prediction (in one case), but wrong theory nevertheless.

38 / 263

Popper… …therefor suggested that scientists should try to falsify theories. He also seems to believe that a single deviant case falsifies a theory.

Lakatos… Suggests that Popper is wrong. Falsification is not as informative as Popper suggests, since every theory makes auxiliary assumptions. Hence, theories can be correct, but make wrong predictions because they rely on wrong auxiliary assumptions.

Bayesian Philosophy of Science… verification is informative and should change your prior. If before you have seen support for a theory you believed that a theory is correct with 80 percent, you should increase this probability, but perhaps not to full certainty (100 %). Likewise, any rejection of a theory does not prove a theory wrong. One should just adjust the prior downwards, but perhaps not to 0 percent.

39 / 263

Summary Theory

- a theory derives predictions from assumptions (usually) relying upon a causal mechanism - predictions are either deterministic or probabilistic - if predictions are deterministic, a single deviant case falsifies them - if predictions are probabilistic, single deviant cases are uninformative - thus: more complicated criteria are needed telling us when a probabilistic theory is falsified (known

to be wrong)

40 / 263

Concepts to Understand - research - science - scientific method - theory probabilistic deterministic - assumptions - causal mechanism - prediction (hypothesis) - methods - verification - falsification

41 / 263

More Complex Questions to Discuss

(please answer one of these questions in one paragraph, the best answers will be published on LEARN) What is our expectation on outcomes in the absence of any causal mechanism, that is: if everything is random? Do causes have identical effects in all cases/circumstances? Are causal effects stable over time and space? How are positive (predictive) and normative (prescriptive) theories related to each other?