the role of statistics in promoting human rights: without data, all you are is just another person...
TRANSCRIPT
The role of statistics in promoting The role of statistics in promoting human rights:human rights:
Without data, all you are is just Without data, all you are is just another person with an opinionanother person with an opinion
AAAS Science and Human RightsAAAS Science and Human RightsMary W. GrayMary W. Gray
January 11, 2111January 11, 2111
[email protected]@american.edu
Why statistics?Why statistics?
There are three kinds of lies …lies, damn lies, and statistics.There are three kinds of lies …lies, damn lies, and statistics.Benjamin DisraeliBenjamin Disraeli
Numbers do not lie, but they have the propensity to tell the truth with intent to Numbers do not lie, but they have the propensity to tell the truth with intent to deceive.deceive.
The death of one man is a tragedy. The death of millions is a statistic. Stalin, The death of one man is a tragedy. The death of millions is a statistic. Stalin, comment to Churchill at Potsdam, 1945comment to Churchill at Potsdam, 1945
If you want to inspire confidence, give plenty of statistics. It does not matter that If you want to inspire confidence, give plenty of statistics. It does not matter that they should be accurate, or even intelligible, as long as there is enough of them. they should be accurate, or even intelligible, as long as there is enough of them.
Lewis Carroll Lewis Carroll
Far better an approximate answer to the right question, than the exact answer to Far better an approximate answer to the right question, than the exact answer to the wrong question.the wrong question.
John TukeyJohn Tukey
Why use statistics?Why use statistics?
What can we say and do to help human rights researchers and What can we say and do to help human rights researchers and activists?activists?
Statistics can provide the information needed to answer questionsStatistics can provide the information needed to answer questionswhere should a new clinic be locatedwhere should a new clinic be located
Statistics can suggest questions that should be askedStatistics can suggest questions that should be askedwhat is the prevalence of infant mortalitywhat is the prevalence of infant mortality
Statistics can tell us how probable certain results are or will beStatistics can tell us how probable certain results are or will beis it credible that figures offered by the government are correctis it credible that figures offered by the government are correct
Statistics can assist us to make predictions or estimatesStatistics can assist us to make predictions or estimatesoutcome of an election, immunization coverageoutcome of an election, immunization coverage
Statistics can help us come to conclusionsStatistics can help us come to conclusions what needs to be done about a problemwhat needs to be done about a problem
Can you believe what you read in the Can you believe what you read in the newspapers?newspapers?
What Happened to Skepticism?What Happened to Skepticism?
New York TimesNew York Times front page headline 21 May 2009: front page headline 21 May 2009: ““1 in 7 Detainees Rejoined Jihad, Pentagon Finds”1 in 7 Detainees Rejoined Jihad, Pentagon Finds”
The first paragraph reported that an unreleased Pentagon study had concluded that about one The first paragraph reported that an unreleased Pentagon study had concluded that about one in seven of the 534 prisoners already transferred abroad from Guantanamo had “returned in seven of the 534 prisoners already transferred abroad from Guantanamo had “returned to terrorism or militant activity.”to terrorism or militant activity.”
The article failed to distinguish between former prisoners suspected of new acts of terrorism—The article failed to distinguish between former prisoners suspected of new acts of terrorism—more than half the cases—and those supposedly confirmed to have rejoined jihad against more than half the cases—and those supposedly confirmed to have rejoined jihad against the West. Had only confirmed cases been considered, 1 in 7 would be changed to 1 in 20. the West. Had only confirmed cases been considered, 1 in 7 would be changed to 1 in 20.
Now the recidivism rate is reduced from 14% to 5%. But is it recidivism at all? Were any of the Now the recidivism rate is reduced from 14% to 5%. But is it recidivism at all? Were any of the released inmates ever engaged in terrorist or militant activity before their imprisonment? released inmates ever engaged in terrorist or militant activity before their imprisonment? Or were they radicalized while in Guantanamo?Or were they radicalized while in Guantanamo?
So, clearly the So, clearly the New York TimesNew York Times reporters and editors should have been skeptical, but so should reporters and editors should have been skeptical, but so should the readers—and in fact, many were and the paper’s Public Editor ended up writing a the readers—and in fact, many were and the paper’s Public Editor ended up writing a critical column on the issue.critical column on the issue.
Could this be why not?Could this be why not?
““DoonesburyDoonesbury” ” Washington Post Washington Post 11 July 2008 11 July 2008
If it seems too good/bad to be true, If it seems too good/bad to be true, it probably isn’tit probably isn’t
The Kuwaiti incubators story.
Were there so many premature babies in Kuwait in 1991 that 312 incubators were in operation?
More skepticism: “The Deadly Toll of More skepticism: “The Deadly Toll of Abortion by Amateurs”Abortion by Amateurs”
This was the headline on 2 June 2009. The story reported that for every 100,000 births in This was the headline on 2 June 2009. The story reported that for every 100,000 births in Tanzania, 950 women die. But nowhere do we see how many of the deaths are Tanzania, 950 women die. But nowhere do we see how many of the deaths are attributable to botched abortions.attributable to botched abortions.
We are told there are 19 million “unsafe” abortions a year and 70,000 deaths attributable to We are told there are 19 million “unsafe” abortions a year and 70,000 deaths attributable to them (presumably worldwide). This is, we learn further, is 13% of all maternal deaths.them (presumably worldwide). This is, we learn further, is 13% of all maternal deaths.
World population 6,800,000,000 World population 6,800,000,000 women 15 t0 49 1,760,000,000women 15 t0 49 1,760,000,000 Rate of “unsafe” abortions worldwide 19/1760 = 1%Rate of “unsafe” abortions worldwide 19/1760 = 1%Is it credible that each year 1 woman in every 100 worldwide has an “unsafe” abortion?Is it credible that each year 1 woman in every 100 worldwide has an “unsafe” abortion?
By extension of the worldwide figures on maternal deaths, we might conjecture that 123 (of By extension of the worldwide figures on maternal deaths, we might conjecture that 123 (of every 100,000 births) of the maternal deaths in Tanzania arise from “unsafe” abortions if every 100,000 births) of the maternal deaths in Tanzania arise from “unsafe” abortions if Tanzania is an “average” country. But is it? Or are all 950 deaths referenced above due to Tanzania is an “average” country. But is it? Or are all 950 deaths referenced above due to unsafe abortions, making the situation particularly dire there?unsafe abortions, making the situation particularly dire there?
Maternal deaths and unsafe abortions, Maternal deaths and unsafe abortions, continuedcontinued
In what may be intended as an explanation of the situation in Tanzania, we are told In what may be intended as an explanation of the situation in Tanzania, we are told that the use of contraception is 25% in Tanzania, but 39% in Kenya, and 60% in that the use of contraception is 25% in Tanzania, but 39% in Kenya, and 60% in South Africa (where abortion is legal). But no mortality rate is given for these South Africa (where abortion is legal). But no mortality rate is given for these countries.countries.
So what are we to conclude about “unsafe” abortion, use of contraception and So what are we to conclude about “unsafe” abortion, use of contraception and death rates?death rates?
Well, “unsafe” abortions are not good, in Tanzania or anywhere else, but without Well, “unsafe” abortions are not good, in Tanzania or anywhere else, but without statistics, I’m just another person with an opinion.statistics, I’m just another person with an opinion.
Advice to researchers: Advice to researchers: Some sources of dataSome sources of data
OnlineOnline
www.worldbank.orgwww.worldbank.org
www.unstats.un.org/demographicwww.unstats.un.org/demographic
www.census.gov, www.bls.gov, www.cdc.gov, www.nih.gov,www.census.gov, www.bls.gov, www.cdc.gov, www.nih.gov,www.stat.can.gc.can. www.statistics.gov.ukwww.stat.can.gc.can. www.statistics.gov.uk
www.who.int/research/en/www.who.int/research/en/
www.nber.org/links/data.htmlwww.nber.org/links/data.html
University data sources, e.g., University of MichiganUniversity data sources, e.g., University of Michiganwww.lib.umich.edu/govdocs/stforeig.html#compwww.lib.umich.edu/govdocs/stforeig.html#compThis source lists general international sources as well as data sources for This source lists general international sources as well as data sources for
individual countriesindividual countries
Other sources of dataOther sources of data
Census dataCensus dataGenerally collected by government agencies (so look for bias)Generally collected by government agencies (so look for bias)Worldwide compilations by IGOs and others Worldwide compilations by IGOs and others
Financial, budget, accounts recordsFinancial, budget, accounts recordsChurch records, Land recordsChurch records, Land records
Survey dataSurvey dataGovernments, IGOs, NGOsGovernments, IGOs, NGOsAcademic researchersAcademic researchersCommercial entitiesCommercial entities
Observational dataObservational dataRetrospective case studiesRetrospective case studies
Experimental dataExperimental dataClinical trialsClinical trials
Considerations when seeking dataConsiderations when seeking data
What is the age of the data? What is the age of the data?
Where did they come from?Where did they come from?
In what medium were they originally produced? In what medium were they originally produced?
What is the geographical coverage of the data?What is the geographical coverage of the data?
Does the data seem logical and consistent?Does the data seem logical and consistent?
In what format are the data kept?In what format are the data kept?
How were the data checked?How were the data checked?
Why were the data compiled?Why were the data compiled?
What is the reliability of the provider?What is the reliability of the provider?
From what population do the data come?From what population do the data come?
When to give upWhen to give up
Consider the possibility that there is no data set that contains exactly the Consider the possibility that there is no data set that contains exactly the information you need in exactly the form you need it.information you need in exactly the form you need it.
In trying to compare data, face the fact that because of different time frames, In trying to compare data, face the fact that because of different time frames, geographic limits, different categories and different definitions of terms, it may geographic limits, different categories and different definitions of terms, it may be impossible to integrate data from various sources.be impossible to integrate data from various sources.
Sample the data sets to check for compatibility, credibility, etc. If there are Sample the data sets to check for compatibility, credibility, etc. If there are glaring inconsistencies, you probably need to abandon the data set.glaring inconsistencies, you probably need to abandon the data set.
Is it possible to construct your own data set?Is it possible to construct your own data set?
Census v. surveyCensus v. survey
In theory, census data includes all of a populationIn theory, census data includes all of a populationUndercountUndercount
OvercountOvercountSurveys rely on sampling to make estimates about a populationSurveys rely on sampling to make estimates about a population
A sampling frame is requiredA sampling frame is requiredProbability samplesProbability samples
Simple random sampleSimple random sampleSystematic sampleSystematic sampleStratified sampleStratified sampleCluster sampleCluster sampleMulti-stage sampleMulti-stage sample
Non-probability samplesNon-probability samplesVolunteer sampleVolunteer sampleConvenience sampleConvenience sample
SamplingSampling
Why sample?Why sample?To estimate something about a populationTo estimate something about a populationTo make a predictionTo make a predictionTo test a hypothesisTo test a hypothesis
A A simple random sample simple random sample of size of size nn is a sample such that any subset of is a sample such that any subset of n n elementselements is is equally likely to be selected.equally likely to be selected.
A A stratified sample stratified sample consists of random samples from several strata of a population.consists of random samples from several strata of a population.
A A cluster sample cluster sample consists of samples from randomly selected subsets of a population.consists of samples from randomly selected subsets of a population.
Sampling and non-sampling errorSampling and non-sampling error
Sampling error in making estimates about the population from a sample or in testing Sampling error in making estimates about the population from a sample or in testing a hypothesis results from the process of sampling itself and can be controlled, a hypothesis results from the process of sampling itself and can be controlled, but not eliminated.but not eliminated.
Non-sampling errorNon-sampling errorSelection biasSelection biasQuestion(er) biasQuestion(er) biasResponse biasResponse bias
Involuntary Involuntary VoluntaryVoluntary
Sample size and variationSample size and variation
Too small a sample: population differences cannot be detected or estimates will not Too small a sample: population differences cannot be detected or estimates will not be very accurate. The smaller the sample the larger the sampling error.be very accurate. The smaller the sample the larger the sampling error.
Too large a sample: costly (in money and in people) and may focus on differences Too large a sample: costly (in money and in people) and may focus on differences too small to be of practical significance.too small to be of practical significance.The larger the sample the larger the sampling error.The larger the sample the larger the sampling error.
If the sample is not too large or too small, the sampling error does not depend on If the sample is not too large or too small, the sampling error does not depend on the size of the population.the size of the population.
But variation within the sample is also critical—the larger the variation ,the larger But variation within the sample is also critical—the larger the variation ,the larger the sampling error and the smaller the variation, the smaller the sampling error.the sampling error and the smaller the variation, the smaller the sampling error.
What might a data set look like?What might a data set look like?ScoreScore gender studygender study1010 00 331010 00 2288 11 2277 00 1188 00 3399 11 5588 11 441111 00 661414 00 551515 00 771414 11 661515 00 881111 11 221111 00 551212 11 441212 11 331212 11 111313 00 661212 00 331313 00 661010 00 221313 00 5577 11 111313 11 331111 00 771414 00 881212 11 331212 00 551010 00 441010 11 331212 00 6699 00 2299 11 331313 00 551010 11 331111 11 44
ScoreScore represents the represents the scores on an examscores on an examadministered to administered to 36 individuals36 individuals
GenderGender is represented is represented by by
0 for males and 0 for males and 1 for females for1 for females forcomputational purposescomputational purposes
StudyStudy represents therepresents thenumber of hours thatnumber of hours thateach individual each individual studied for the examstudied for the exam
Organizing dataOrganizing data
Each line represents an observationEach line represents an observationIndividualsIndividualsGroups of people (averages)Groups of people (averages)CountriesCountriesProvincesProvincesRegionsRegionsCitiesCitiesPeriod of timePeriod of time
Each column represents values of a variableEach column represents values of a variable
Kinds of variablesKinds of variablesCategorical (qualitative)—gender, country, “dummy” variablesCategorical (qualitative)—gender, country, “dummy” variablesOrdered—Likert scales (but be careful)Ordered—Likert scales (but be careful)Quantitative—age, income, number of newspapers readQuantitative—age, income, number of newspapers read
Centrality measuresCentrality measures
MeanMeansum of all observations/number of observations (“average”)sum of all observations/number of observations (“average”)
MedianMedianif there are an odd number of ordered observationsif there are an odd number of ordered observations
median = middle observationmedian = middle observationif there are an even number of ordered observationsif there are an even number of ordered observations
median = mean the two middle observationsmedian = mean the two middle observations
ModeModethe most frequent observationthe most frequent observation
Normal curveNormal curve
Data distributionData distribution
SD
Average tax cutAverage tax cut
1. Bill Gates 1. Bill Gates $999,010$999,010
2. Taxpayer 22. Taxpayer 2 1010. .. .. .. .. .. ... . .
1000. Taxpayer 1000 1000. Taxpayer 1000 10 10 $1,000,000$1,000,000
Average = $1,000,000/1000 = $1000Average = $1,000,000/1000 = $1000
Bill Gates is an “outlier.”Bill Gates is an “outlier.”
From description to inferenceFrom description to inference
From descriptive statistics we would like to make inferencesFrom descriptive statistics we would like to make inferences
What can we say about a population from sample data?What can we say about a population from sample data?
How likely is an event?How likely is an event?Test an hypothesis about a populationTest an hypothesis about a population
Can we make predictions?Can we make predictions?Outcome of election, need for facilitiesOutcome of election, need for facilities
What is the relation between variables?What is the relation between variables?Correlation v. causationCorrelation v. causation
Important considerationsImportant considerations
Distinguish between
Uncertainty is a lack of knowledge about specific factors, models, parameters, measurement, sampling, systematic errors, oversimplification of real world processes, misspecification of model structure, inappropriate proxy variables, descriptive or aggregation errors, misjudgment,incomplete analysis
Variability arises from real heterogeneity, diversity, results of natural random processes
Summary advice: Know when you need a Summary advice: Know when you need a statistician and get one as early as possiblestatistician and get one as early as possible
THE ROLE OF A STATISTICIAN IN A PROJECTParticipate early and often to provide guidance in designing studies and collecting data
Formulate the questions to be askedConsider the data needed to answer the questions
Resist unrealistic expectationsMonitor the execution of studies and collection of data to provide a basis for accountability and cost-effectiveness From existing sources—creating methods to improve the quality of available data Through retrospective case studies Through surveys—sampling planGrapple with the data
Clean the data Design and carry out the analysis
Interpret the data and results and draw conclusions
Pitfalls for which statisticians may share Pitfalls for which statisticians may share responsibilityresponsibility
1) Failure to gain the support of key decision-makers
2) Unrealistic goals and expectations
3) Failure to develop a clear map of the process
4) Building of scenarios that are not credible
5) Inappropriate time frames and scopes
6) Failure to design a survey process to address the questions asked
Useful referencesUseful references
SoftwareSoftwareSPSS SAS STATA RSPSS SAS STATA R
Basic textBasic textMoore, McCabe and Craig, Moore, McCabe and Craig, Introduction to the Practice of Statistics, Introduction to the Practice of Statistics, 6 6thth ed., Freeman, 2009 ed., Freeman, 2009
More advancedMore advancedAgresti and Finlay, Agresti and Finlay, Statistical Methods for the Social Sciences, Statistical Methods for the Social Sciences, 44thth ed., Pearson, 2009 ed., Pearson, 2009Finkelstein and Levin, Finkelstein and Levin, Statistical Methods for Lawyers,Statistical Methods for Lawyers, 2 2ndnd ed., Springer, 2001 ed., Springer, 2001
SamplingSamplingLevy and Lemeshow, Levy and Lemeshow, Sampling of Populations: Methods and Applications, Sampling of Populations: Methods and Applications, 44thth ed., Wiley, ed., Wiley, 20082008
ApplicationsApplicationsAsher, Banks and Scheuren, eds., Asher, Banks and Scheuren, eds., Statistical Methods for Human Rights, Statistical Methods for Human Rights, Springer, 2007Springer, 2007Ball and Asher, “Statistics and Slobodan,” Ball and Asher, “Statistics and Slobodan,” ChanceChance, 15, pp. 17-24, 2002, 15, pp. 17-24, 2002Ball, Scheuren, Seltzer and Spirr, “Multiple or Ball, Scheuren, Seltzer and Spirr, “Multiple or NN-system estimates of the number of political -system estimates of the number of political killings in Guatemala,” killings in Guatemala,” ASA Proceedings in Social Statistics, ASA Proceedings in Social Statistics, pp. 156-160, 1999pp. 156-160, 1999Burnham, Lafta, Doocy and Roberts, “Mortality after the 2003 invasion of Iraq,” Burnham, Lafta, Doocy and Roberts, “Mortality after the 2003 invasion of Iraq,” The The Lancet, Lancet, 368, pp. 1421-1428, 2006368, pp. 1421-1428, 2006Gray, “Statisticians discuss mortality in Iraq survey,” Gray, “Statisticians discuss mortality in Iraq survey,” Amstat News, Amstat News, April 2007, pp. 7-9April 2007, pp. 7-9