margins of error: public understanding of statistics in an era of big data

MarginsMargins f Eof Error

John PullingerJohn Pullinger,President of the Royal Statistical Society

© Ipsos MORI / King’s College London

Public trust andPublic trust and understandingg

Bobby DuffyBobby DuffyDirector, Ipsos MORI Social Research Institute, Visiting Senior Fellow, King’s College London


Focus on understanding

d l b tand value – but firstly on trustfirstly on trust…


Scientists and academics win...

How much trust do you have in information provided by the following types of people?

28 46 3

A great deal A fair amount None at all

Scientists

18

13

45

42

6

10

Academics

Accountants

12

8

37

31

12

15

Statisticians

Economists 8

9

31

28

15

8

Economists

Actuaries

2

1

21

7

23

54

Pollsters

Politicians


Base: 1,034 British adults aged 16-75 Source: RSS/Ipsos MORI 2013

Trust in scientists vs trust in clergy – a new age of reason?

...would you generally trust them to tell the truth, or not?

90 Clergymen/Priests Scientists

75

8085

Clergymen/Priests Scientists

% Yes

65

7075

50

5560

40

4550

3035

98 99 00 01 02 03 04 05 06 07 08 09 10 11 12* 13**


Base: c.1,000-2,000 Source: Ipsos MORI most years face-to-face in-hom, *2012 ICM telephone ** 2013 IM telephone

Trust in civil servants vs politicians – views have diverged...

...would you generally trust them to tell the truth, or not?

Civil Servants Government Ministers Politicians Generally Journalists

60

70% Yes

40

50

30

40

10

20

0


Base: c.1,000-2,000 Source: Ipsos MORI most years face-to-face in-hom, *2012 ICM telephone ** 2013 IM telephone

But government less trusted with our data than online retailers?

5A greatCompanies such as supermarkets and online5

38

2A great

deal

A fair

supermarkets and online retailers collect a lot of data on their customers (for example through loyalty 38

40

30A fair

amount

Not very

cards). To what extent, if at all, do you trust companies to use the data they collect about you appropriately40

12

41Not very

muchabout you appropriately

The government collects a lot of data on citizens (for

l th h t12

6

20Not at all example through tax

returns). To what extent, if at all do you trust the government to use the data 6

6Don't know

gthey collect about you appropriately?


Base: c. 500 British adults aged 16-75 Source: RSS/Ipsos MORI 2013

Big, technical g,issues for people to come to view on…


...not least, debt vs deficit...

As you may know there is currently a lot of discussion about our national “debt” and “deficit”. Can you tell me what these words mean when talking about government finances?

The difference between

3

Debt means Deficit meansThe difference between what government spends and the income it receives each year3

78

47

8receives each year

The total amount of money that the government owes

4Both mean the same

Don’t know

82 8282 82



...it is a tricky one...


...but public also not so clear when “use it in a sentence”...

And can you tell me whether the following statement is true or false?

“The national debt will always go down if the deficit is decreasing”

20 Those who got definitions right

The national debt will always go down if the deficit is decreasing

2820

TRUE

Those who got definitions right no more likely to get this right

FALSEDon't know

Public think 40% of planned cuts already been made

52already been made...



Basic understandingunderstanding of numbers isof numbers is key to statistical yliteracy – and it i i dis mixed…


Most get very simple questions correct...

What is 50 expressed as a percentage of 200?

2010 89% t

92

10%

25%

2010: 89% correct

92

3

25%

50%

175%

1Other

Don't 3Don t know



...and slightly trickier...

What is the average of the following three numbers – 5, 10 and 15?

2010: 71% correct16

70

5

10

2010: 71% correct

70

1

10

12

515

3Other

5Don't know



...but real difficulties with probabilities...

If you spin a coin twice what is the probability of getting two heads?

1 % 2010 30% t1

26

15%

25%

2010: 30% correct

240%

58

1

50%

75% 1

2

75%

Other Strong relationship with education (A-level+),

10Don't know

Strong relationship with education (A level ), but also big differences by age, younger groups more likely to get right...



There are alsoThere are also known biases in how we consider

t ti tistatistics…


A personal optimism bias...

What do you think the chance or probability is of the following being injured or killed in a road accident this year (whether as a road user or a pedestrian)?

S i G t B it i Y2

31About 1 in 2

About 1 in 5

Someone in Great Britain You

8

6

2

1

About 1 in 5

About 1 in 10

Ab t 1 i 20

Mean probability:Someone = 4.1%Y 1 6%6

7

7

2

3

About 1 in 20

About 1 in 50

You = 1.6%

Actual probability = c1.2%?7

20

24

5

19

About 1 in 100

About 1 in 100024

2340

27

About 1 in 10,000

Don't know



...but focus on negative information

Imagine you have a life-threatening illness and your doctor has told you that you need an operation to treat it. How likely, if at all, are you to have this operation if your doctor tells you that...y

90% of people who have the operation are alive for at least 5 years following the operation10% of people who have the operation die within 5 years of the operation

56

33

39Very likely

33

3

38

6

Quite likely

Not very likely

1

6

2

y y

Not at all likely Avoid targets on “negatives”, even if hit them? Waiting

716Don't know

even if hit them? Waiting times, immigration...



But does it matter? Do

l idpeople consider evidence – orevidence or think their leaders do?


Principle-based policy-making...

Politicians will take decisions partly based on what they think is right, and partly on evidence of what works. Do you think they base their decisions more on what they think is right than on evidence, more on evidence than on what they think is right, or do you think they consider them b th i l ?both in equal measure?

More on what they think is right than on evidence

18than on evidence

More on evidence than what they think is right

16

y g

On evidence and what they think is right about the same

t5216 amountDon't know

13



...but mirrors people’s own use of evidence

People have different attitudes towards statistics. Which of the following do you agree with most?

My own experiences or those of my family and friends are more important than statistics in helping

26Statistics are more important than

me keep track of how the government is doing

46my own experiences or those of my family and friends in helping me keep track of how the government is doing

18

Both equally

Neither/Don’t know

9

Neither/Don t know



More broadlyMore broadly, understanding gnumbers is undervalued?


We’re not embarrassed about lack of understanding of numbers...

Which of the following things would you feel most embarrassed about admitting to friends and family?

6I'm not very good with numbers

15I'm not very good at reading and writing

75

y g g g

Neither 75Neither

5Don't know


Base: 516 British adults aged 16-75, interviews conducted online 9th-15th April 2013 Source: RSS/Ipsos MORI 2013

...and there’s little pride in doing it well

Thinking about your child/if you had a child, which of the following would make you most proud?

13If they were very good with numbers

55If they were very good at reading and iti 55

16

writing

N ith 16Neither

15Don't know


Base: 516 British adults aged 16-75, interviews conducted online 9th-15th April 2013 Source: RSS/Ipsos MORI 2013

We’ve got a long way to go...

I keep saying that the sexy job in the next 10 years will be statisticians. And I’m not kidding.Hal Varian, chief economist at Google

Statistical thinking will one day be as necessary for efficient citizenship as the ability to read or writeHG Wells

Value of statisticsNumber of people reachedQuantity of statistical infoMedia affectRelevanceTrustNumeracy

VAS = N * [(QSA * MF) * RS * TS * NL]Enrico Giovannini, Former Chief Statistician, OECD


Thank youybobby duffy@ipsos [email protected]@BobbyIpsosMORI


Understanding andUnderstanding and Trust in StatisticsTrust in StatisticsAndrew Dilnot CBE,Andrew Dilnot CBE,Chair of the UK Statistics Authority


GDP 1948-2012 (Index 2009=100)

120

100

)

60

80

009=

100)

40

60

ndex

(20

20

I

0


GDP 2000-2013

120

100

60

80

009=

100)

40

60

Inde

x (2

0

20

0


GDP 2000-2013

110

100

105

90

95

009=

100)

85

90

Inde

x (2

0

75

80

70


GDP Revisions

Available data( )

Preliminary Estimate Second Estimate Quarterly National Accounts

(output measure)

25 days approx 55 days approx 3 months


Former minister slams 'national catastrophe' of 'national catastrophe' of teenage mothers addicted to benefits UK has highest teen

pregnancy rate in p g yEuropeTEENAGE PREGNANCY

SOARS

NO SET OF VALUES SOARS

FOR GYM-SLIP MUMS


MUMS

Under 18 conception rate for England and Wales



Norovirus

Norovirus lab reports


Source: Health Protection Agency

Norovirus confidence intervals

• 1:1500 (1 lab case = 1500 in community). • 2000 lab cases = 3million in community

• But maybe 1:140But maybe 1:140 • (=280,000 cases)

• Or maybe 1:17,000 • (=34 million cases)

• Community study lab cases…• =1


The 2011 Census and uncertainty

1 4Relative Confidence Interval width

1.2

1.4t)

0 8

1

(per

cen

t

0.6

0.8

al w

idth

(

0 2

0.4

Inte

rva

0

0.2

England South East Kent Canterbury


England South East Kent Canterbury

Trends in police recorded crime and CSEW


Lies, damn lies

Crime and crime statistics

statistics were

statistics

POLICE FAIL were distorted by

POLICE FAIL TO RECORD y

politicsTO RECORD CRIME PROPERLY


Crime falls to new low despite recession and unemploymentrecession and unemployment

...The 6% fall in crime reported in the latest quarterly p q yfigures by both the Crime Survey for England and Wales and the separate police recorded crime figures means that crime has now dropped by more than 50 % since it peaked in the mid‐1990s...


The Guardian, 19 October 2013

Public Understanding of t ti ti i f bistatistics in an era of big

datadataDenise Lievesley,Denise Lievesley,Head of School of Social Science and Public Policy,King’s College London


Challenges facing statisticians

H ilit C fid

R l

Humility

A t

Confidencevs

Relevance

T t

Autonomy

S i i

vs

Trust Scepticismvs

Measurement Qualityvs

Pragmatism Purismvs


Humility vs. Confidence

Being a statistician meansnever having to say

you’re certain


Humility – being aware of our limitations

“Good science should not turn a blind eye to known imperfections –nor should these be concealed from users”

Sir Roger Jowell 2007

“The absence of excellent evidence does not make evidence-based decision making impossible: what is required is the best evidence available not the best evidence possible”

Si M i G 199Sir Muir Gray 1997


ISI declaration on professional ethics 1985

• One of the most important but difficult responsibilities of the t ti ti i i th t f l ti t ti l f th i d t t thstatistician is that of alerting potential users of their data to the

limits of their reliability and applicability. The twin dangers of either overstating or understating the validity or generalisabilityeither overstating or understating the validity or generalisabilityof data are nearly always present.

• Confidence in statistical findings depends critically on their faithful representation. Attempts by statisticians to cover up

i i i i l b derrors, or to invite over- interpretation, may not only rebound on the statisticians concerned but also on the reputation of statistics in generalstatistics in general.


Confidence –using the data to make a difference

•We need to provide information of high quality,We need to provide information of high quality, integrity and robustness which can be relied on.

•We should be confident about our findings and prepared to account for them. p p


Communication

We need to improve our communication skills and think about impact.We should learn how to tell a story with datayand remember that communication is not what is delivered but what is receivedis delivered but what is received.e.g.

• Bill Gates has a personal fortune greater than the combined wealth of the 106 million poorest Americans.

• The cost of putting all children into school is less than is spent on icecream in Europe each year


Sir Gus O’Donnell(former UK Cabinet Secretary)

“I want [the ONS] to be boring, to put out the plain facts, and nothing but the facts and on clear predictable deadlines ” henothing but the facts, and on clear, predictable deadlines, he said. It would then be for politicians and government press officers to interpret the figures, he added.p g ,


Response of the Royal Statistical Society

• it is clearly the task of statisticians to interpret the figures in a statistical context, to facilitate understanding and avoid misunderstanding.

• The Code of Practice of the UK Statistics Authority explicitly states that Official statistics accompanied byexplicitly states that Official statistics, accompanied by full and frank commentary, should be readily accessible to all users and that all UK bodies that are responsibleto all users and that all UK bodies that are responsible for official statistics should prepare and disseminate commentary and analysis that aid interpretation andcommentary and analysis that aid interpretation, and provide factual information about the policy or operational context of official statistics


operational context of official statistics.

Relevance vs. Autonomy

UN Fundamental Principles of Official Statistics

Principle 1

“Offi i l t ti ti id i di bl l t i th“Official statistics provide an indispensable element in theinformation system of a democratic society, serving theGovernment the economy and the public To this end officialGovernment, the economy and the public ... To this end, officialstatistics that meet the test of practical utility are to be compiledand made available on an impartial basis by official statisticalagencies..”


Impartiality

• The role of statisticians: to inform political debate and decisions without taking partt out ta g pa t

• Fear that enhancing statistical utility will compromise impartiality

• There must be no political interference with the data and no perception that there isperception that there is

But does this mean we are too cautious?

Are statisticians so afraid of being accused of political motives gthat they dare not make reports useful for the public debate?


The value of statistics to society must not just be asserted; it must be demonstrated“Were a balance sheet for official statistics to be prepared the“Were a balance sheet for official statistics to be prepared, the

costs would be clear enough. The benefit, or value, would however be found to be much more diffuse and harder to treat inhowever be found to be much more diffuse and harder to treat in

traditional accounting terms. Given this, it is possible that the vital asset that official statistics represent is undervalued in public sector planning processes. And we observe that little

systematic consideration is given to how the public value could be maximised”be maximised .

(UK Statistics Commission, The Use Made of Official Statistics,


(UK Statistics Commission, The Use Made of Official Statistics, 2007)

Trust vs. Scepticism

• Pre-requisite for evidence based policy and for managing for results is that the data must be trustworthy


But it is not enough that the data are trustworthy they must also be trusted

• Otherwise they won’t be used• There will be fights about the data rather than about the issues

• Data need to be the currency of public debates


Evidence sometimes resisted...

“There is nothing a governmentThere is nothing a government hates more than to be well-

informed: for it makes the process of arriving at decisions much moreof arriving at decisions much more

complicated and difficult.”pJohn Maynard Keynes


Inconvenient truths

• Governments prefer good news stories

• Bad news stories may be delayed or buried

• They are often too focussed on populism• They are often too focussed on populism

• The government’s horizons can be shorter than those of i i i !statisticians!

• They can prefer their own spin to that of the statisticiany p p


Important aspects of building trust

• Autonomy of statisticiansSt ti ti l l i l ti• Statistical legislation

• Existence of an independent statistical boardD l t f d f d t• Development of codes of conduct

• Breaches of the code identified, investigated and publicised• Appointment of senior statisticians removed from the political

processU h ld b i l d i tti th d ( ki th• Users should be involved in setting the agenda (asking the awkward questions)

• External audits of the statistical processes should be employed• External audits of the statistical processes should be employed• Audit body should report to Parliament


Measurement vs. Quality

• Statisticians need to guard against “what can’t be measured isn’t real”

• The danger with a measurement culture is that excessive attention i i t h t b il d t th f h t iis given to what can be easily measured, at the expense of what is difficult or impossible to measure quantitatively even though this may be fundamentalmay be fundamental.


Challenges to integrity –the rise of performance monitoring

• Performance data can be used in establishing 'what orks' among polic initiati es to identif ell performingworks' among policy initiatives; to identify well-performing

or under-performing institutions and public servants; and, equally important to hold Ministers to account for theirequally important, to hold Ministers to account for their stewardship of the public services

H t i b th it i th bli i• Hence, government is both monitoring the public services, and being monitored, by performance indicators.

• Because of government's dual role, performance monitoring must be done with integrity and shielded from undue political influence


Hitting the target but missing the point

htt // k/PDF/P f M it i df


http://www.rss.org.uk/PDF/PerformanceMonitoring.pdf

Audit Commission report

“What makes a target ‘good’ is not just the way a target isWhat makes a target good is not just the way a target is expressed—it’s about the way it was derived, the extent to which service users were involved in its developmentto which service users were involved in its development, the extent to which it helps to achieve policy objectives, the extent to which it has the support of the staff whosethe extent to which it has the support of the staff whose efforts will achieve it, the quality of the data used to measure its achievement and the clarity andmeasure its achievement, and the clarity and transparency of its definition”


Pragmatism vs. Purism

• To what extent should we exploit data from a widerTo what extent should we exploit data from a wider range of sources?

• May allow us to produce more timely data at lower cost

• Opportunities provided by BIG data


Fundamental changes to data sources might need to involve review as to the nature of evidence

• Use of ‘free form’ data raises questions about how to i h li d i i d i h hcommunicate the quality and uncertainty associated with the

evidence

• In the context of some moves towards greater formalisation of evidence (such as randomised control trials)

It does not remo e the need for SCIENCE• It does not remove the need for SCIENCE


The use of big data brings challenges?

• Need programmes of work on the technical and analytic challenges especially relating to data qualitychallenges especially relating to data quality• But also on

• Communication and dissemination of statistics

• Culture of statistical agencies• Culture of statistical agencies

• Trust of the public

• Changing relationships with users and providers

• The responsibilities of official statisticians

• The meaning of privacy in this new world

• etc


• etc.

Develop statisticians for the future

• Foster adaptability

• Transferable skills

• Build research and innovation skills

• Create a cadre of people who challenge pre-Create a cadre of people who challenge pre-conceptions

• Not to mould them in our own image

• Nor to create homogeneous communities• Nor to create homogeneous communities

• Education is about opening minds not closing them


margins of error: public understanding of statistics in an era of big data

Design

rssipsos mori

british adults

scientists vs trust

debt vs deficit

appropriatelythe government

government owes4both

government finances

icm telephone