dr. g. johnson, introduction to research methods research methods for public administrators dr....

Dr. G. Johnson, www.researchdemystified.org

1

Introduction to Research Methods

Research Methods for Public Administrators

Dr. Gail Johnson


2

Numbers and Facts

Any policy debate is awash in numbers that are presented as absolute fact.

But are they?Or are we being tricked into “premature

certainty” because numbers appear concrete?


3

In the News According to a 2009 study done for the American

Health Insurance Plans by Price Waterhouse and Coopers:

The average family health insurance coverage costs approximately $12,300 today

Key finding: The overall impact of the proposals for health care reform will be to increase the cost of private insurance coverage for individuals, families, and businesses above what these costs would be in the absence of reform.


4

Their Numbers

Health Insurance premiums could be expected to increase to approximately: $15,500 in 2013 under current law and to

$17,200 if these provisions are implemented. $18,400 in 2016 under current law and to

$21,300 if these provisions are implemented. $21,900 in 2019 under current law and to

$25,900 if these provisions are implemented.


5

Policy Debate What does this data mean in terms of

Congressional proposals to require everyone purchase health insurance?

What would you conclude based on this research result?

What do you want Congress to do based on these numbers?

What do you want Congress to do based on your beliefs?


6

Paradox

Research: the search for rationalityBut:“We must be aware that our knowledge and

experiences are finite, and always imperfect.”

Hitoshi Kume


7

Assessing Credibility

Requires Basic knowledge of research methods

Maintaining a research perspective Knowing how data was collected is necessary to help

assess whether the results are believable Critical thinking skills

A hint of detachment and skepticism Being aware of our beliefs yet setting them aside to

see things clearly


8

The Tough Questions

Is it true? How do you know? What is the logic of the argument? What is the evidence?

Remember: Much of what is presented as fact melts away

under scrutiny.Beware of premature certainty.


9

Critical Thinking Ask Yourself:

“How do I know that the things I believe are true?” “What would it take to convince me otherwise?”

The greatest challenge comes when looking at research that reflects what we already believe

Critical thinking challenges our most cherished beliefs

To see clearly, we must be aware of our beliefs while also remaining detached from them


10

The Role of Research in the Public Sector Guide policy Target programs Find out what works and what doesn’t Convince funders Provide feedback from customers Help plan interventions Advocate for change


11

Research in the Public Sector

The Intersection of Data, Values and Beliefs and Politics

Does welfare help or hurt people? Should oil companies be prevented from

earning too much profit? Is investment in primary prevention more

effective in reducing crime than spending on law enforcement?


12

Different Research Approaches

Qualitative: Stories Observations No numbers

Quantitative: What you can count.


13

Qualitative Data: Stories

AnecdotesExample: Michael Moore’s “Sicko”

Example: Barbara Erhen “Nickled and Dimed”

Paints a picture, makes it feel real Limited in scope:

Is this a small or large problem?


14

Quantitative Data: Numbers

Percents, Counts, Correlations Example: Study: U.S. Ranking for Preventable

Deaths: U.S. lowest of industrialized countriesBut need to check how they defined and

measured preventable deaths


15

Quantitative Data: Numbers

Be aware that some things are not really knowable—so how did they come up with the numbers? Number of illegal immigrants Amount of money spent on illegal drugs 10-year projection of annual U.S. budget deficits (even

1-year is not precise) The true cost of mandatory health insurance Percent of carbon from raising animals


16

As An Advocate

Which would be more effective?

Voice your concerns about a problem? Or

Show data documenting the problem?


17

As a Citizen

What do you base decisions on? Data? Emotional appeals? Personal opinion? Political demands?

Do some work better than others in different situations?


18

As a Decision Maker

What do you base decisions on? Data? Emotional appeals? Personal opinion? Political demands?

Do some work better than others in different situations?


19

The Tough Question

Generally, we prefer data, facts, hard evidence

But the question is: How good is the research? Meaning: does the research yield accurate,

reliable, valid, and unbiased information?


20

What is Research?

A systematic search for answers to questions.

Search: to uncover, examine, find by exploration, to investigate, to inquire.

Research: "the systematic inquiry into a subject in order to discover or revise facts, theories


21

What is Research?

Research: Root of work means “to know.”Re---Search:

conveys the idea of searching again, from a different perspective, using different approaches, looking at different data.


22

What is Research?

Empirical: derived from experience or experiment; observation and experience.

Science: "a branch of knowledge or study dealing with a body of facts or truths systematically arranged and showing the operation of general laws; systematic knowledge"


23

This Thing Called Science

Scientific Method Curiosity Systematic observation Systematic experimentation

Theories and Hypotheses: Used to Explain Relationships


24

Different Goals

Scientists: More interested in creating knowledge Not concerned about utility or application of

the results Practitioners:

More interested in application and problem solving

If the research isn’t useful, then so what?


25

Research Process: Simplified 3-step Plan Do Report


26

Research Planning Process

Determine the issues Decide on the research question(s) Select measures Identify the ideal design given the type of

research question Develop data collection methods and

instruments


27


Decide on sampling frame and strategy Develop analysis plan Articulate your entire research plan Review research plan and test all data

collection instruments Prepare work plan with resource and time

requirements


28

Planning Matters

When people think about research, they tend to immediately focus on statistics

But no amount of statistical wizardry can save mistakes made in the planning process

Planning is harder and takes longer than most people expect


29

Doing Research

Gathering the data Preparing data for analysis Analyzing and interpreting the data


30

Reporting

Executive Summary Reports Charts and Tables Oral Briefings


31


Presented as linear but in practice, it is not The word is iterative: meaning researchers make

some initial assumptions about how to proceed and then go back and make changes to the plan as new information shows that their initial plans will not work

There is a lot of back and forth before it all clicks into place (and researchers live with the limitations of the situation)


32

Ethical Research Do No Harm

Protect participants from being harmed by the research Guard the confidentiality of participants Do not coerce people to participants: they must freely

consent No one should be denied benefits to which they are

entitled because they refuse to participate in research project

Do not quickly conclude a program does not work just because there does not appear to be an impact


33

Objective: Strive to See Clearly

Objectivity is a worthy goal even if not 100% attainable Think in terms of being free of ideological or

political blinders or desired outcomes Be aware of biases

Researchers should state biases and build in quality controls to minimize bias

Quality Control: Have others with different perspectives review the research design, analysis and results.


34

Principles of Good Research

Honest Do not lie Do not distort or “spin” the data Fully disclose methods, definitions,

assumptions, biases Fully disclose limitations of the research and

implications for making any conclusions


35


Accurate: Strive to be error free Build quality control procedures into data

collection, analysis, interpretation, and written product

Verify data entry and analysis. Have someone check your numbers.


36


Technically Correct Use appropriate designs, data collection

methods, analysis, statistics and charts The right data collection instruments. The right statistics.


37


Reliable and Valid Measures Use reliable and valid measures Measure what matters using systematic

approaches with fixed measurement rules Ask: do they measures actually measure what

they claim to measure? Are they measuring reported crime but claiming to

be measuring all crime?


38

Research Flaws

Perfection is not the standard to assess the credibility of research results. All research is flawed, so the trick is to

distinguish minor flaws from major ones. Skill: to assess the relative strengths and

limitations of research to determine the credibility of the results.


39

Research Flaws

It means working in the gray area. If the research is the best given the

circumstances, flawed though it may be, it should be considered.

However, decisions “to do something” based on that data should be made with caution, and implemented incrementally with built-in feedback.


40

Seriously Flawed Research

While you might not want to make decisions based on seriously flawed research, it still may actually be correct, provide useful insights, or some guidance about how to do the next study. It might provoke important conversations.

Remember: perfection is not a useful standard.


41

Takeaway Lesson

Sophisticated Users of Research Question Numbers!“Many a statistic is false on its face. It gets by only because the magic of numbers brings out a suspension of common sense."

---How to Lie With Statistics, Huff, p. 138.


42

Ask Tough Questions

Does the researcher have axe to grind?Particular political ideology? Desire to

get more funding? Who was included and excluded in the

study? Is the data really knowable?

Number of actual crimes is not the same as the number of reported crimes.


43

Ask Tough Questions

Is reported behavior based on self-reports?How many people used illegal drugs in

the past week? Does the research make a giant leap to

conclusion? Did a study with all male participants but

generalizes to everyone?


44

Creative Commons

This powerpoint is meant to be used and shared with attribution

Please provide feedback If you make changes, please share

freely and send me a copy of changes: [email protected]

Visit www.creativecommons.org for more information

mailto:[email protected]

http://www.creativecommons.org/

dr. g. johnson, introduction to research methods research methods for public administrators dr....

Documents