dr. g. johnson, research at the intersection of politics and administration research methods for...

Dr. G. Johnson, www.ResearchDemystified.org

1

Research at the Intersection of Politics and Administration

Research Methods for Public Administrators

Dr. Gail Johnson


2

Research Process Revisited

Key points Planning is the key

No statistical magic can fix planning mistakes

What are the assumptions?What those particular assumptions?Do the assumptions change the results?


3

Research Process Revisited

Key points What are the boundaries of the study?

This defines the limitations: if they only survey people in NYC, they cannot make generalizations about people in Detroit

It is easy to conclude more than the research can actually support


4

Why Research Matters

Challenges what we think we know Explores what we do not know Provides information that can engage people in

discussion and debate to solve public problems Ideally, provides information as a counterweight

to emotion and rhetoric in a super-hot political environment


5

Research Can Help Managers

When seeking information rather than relying on assumptions or beliefs Customer, citizen or employee feedback Program/financial auditing, efficiency audits Monitoring pilot program Measuring results Program evaluation, effectiveness assessment Budget, cost-benefit analysis, unit price Best practices Testing performance


6

Managing Research Projects

Scope of work (SOW) Details of the research project Time Budget Requisite skills

Gantt chart Visual way to track key tasks Helps identify consequences of tasks that take more

time than allocated


7

Managing Research Projects

Build in quality Engage key stakeholders early to sharpen the focus of

the research and to develop buy-in for the reslts Use experts to review: measures, design, assumptions,

data collection, data analysis, review of draft report Require an audit trail to verify accuracy of data,

analysis and conclusions Make sure the research methodology is appropriate for

the situation and will support the conclusions


8

Using Research Results Requires the ability to:

Understand the results Assess the credibility of those results based on the

research methods used Sophisticated users of research results need

An understanding of the research process Critical thinking abilities A willingness to consider information that is different

from from our beliefs The ability to change our minds in light of credible

information


9

First Take On Research Results

Our first filter: do the results make sense on a gut level? Based on our experience Based on our “common sense” Based on our beliefs about how the world works

But we all have biases and blinders Remember: we do not actually know as much as we

believe we know And that is true for experts as well


10

Second Take On Research Results Our second filter: a researcher’s perspective that

seeks to see clearly For any conclusion, ask: “Where’s the evidence?”

Opinions—no matter how persuasive—are not evidence

Judgment—no matter how expert—is not evidence

Begin the process of systematically assessing credibility


11

Assessment Requires Critical Thinking Abilities Necessity to distinguish fact from fiction, even if

it has the appearance of science To maintain a researcher's perspective—which is a

detached critique of the methods used, assumptions made, and the conclusions drawn Resist the temptation to be overly critical of results that

challenge what we believe Challenge ourselves to critically look at results that

support what we believe


12

Assessing Research Credibility

Do the researchers (or their sponsors) have a particular policy position they want to support?

Are the seeking to understand or to win an argument?

Is there bias? Remember though—they may have an interest

in a particular outcome but that does not necessarily mean their research is biased.You still need evidence of bias.


13

Assessing Credibility

Is the data knowable?If not, how did they come up with it?Are the talking about actual behaviors or

self-reported behaviors?Remember: actual crime is not the

same as reported crime What makes the data convincing?


14

Assessing Credibility

What are their assumptions—especially when looking at research which tries to predict the future: Budget forecasting always assumes something

about the economy“Rosy scenarios”-where the economy is

growing-- will lead to different results than “economic collapse” scenarios


15

Assessing Credibility: The Methods Checklist Types of research questions

Be on the lookout for cause-effect questions, even if they do not sound like one initially

Measures: are they valid and reliable? Design: experimental, quasi, or non-

experimental? If cause-effect question, what design did they use?

Data collection systematic? Controls for error and bias?


16

Assessing Credibility: The Methods Checklist Sample: random or non-random?

Is size and response rate appropriate? Appropriate analysis?

Did they do tests for statistical significance if they used a random sample?

Did they show measures of association if answering a relationship question.

Conclusions within the framework of the study? Does the conclusions stay within the limitations of the

research methodology?


17

Process for Assessing Credibility

Break the research apart: Read introduction and conclusion to get the

point of the research Read the methodology Look at the data analysis

Make your own assessment: Does the research answer the questions? Does the analysis support the conclusions?


18

Final Words About Assessing Credibility Be mindful of your own biases when you

review research results Find a balance between trusting your own

instincts and being blinded by your biases Do not accept research results just because

they agree with what you believe


19

Final Words About Assessing Credibility Do not dismiss a study because it is flawed.

All research is flawed. The issue is whether it was the best that could

have been done in the situation

Do not dismiss a study out of hand because you do not like the results.


20

The Limitations of Science

Not everything is knowable Not everything can be reduced to numbers Bias can be embedded in how questions are

framed Social science is not set up to prove

anything It is the search for knowledge Typical conclusion: more research is needed


21

The Limitations of Science

Paradigm paralysis Cling to earth-is-flat theories

Market Equilibrium Theory: assumes markets respond to new information

But research shows that markets change even when there is no financial news

Ecology of beliefs, expectations, perceptions, misperceptions, rumors, gossip, fears, speculation can drive market swings


22

Mark Buchanan

“It’s these internal dynamics that make it possible for billions to evaporate from portfolios in a few short months just because people suddenly begin remembering that housing values do not always go up.”

“Equilibrium theory assumes incremental change but data suggests that markets can turn from ice to water in an instant. A new theory is needed.”

NY Times: This Economy Does Not Computer, Op-Ed, October 1, 2008


23

Intersection of Science and Administration Social science often is too weak to

demonstrate cause-effect relationships Programs can appear not to work because

their impact can not be proven Stokes cynicism about government efficacy

Pressure to take action but the information is far from certain


24

Working at the Intersection

If we accept that the information is imperfect and that we cannot wait for perfect information, then incremental steps make sense Pilot test programs best on research data

Build in feedback processes so more information is gathered as we go along Continuous improvement Incremental adjustments


25

Working At The Intersection

Welcome new information even if it shows that we made an incorrect decision Sometimes data from the past is a good predictor of the

future But sometimes it is like trying to drive forward while

looking through our rear view mirror Compassion: There is no way to know the

decision was incorrect at the time it was made It was necessary to try something so it generated more

–and better—information


26

Challenges Of Working At The Intersection Controversial issues—vocal competing groups Beliefs intertwine with policy Ideology clashes with science Public administrators are often at the center of this

intersection and must find a middle way between the competition of ideas, beliefs and conflicting policy preferences of the various stakeholders


27

The Joys Of Working At The Intersection The research perspective—the discipline of

detachment and honesty—will enable us to engage in deep conversations that might bring us closer to solutions that can solve very difficult problems

It is typically difficult, messy, and frustrating—but is also exciting to be doing work that matters

Finding that middle way is a way to support the democratic process.

It is one way to make a positive difference in the world


28

Takeaway Lessons At The Intersection Remember: the toughest problems wind up

on the government’s doorstep Simple solutions to complex problems do

not exist Single interventions are not likely to be

sufficient for complex problems


29

Final Words: Sophisticated Users of Research Results Value the search for new ways of looking at the

world, ourselves and others Are open to the emerging situation Are able to consider at least three alternatives or

possible explanations or strategies to any given problem

Manage uncertainty by moving forward in incremental steps while continuing to gather information


30

Final Words About Social Justice Be mindful of the benefits as well as the limits of

social science An unintended consequence of demanding that

that social service programs are effective but not tax loopholes, mortgage deductions, payments to corporations is an issue of social justice

Given the difficulty in “proving” things work using social science tools, we run the risk of wrongly assuming that programs don’t work just because we can’t measure an effect.


31

Final Words About Assessing Research Results Research methods can be far more complex than

the basics presented here I would not hesitate to call in experts if asked to

make a decision based on a complicated study or a study using unfamiliar methods or statistics

I advise you to do the same


32

Takeaway Lesson

We often know far less than we think we know. It might not seem that way as different sides offer

up statistics that appear to be rock solid

Remember:

"Many a statistic is false on its face. It gets by only because the magic of numbers brings out a suspension of common sense."

How to lie with statistics, Darrel Huff, p. 138.


33

Research Methods As Life Lessons Approach each situation with calm confidence Be open to new information that pushes you out of

your comfort zone Each step beyond your comfort zone expands it

Ask questions and don’t be intimidated by statistics, social science jargon or experts


34

Takeaway Lessons

Social science is rarely conclusive and public administrators have to n to work in the middle, where things often depend on the context

More research is needed is not a full-employment program for researchers! It merely reflects that fact that research is a

process of discovery and a work in progress


35

Creative Commons

This powerpoint is meant to be used and shared with attribution

Please provide feedback If you make changes, please share freely

and send me a copy of changes: [email protected]

Visit www.creativecommons.org for more information

dr. g. johnson, research at the intersection of politics and administration research methods for...

Documents

research resultsour

research methodology

research processcritical

research projectsbuild

research resultsrequires

research matterschallenges

data analysis

particular assumptions