research methods in crime and justice chapter 13 evaluation research methods

Research Methods in Crime and Justice

Chapter 13Evaluation Research Methods

Evaluation Research Methods

• Evaluation research has become a critical part of criminal justice administration.

• The analysis of a program’s efficiency and effectiveness is an important part of a criminal justice practitioner’s job.

• Evaluation research considers the effectiveness, efficiency, and the unintended consequences of a program, process or policy.

Evaluation Research Basics

• Like the qualitative and non-reactive research traditions, evaluation research is not a single methodology.

• Instead, evaluation research is any applied research method that considers the effectiveness, efficiency, and the unintended consequences of a program, process or policy.

Types of Evaluation Research

• There are two commonly used methods in evaluation research.– A cost/benefit analysis refers to a technique that

compares the costs and benefits of an existing or future program, process or policy.

– The multiple methods approach involves the use of multiple data gathering techniques to evaluate a program, process or policy.

The Benefits of Evaluation Research

• Evaluation research is effective at;– Demonstrating the effectiveness or efficiency of

an existing program, process or policy.– Monitoring the effectiveness or efficiency of new

programs, processes or policies,– Evaluating the unintended consequences of a

program, process or policy, and/or– Informing the debate on controversial programs,

processes and policies.

The Limitations of Evaluation Research

• Evaluation research;– Can evaluate but cannot determine the underlying

outcome or objective of a program, process or policy.

– Is pointless when a program, process or policy is mandatory or legally required.

The Evaluation Research Process

• A Case Study in Evaluation Research (The Police Protective Custody Process)– In most states, there is a process by which the police

can remove abused and/or neglected children from their homes and place them in temporary housing.

– This process is expensive and in most cases results in considerable physical and psychological disruption.

– Withrow (2001) and Withrow and Bolin (2003) were asked to conduct an evaluation of the Police Protective Custody process in Kansas.


• Asking a Research Question in Evaluation Research– The research question in evaluation research is

whether the program or policy is performing as it was intended to.• Is it effective?• Is it efficient?• Are their unintended consequences or opportunities?


• Asking a Research Question in Evaluation Research – cont’d– The PCC evaluation had the following research

objectives (questions).• To document the PPC process.• To determine what factors affect a child’s length of stay.• To make recommendations for reducing the number of

PPC referrals.• To make recommendations for reducing a child’s length

of stay (Withrow, 2001 p. i).


• Conducting a Literature Review in Evaluation Research– The primary literature in evaluation research is

other evaluation studies on the same topic.– Many of these studies are in the possession of

criminal justice and allied agencies.


• Conducting a Literature Review in Evaluation Research – cont’d– There were actually two versions of the PPC

report.• The scholarly journal included literature from scholarly

sources, most notably research on the effect of separation during the PPC process.• The version written for the Wichita Children’s Home

included only a nominal amount of scholarly literature because it was intended to be consumed by practitioners.


• Refining the Research Question in Evaluation Research– Most evaluation research projects do not include

formal hypotheses.– Instead they ask;

• Did the policy, process or program have its intended effect?

• Was the policy, process or program implemented efficiently?

• Did the policy, process or program result in unintended consequences?


• Refining the Research Question in Evaluation Research – cont’d– Explicit hypotheses were not developed for the

PPC study.– Instead Withrow and Bolin made several

predictions about the relationships between the variables used in the study.

– These predictions enabled then to at least begin the analysis.


• Defining Concepts and Creating Measures in Evaluation Research– In evaluation research, conceptualization and

operationalization involve developing measures for policy or program success, efficiency and/or effectiveness.

– Evaluation researchers must be mindful that stakeholders may have a different view on how to measure a program’s success.


• Defining Concepts and Creating Measures in Evaluation Research – cont’d– Many of the concepts in the PPC study (e.g. foster

care, respite, etc.) were defined by law.– Most of the other variables (e.g. parents drug use)

were dichotomous (i.e. yes or no).– This is common in criminal justice research.


• Defining Concepts and Creating Measures in Evaluation Research – cont’d– There were competing perspectives on what constitutes the

success of the PPC process.• Police officers associated success with the ability to refer a child to

the WCH without restriction. • Social workers defined success in terms of whether they were able

to find a suitable long term living situation for the child. • Prosecutors defined success in terms of their ability to convince a

judge to approve a child in need of care petition. • Representatives from social services agencies who were concerned

about expenditures were defined success as the ability to limit services to children who did not need them.


• Defining Concepts and Creating Measures in Evaluation Research – cont’d– To overcome the inevitable conflict associated

with alternative views of ‘success’ the researchers chose to focus on producing accurate descriptions of how the system worked.

– This, they believed, would enable policy makers to make more informed decisions


• Designing a Method in Evaluation Research– The actual method or methods used by evaluation

researchers depends on the type of research question asked.

– The use of multiple methods has become particularly popular in evaluation research.


• Designing a Method in Evaluation Research – cont’d– Because the PPC had four objectives, no single

research method would have sufficed. – The researchers used a multiple methods in our

evaluation. • focus group• Secondary analysis• Archival analysis• Interviews


• Designing a Method in Evaluation Research – cont’d– Focus groups with representatives from the

affected agencies allowed us to;• Document the PPC process, and• Identify redundancies in the process.

– Secondary analysis of data on admitted children allowed us to identify the factors that affected;• A child’s length of stay, and• Whether a child would be admitted to state custody.


• Collecting Data in Evaluation Research– The process by which evaluation researchers

collect data depends on the specific methods they use.

– There are a two important considerations specific to evaluation research.• Gaining full access to data during an evaluation is

sometimes difficult.• Criminal justice data is notoriously incomplete,

inaccurate, and inconsistent.


• Collecting Data in Evaluation Research – cont’d– In the PPC study the focus group required considerable

coordination. • The practitioners involved in the PPC process are busy. • So finding a single day when they all could meet was a rather

daunting task.

– The data set for the secondary analysis included information on 6,600 children.• This data set had to be ‘cleaned’ to facilitate the analysis.• In order to avoid the possibility of violating a child’s privacy, the

data set did not contain (at the researcher’s request) names and other personal identifiers.


• Analyzing and Interpreting Data in Evaluation Research– The analyzes and interprets of the evaluation’s

results depends on;• the nature of the research methods used, • the data produced by the research methods, and • the overall objectives of the study.


• Analyzing and Interpreting Data in Evaluation Research – cont’d– The PPC study produced (for the first time) a flow chart

of the PPC process that enabled policy makers to ‘see’ system redundancies.

– The secondary analysis found that children were more likely to be taken from their parents if;• they had previously received state social or welfare services, • they had parents who used illegal drugs, • they had previously been abused, and/or• they had limited parental supervision.


• Analyzing and Interpreting Data in Evaluation Research – cont’d– The PPC study made two primary recommendations.

• The PPC process should be managed by a team of professionals.

• Police officers should only use the PPC process for children who;– are being or have been abused or neglected,– have received any type of social or welfare services,– Have limited parental supervision, and– Are younger than 16 years of age.


• Communicating the Findings from Evaluation Research– Most evaluation studies are published ‘in-house’.– Prior to the release of the report it is prudent to allow

affected individuals to review the report for accuracy.– Often evaluation researchers are asked to present

their findings in multiple formats to different audiences.

– On occasion the results of an evaluation report will cause a public controversy.


• Communicating the Findings from Evaluation Research - cont’d– The initial report (for agency consumption) was

brief to accommodate busy practitioners.– Another version of the report was published in a

scholarly journal.– In both situations the reports clearly identified the

limitations of the study.

Getting to the Point

• Evaluation research is designed to determine whether a program, process or policy is achieving its intended outcome and/or resulting in any unintended consequences.

• Evaluation research is becoming increasingly critical to criminal justice practice.


• Evaluation research typically involves either;– a cost/benefit analysis, in which the costs

associated with a policy or program are weighed against the benefits, or

– a multiple methods approach, in which various data collection methods are used to evaluate a policy or program.


• Evaluation research is effective at demonstrating and monitoring the effectiveness or efficiency of new and existing programs, processes or policies.

• It can also identify the unintended consequences of and provide objective information about controversial programs, processes and policies.


• Evaluation research is not effective; – without agreed upon objectives, or – when a program is mandated by law, policy or

higher authority.


• Evaluation research can be exploratory, descriptive or explanatory depending on the researcher’s objective.

• It is almost always applied because the researcher usually intends for the results to influence policies and programs.


• Although evaluation research reports can be found in scholarly journals, the overwhelming majority of them are in the possession of; – various government agencies, – privately sponsored research organizations, and – individual criminal justice agencies.

• Such reports are typically available through the internet or can be made available through a direct request to the individual agency in possession of the report.


• Evaluation research projects may begin with general research questions or explicit hypotheses depending on the nature of the project.

• In either case, the primary purpose is to investigate whether a program, process or policy is having its intended results.


• In evaluation research, researchers typically start by defining and operationalizing policy or program effectiveness.

• Because criminal justice agencies are publically funded, the researchers who evaluate criminal justice programs should be mindful of the various political objectives when developing measures of program effectiveness.


• The actual method or methods used by evaluation researchers to collect data depends on the type of research question(s) asked.

• The use of multiple methods has become particularly popular in evaluation research.


• Gaining access to data during evaluation research is sometimes difficult.

• In many cases, individuals who make the policy or work in the program are less than willing to give information, lest the policy or program be found to be unsuccessful or inefficient.


• Criminal justice data is also notoriously incomplete, inaccurate and inconsistent.

• As a result, evaluation researchers should expect that they will need to spend considerable time preparing the data for analysis.


• The interpretation process in evaluation research is not much different than that required in any research project.

• But evaluation researchers should bear in mind that criminal justice data are notoriously incomplete and inconsistent.

• This limitation should always been acknowledged in the interpretation of the results.


• Evaluation researchers are often asked to present their findings in multiple formats and to multiple audiences.

• When the media get involved with disseminating the results of the research, evaluation researchers should be mindful of the complicated dynamics of public opinion and messaging.

Research Methods in Crime and Justice

Chapter 13Evaluation Research Methods

research methods in crime and justice chapter 13 evaluation research methods

Documents

evaluation research

evaluation research

evaluation research

evaluation research

types of evaluation

research question

evaluation studies

pcc evaluation