summative evaluation

Summative EvaluationSummative Evaluation

The Evaluation The Evaluation after after implementationimplementation

Involves _______ dataInvolves _______ data

CollectingCollectingAnalyzingAnalyzing

SummarizingSummarizing

For the purpose ofFor the purpose of

Giving decision makers Giving decision makers information on the information on the

effectiveness and efficiency effectiveness and efficiency of instructionof instruction

Effectiveness of Content Effectiveness of Content

Instruction solve the problem?Instruction solve the problem? Criterion created prior to evaluation?Criterion created prior to evaluation? Was the criterion established Was the criterion established

in conjunction within conjunction withthe needs assessment?the needs assessment?

SpecificallySpecifically

Did learners achieve the objectives?Did learners achieve the objectives? Learners feeling about instruction?Learners feeling about instruction? What were the costs?What were the costs? How much time did it take?How much time did it take? Was instruction implemented Was instruction implemented

as designed?as designed? What unexpected outcomes?What unexpected outcomes?

Alternative Approaches to Alternative Approaches to Summative EvaluationSummative Evaluation

ObjectivismObjectivismSubjectivismSubjectivism

ObjectivismObjectivism

Based on empiricismBased on empiricism Answering questions on the bases of Answering questions on the bases of

observed dataobserved data Goal based and replicable, uses the Goal based and replicable, uses the

scientific methodscientific method

SubjectivismSubjectivism

Employs expert judgmentEmploys expert judgment Includes qualitative methodsIncludes qualitative methods

observation and interviews observation and interviews evaluate contentevaluate content

““Goal Free”Goal Free” evaluators haven’t a evaluators haven’t a

clue about the goalsclue about the goals

Objectivism Objectivism (limitations)(limitations)

Examine only a limited number of Examine only a limited number of factorsfactors

May miss critical effectsMay miss critical effects

Subjectivism Subjectivism (limitations)(limitations)

Are not replicableAre not replicable

Biased by idiosyncratic experiences, Biased by idiosyncratic experiences, perspectives, or the people who do perspectives, or the people who do the evaluationthe evaluation

May miss critical effectsMay miss critical effects

Designer Role in Summative Designer Role in Summative Evaluation?Evaluation?

Somewhat controversialSomewhat controversial

Timing of Summative Timing of Summative Evaluation?Evaluation?

Not in the first cycleNot in the first cycle

Summary DiagramSummary Diagram FormativeFormative

Design ReviewsDesign ReviewsExpert ReviewsExpert ReviewsOne-to-one Eval.One-to-one Eval.Small Group Eval.Small Group Eval.Field TrialsField TrialsOngoing Eval.Ongoing Eval.

SummativeSummativeDetermine Goals of Determine Goals of the Evaluationthe EvaluationSelect OrientationSelect OrientationSelect DesignSelect DesignDesign or Select Design or Select Evaluation MeasuresEvaluation MeasuresCollect DataCollect DataAnalyze DataAnalyze DataReport ResultsReport Results

Goals of the EvaluationGoals of the Evaluation

What decisions must be made?What decisions must be made? What are the best questions?What are the best questions? How practical is it to gather data?How practical is it to gather data? Who wants the answer to a question?Who wants the answer to a question? How much uncertainty?How much uncertainty?

Orientation of EvaluationOrientation of Evaluation

Goal-based or goal-freeGoal-based or goal-free A middle ground?A middle ground? Quantitative or qualitative Quantitative or qualitative

appropriate?appropriate? Experimental or naturalistic Experimental or naturalistic

approach?approach?

Select Design of EvaluationSelect Design of Evaluation Describes what data to collectDescribes what data to collect When the data will be collectedWhen the data will be collected And under what conditionsAnd under what conditions Issues to consider:Issues to consider:

How much confidence must we have that the How much confidence must we have that the instruction caused the learning? (internal instruction caused the learning? (internal validity)validity)

How important is the generalizability? How important is the generalizability? (external validity)(external validity)

How much control do we have over the How much control do we have over the instructional situation?instructional situation?

Design or Select Evaluation Design or Select Evaluation MeasuresMeasures

Payoff outcomesPayoff outcomes Is the problem solved?Is the problem solved?

Costs avoidedCosts avoided Increased outputsIncreased outputs Improved qualityImproved quality Improved efficiencyImproved efficiency

Design or Select Evaluation Design or Select Evaluation Measures Measures (2)(2)

Learning OutcomesLearning Outcomes Use instrument you’ve already Use instrument you’ve already

developed for the summative evaluationdeveloped for the summative evaluation But measure the entire programBut measure the entire program


AttitudesAttitudes Rarely the primary payoff goalsRarely the primary payoff goals Ask about learner attitudes towardAsk about learner attitudes toward

learninglearning instructional materialsinstructional materials subject matter subject matter

Indices of appealIndices of appeal attention, likeableness, interest, relevance, attention, likeableness, interest, relevance,

familiarity, credibility, acceptability, and familiarity, credibility, acceptability, and excitementexcitement


Level of ImplementationLevel of Implementation degree to which the instruction was degree to which the instruction was

implementedimplemented CostsCosts

Cost-feasibilityCost-feasibility Cost-effectivenessCost-effectiveness

Alternative DesignsAlternative Designs

Instruction then posttestInstruction then posttest Pretest then instruction then posttestPretest then instruction then posttest

The ReportThe Report

SummarySummary BackgroundBackground

Needs assessment, audience, context, Needs assessment, audience, context, program descriptionprogram description

Description of evaluation studyDescription of evaluation study Purpose of evaluation, evaluation of the Purpose of evaluation, evaluation of the

design, outcomes measured, design, outcomes measured, implementation measures, cost-implementation measures, cost-effectiveness info., analysis of effectiveness info., analysis of unintentional outcomesunintentional outcomes

The Report The Report (continued)(continued)

ResultsResults Outcomes, implementation, cost-Outcomes, implementation, cost-

effectiveness info., unintentional effectiveness info., unintentional outcomesoutcomes

DiscussionDiscussion causal relationship between program & causal relationship between program &

resultsresults Limitation of studyLimitation of study

Conclusion & RecommendationsConclusion & Recommendations

SummarySummary

Summative evaluation is after Summative evaluation is after implementationimplementation

Limitations of subjective and Limitations of subjective and objective evaluationobjective evaluation

What to include in the reportWhat to include in the report

summative evaluation

Education