architecting evaluation and assessment using a design ... · architecting evaluation and...
TRANSCRIPT
Architecting Evaluation and Assessment—Using a Design-based
Approach to Develop a Complete and Systematic Approach
Presenters:
Dr. Douglas Gilbert – University of the Rockies
Dr. Peter Pintus – University of the Rockies
Dr. Olin Oedekoven – Peregrine Academic Services
@ACBSPAccredited #ACBSP2015
Agenda
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
2 @ACBSPAccredited #ACBSP2015
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
3 @ACBSPAccredited #ACBSP2015
The Assessment Challenge: Four Key Questions
1. Do we know what graduates need to know and be able to do?
2. Does the curriculum provide what graduates need to know and be able to do?
3. Are students on a learning path for relevant knowledge, skills, and abilities?
4. Do graduates have the competencies demanded by employers?
4 @ACBSPAccredited #ACBSP2015
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
5 @ACBSPAccredited #ACBSP2015
Key ACBSP Assessment Requirements
• Criterion 4.1 – Must have a “learning outcomes assessment program”
a. Program learning objectives (PLOs) b. An assessment process c. Internal assessment approach d. External assessment approach e. Formative and summative assessment approach
• Criterion 4.2 – Trends
• Criterion 4.3 – Time series benchmarked comparisons
and also…
6 @ACBSPAccredited #ACBSP2015
Setting strategic direction
•Standard 2 – “Process … to better
address key student and program
performance requirements”
7 @ACBSPAccredited #ACBSP2015
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
8 @ACBSPAccredited #ACBSP2015
What is a “complete and systematic approach”?
• Complete — “[H]aving all necessary parts : not lacking anything”
(Merriam-Webster.com. 2015. http://www.merriam-webster.com)
• Systematic — “The term "systematic" refers to approaches that are repeatable
and use data and information so learning is possible.” (ACBSP Bac/Grad Standards and Criteria—Rev J., December 2014, p. 71).
— “Well-ordered, repeatable, and exhibiting the use of data and information so that learning is possible.” (Baldrige Excellence Framework 2015-16, p. 53).
• Approach — “[T]he methods used by an organization to address the criteria
requirements. Approach includes the appropriateness of the methods to the requirements and the effectiveness of their use.” (ACBSP Bac/Grad Standards and Criteria—Rev J., December 2014, p. 62).
9 @ACBSPAccredited #ACBSP2015
Is this your assessment design?
10 @ACBSPAccredited #ACBSP2015
A Problematic Assessment System
11
Data: Student surveys and
classroom work, accumulated by
faculty in classroom
Uses: Individual Faculty
Efforts
Data: Departmental studies, e.g.
accounting, OB, etc.
Uses: Departmental Improvement
Data: Institutional res--
• Surveys (NSSE, EOCS, etc.)
• Std-ized tests • IPEDS • Alumni
Uses: Reporting
Uses: Program reviews
Data: Programmatic
assessments, e.g. portfolios.
Uses: Program reviews
Data: Academic support
and co-curricular, e.g. library, IT, student
affairs, etc.
Uses: Individual dept. improvement
Uses: Program reviews
Faculty Governance
Curriculum approval
Provost / VPAA
Assessment Advisory Committee
Dean / Dept. Chair
Program Chairs
Strategic Planning
VPs
Institutional Research
Adapted from Wolvoord (2010) @ACBSPAccredited #ACBSP2015
Common “Disconnects”
• Structure — Academic data islands of classroom, departments, and programs
— Institutional and instructional data not integrated
• Process — Blurred or non-existent assessment roles and responsibilities
— Non-systematic or isolated cycles of improvement
• Technology — “One-off” faculty classroom technology approaches
— Wide range of surveys and assessment automation
• People — Separated, disjointed assessment responsibilities
— Holistic knowledge management competencies lacking
— Little cross-functional / inter-departmental perspective
12 @ACBSPAccredited #ACBSP2015
A Design-Based Assessment Architecture
13
Business Unit Strategy
Mission Vision Values
Structure
Data Definitions
KPIs / SLOs
Control Limits
Technology People
Architecture Enablers
Policies Procedures Instruments
Process
Performance
Measurement
Performance
Analysis / Review
Performance
Improvement
Technology
Standards
Organizational
Learning Focus
Analysis
Technology
Knowledge Mgmt
Competencies
Collection
Technology
Process-based
Organization
@ACBSPAccredited #ACBSP2015
The Common Singular Focus
14
Business Unit Strategy
Mission Vision Values
Structure
Data Definitions
KPIs / SLOs
Control Limits
Technology People
Architecture Enablers
Policies Procedures Instruments
Process
Performance
Measurement
Performance
Analysis / Review
Performance
Improvement
Technology
Standards
Organizational
Learning Focus
Analysis
Technology
Knowledge Mgmt
Competencies
Collection
Technology
Process-based
Organization
@ACBSPAccredited #ACBSP2015
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
15 @ACBSPAccredited #ACBSP2015
• Exams / Standardized tests
• Case study analyses
• Rubric scores for assignments, reports, and research
projects
• Rubric scores for writing, oral presentations, and
performances
• Observations of field work, internship performance,
service learning, consulting projects
• Quizzes
• Class discussion participation
Course evaluations
Grades not based on explicit criteria related to clear
learning goals
Test blueprints (outlines of the concepts and skills
covered on tests)
Percent of class time spent in active learning
Number of student hours spent on service learning
Number of student hours spent on homework
Number of student hours spent at intellectual or cultural
activities related to the course
Cours
e
Academ
ic
Pro
gra
m
Busi
ness
Unit
F
orm
ativ
e
Su
mm
ativ
e
Fo
rma
tive
S
um
ma
tive
Perfo
rmance
Managem
ent
Asse
ssment
Financial Focus
Student
Focus
Customer
Focus
Workforce
Focus
Leadership and Governance Focus
Yesterday
Today
Tomorrow
Operational Environment
Direct Indirect
Capstone projects, senior theses, exhibits, or
performances (internal to school only)
Student publications or conference presentations
Employer and internship supervisor ratings of students'
performance
Externally normed assessments, e.g. Peregrine, ETS,
etc.
Pass rates or scores on licensure, certification, or
subject area tests
Job Placement
Department or program review data
Employer or alumni surveys
Focus group interviews with students, faculty members,
or employees
Student perception surveys
Internal External
Sta
ndard
4.1
a,b
Sta
ndard
2
or
Outcome / Instrumentation Overview
16
Sta
ndard
4.1
c,d
,e
Course Level Elements
17
Program Level Elements
18
Business Unit Elements
19
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
20 @ACBSPAccredited #ACBSP2015
Overview of Examples
1. College-level KPI Reports (Standard 2)
2. Signature assignments using standard rubric (Standard 4.1c)
3. NPS – External Measure (Standard 4.1d)
4. Trending (longitudinal) dashboard (Standard 4.2)
5. External test benchmarking approach (Standard 4.3)
21 @ACBSPAccredited #ACBSP2015
College Dashboard - Layer 1 (National Baldrige Recipient Example)
22 Source: Dallas Community College District
ACBSP Standard 2
@ACBSPAccredited #ACBSP2015
College Dashboard - Layer 2 (National Baldrige Recipient Example)
23 Source: Dallas Community College District @ACBSPAccredited #ACBSP2015
College Dashboard - Layer 3
(National Baldrige Recipient Example)
24 Source: Dallas Community College District @ACBSPAccredited #ACBSP2015
Signature Assignments Using a Standardized Rubric (Standard 4.1c)
25
Standardized Rubric
@ACBSPAccredited #ACBSP2015
NPS as an External Measure (Standard 4.1d)
26
Source: Bain & Co.
http://www.netpromotersystem.com/about/measuring-your-net-promoter-score.aspx
@ACBSPAccredited #ACBSP2015
Trending (longitudinal) dashboard (Standard 4.2)
27 @ACBSPAccredited #ACBSP2015
External test benchmarking approach (Standard 4.3)
28 @ACBSPAccredited #ACBSP2015
The Challenge of Assessment
ACBSP Assessment Requirements
Using a Design Approach to Architect a Model
Multi-Tiered Assessment Instrumentation
Examples
Conclusions
29 @ACBSPAccredited #ACBSP2015
Putting it all together Signature Assignment Example
30
Signature assignment aligned to Mission, Vision, & Values
Structure
• Rubric structure
• Relevant student
learning
outcomes
• Targets
Technology
• LMS
• Rubric software
• Analysis
software
People
• Knowledge mgt
competencies
• CQI approach
• SIPOC for fit of
process
Architecture Enablers
Assessment Policies Overall assessment
cycle Rubric types
Process
• Signature
assignment
selection
• Scoring / rubric
process
• Cycles of review /
improvement
@ACBSPAccredited #ACBSP2015
Additional Resources
•Presentation available at
http://bitly.com/ACBSP2015
•White paper to be available at same
location
31 @ACBSPAccredited #ACBSP2015