1

Annual Outcomes Assessment Report Workshop:

Support, Research and Service UnitsOffice of Institutional Planning, Research and Effectiveness (OIPRE)

October 6 – 8, 2015Drake LRC

2

Workshop Objectives

1. Provide a rationale for engaging in outcomes assessment2. Describe the concept of institutional effectiveness3. Outline steps in the outcomes assessment process4. Describe AAMU outcomes assessment process requirements5. Outline annual outcomes assessment report timeline

3

Rationale for Assessment

• Outcomes assessment is a formative process• Outcomes assessment is not an end; rather, it is a means for improving unit

processes/functions

• Motivation to engage in outcomes assessment• Intrinsic value• External accountability demands

4

Rationale for Assessment

• External accountability demands• Outcomes assessment is a requirement of our regional accreditor – SACSCOC• Without regional accreditation our students are not eligible for federal

student aid

• Intrinsic value• Outcomes assessment allows us to determine how well we are fulfilling our

role at the university and where appropriate, make the necessary changes to our operations to ensure students reach the expected performance levels

5

Institutional Effectiveness

• Institutional effectiveness is the systematic, explicit, and documented process of measuring institutional performance against mission in all aspects of an institution (p. 16).• A commitment to continuous improvement is at the heart of an on-

going planning and evaluation process. It is a continuous, cyclical process that is participative, strategic, flexible, relevant and responsive (p. 16).

Source: Southern Association of Colleges and Schools Commission on Colleges, Resource Manual for the Principles of Accreditation

6

University Mission StatementAlabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, of knowledge

and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The university offers contemporary baccalaureate, master’s educational specialist and doctoral level degrees to prepare student for careers in the arts, sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered

educational environment for the emergence of scholars, scientist, leaders and critical thinkers, who are equipped to excel through their contribution and leadership in a 21st century national and global society.

Strategic Priority 2: Strengthen Structures,

Operations, and Systems to

Promote and Support

Organizational Excellence and

Stewardship

Strategic Priority 6:

Enhance university Engagement

through Expanded Outreach

Strategic Priority 5:

Enhance the University’s Image and Recognition

Strategic Priority 4:

Secure the University’s

Financial Future

Strategic Priority 3:

Upgrade University

Infrastructure and Facilities

Strategic Priority 1:

Enhance AAMU’s Distinctiveness

and Competitiveness

UNIT 1MISSION

UNIT 2MISSION

UNIT 3MISSION

UNIT 4MISSION

UNIT 5MISSION

7

Institutional Effectiveness

• Comprehensive Standard 3.3.1 (Institutional Effectiveness – unit level)

The institution 1) identifies expected outcomes*, 2) assesses the extent to which it achieves these outcomes, and 3) provides evidence of improvement based on analysis of results in the following areas…

*For academic programs this includes student learning outcomes.

8

Institutional Effectiveness

… in each of the following areas:• 3.3.1.1 Educational Programs• 3.3.1.2 Administrative Support Services• 3.3.1.3 Academic and Student Support Services• 3.3.1.4 Research• 3.3.1.5 Community/Public Service

9

Assessment Steps

1. Develop unit mission statement/purpose2. Develop outcomes3. Develop appropriate assessment measures4. Develop desired performance targets5. Analyze results6. Develop and implement improvement plans

Mission Statement

Step 1

OutcomesStep 2

Assessment Measures

Step 3

Performance TargetsStep 4

AnalysisStep 5

Improvement PlansStep 6

Assessment Steps

11

Unit Mission Statements

• Mission statements should…

• Be specific, distinctive• Clearly state the purpose of the unit• Identify the unit’s stakeholders• State the values and principles that guide the unit• Speak to what the unit is currently doing• Align with AAMU’s strategic goals

12

Mission Statement Template

• The mission of <<enter unit name>> is to <<unit’s primary

purpose(s)>> by providing <<unit’s primary functions or activities>> to

<<unit’s primary stakeholders>>.

13

Sample Mission Statements

• The Accounting Office seeks (1) to provide administrators with accurate and timely financial data to assist them in the management of the institution’s resources, and (2) to ensure that financial records are maintained in accordance with generally accepted accounting principles and guidelines as established by State and Federal agencies.• The Office of Assessment serves to promote excellence in student

learning, educational practices and quality of service by establishing and maintaining a campus culture of self evaluation and improvement all levels of the institution.

14

Sample Mission Statements

• The library strives to support and stimulate teaching and learning by providing an environment in which instruction and research can flourish. The college’s libraries aim to acquire, preserve, provide access to, and disseminate recorded knowledge in all its forms. Access will be provided through traditional and technological methods. The library will provide bibliographic, reference, and instructional support to student faculty, staff and the community. • The Registrar’s Office is responsible for processing applications for

admission, processing and reporting student enrollment, maintaining registered student records, receiving grades, and the processing of transcript information.

15

Unit Outcomes

Unit outcomes should…• Focus on current services, processes or instruction• Be clearly and succinctly stated• Be under the control or responsibility of the unit• Be meaningful, not trivial• Be measurable• Be actionable; lead to improvements• Lead to non “yes/no” results

16

Unit Outcomes

Tips for developing unit outcomes:

1. Develop a master list of what the unit does (services or functions)

2. From this list, identify the three most important services or

functions

17

Outcomes and the Annual Assessment Report• For the annual assessment report, support, research and service units

must report on:

• Three (3) outcomes

18

Types of Outcomes

• Efficiency• Accuracy• Effectiveness• Client satisfaction• Quality• Comprehensiveness• Compliance with standards

19

Unit Outcome Example

Efficiency Example

The Registrar’s Office processes transcript requests in a timely manner.

20

Unit Outcome Example

Accuracy Example

Purchasing accurately processes purchase orders.

21

Unit Outcome Example

Effectiveness Example

The Human Resources office provides effective new employee orientation services.

22

Unit Outcome Example

Quality Example

The Honors College provides high quality advising to its students.

23

Unit Outcome Example

Comprehensiveness Example

Financial Aid provides comprehensive customer service.

24

Unit Outcome Example

Compliance with Standards Example

The Finance Department complies with standard accounting practices.

25

Critiquing Unit Outcomes

• Information Technology will procure new networking equipment.

• This is not a current function/process of the office. Also, this outcome cannot lead to improvements.

• Information Technology provides reliable networking service to the campus.

26

Critiquing Unit Outcomes

• The Career Center will ensure that 90% of graduates obtain jobs or pursue a higher degree.

• This outcome is not under the unit’s control. As written, the outcome cannot lead to improvements.

• The Career Center effectively informs students about career or higher education opportunities.

27

Critiquing Unit Outcomes

• Athletics will ensure fans enjoy an attractive and entertaining football schedule.

• This outcome is not under the unit’s control. Also, this outcome is difficult to measure.

• Athletics provides athletic facilities that are safe.

28

Critiquing Unit Outcomes

• Payroll will always issue paychecks on time.

• This outcome leaves no room for improvement (100% target).

• Payroll accurately reconciles and remits benefit deductions.

29

Critiquing Unit Outcomes

• The Library will offer prompt, courteous, professional service to students, faculty, and staff regarding books, periodicals, ebooks, and interlibrary loans.

• This outcome is bundled – the statement includes multiple outcomes.

• The Library offers satisfactory service to its constituents.

30

Assessment Measures

• Now that you have developed sound outcomes, you must

1. Indicate how the outcome will be assessed

31

Assessment Measures

• Assessment measures should be designed to yield meaningful,

actionable data that can lead to improvements.

32

Assessment Measures

• Should…• Be directly related to the outcome• Address all aspects of the outcome over time• Provide measurable results• Provide adequate data for analysis• Provide actionable data

33

Assessment Measures

• Should…• Outline a systematic way to assess the outcome• Include the following details:• What will be collected• When the data will be collected• What assessment tool will be used• How will the data be collected

34

Choosing Assessment Measures

• Direct Assessments

• Are based on direct analysis of the service, process or function in question

• Indirect Assessments

• Are based on stakeholder attitudes and perceptions of the service, process or function of interest

35

Choosing Assessment Measures

• Where possible, multiple direct assessment methods should be used.

36

Assessment Measures

• Quantitative Assessment Measures

• Response time

• Accuracy

• Cost savings

• Efficiency

37

Assessment Measures

• Client Satisfaction Assessment Methods

• Surveys

• Comment cards

• Focus groups

• Observation of behavior

38

Assessment Measures

• External or Peer Comparison Assessment Methods

• Auditors

• Fire Marshall

• Peer institutions

• Other outside agencies

39

Performance Targets

• Why do we need performance targets?

• Allows the unit to identify desired performance levels

• Provides standard for determining success

• Puts data analysis in perspective

40

Performance Targets

• Targets should…

• Use component (not respondent) as unit of analysis

• Relate to outcome and assessment method

• Be specific (“x% of responses will be correct for each question” or “y% of

scores for each item on rubric or survey will be at least 4”

• Avoid vague words such as “most” or “majority”

• Avoid “all” or “100%” targets

41

Critiquing Measures and Targets

• The Travel Office will conduct a training session each month on the institution’s travel policies. At least 75% of all offices will send a representative to the training. • As stated, this outcome cannot lead to

improvements. Also, attendance at the training sessions is out of the unit’s control.

• The Travel Office will conduct a training session each month on the institution’s travel policies and administer a five-question survey at the end.

• At least 75% of all responses to each question will be correct.

42

Critiquing Measures and Targets

• The University Police Office will collect comment cards from its walk-up service window. The average score for each card will be at least a 3.5 out of a possible score of 5.

• As stated, this outcome cannot lead to improvements. Also, the outcome uses averages which focus on the respondent rather than components. Instead, we want to know what proportion of respondents reached the target score for each item on the card.

• The University Police Office will collect comment cards from its walk-up service window.

• For each item on the card, at least 80% of responses will be a 4 of 5 (Satisfied or Very Satisfied).

• The items on the comment card represent this measure’s components.

43

Critiquing Measures and Targets

• Custodial Services will use a checklist to monitor the cleanliness of the Administration Building’s conference room for a one-month period. No more than four of the ten items on the checklist will be deficient.• This measure doesn’t assess overall

effectiveness of custodial services. Need to look more broadly. Also, the time frame for the assessment is too limited. Finally, the measure as stated doesn’t readily lend itself to targeting improvement efforts.

• Custodial Services will use a checklist to monitor the cleanliness of classrooms for a two-month period.

• No item on the checklist will be found to be deficient more than 10% of the time.

44

Critiquing Measures and Targets

• The Cafeteria plans to expand its salad bar substantially in December. The Cafeteria will count the number of people who visit its salad bar before and after the expansion. The target is an increase of 10%.• This measure will not yield actionable

data. Because the performance target is based on attendance it may measure things other than the impact of the new salad bar (may reflect seasonal changes in attendance patterns).

• The Cafeteria plans to expand its salad bar substantially in December. At the end of the spring semester, a survey will be conducted to assess satisfaction with the expansion. Also, three focus groups will be conducted to identify additional potential improvements.

• For each item on the survey, at least

75% of responses will be Satisfied or Very Satisfied.

45

Critiquing Measures and Targets

• Student Support Services has created a new website. The number of hits to the website will be tracked for two months. The target is an average of ten hits per day.• This measure doesn’t provide

actionable data as it simply tracks website traffic, not the quality of the website.

• Student Support Services has created a new website. Several focus groups representing various constituencies on campus will be conducted to determine how useful, attractive, and functional the new website is.

• Since focus groups do not yield quantitative data, no target is set.

46

Analysis of Results

• Here we move from planning the assessment to conducting it

• Provides data on which you can base changes

47

Analysis of Results

Analysis should…

• Provide detailed data (avoid “a majority” or “most”)• Use specific numbers (no rounding)• Avoid technical language• Be clearly and succinctly presented• Align with outcome and target• Support actions taken later to foster improvement

48

Improvement Plans

• Should…

• Address outcomes where the performance target was not met

• State what future actions will take place to address the outcome

49

Types of Improvement Plans

• Procedure, process change• Technology update• Assessment method change• Target adjustment

50

Class Example

Outcome

• Institutional Planning, Research and Effectiveness (OIPRE) provides a website that is informative and useful.

51

Class Example

Assessment Measure• OIPRE staff will develop a series of 12 questions representing

information that should be readily available from the OIPRE website. Fifty people from outside of OIPRE will be asked to use the website to answer the 12 questions and make any suggestions for enhancing the site.• OIPRE will create a spreadsheet to record the number of respondents

who found the correct answer to each of the 12 questions. They will also note any suggestions for enhancing the website.

52

Class Example

Target

• For each of the 12 questions, at least 40 of the 50 respondents will find the answer.

53

Class Example

Analysis

Question Topic Area Answers Found Total Respondents1 Enrollment 48 502 Retention 42 503 Graduation 45 504 Majors Offered 34 505 Academic Leadership 32 506 Administrative Officers 41 507 Faculty and Staff 44 508 Freshmen Profile 47 509 Salary Information 40 5010 General Campus Info 48 5011 Peer Group 35 5012 OIPRE Contact Information 33 50

54

Class Example

Analysis

50 respondents were tasked with finding twelve pieces of information from the OIPRE website. At least 40 respondents were able to identify eight of the twelve items. Fewer than forty respondents were able to identify four of the twelve items (majors offered, academic leadership, peer group and OIPRE contact information).

55

Class Example

Improvement plans

Respondents indicated difficulty finding OIPRE contact information on the website. A ‘Contact Us’ banner will be placed at the top of each page so site visitors are always one click away from OIPRE contact information. Respondents indicated having to search through multiple pages to find Deans and Department Chairs and majors offered within each school. OIPRE will re-format the web page to display a comprehensive listing of academic leaders and majors on a single page rather than showing this information for the respective colleges on college-specific pages.

56

Annual Assessment Report TemplateUniversity Mission: Alabama Agricultural and Mechanical University is a public, comprehensive 1890 Land-Grant institution, committed to access and opportunity, and dedicated to intellectual inquiry. The application of knowledge and excellence in teaching, research and service is responsive to the needs of a diverse student population and the social and economic needs of the state and region. The University offers contemporary baccalaureate, master’s, educational specialist and doctoral level degrees to prepare students for careers in the arts, sciences, business, engineering, education, agriculture and technology. As a center of excellence, the University is dedicated to providing a student-centered educational environment for the emergence of scholars, scientists, leaders and critical thinkers, who are equipped to excel through their contributions and leadership in a 21st century national and global society.

Unit Mission/Purpose: Step 1 Unit Outcome Assessment Methods Performance

TargetsAssessment Results Improvement Plans Improvements

ImplementedStep 2

Step 3 Step 4 Step 5 Step 6 Step 6

57

Assessment Dates

• Assessment Plans Due• Steps 1 – 4• Mission, outcomes, measures and targets• Friday, November 13, 2015

• Assessment Reports Due• Steps 5 – 6• Findings, improvement plans• May ??, 2016

58

OIPRE Contact Information

Office of Institutional Planning, Research and Assessment306 Patton Hall

James Walke, Directorjames.walke@aamu.edu 256-372-8876

Dianne Bozeman, Assessment Coordinatordianne.bozeman@aamu.edu 256-372-8644