www.accjc.org october 28, 2010 use of research data to assess and improve educational quality and...

www.accjc.org October 28, 2010

Use of Research Datato Assess and Improve Educational Quality

and Institutional Effectiveness

Presentation at California Community Colleges Chief Instructional Officers Workshopby

Dorte KristoffersenAssociate Vice President, ACCJC

1


Presentation Outline

•Some Definitions

•Characteristics of Good Data and Data Sources

•Data-driven Decision-making

•Purposes of Accreditation

•Types of Data and Data Requirements

•Examples of Effective Use of Data

•Concluding Remarks

2


Definitions of Evidence and Data

General definitions:

•Evidence includes everything that is used to determine or demonstrate an assertion

•Data refers to groups of information that represent the qualitative or quantitative attributes of a variable or set of variables.


Use of the Terms Evidence and Data in Accreditation

•Evidence is all information included in the accreditation process, i.e. Self Evaluation Reports of Educational Quality and Institutional Effectiveness, information supporting the claims and descriptions in the Self Evaluation Report, interviews during the site visit, and previous External Evaluation Reports.

•Data is information deliberately collected by the institution to provide it with clear information about the quality of its education and services.


Characteristics of Good Data

•For data to be a useful and reliable source of information and for decision-making, it needs to have the following characteristics. It should be:

Accurate and tested for validity and significance Up-to-date and complete in terms of aspects of analysis it is

meant to coverConsistently used From reliable sourcesLongitudinal and disaggregated as appropriate

I5


Data Sources

•There are several sources of data than an institution can draw information from. Data sources can be internal, external or both. Examples of data sources are:

Institutional data, such as statistics and survey resultsDemographics at the local/district, regional or federal level Data reported to State Government and/or relevant

associations, such as Cal-PASSOther sources relevant for the institution

6


Data-driven decision-making

•It is a sound principle when making judgments to act on what we know. Judgments should not be based on assumptions and feelings.

•Data is a necessary knowledge-base for decision-making

•The information that an institution collects, analyzes, and reflects upon should be designed to answer questions the institution has raised against its mission and institutional objectives, and the information will thus determine what action the institution takes to support the improvement of educational quality and support student success

7


Data-driven decision-making continued

•‘No News; is normally ‘Good News’

But ‘No data’ does not mean ‘good news’.

It means no sense of bearings and no direction for the future.

8


Purposes of Accreditation

•To provide assurance to the public that education provided by institutions meets acceptable levels of quality.

•To promote continuous institutional improvement.

•To maintain the high quality of higher education institutions in the region/nation.

9


Types of Data

•The principle of data-driven decision-making should apply to an institution’s reflections on its achievement of its mission and institutional objectives and its compliance with the Accreditation Standards

•The Commission requires various kinds of specific data

•Most of the data is required to support institutions’ claims of compliance with the Accreditation Standards and some of the data relate to the U.S.D.E requirements

•The Commission also expects that institutions collect data which is relevant for the institutions’ internal continuous quality improvement activities

10


Commission Data Requirements

•Data about incoming students, i.e.:

Student demographics

Student enrolment data across instructional programs

Student educational goals

Student development needs

11


Commission Data Requirements Continued

•Data about incoming students should include information about students studying in Distance Education or Correspondence Education mode, such as:

Account of programs, courses, certificates where 50% or more is offered in distance education or correspondence education mode

Annual growth in headcount enrollment

12



•Data about Program Review, such as:

Number of enrolled students

Student learning outcomes at program and course level

Assessment results

13



•Data on student grievances and complaints, including follow-up

•Data on Internationalization, i.e.:

Numbers of international campuses

Numbers of programs for non-U.S. students

Numbers of non-U.S. students enrolled

14



•Two types of data are particularly important as they relate directly to student success, i.e.

Student Achievement Data

Student Learning Outcomes Data

•These two types of data will be explored in more detail as examples

15


Student Achievement Data

•Data on student achievement has been long required by the ACCJC. Institutions already use some common measures to examine student achievement, and some of these measures derive from federal regulatory language.

16


Student Achievement Data Continued

•Required Student Achievement Data:

Retention rates from term to term

Student progression to the next course/level of course, including from pre-collegiate to collegiate level

Graduation rates (certificate or degree)

Job placement rates

Licensure pass rates

Transfer rates to four-year institutions

17


Value of Student Achievement Data•Student Achievement Data is end-point data providing an

institution with key information about the achievement of its institutional mission and in terms of student success

•Such data will:If collected longitudinally and analyzed continuously provide

the institution with indications as to whether improvements in pedagogy or services may be required to improve student progression and completion

Keep institutions informed about fluctuations and serve as a warning if rates decrease and trends need to be reversed

18


Value of Student Achievement Data Continued

•Student achievement data will also: Provide information to the institutions about barriers to

completion and transfer

Present trends that identify the need for institutions to explore the barriers through the collection of additional data and/or conduct separate surveys

If collected in disaggregated form inform institutions as to whether all students progress and complete their studies or if attention needs to be given to special groups

19


New Commission Requirement

•As of Fall 2012, Self Evaluation Reports of Educational Quality and Institutional Effectiveness should include data on key student achievement measures in disaggregated form by age, gender, race/ethnicity, socio-economic status and other measures that the institution considers relevant for its population

20


An Example of Student Achievement Data

•Research study by Institute for Higher Education Leadership and Policy (IHELP). Divided we Fail: Improving Completion and Closing Racial Gaps in California’s Community Colleges, October 2010

•IHELP followed 250.000 degree seeking Californian community college students over the course of six years

•The goal of the study was to use it to identify solutions to improve completion rates for all students and closing the disparities across racial/ethnic groups

•The study is conducted at a generic level, but the findings and how the can be used can serve as an example for individual colleges

21


An Example Continued

• Too few students reach the milestones: racial/ethnic disparities abound:

Too many students fail to complete

Six years after enrolling 70% of degree-seeking students had not completed a certificate or degree and had not transferred to a university (most had dropped out)

Failure rates among black students (75%) and Latinos (80%) are higher

22


An Example Continued

•Critical milestone is missed:

Only 40% of degree-seeking students had earned at least 30 college level credits at a CCC.

35% of Latino students and 28% of black students reached this milestone

Fewer Latinos who reach the above milestone complete a certificate, degree or transfer (47%) as compared to white (60%), Asian-Pacific Islander (58%) and black (53%) students

23


Student Learning Outcomes Data

•The Commission also requires institutions to demonstrate educational effectiveness by collecting and analyzing student learning outcomes data

•Student Learning Outcomes data:Is end-point data

Consists of learning outcomes at course, program and degree level

Is derived from summative assessment

24


Value of Student Learning Outcomes Data

•Student Learning Outcomes data:

Provides information about how well students are able to achieve institutional and programmatic missions and objectives, and

Is therefore a key source of data for Program Review

May help institutions identify barriers to student learning and student success and how these barriers can be addressed

May identify whether improvements in curriculum and teaching and learning methods are warranted to increase the achievement of learning outcomes

25


Key Questions to Ask about Data Collection and Data Use

•What is the mission of the institution (Institutional, Programmatic)?

•What are the institutional objectives?

•What is the institution trying to achieve in terms of its institutional missions and educational quality, including possible goals and targets?

•What data is relevant to collect to gain information on institutional achievements?

26


Key Questions to Ask continued

•What are the data sources, i.e. internal and/or external?

•Who should be involved in the data analysis?

•How should the outcomes of the analysis be reported?

•Who is responsible for taking action and ensuring that improvements are made?

27


Concluding Remarks

• Data is collected intentionally and its meaning and relevance has been considered

• It is purposeful and designed to provide information on institutional achievements

• It is collected systematically and continuously

• Data is interpreted, reflected and acted upon

• It is used for improvement

28


‘Learning without thought is labor lost;

thought without learning is perilous.’

(Confucius 551-470 BC)

29


Thank you

ACCJC/WASC10 Commercial Blvd, Suite 204

Novato, CA 94949Tel: 415-506-0234Fax: 415-506-0238

Website: www.accjc.orgEmail: [email protected]

30

www.accjc.org october 28, 2010 use of research data to assess and improve educational quality and...

Documents