Evaluation Research into the effectiveness of university learning environments

Rob PhillipsAcademic Consultant

Educational Design Project, School of Education, Curtin University, PerthAdjunct Associate Professor, School of Education, Murdoch University, Perth

My Background

• 1980s– Research scientist – theoretical chemistry and computer

science – Germany and Australia• 1990s

– Interactive multimedia designer and project manager• 2000s

– Educational designer• 2010s

– Learning and Teaching Policy work– Academic consultant – mentoring and educational design

Introductions

• Why are you here?

Book advert

Disclaimer

• The book is largely about e-learning• The ideas apply well to any learning

environment– almost all include technology these days

Overview

• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project

Fundamentals of Research

• Inquiry into the behaviour of phenomena• Exercise of human curiosity• Attempting to solve problems in the world• Attempting to understand the world

In Broad Terms

• What sort of research do you do?• What do you investigate?• What are the main characteristics of this

research?

• Work in pairs

Fundamentals of Research

• Inquiry into the behaviour of phenomena• Exercise of human curiosity• Attempting to understand the world• Attempting to solve problems in the world

What is a phenomenon?

• Inquiry into the behaviour of phenomena• What is a phenomenon?

– an instance of something – an artefact or event that is known through the senses

Natural Phenomena

• They exist naturally in our world– the seasons, gravity and waves on the ocean– we can observe and measure them

• Other natural phenomena cannot be directly observed– the inner workings of atoms and molecules– their behaviour can be measured

Artificial Phenomena

• Phenomena/ artefacts which we create through a design activity

• Tangible – aircraft, cars, etc.• Intangible – computer software

Event Phenomena

• An artefact or event that is known through the senses

• In Education– a coming-together of teacher, learners and various

resources in a particular setting, usually following some design activity

– Learning Environment

Summary

• Natural phenomena• 1 phenomenon – many instances

• Artificial phenomena– Many phenomena – many instances

• Event Phenomena– Many phenomena – single instances

Paradigms vs Ideology

Paradigms• The set of practices that define a scientific

discipline at any particular period of time (wikipedia)

• A world view underlying the theories and methodology of a particular scientific subject (http://oxforddictionaries.com)

Ideology

• The set of beliefs characteristic of a social group or individual (http://oxforddictionaries.com)

• A comprehensive and coherent set of basic beliefs about political, economic, social and cultural affairs that is held in common by a sizable group of people within a society. (http://www.socialsciencedictionary.com)

Paradigms vs Ideology

• Is your research approach based on a paradigm or an ideology?

Overview

• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project

E-learning Artefacts and Environments

• We design and use e-learning artefacts (intangible artificial phenomena)– learning objects, learning management systems,

e-portfolios• e-learning environments (event phenomena)

add the human context– an interaction between learners, teachers and

resources in a set context

Inquiring About the World (IATW)

• New science unit for a new primary and early childhood degree– First offered in 2013

• Teaching problem – How to make science interesting and engaging

• Design– Authentic learning approach– Students investigate a self-chosen environmental issue over

the whole semester

Learning Outcomes• Develop lifelong learning skills including: analytical thinking,

information management, technology skills and communication skills

• Apply processes of inquiry to carry out a scientific investigation

• Report on the outcomes of the processes of inquiry using a range of Information and Communication Technologies

• Evaluate the outcomes of the inquiry processes in the context of the science and humanities content from the Australian Curriculum

Assessment1 Project Plan 15%

2 Report Draft 20%

3 Report 45%

4 Presentation 20%

Video

• IATW Flash video

Baseline

The Learning LifecyclePhase

Design

Design

Develop

Evaluate

Trial learning env’t

Learning environment

Mature Learning environment

Learning lifecycle

The IATW Lifecycle

• On- campus and Open Universities Australia (OUA)

Date Offering

July 2013 Curtin Semester 2

May 2014 OUA Study Period 2

July 2014 Curtin Semester 2

Nov 2014 OUA Study Period 4

May 2015 OUA Study Period 2

Overview

• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project

My Invention

• The Kugelschreiber• A better way to write!

Researching the Kugelschreiber

• I know that this technology is better than old, pencil technology

• Compare the two, in writing a test• Treatment and control groups

Researching the Kugelschreiber

• I know that this technology is better than old, pencil technology

• Compare the two, in writing a test• Treatment and control groups

Kugelschreiber results were slightly worse than pencil results, but the difference was not significant at the 90% confidence level. The Kugelschreiber is no worse than a pencil.

Inquiry into Phenomena

• Natural– Goal: observing how natural phenomena work

• and ultimately understand them

• Artificial– Goal: design artefacts, improve their design and use

them • and ultimately understand them

• Event– Goal: design events, improve their design

• and ultimately understand them

Goals of Research

• Be clear about the phenomenon you are investigating

• Clearly define the goals and questions of your research

• Choose the research approach accordingly

Summary so Far

• We have thought about the phenomenon of e-learning

• We have different research goals at different stages of the e-learning lifecycle

• How will we conduct this research?

General Research Approaches

• Pure versus applied research• Hard versus soft research• Research and development• Research then development

Becher, T. (1989). Academic tribes and territories: intellectual enquiry and the cultures of disciplines. Buckingham: Society for Research into Higher Education.Jones, C., Zenios, M., & Griffiths, J. (2004). Academic use of digital resources: Disciplinary differences and the issue of progression Proceedings of the 4th Networked Learning Conference. Lancaster University, England, UK.

Boyer’s Four Scholarships

• Scholarship of discovery • Scholarship of integration• Scholarship of application• Scholarship of teaching

Boyer, E. L. (1990). Scholarship reconsidered - Priorities of the professoriate. New Jersey: Carnegie Foundation for the Advancement of Teaching.Boyer, E. L. (1996). The scholarship of engagement. Journal of Public Outreach, 1(1), 11-20.

Pasteur’s Quadrant

Stokes, D. E. (1997). Pasteur's Quadrant: Basic science and technological innovation. Washington, D.C.: The Brookings Institution.

What is evaluation?

• The term is used imprecisely• Evaluation implies judgement and decisions• Formative evaluation:

– judgements about improving something• Summative evaluation:

– judgements about the merit or worth of something

Investigating a New Learning Environment

• Questions:– Does the environment function as designed?– How can the environment be improved?

• These are evaluation judgements• Is it any good???

Investigating a Mature Learning Environment

• A study of a mature learning environment may shed light on – how learners engaged with the designed learning

environment – why some learners achieved at different levels– how some learners used the learning environment

to achieve a deeper understanding• These are educational research questions• The goal is increased understanding

Evaluation vs Research

• Studies of learning environments involve a mixture of– Evaluation: making judgements about improvement and

value and worth– Research: increasing understanding

• Continuum between judgements and understanding– Does it function as designed?– How can it be improved?– How does this lead to improved student learning?

• Similar methods – different goals

Judgments

Understanding

IATW - currently

• Pre- and post-test– Confidence with ICT– Knowledge of scientific inquiry processes

• Anonymous survey of students– Did they like it?

IATW questions to ask

• Was the website usable?• What things didn’t work as well as they should?• Was there evidence that students engaged with the

authentic task?• Will the learning design scale up for 2000 OUA

students?

Overview

• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project

Baseline

The Learning LifecyclePhase

Design

Design

Develop

Evaluate

Trial learning env’t

Learning environment

Mature Learning environment

Learning lifecycle

Baseline analysis

Evaluation Research across the Learning Lifecycle

Design evaluation

Design

Develop

Evaluate

Formative evaluation

Formative evaluation

Summative evaluationEffectiveness research

Learning lifecycle

e-learning Research Approaches

• Need to apply to an artificial phenomenon• Reflect the e-learning life cycle

• Cyclical research approaches– Design-based research – Action Inquiry

Design-based Research• Focus on a designed learning environment• A cyclical approach that originated in engineering and

other design fields

• Consistent with – a mixture of evaluation and research– Stokes’ views on quest for understanding and consideration

of use

Action Inquiry

• Focus on personal practice• Local focus• Little concern with theory

Drawing the Threads Together

• e-learning environments are designed phenomena– they follow a lifecycle

• Cyclical research approaches• Mix of evaluation and research

– make sure it works first• Now – onto the practical stuff

Overview

• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project

Breaking Down the Complexity

• How to use all this information:– It’s a complex situation, with– many elements, which– evolves over time

• It requires multiple research methods• Use divide and conquer techniques and a

template to simply the issue

Divide and Conquer Techniques

• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle

– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-

research plan

Evaluation-research Process

Evaluation-research Process 1

Design Principles

• relatively humble ‘proto-theories’• not ‘grand’ theories of learning • Two roles for theory:

– Drawing on theory to support evaluation-research goals and questions across the e-learning lifecycle

– Building theory from research outcomes

Builds theory

Draws on theory

Baseline analysis

Evaluation Research across the Learning Lifecycle

Design evaluation

Design

Develop

Evaluate

Formative evaluation

Formative evaluation

Summative evaluationEffectiveness research

Learning lifecycle

Evaluation-research Process 1

The LEPO Framework

• Learning Environment, Process, Outcomes

Educational Context

Learning environments

Learning processes

Learning outcomes

Students Teachers

facilitate

design facilitate

assess

lead todetermine

work within

engage in

demonstrate

interact with

IATW Conceptual Framework

• Model for inquiry-based learning• Model for information literacy

• Herrington’s 9-stage authentic learning model

Broad Evaluation-Research Questions

• The broadest questions that can be asked in the study

• Open-ended, exploratory questions • IATW examples:

– How can the unit design be improved for a large, online OUA cohort?

– How effective is the unit at developing inquiry skills in students?

Evaluation-research Process 2

IATW

• Stakeholders– Centre for Teaching and Learning– Head of School– Standards bodies

• Participants– Students– Teaching staff– LMS support staff

Activity

• Fill in page 1 of the template• Discuss it with your partner

Divide and Conquer Techniques

• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle

– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-

research plan

Evaluation-research Forms• Baseline analysis: Document the teaching and learning

context; needs analysis; assumptions, literature• Design evaluation: Evaluate the documented design of

the learning environment• Formative evaluation: Make judgments about, and

suggest improvements to, the e-learning environment• Effectiveness research: Develop understanding of how

learners engage with learning tasks to demonstrate learning outcomes

• Project management evaluation: judgements about the conduct of a project

Evaluation-research Process 3

Specific Evaluation-research Questions

• Broad evaluation-research questions are hard to answer

• Specific evaluation-research questions should be answerable– identify the evidence which is needed to answer

them• Synthesise the evidence from the specific

questions to answer the broad questions

Examples of Specific Research Questions

Formative evaluation of the e-learning environment • How do learners use the learning environment?• To what extent do learners engage with learning

tasks as intended?• Are the learning tasks appropriate for the target

learners?• How useful do learners find the e-learning

environment?

Divide and Conquer Techniques

• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle

– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-

research plan

Evaluation-research Process 3

Evaluation-research Matrices

• To map data sources and methods against the research questions

• Examples from IATW

Baseline Analysis

Design Evaluation

Formative Evaluation

Effectiveness Research

Divide and Conquer Techniques

• Evaluation research process diagram• Five evaluation-research ‘forms’• Break down the e-learning lifecycle

– To develop appropriate research questions• Use an evaluation-research matrix• Use template to combine into an evaluation-

research plan

Evaluation-research Plan

• Use the template• Work through the evaluation matrices and

develop questions which are relevant to your project

• Specify your data sources– Interview schedules– Survey questions

Do the evaluation research

Overlap between Forms

Overview

• Phenomena• The e-learning lifecycle• e-learning evaluation research• Cyclical evaluation research approaches• Divide and Conquer techniques• Work on own project

Evaluation-research Plan

• Use the template to start to develop your own evaluation research plan

• Work through the evaluation matrices and develop questions which are relevant to your project

Application of the Approach

• PhD research• Teaching improvement• Preparation for grant applications

Where to publish your work?Journal of Perspectives in Applied Academic Practicehttp://jpaap.napier.ac.uk/index.php/JPAAP International Journal for SoTLhttp://academics.georgiasouthern.edu/ijsotl/v5n1.htmlInvited essays, Research papers, essays MERLOT Journal of Online Learning and Teachinghttp://jolt.merlot.org Journal of University Teaching and Learning Practice http://ro.uow.edu.au/jutlp/ International Journal of Innovation in Science and Mathematics EducationIJISME is dedicated to encouraging and supporting early career researchers as well as academics in science and mathematics who are new to education research.http://ojs-prod.library.usyd.edu.au/index.php/CAL/index