Presentation

E-Learning in modern Learning Settings: Recent Research Activities

Institute for Information Systems and Computer Media (IICM)Faculty of Informatics – Graz University of Technology, Austria

Christian Gütl

Version June 30th, 2009

© 2009 - Christian Gütl19/04/23 2

Agenda

• Introduction

• Graz University of Technology

• Institute for Information Systems and Computer Media

• Recent and Active Research

• Learning Ecosystem

• E-Assessment

• Virtual 3D Worlds for Learning & Training

© 2009 - Christian Gütl19/04/23 3

About Graz

Graz

Vienna

© 2009 - Christian Gütl19/04/23 4

About Graz University of Technology 1

Graz

© 2009 - Christian Gütl19/04/23 5

About Graz University of Technology 2

• History– 1811 Joanneum founded by Archduke Johann– 1901 doctorate program(s)– 2004 legal entity in public law

• Some Statistics– Seven faculties that comprises 104 institutes – Some 10,000 students total, 15 % foreigners– 1080 academic and 670 non-academic staff

• Some Success Stories– Nikola Tesla (AC, wireless signal transmission, …)

– Otto Nußbaumer (speech and music transmission in 1904)

– Richard Zsigmondy (Nobel prize in chemistry, 1926)

© 2009 - Christian Gütl19/04/23 6

About the IICM

Institute for Information Systems and Computer Media (IICM)

• Some 25 Full Time Members

• Research Areas– Adaptive Systems– Cross Media Information Retrieval– Digital Libraries and Information Systems– e-Learning– Information Visualization– Innovative Media Technologies– Virtual 3D Environment– Usability

© 2009 - Christian Gütl19/04/23 7

About Christian Gütl• Teaching

– ISR, KM, Information Systems, PM– Supervising Master & PhD Theses

• Active Research Areas– IR & Visualization– E-Learning and e-Assessment– 3D Worlds for learning & knowledge transfer

• Organizational Work– Student Exchange Programs– …

• Community Work– Reviewer for research projects (EC, NSF, …)– E-Learning Groups (national & international)– Program Committees & Editorial Boards

© 2009 - Christian Gütl19/04/23 8

Modern Learning Settings

• Modern society is characterized by rapid developing and ever-changing situations.

• Educational approaches have changed …– From remedial repetitive learning– To learning with understanding & collaboration

• ICT-based learning changed from …– Content-centered, centralized and static EL 1.0– People-centered, enhanced approach EL 2.0

• Characteristics include ... (Hart 2008; Drášil et al. 2007)

• Blurring roles of teachers and students• Collaborative nature of learning• Content sharing, syndication, reuse & re-purposing

© 2009 - Christian Gütl19/04/23 9

The Concept of Ecosystem

Ecosystem (Chang & Gütl, 2007)

– “community or assemblage and its associated environment in a specific place” (Tansely 1935)

– The model of the ecosystem emphasizes a holistic approach where it highlights

• the significance of each component,• the behavior,• the relationship and interactions,• the environmental borders

– Ecosystem can be applied to • humans and human-generated processes and

structures – in order to examine an existing system – or form an effective and successful new system

© 2009 - Christian Gütl19/04/23 10

E-Learning Ecosystem (ELES)

• Model from Chang & Gütl (2007)– Based on the notion of ecosystem– has been transformed in learning domain

• E-Learning Ecosystem (ELES)– Biotic or living units

• Learning community and other stakeholders– teachers, students, content provider, instructional

designers, pedagogical experts

– Abiotic or non-living units• Learning environment

– Learning media, technologies and tools

– Learning environmental boundaries• Defines physical and logical borders

– Learning ecosystem conditions• internal and external influences

– Knowledge evolution, educational goals, learning tasks, cultural & sociological aspects, ….

© 2009 - Christian Gütl19/04/23 11

E-Learning Ecosystem (ELES)(Chang & Gütl, 2007)

Lear

ning

Env

ironm

enta

l Bor

ders

Learning Ecosystem Conditions

External Influences

Learning Stakeholders (Biotic Units)

Learning Utilities or Learning Environment

(Abiotic Units)

Internal Influences

© 2009 - Christian Gütl19/04/23 12

Adaptive E-Learning

Server-side

User Modeling System

Eye-Tracking System

Web Client

Adaptation System

Learning Management

System

Client-side

Inte

rfac

e

Information Retrieval System

Concept Modeling System

Content-Tracking System

Inte

rfac

e

Dynamic E-Learning Knowledge Repository

AdeLE (Gütl et al. 2005; Gütl 2007)

• Distributed System• Service-Based Approach• Enhanced User Profile• Applies Eye-Tracking

© 2009 - Christian Gütl19/04/23 13

Content & Gaze Tracking(Gütl et al., 2005)

© 2009 - Christian Gütl19/04/23 14

Assessment• Assessment (Shepard, 2000)

– is not a tool to force students to learn and punish them on their lack of knowledge

– rather it should be a means to enhance learning• Society requires … (Dochy & McDowellb, 1997)

– cognitive competencies (such as problem solving, critical thinking, efficient use of information)

– meta-cognitive competencies (such as self-reflection and self-evaluation)

– social competencies (such as group working and communication skills)

– affective dispositions (such as flexibility and independency)

Summative & Formative Assessment Involvement of Teachers & StudentsFeedback is key in the learning process

© 2009 - Christian Gütl19/04/23 15

Computer-supported Peer Assessment

• Peer Assessment (Dochy & McDowell, 1997)

– … students carry out assessment and grading on their peers’ performance; assessment criteria are either pre-defined or will be defined as part of the peer assessment activity

• Advantages– students develop several learning and meta-

learning skills– secure knowledge they already have and acquire

additional knowledge

• Problems– may be biased– grade often more leniently than instructors– Lack of motivation

Grade = Test + Assessment PerformancePre-selected Candidate Answers

© 2009 - Christian Gütl19/04/23 16

Peer Assessment ResultsStudent Assessment Performance Comparision

0,00

2,00

4,00

6,00

8,00

10,00

12,00

Candidate Answer

Tutor Reference

Peer Assessment(best Performance)

Peer Assessment(worst Performance)

Assessment Performance Comparision Test Item 2

0,00

2,00

4,00

6,00

8,00

10,00

12,00

1 3 5 7 9 11 13 15 17 19 21 23 25

Candidate Answer

Tutor Reference

Peer Assessment(arithmetic mean)

Assessment Computation Mean σ

Arithmetic Mean 0.88 0.73

Weighted Average (PA) 0.86 0.71

Weighted Average (Pre-marked Samples) 0.78 0.71

© 2009 - Christian Gütl19/04/23 17

Students‘ Performance & OpinionKnowledge Acquisition

0

2

4

6

8

10

0 (disagree) 1 2 3 4 5 (agree)

Level of Agreement

Nu

mb

er o

f S

ub

ject

s ... by refence answ erpreparation (SQ03)

... by peer assessmenttask (SQ04)

… of details by answ erreview s (SQ05 )

I like Peer Assessment

0

2

4

6

8

10

12

14

0 (disagree) 1 2 3 4 5 (agree)

Level of Agreement

Nu

mb

er o

f S

ub

ject

s

… as part of learningactivity (SQ08)

… as part of performancegrading (SQ09)

… as part of futurelearning settings (SQ10)

Performance

Scale1 bad … 5 good

Test3.20 (σ = 1.00)

Reference Answer

4.80 (σ = 1.85)

Assessment3.64 (σ = 1.02)

© 2009 - Christian Gütl19/04/23 18

Computer-based Assessment

• Problem 1– Huge effort to prepare proper tests

• Problem 2– Additional effort to keep up-to-date with changing

learning content– Multiple effort for personalized learning content– Dealing with unpredictable learning content

(student assignments or background knowledge)

• Problem 3– Different levels of knowledge requires different

assessment methods– Lower levels: limited choice, completion exercise– Higher levels: free-text answers, essays

How can we overcome these problems ?

© 2009 - Christian Gütl19/04/23 19

The Big Picture

e-Examiner

Assessment Management

QuestionGeneration

Inte

rfac

e(I

MS

QT

I)

Answer Assessment

LMS

Other Systems

Assessment Service

Assessment Feedback

II

I

Question & Reference Answers

Repository

Our long-term goal: move towards a computer-based assessment systems– Automatic question generation

(multiple choice, completion exercise, ….– Automatic assessment

(challenging short free-text answer assessment)– Automatic Feedback

© 2009 - Christian Gütl19/04/23 20

Automatic Question Generation– Different file formats from local file system,

remote (Web servers) and from ext. applications– Automatic Summarizer from (Visser & Wieling)

– Word extraction by GATE (General Architecture for Text Engineering) system (Sheffield NLP group)

– Candidate words used for exercise creation– Interface to be used in various application scenarios

Ass

essm

ent S

yste

m

LMS

Tes

t Wor

ds Id

entif

icat

ion

GA

TE

Cor

e &

Plu

gins

Tokeniser

Sentence Splitter

POS Tagger

Morphological Analyzer

Stopword Finder

Test Word Extractor

Exe

rcis

e C

reat

or

Com

mun

icat

ion

Inte

rfac

e

Aut

omat

ic S

umm

ariz

er

Doc

umen

t Filt

er

Doc

umen

t Fet

cher

Java

GU

I

© 2009 - Christian Gütl19/04/23 21

Short Text Answer Assessment

• Principle (Gütl, 2008)

– Comparison between candidate answer and reference answer(s)

ROUGE = Recall-Oriented Understudy for Gisting Evaluation

• It defines a set of statistical measures – ROUGE-N

word n-gram co-occurence– ROUGE-L

longest common subsequence of words– ROUGE-W (= weighted ROUGE-L)– ROUGE-S

skip-bigrams = pair of words in sentence order– ROUGE-SU (= ROUGE-S + Word Unigrams)

© 2009 - Christian Gütl19/04/23 22

Prototype Architecture

User Management

Test Management

Control and View

Data Storage and Retrieval

(Struts)

(Hibernate)

Web Application(Tomcat)

Database(MySQL)

JDB

CHTTP

Web

Clie

ntAnswer Assessment(GATE)

GA

TE

Cor

e S

yste

m

GA

TE

Plu

gins

JAVA

ROUGE Statistics

COSIN Text Similarity

Tokeniser

Sentence Splitter

POS Tagger

Morphological Analyzer

Stopword Finder

Token Normalizer

Assessment Score Builder

ROUGE-based short text answer assessment

© 2009 - Christian Gütl19/04/23 23

Experiment Results 1

Automated vs. Manual Assessment (ROUGE-1-R and NoT)

0

2

4

6

8

10

12

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79

Number of Test Item

Sco

re

MANUAL_SCORE CALCULATED_SCORE

Results for experiment setup 1– ROUGE-1-R (recall of word unigram)– NoT (number of tokens)

© 2009 - Christian Gütl19/04/23 24

Experiment Results 2

Experiment Setup Average Absolute Error Correlation

No.Description of applied Similarity Measures

Training Dataset

Test Dataset

Total Dataset

Training Dataset

Test Dataset

Total Dataset

1 Number of Tokens and Recall Measure of

ROUGE-1

1.47 1.51 1.49 0.80 0.80 0.80

2 Number of Tokens, COSIN Similarity and

ROUGE Recall and Precision Measures

( ROUGE-1, ROUGE-2, ROUGE-3, ROUGE-L, ROUGE-S, ROUGE-

SU, ROUGE-W)

1.03 2.00 1.54 0.84 0.79 0.81

3 Number of Tokens, Recall of ROUGE-L,

Recall and Precision of ROUGE-1

1.32 1.46 1.39 0.82 0,81 0,81

Interest in Virtual 3D Worlds

• Infrastructure• Interlinked Worlds 3D Web

• E-Commerce• Virtual Currency and Payment system• Shop System usable in multiple worlds• Plagiarism Detection

• Learning & Training

Plattforms of Interest• Second Life (http://secondlife.com/)

• RealXtend (http://www.realxtend.org/)

• Sun Wonderland (https://lg3d-wonderland.dev.java.net/)

• OGRE (http://www.ogre3d.org/)

© 2009 - Christian Gütl19/04/23 25

© 2009 - Christian Gütl19/04/23 26

Virtual 3D World for Learning

(Kappe & Gütl, 2009)

Multiple communication channels both verbal communication and non-verbal

Presence (feeling to be part of the virtual environment)

Awareness of other avatars, the environment and activities

Facilitating collaboration which is important in modern working and learning processes

Reducing barriers between students, tutors and instructors(Kemp & Livingstone, 2006)

Belonging to a community creates a virtual social space

© 2009 - Christian Gütl19/04/23 27

New Architectural Approaches 1

Based on Second Life

• … new learning virtual spaces without “physical constraints” of real world

© 2009 - Christian Gütl19/04/23 28

New Architectural Approaches 2

• … dynamic & temporarily learning spaces created according specific needs

© 2009 - Christian Gütl19/04/23 29

Learning in Small Groups 1

• … with Curtin University of Technology• Collaborative Learning and Writing

Based on Second Life

© 2009 - Christian Gütl19/04/23 30

Learning in Small Groups 2

• Private group learning areas • Presentation slides, Google Docs, White Board

© 2009 - Christian Gütl19/04/23 31

Learning in Small Groups 3

• … based Sun Wonderland V 0.4

© 2009 - Christian Gütl19/04/23 32

Adaptive Learning Path(Nussbaumer, Gütl, Albert, & Helic, 2009)

• … with Graz University, Cognitive Science • Competence-based Adaptation in 3D Space

© 2009 - Christian Gütl19/04/23 33

Virtual Experiment 1

• … with Faculty of Architecture• Experiments concerning firmness of

reinforced concrete• … based in Sun Wonderland

© 2009 - Christian Gütl19/04/23 34

Virtual Experiment 2(Scheucher, Bailey, Gütl, & Harward, 2009)

• … with CECI at MIT• Virtual & remote physical experiments

© 2009 - Christian Gütl19/04/23 35

Virtual Experiment 2(Scheucher, Bailey, Gütl, & Harward, 2009)

• Based on Sun Wonderland

© 2009 - Christian Gütl19/04/23 36

Immersive Interaction with Worlds

• … with Computer Graphics and Knowledge Visualization, Graz University of Technology

• Representation of Worlds in DAVE (Definitely Affordable Virtual Environment)

© 2009 - Christian Gütl19/04/23 37

ReferencesVanessa Chang, and Christian Gütl (2007). E-Learning Ecosystem (ELES) – A

Holistic Approach for the Development of more Effective Learning Environment for Small-to-Medium Sized Enterprises (SMEs). In IEEE-DEST 2007, Cairns, Australia.

Drášil, P., Pitner, T. E-Learning 2.0: Methodology, Technology and Solutions, Retrieved on 13 Jan 2007 from http://www.fi.muni.cz/~tomp/l2/e-learning2.0/icte/l2_platform.pdf.

Dochy, F. J.; & McDowell, L. (1997) Introduction. Assessment as a tool for learning. Studies in Educational Evaluation, 23 (4), 279-298.

Hart, J. (2008). What is E-learning 2.0? E-LEARNING HANDBOOK, Centre for Learning & Performance Technologies, last edited March 2nd, 2008, last retrieved March 5th , 2008 from http://www.c4lpt.co.uk/handbook/elearning20.html

Kappe, F., Gütl, C. (2009). Enhancements of the realXtend framework to build a Virtual Conference Room for Knowledge Transfer and Learning Purposes. EDMEDIA 2009, accepteded.

Shepard, L. (2000). The Role of Assessment in a Learning Culture. Educational Researcher, 29 (7), 4-14.

Dochy, F. J.; & McDowell, L. (1997) Introduction. Assessment as a tool for learning. Studies in Educational Evaluation, 23 (4), 279-298.

GATE. Overview. Natural Language Processing Research Group, University of Sheffield, UK, last retrieved Nov. 29th, 2007 form http://gate.ac.uk/sale/gate-flyer/2007/gate-flyer-4-page.pdf

© 2009 - Christian Gütl19/04/23 38

ReferencesChristian Gütl (2008). Moving towards a Fully-Automatic Knowledge Assessment

Tool. iJET International Journal of Emerging Technologies in Learning, to be published.

Christian Gütl (2008a). Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes, accepted paper, .LRN Conference 2008, Guatemala, February 12th - 16th, 2008.

Christian Gütl, Maja Pivec, Christian Trummer, Victor Manuel García-Barrios, Felix Mödritscher, Juergen Pripfl, and Martin Umgeher (2005). AdeLE (Adaptive e-Learning with Eye-Tracking): Theoretical Background, System Architecture and Application Scenarios. Journal ERODL, issue 2005/II, 2005.

Christian Gütl (2007). Moving Towards a Generic, Service-based Architecture for Flexible Teaching and Learning Activities. In C. Pahl (Ed.) Architecture Solutions for E-Learning Systems (peer-reviewed), Peer-reviewed book chapter, Idea Group Inc., Hershey, USA.

Visser, W.T., & Wieling, M.B. Sentence-based Summarization of Scientific Documents. The design and implementation of an online available automatic summarizer. Report, last retrieved Nov. 29th, 2007 form http://home.hccnet.nl/m.b.wieling/files/wielingvisser05automaticsummarization.pdf

Nussbaumer, N., Gütl, C., Albert, D., Helic, D. (2009). Competence-based Adaptation of Learning Environments in 3D Space. Track on Virtual Worlds for academic, organizational, and life-long learning (ViWo 2009), IMCL 2009, April 2009, Amman, Jordan, pp103-108.

Scheucher, T, Bailey, P. H., Gütl, C., Harward, V. J. (2009). Collaborative Virtual 3D Environment for Internet-accessible Physics Experiments. REF 2009.

© 2009 - Christian Gütl19/04/23 39

Questions & Contact Information

Thank you for your Attention!Questions are welcome!

Further Information:

Christian Gütl

http://www.iicm.edu/guetl

cguetl@iicm.edu