university of duisburg-essen information systems for production and operations research dr. jan m....

University of D

uisburg-Essen

Information Systems for Production and Operations ResearchDr. Jan M. Pawlowski

2004-11-25E-Learning Quality

E-Learning Quality and the International Standardization Perspective

Piraeus, 2004-11-25

Dr. Jan M. Pawlowski

2004-11-25, Slide 1.2E-Learning Quality

Univ

ersity

of D

uisb

urg

-Esse

nInformation Systems for Production and Operations Research


Contents

Part 1: Quality Research and Standardization: State of the Art• Perspectives of Quality• Quality Approaches in Europe

• European Quality Observatory

• Quality Standardization

Part 2: Adaptation of Quality Approaches• The Quality Decision Cycle• ISO 19796-1• Adaptation / Profiles


Univ

ersity

of D

uisb

urg

-Esse



University of Duisburg-Essen: Where we are…


Univ

ersity

of D

uisb

urg

-Esse



Individual Quality Through Standardisation?!

Return on Investment

Learner Satisfaction

Strategic Impact

ProductivityPersonality

Development

Process Optimization

Customer Satisfaction

Organization‘sSuccess

Market Share

Media QualityMethod MixIndividualization/

Personalization

Interoperability Ergonomics

Learning Culture

Availability

Flexibility

Accessibility

Individual Quality Profile


Univ

ersity

of D

uisb

urg

-Esse



Diversity of Quality Approaches & Standards

LOMLOM

SCORMSCORM

SGML, XML, RDF, XHTML SGML, XML, RDF, XHTML

Author GuideAuthor Guide

ISO 9241ISO 9241

Accessibility Guidelines

Accessibility Guidelines

DIN CriteriaDIN Criteria

ASTD CriteriaASTD Criteria

Organization’s Criteria

Organization’s Criteria

ImplementLearning Scenario

Design Learning Scenario

Evaluate Learning Scenario

… …

ISO 9000ISO 9000 EFQMEFQM EFQM QAAEFQM QAA ……Management Approaches

Management Approaches

ECTSECTSQoL

Bench-marks

QoL Bench-marks

European DE Guide-

lines

European DE Guide-

lines

Media Association Certificate

Media Association Certificate

Auto Industry Standard

Auto Industry Standard

……


Univ

ersity

of D

uisb

urg

-Esse



Quality Research Classification

• Context• Branch• Sector• Culture• …

• Objectives: • Learning performance, cost-benefit,

financial objectives, motivation, …

• Dimension: • Processes• Products• Competencies

• Perspective• Learner• Provider• Author• Administrator• …

• Methodology:• QM, QA• Evaluation • Benchmarking• …

• Metrics;• Indicators• Criteria

Which quality approach will suit my organization?

Individual QualityNational, regional, local requirementsOrganizational requirementsContextQuality Objectives & GoalsQuality Instruments …

Support by Standards


Univ

ersity

of D

uisb

urg

-Esse



Quality Standardization

CEN/ISSS

European Quality

Observatory

State of the Art :•Collection & Analysis of Quality Approaches•Conceptual Model

DIN

State of the Art :•Extended Process Model•PAS 1032-1•Harmonized QM Model

ISO

State of the Art:•Description Scheme•Process Model•FCD 19796-1

New Items:•ISO 19796-1 Profiles•Learning Ressource Transparency•Conceptual Model


Univ

ersity

of D

uisb

urg

-Esse



The Quality Decision Cycle: A Framework for Diversity

Analysis ofQuality Needs

Analysis & Comparison of

Approaches

Decision Support

Adaption/ Implementation Recommend-

ations


Univ

ersity

of D

uisb

urg

-Esse



Decision Cycle: Tools and Models

EQO

RFDQ/RQC

Analysis of Quality Needs

Analysis & Comparison of

Quality Approaches

Decision Support /

Harmonisation

Adaptation / Implementation

QualityManagement Approaches

Specific Standards /

Criteria

QualityAssurance

Approaches

Common Criteria

Common Process Approach

Vocabulary

Input

Use as Analysis Model

Consensus

Common Process Approach Common Criteria

Country Profiles

Sector Profiles

…Country Profiles

Sector Profiles

…

Specific Profiles

ISO/IEC JTC1 SC36 WG5

EQO

ISO 19796-1: RFDQ


Univ

ersity

of D

uisb

urg

-Esse



ISO/IEC 19796-1

• To provide a framework for interoperable description of quality approaches

• To provide a framework for harmonization• Abstract framework (Reference Framework for the Description of Quality

Approaches)• Description Scheme• Process Model

• Profiles• National instantiations• Specific objectives• Reference Quality Criteria

• Use as • Guideline• Quality Development Tools• Questionnaire


Univ

ersity

of D

uisb

urg

-Esse



Description Scheme: RFDQAttribute Description Example

ID Unique Identifier AB1234

Name Process name Method selection

Category Main Process Course Development

Sub-processes Sub-processes / tasks Method identification, method alternatives, method priorization

Objective Objective of a Process Adequate selection of one or more didactic con-cepts

Description Description of the process Within this process the didactic concept and methods are evaluated and selected

Method Methodology for this process

Reference to guideline / docu-ments

Method selection shall be based on the tar-get group.

Methods are selected based on the teach-ers’ experience

See Method Guidelines Handbook

Relation Relation to other processes Before the method selection a target group analysis must be performed;

Result Expected result of a process Method specification

Documents

Actor Responsible / participating actors Team Didactical Design

Metrics/Criteria Evaluation and Metrics for this process

Criteria catalogue 3.2.2-3.2.6

Standards Standards used DIN EN ISO 9241, LOM

Annotations / Example

Further Information, Examples of usage


Univ

ersity

of D

uisb

urg

-Esse


Dr. Jan M. PawlowskiReference Framework for the Description of Quality Approaches: ISO/IEC 19796

NANeeds Analysis

FAFramework Analysis

CDConception / Design DP

Development/Production

IMImplementation

LPLearning Process /

Realization

EOEvaluation/ Optimization


Univ

ersity

of D

uisb

urg

-Esse


Dr. Jan M. PawlowskiReference Framework for the Description of Quality Approaches: ISO/IEC 19796

ID Category Process Name

Description Relations

3.2 Conception / Design

Concept of the contents

Concept of learning and teaching contents

1.1 Demand analysis

2.2 Qualifications

Sub proc-ess(es)

Content selection

Content Design

Objective

1. Learner Demand: The goal is to provide contents adapted to the needs and demand of the learner.

2. Adaptation: Each course shall provide different content presentation formats and entry points based on the user experience.

Method

1. A prototype of the content shall be provided to a group of learners’ representa-tives. In a consensus process, the contents shall be prioritized and agreed on.

2. For each course, classify groups of learners according to their learning type. Adapt presentation format and methods according to these learning types.

Result

1. Documentation of planned and agreed contents

2. Periodically, evaluate learning performance of different learners (test groups).

Actors Curriculum designer, didactic experts, institution accreditation authority, teacher, learn-ers’ representatives

Metrics / Criteria

The content are measured based on their relevance, importance, exemplaricity, …

Standards Higher Education Standards

Annotation / Example


Univ

ersity

of D

uisb

urg

-Esse



Reference Quality Criteria

SectionNo. Section/Category No of criteria No of descriptive criteria

2 General Conditions 101 32

3 Technical Aspects 103 23

4 Data storage and Data processing 37 14

5 Functionalities 69 29

6 Theoretical Aspects 80 17

7 Encoding of Information 59 3

8 Special modes of presentation 31 0

Total 480 118


Univ

ersity

of D

uisb

urg

-Esse



Reference Quality Criteria

No Criterion Description/ Operationalization

Value space Comments, Explanations

5.4.2.4 Support of asynchronous communication by enabling learners to share/publish data

Which type of data can be published/shared by learners?

For which type of data are publication/ authoring tools offered for learners?

a) selection (none, plain text, formatted text, images, audio, video, other)

b) selection (none, for plain text, for formatted text, for Web-pages, image editing, audio editing, video editing, other)

The possibility to share different kind of data among learners supports the communication and learning process, especially for group and project oriented learning styles. For example learner web-pages or shared file directories facilitate the exchange of data. A basic form of data exchange would be the exchange of ascii-text through Email.

Standard Authoring and publication tools facilitate the exchange of data among learners. For example conflicts of data formats are avoided, when learners use the tools offered by the learning product instead of using a variety of tools.


Univ

ersity

of D

uisb

urg

-Esse



Individual Quality Based on Standards (1)

• Analysis of Quality Needs• Context (EQO Conceptual Model & Vocabulary)

• Requirements

• Actors (ISO/IEC 19796-1)

• Objectives (ISO/IEC 19796-1)

• Processes (ISO/IEC 19796-1)

• Comparison & Recommendation• EQO Tools

• Adaptation• Using EQO & ISO 19796-1


Univ

ersity

of D

uisb

urg

-Esse



Individual Quality: Adaptation

ID Category Process Description Relation

NA Needs Analysis Identification and description of requirements, demands, and

constraints of an educational project

Sub-processes / Sub-aspects

NA.1 Initiation

NA.2 Stakeholder Identification

NA.3 Definition of objectives

NA.4 Demand Analysis

Objective To describe the needs and demand leading to an educational project

Method Quality Function Deployment

Result Documentation of goals, objectives, needs, and requirements of an educational project

Actors Project manager; specialists, learners, sponsors

Metrics / Criteria Indicators

Standards ISO 9000:2000


Involve relevant actors

Identify relevant processes

Specify quality objectives

EQO Vocabulary•CEO/ Official•Quality manager•Consultant•Evaluator• Learner•Teacher•developers (software developer, content broker, authors etc.)

•Policy Maker•Researcher


Univ

ersity

of D

uisb

urg

-Esse



Individual Quality: Adaptation

ID Category Process Description Relation

NA Needs Analysis Identification and description of requirements, demands, and

constraints of an educational project

Sub-processes / Sub-aspects

NA.1 Initiation

NA.2 Stakeholder Identification

NA.3 Definition of objectives

NA.4 Demand Analysis

Objective To describe the needs and demand leading to an educational project

Method Quality Function Deployment

Result Documentation of goals, objectives, needs, and requirements of an educational project

Actors Project manager; specialists, learners, sponsors

Metrics / Criteria Indicators

Standards ISO 9000:2000


Define Criteria and Standards CONSENSUSDefine Methods

EQO Vocabulary• Benchmarking• Evaluation• Management approaches• Quality Criteria• Standards• Certification• Accreditation process• Quality Policy

Reference Quality Criteria• ISO 9241• General Conditions• Technical Aspects• Data storage and Data processing

• Functionalities• Theoretical Aspects• Encoding of Information• Special modes of presentation


Univ

ersity

of D

uisb

urg

-Esse



Conclusion

Standards support the full decision cycleStandards are the basis for further harmonizationTools are available for description, search, and recommendations

⇨Individual Quality adaptation needs to be supported⇨Quality as a selection and adaptation process⇨Long-term performance evaluation


Univ

ersity

of D

uisb

urg

-Esse



Next Steps

To provide a complete harmonized quality model

• Providing • Objectives• Methods• Specific Criteria• Metrics

• Including worldwide experts• Start of ISO/IEC 19796-xxxx

Input from Germany / CEN/ISSS• Good Practice: ISO/IEC 19796-1

in Europe• Learning Resource Transparency• Harmonized German Quality

Model

?Develop working draft on a workplan towards a harmonized quality model

?Involving European experts

?WG Meeting in January/February in Essen, Germany


Univ

ersity

of D

uisb

urg

-Esse



Contact

http://wip.wi-inf.uni-essen.de

[email protected]++49 (0) 201 1834061


Univ

ersity

of D

uisb

urg

-Esse



References: Standards

• DIN Workshop „Quality and Standardization“

•Process-Model, Description Model

•http://www.beuth.de/sc/pas1032-1

• CEN/ISSS Workshop Learning Technologies•European Process Description Model

•http://www.cenorm.be/isss/cwa_download_area/cwa14644.pdf

• ISO/IEC JTC1 SC36•Generic Description Model

•http://jtc1sc36.org/

university of duisburg-essen information systems for production and operations research dr. jan m....

Documents

elearning quality isoiec

elearning quality contents

pawlowski slide

individual quality national

quality decision cycle

operations research

rfdq slide

essen information systems