University of D
uisburg-Essen
Information Systems for Production and Operations ResearchDr. Jan M. Pawlowski
2004-11-25E-Learning Quality
E-Learning Quality and the International Standardization Perspective
Piraeus, 2004-11-25
Dr. Jan M. Pawlowski
2004-11-25, Slide 1.2E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Contents
Part 1: Quality Research and Standardization: State of the Art• Perspectives of Quality• Quality Approaches in Europe
• European Quality Observatory
• Quality Standardization
Part 2: Adaptation of Quality Approaches• The Quality Decision Cycle• ISO 19796-1• Adaptation / Profiles
2004-11-25, Slide 1.3E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
University of Duisburg-Essen: Where we are…
2004-11-25, Slide 1.4E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Individual Quality Through Standardisation?!
Return on Investment
Learner Satisfaction
Strategic Impact
ProductivityPersonality
Development
Process Optimization
Customer Satisfaction
Organization‘sSuccess
Market Share
Media QualityMethod MixIndividualization/
Personalization
Interoperability Ergonomics
Learning Culture
Availability
Flexibility
Accessibility
Individual Quality Profile
2004-11-25, Slide 1.5E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Diversity of Quality Approaches & Standards
LOMLOM
SCORMSCORM
SGML, XML, RDF, XHTML SGML, XML, RDF, XHTML
Author GuideAuthor Guide
ISO 9241ISO 9241
Accessibility Guidelines
Accessibility Guidelines
DIN CriteriaDIN Criteria
ASTD CriteriaASTD Criteria
Organization’s Criteria
Organization’s Criteria
ImplementLearning Scenario
Design Learning Scenario
Evaluate Learning Scenario
… …
ISO 9000ISO 9000 EFQMEFQM EFQM QAAEFQM QAA ……Management Approaches
Management Approaches
ECTSECTSQoL
Bench-marks
QoL Bench-marks
European DE Guide-
lines
European DE Guide-
lines
Media Association Certificate
Media Association Certificate
Auto Industry Standard
Auto Industry Standard
……
2004-11-25, Slide 1.6E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Quality Research Classification
• Context• Branch• Sector• Culture• …
• Objectives: • Learning performance, cost-benefit,
financial objectives, motivation, …
• Dimension: • Processes• Products• Competencies
• Perspective• Learner• Provider• Author• Administrator• …
• Methodology:• QM, QA• Evaluation • Benchmarking• …
• Metrics;• Indicators• Criteria
Which quality approach will suit my organization?
Individual QualityNational, regional, local requirementsOrganizational requirementsContextQuality Objectives & GoalsQuality Instruments …
Support by Standards
2004-11-25, Slide 1.7E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Quality Standardization
CEN/ISSS
European Quality
Observatory
State of the Art :•Collection & Analysis of Quality Approaches•Conceptual Model
DIN
State of the Art :•Extended Process Model•PAS 1032-1•Harmonized QM Model
ISO
State of the Art:•Description Scheme•Process Model•FCD 19796-1
New Items:•ISO 19796-1 Profiles•Learning Ressource Transparency•Conceptual Model
2004-11-25, Slide 1.8E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
The Quality Decision Cycle: A Framework for Diversity
Analysis ofQuality Needs
Analysis & Comparison of
Approaches
Decision Support
Adaption/ Implementation Recommend-
ations
2004-11-25, Slide 1.9E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Decision Cycle: Tools and Models
EQO
RFDQ/RQC
Analysis of Quality Needs
Analysis & Comparison of
Quality Approaches
Decision Support /
Harmonisation
Adaptation / Implementation
QualityManagement Approaches
Specific Standards /
Criteria
QualityAssurance
Approaches
Common Criteria
Common Process Approach
Vocabulary
Input
Use as Analysis Model
Consensus
Common Process Approach Common Criteria
Country Profiles
Sector Profiles
…Country Profiles
Sector Profiles
…
Specific Profiles
ISO/IEC JTC1 SC36 WG5
EQO
ISO 19796-1: RFDQ
2004-11-25, Slide 1.10E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
ISO/IEC 19796-1
• To provide a framework for interoperable description of quality approaches
• To provide a framework for harmonization• Abstract framework (Reference Framework for the Description of Quality
Approaches)• Description Scheme• Process Model
• Profiles• National instantiations• Specific objectives• Reference Quality Criteria
• Use as • Guideline• Quality Development Tools• Questionnaire
2004-11-25, Slide 1.11E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Description Scheme: RFDQAttribute Description Example
ID Unique Identifier AB1234
Name Process name Method selection
Category Main Process Course Development
Sub-processes Sub-processes / tasks Method identification, method alternatives, method priorization
Objective Objective of a Process Adequate selection of one or more didactic con-cepts
Description Description of the process Within this process the didactic concept and methods are evaluated and selected
Method Methodology for this process
Reference to guideline / docu-ments
Method selection shall be based on the tar-get group.
Methods are selected based on the teach-ers’ experience
See Method Guidelines Handbook
Relation Relation to other processes Before the method selection a target group analysis must be performed;
Result Expected result of a process Method specification
Documents
Actor Responsible / participating actors Team Didactical Design
Metrics/Criteria Evaluation and Metrics for this process
Criteria catalogue 3.2.2-3.2.6
Standards Standards used DIN EN ISO 9241, LOM
Annotations / Example
Further Information, Examples of usage
2004-11-25, Slide 1.12E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. PawlowskiReference Framework for the Description of Quality Approaches: ISO/IEC 19796
NANeeds Analysis
FAFramework Analysis
CDConception / Design DP
Development/Production
IMImplementation
LPLearning Process /
Realization
EOEvaluation/ Optimization
2004-11-25, Slide 1.13E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. PawlowskiReference Framework for the Description of Quality Approaches: ISO/IEC 19796
ID Category Process Name
Description Relations
3.2 Conception / Design
Concept of the contents
Concept of learning and teaching contents
1.1 Demand analysis
2.2 Qualifications
Sub proc-ess(es)
Content selection
Content Design
Objective
1. Learner Demand: The goal is to provide contents adapted to the needs and demand of the learner.
2. Adaptation: Each course shall provide different content presentation formats and entry points based on the user experience.
Method
1. A prototype of the content shall be provided to a group of learners’ representa-tives. In a consensus process, the contents shall be prioritized and agreed on.
2. For each course, classify groups of learners according to their learning type. Adapt presentation format and methods according to these learning types.
Result
1. Documentation of planned and agreed contents
2. Periodically, evaluate learning performance of different learners (test groups).
Actors Curriculum designer, didactic experts, institution accreditation authority, teacher, learn-ers’ representatives
Metrics / Criteria
The content are measured based on their relevance, importance, exemplaricity, …
Standards Higher Education Standards
Annotation / Example
2004-11-25, Slide 1.14E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Reference Quality Criteria
SectionNo. Section/Category No of criteria No of descriptive criteria
2 General Conditions 101 32
3 Technical Aspects 103 23
4 Data storage and Data processing 37 14
5 Functionalities 69 29
6 Theoretical Aspects 80 17
7 Encoding of Information 59 3
8 Special modes of presentation 31 0
Total 480 118
2004-11-25, Slide 1.15E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Reference Quality Criteria
No Criterion Description/ Operationalization
Value space Comments, Explanations
5.4.2.4 Support of asynchronous communication by enabling learners to share/publish data
Which type of data can be published/shared by learners?
For which type of data are publication/ authoring tools offered for learners?
a) selection (none, plain text, formatted text, images, audio, video, other)
b) selection (none, for plain text, for formatted text, for Web-pages, image editing, audio editing, video editing, other)
The possibility to share different kind of data among learners supports the communication and learning process, especially for group and project oriented learning styles. For example learner web-pages or shared file directories facilitate the exchange of data. A basic form of data exchange would be the exchange of ascii-text through Email.
Standard Authoring and publication tools facilitate the exchange of data among learners. For example conflicts of data formats are avoided, when learners use the tools offered by the learning product instead of using a variety of tools.
2004-11-25, Slide 1.16E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Individual Quality Based on Standards (1)
• Analysis of Quality Needs• Context (EQO Conceptual Model & Vocabulary)
• Requirements
• Actors (ISO/IEC 19796-1)
• Objectives (ISO/IEC 19796-1)
• Processes (ISO/IEC 19796-1)
• Comparison & Recommendation• EQO Tools
• Adaptation• Using EQO & ISO 19796-1
2004-11-25, Slide 1.17E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Individual Quality: Adaptation
ID Category Process Description Relation
NA Needs Analysis Identification and description of requirements, demands, and
constraints of an educational project
Sub-processes / Sub-aspects
NA.1 Initiation
NA.2 Stakeholder Identification
NA.3 Definition of objectives
NA.4 Demand Analysis
Objective To describe the needs and demand leading to an educational project
Method Quality Function Deployment
Result Documentation of goals, objectives, needs, and requirements of an educational project
Actors Project manager; specialists, learners, sponsors
Metrics / Criteria Indicators
Standards ISO 9000:2000
Annotation / Example
Involve relevant actors
Identify relevant processes
Specify quality objectives
EQO Vocabulary•CEO/ Official•Quality manager•Consultant•Evaluator• Learner•Teacher•developers (software developer, content broker, authors etc.)
•Policy Maker•Researcher
2004-11-25, Slide 1.18E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Individual Quality: Adaptation
ID Category Process Description Relation
NA Needs Analysis Identification and description of requirements, demands, and
constraints of an educational project
Sub-processes / Sub-aspects
NA.1 Initiation
NA.2 Stakeholder Identification
NA.3 Definition of objectives
NA.4 Demand Analysis
Objective To describe the needs and demand leading to an educational project
Method Quality Function Deployment
Result Documentation of goals, objectives, needs, and requirements of an educational project
Actors Project manager; specialists, learners, sponsors
Metrics / Criteria Indicators
Standards ISO 9000:2000
Annotation / Example
Define Criteria and Standards CONSENSUSDefine Methods
EQO Vocabulary• Benchmarking• Evaluation• Management approaches• Quality Criteria• Standards• Certification• Accreditation process• Quality Policy
Reference Quality Criteria• ISO 9241• General Conditions• Technical Aspects• Data storage and Data processing
• Functionalities• Theoretical Aspects• Encoding of Information• Special modes of presentation
2004-11-25, Slide 1.19E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Conclusion
Standards support the full decision cycleStandards are the basis for further harmonizationTools are available for description, search, and recommendations
⇨Individual Quality adaptation needs to be supported⇨Quality as a selection and adaptation process⇨Long-term performance evaluation
2004-11-25, Slide 1.20E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Next Steps
To provide a complete harmonized quality model
• Providing • Objectives• Methods• Specific Criteria• Metrics
• Including worldwide experts• Start of ISO/IEC 19796-xxxx
Input from Germany / CEN/ISSS• Good Practice: ISO/IEC 19796-1
in Europe• Learning Resource Transparency• Harmonized German Quality
Model
?Develop working draft on a workplan towards a harmonized quality model
?Involving European experts
?WG Meeting in January/February in Essen, Germany
2004-11-25, Slide 1.21E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
Contact
http://wip.wi-inf.uni-essen.de
[email protected]++49 (0) 201 1834061
2004-11-25, Slide 1.22E-Learning Quality
Univ
ersity
of D
uisb
urg
-Esse
nInformation Systems for Production and Operations Research
Dr. Jan M. Pawlowski
References: Standards
• DIN Workshop „Quality and Standardization“
•Process-Model, Description Model
•http://www.beuth.de/sc/pas1032-1
• CEN/ISSS Workshop Learning Technologies•European Process Description Model
•http://www.cenorm.be/isss/cwa_download_area/cwa14644.pdf
• ISO/IEC JTC1 SC36•Generic Description Model
•http://jtc1sc36.org/