Learnometrics:Metrics for Learning

ObjectsXavier Ochoa

Learning Object

Any digital resource that can be reused to support learning

(Wiley, 2004)

Share and Reuse

Sharing

Sharing

Repository

Metadata

Book Metadata

Learning Object Metadata

GeneralTitle: Landing on the Moon

TechnicalFile format: Quicktime MovieDuration: 2 minutes

EducationalInteractivity Level: LowEnd-user: learner

RelationalRelation: is-part-ofResource: History course

Learning Object

LOM

Learning Object Repository

Object Repository Metadata Repository

and /or

Learning Object Economy

Market Makers

Producers Consumers

Policy Makers

Market

How it works?How it can be improved?

Purpose

Generate empirical knowledge about LOE

Test existing techniques to improve LO tools

Quantitative Analysis

Metrics Proposaland Evaluation

Quantitative Analysis of the Publication of LO

• What is the size of Repositories?

• How do repositories grow?

• How many objects per contributor?

• Can it be modeled?

16

Size is very unequal

17

Size Comparison

Repository Referatory OCW LMS IR

Growth is Linear

Bi-phase Linear ln(a.exp(b.x)+c)

Objects per Contributor• Heavy-tailed distributions (no bell curve)

LORP - LORFLotka with cut-off

“fat-tail”

Objects per Contributor• Heavy-tailed distributions (no bell curve)

OCW - LMSWeibull

“fat-belly”

Objects per Contributor• Heavy-tailed distributions (no bell curve)

IRLotka high

alpha“light-tail”

Engagement

Model

Analysis Conclusions–Few big repositories concentrate most of the material

–Repositories are not growing as they should–There is not such thing as an average contributor

–Differences between repositories are based on the engagement of the contributor

–Results point to a possible lack of “value proposition”

Quantitative Analysis of the Reuse of Learning Objects

• Which percentage of learning objects is reused?

• Does the granularity affect reuse?

• How many times a learning object is reused?

26

Reuse Paradox

Measuring Reuse

Measuring Reuse

Measuring Reuse

~20%

Distribution of Reuse

Analysis Conclusions–Learning Objects are being reuse with or without the help of Learning Object technologies

–Reuse paradox need to be re-evaluated

–Reuse seems to be the results of a chain of successful events.

Quality of Metadata

Quality of Metadata

Title: “The Time Machine”Author: “Wells, H. G.”Publisher: “L&M Publishers, UK”Year: “1965”Location: ----

Metrics for Metadata Quality–How the quality of the metadata can be measured? (metrics)

–Does the metrics work?• Does the metrics correlate with human evaluation?

• Does the metrics separate between good and bad quality metadata?

• Can the metrics be used to filter low quality records?

Textual Information correlate with human evaluation

Some metrics could filter low quality records

Study Conclusions–Humans and machines have different needs for metadata

–Metrics can be used to easily establish some characteristics of the metadata

–The metrics can be used to automatically filter or flag low quality metadata

Abundance of Choice

38

Relevance Ranking Metrics–What means relevance in the context of Learning Objects?

–How existing ranking techniques can be used to produce metrics to rank learning objects?

–How those metrics can be combined to produce a single ranking value?

–Can the proposed metrics outperform simple text based ranking?

Metrics improve over Base Rank

RankNet outperform Base Ranking by 50%

Relevance Ranking Metrics• Implications

–Even basic techniques can improve the ranking of learning objects

–Metrics are scalable and easy to implement

• Warning:–Preliminary results: not based in real world observation

Applications - MQM

44

Applications - RRM

Applications - RRM

45

General Conclusions• Publication and reuse is dominated by heavy-tailed distributions

• LMSs have the potential bootstrap LOE

• Models/Metrics sets a baseline against which new models/metrics can be compared and improvement measured

• More questions are raised than answered46

Publications• Chapter 2

– Quantitative Analysis of User-Generated Content on the Web. Proceedings of the First International Workshop on Understanding Web Evolution (WebEvolve2008) at WWW2008. 2008, 19-26

– Quantitative Analysis of Learning Object Repositories. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications ED-Media 2008, 2008, 6031-6040

• Chapter 3– Measuring the Reuse of Learning Objects. Third

European Conference on Technology Enhanced Learning (ECTEL 2008), 2008, Accepted.

Publications• Chapter 4

– Towards Automatic Evaluation of Learning Object Metadata Quality. LNCS: Advances in Conceptual Modeling - Theory and Practice, Springer, 2006, 4231, 372-381

– SAmgI: Automatic Metadata Generation v2.0. Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications ED-Media 2007, AACE, 2007, 1195-1204

– Quality Metrics for Learning Object Metadata. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2006, AACE, 2006, 1004-1011

Publications• Chapter 5

– Relevance Ranking Metrics for Learning Objects. IEEE Transactions on Learning Technologies. 2008. 1(1), 14

– Relevance Ranking Metrics for Learning Objects.LNCS: Creating New Learning Experiences on a Global Scale, Springer, 2007, 4753, 262-276

– Use of contextualized attention metadata for ranking and recommending learning objects. CAMA '06: Proceedings of the 1st international workshop on Contextualized attention metadata at CIKM 2006, ACM Press, 2006, 9-16

My Research Metrics (PoP)• Papers: 14• Citations: 55• Years: 6• Cites/year: 9.17• Cites/paper: 4.23• Cites/author: 21.02• Papers/author: 6.07• Authors/paper: 2.77

• h-index: 5• g-index: 7• hc-index: 5• hI-index: 1.56• hI-norm: 3• AWCR: 13.67• AW-index: 3.70• AWCRpA: 5.62

Thank you for your attention

Questions?

51