data-driven learning strategy
Post on 19-Oct-2014
585 views
DESCRIPTION
Education must capitalize on the trend within technology toward big data. New types of data are becoming available. From evidence approaches to xAPI and the whole Training and Learning Architecture(TLA) big data is the foundation of all.TRANSCRIPT
Data-Driven Learning Strategy
Jessie ChuangClassroom Aid Inc.
Data is
the only hard evidence of what works in education,
and
the compass for personalized learning journey.
Data, Data, Data
Amazon: data driven engine accounts for 1/3 of all sales
Netflix: recommendation engine accounts for 3/4 of all new orders
How about learning and training segment?
Evidence Framework A report by U.S. Dep. of Ed.
- discusses the promise of sophisticated digital learning systems for collecting and analyzing very large amounts of fine-grained data (“big data”) as users interact with the systems.
- describes an iterative R&D process, with rapid design cycles and built-in feedback loops—one familiar in industry but less so in education (however, the report provides numerous examples of applications in education).
Two critical steps - 1● Education must capitalize on the trend within
technology toward big data.
● New types of data are becoming available. ● broad, static categories(demographic, end-of-year grades and test
scores) => more dynamic, fine-grained data in the process of learning
● data in a single context => linked across different parts of learning
● collected through different methodologies and reported in isolation =>
aggregating evidences from different data sources
Two critical steps - 2● A revitalized framework for analyzing and using evidence
○ Gold standard evidence - slow and expensive, poor matches to the
rapid pace of digital development practices
○ An evidence framework should help educational stakeholders align their
methods of obtaining evidence with their goals, the risks involved, the
level of confidence needed, and the resources available.
Confidence
Risk
The need of evidencebefore implementation
collection of data should be an ongoing task
Challenges addressed by evidence approaches
● Making sure learning resources promote deeper learning● Building adaptive learning systems that supports
personalized learning● Combining data to create support systems more
responsive to student needs● Improving the content and process of assessment with
technology● Finding appropriate learning resources and making
informed choices
Evidence Approaches w/i Great Potential1. Educational data mining and learning analytics applied to data gathered from digital
learning systems implemented at scale
2. Rapid A/B testing conducted with large numbers of users within digital learning systems
(example: Khan Academy)
3. Design-based implementation research (DBIR) supported by data gathered from digital
learning systems (collaborative design from developers, researcher, teachers)
4. Large datasets of different types from multiple sources, combined and shared across
projects and organizations
5. Technology-supported evidence-centered design (ECD) of measures of student learning
6. Data gathered from users about a learning resource, how users have used it and their
experiences using it. => Example: enable “Rapid Prototyping” by making early version of a
product freely available online, collect data and mine data for insights; ratings and reviews
from users …...
What Works Clearinghouse (by U.S. Department of Education)- “The high volume of research on different programs, products, practices, and policies in education can make it difficult to interpret and apply the results. We review the research. Then, by focusing on the results from high-quality research, we try to answer the question “What works in education?”
- The goal is to provide educators with the information they need to make evidence-based decisions.
iZone - Researchers and Schools collaborate on an Education Test Bed● Partnership from 250 schools, Research Alliance (based at NYU), EdSurge,
ChallengePost, and IDEO● Supporting from U.S. Dep. of Education, Bill&Melinda Gates Foundation, New
York City Dep. of Education● To support developers in rapidly developing and testing selected technology
based instructional supports and featuring test results on EdSurge● To address unmet needs identified by a diverse group of school stakeholders● Launch prize competitions for developers, winners will be invited to beta-test
products in iZone classrooms● Researcher have access to resources necessary to a given study; test bed data
are available for secondary analysis to continue relevant study (also to develop analytic capability and let good algorithms bubble up)
● The collaboration will produce Consumer Reports-style guide for learning tech.● Silicon Valley also initiated an iZone to create a similar ecosystem
collegestats.org
Learning Analytics Applications ● Learner modeling: model a learner’s knowledge, behaviors, motivation,
experience…● Learner profiling: cluster users into similar groups● Domain modeling: decompose content into components and sequences● Effectiveness: test learning principles, pedagogies etc.● Trend analysis: understand changes over time● Recommendation and adaptation: suggest resources and actions to learners
or instructors, adapt systems to learners...
Research examplesCombine observation data with click streams to detect boredom, frustration, gaming the system…
● Off-task behaviors patterns can be identified like completing module too fast or too slow …
● Evidences of gaming the systems like systematic and rapid incorrect answers, systematic guessing …
Experience API, xAPI, can give us more because …
Learner
Learning DesignerData Analyst
StandardsInteroperabilityAnalysis (data mining)
Learning theoriesPsychologyDesign (experience design, UI)Subject Matter Expertise
*Motivator*Navigator (content brokering)*e-Portfolio(or digital badge backpack)
inform iterateco-design
integrate
(developers are constructors)
*Connecting Learner Community*Learner Attribute (style)
*Narrator (learner created context, reflection)
Gamified layeron data from across platforms
Big Data
[Data-Driven Learning Strategy]
Intelligent learning environment is to match a learner with the right content at the right time.
About Learner Profiling
Image credit: DARPA
Recording Learning EventsLearning happens in interactions:
openclipart.org
Instructors, Peers, Experts….
Contents: Courses, Books, Web pages, Games, AR ….
Activities(making, exercises, researching, online, offline ….)
Learner
Recording Learning Events
Social LearningGroup Learning
image credit: Search Engine People Blog
Learning
is
different
image credit: “A New Architecture for Learning”, published by Educause
xAPI tracking all kinds of learning experiences
The xAPI specification has two primary parts
1. defines the syntax of the xAPI data format a. the vocabularies should be community-drivenb. all activities and context can be trackedc. any enabled application/device can send statements
2. defines the characteristics of “learning record stores” (LRS) - a crucial component of xAPIa. data can be exchanged between LRSs (set free from LMS)b. learner can have life-long “personal learning locker”c. LRSs need to validate xAPI statements
Course Webpage Game Simulator
CoachingSocial LearningProjectMobile
Apps
LRS Learning data is sent to LRS
Other activities
LMS LRS
LRS
ReportingTool
Learning records can be delivered to LMSs,LRSs or Reporting Tools.
Recording Learning Events
This is only the basic idea. Crafting the statements with more context and related information is necessary to support analytics and reporting.
ActivityStreams: stream of activity data statements, borrowed from social analytics
11 Attributes in xAPI Data Format★ Unique Identifier
★ Actor (required)
★ Verb (required)
★ Object (required)
★ Result
★ Context
★ Timestamp
★ Stored (internal recording
timestamp)
★ Authority
★ (Protocol) Version
★ Attachments
All information in XAPI statements can
be separated into :
● meta-data,
● descriptive information, and
● complementary data.
Syntax-1
Actor:
Agent (= persona) or group (multiple IDs allowed)
Verb:
ID = an IRI(URL) = a specific semantic meaning +
human readable display
Object:
an agent, a group, a statement or an activity(most common)
Syntax-2
Activity:
ID = an IRI(URL) = with a specific boundary (granularity)
Definition:
Name
Description
objectType
Extentions (useful to customize reporting)
Syntax-3
Result:
Score
Success
Duration
Completion
Response (learner’s response to the experience)
Extentions
Syntax-4
Context:
Registration (differentiate multiple attempts)
Instructor
Team
ContextActivities (parent, grouping, category, other - like related lesson)
Revision
Platform
Language
Statement (refer to one other statement for a whole experience)
Extentions
Context, Context, ContextThe XAPI differentiates between the core context and the wider context.
The core context includes the instructor(s), the direct peers involved in an activity (team), the learning environment (platform), the language that was used in the performance, and a framing statement for an activity (e.g., the course that relates to the activity).
The extended context includes a set of data-records about the wider context of a learning activity. This wider context is not explicitly specified and can include the location of the learner, the wider (social) relations, the duration of an activity, environmental factors (e.g., temperature or noise level) etc. The format and the content of the wider context is specific to the AP and is not subject to the interoperability of the data format.
Ask the right questions!
LRSAn LRS is defined by two interfaces:
● Statement interface (statement API)● Document interface - this interface handles three types of documents
● State interface (state API)● Activity profile interface (activity API)● Agent profile interface (agent API)
The LRS is responsible for
1. validating that the system sending data is authorized,2. checking that the data being sent is xAPI-compliant,3. storing the data properly,4. making that data available to any other authorized system or activity provider
when asked.
Advanced Applications of xAPI - Data Transfer based on RESTful HTTP w/i LRS● Agent Profile API
○ personal info., learner profile and modeling, user settings, learning journal, career plan & goal
○ an integrated picture of a learner activities across systems and devices with multiple identities
● Activity Profile API (for activity provider)○ interactions between learners (collaboration, social or competition)○ learning planning tool (access to or update the LRS internal definition of
a given activity id, even before the activity sends any statement)● State API
○ persist state across devices● Authentication services, querying services, visualization services, and
personal data services are some examples.
xAPI tracking is...
Semantic
Contextualized
From any device and sensor
LRS frees the learning data so they can be put together, analyzed, modeled, reused, carried with learners and accumulated life-long
xAPI + Open Badges => Learner ePortfolio
Both are representing learner data by exploiting HTTP, JSON, and REST - simple, lightweight method that lowers entry barrier for developers.
Together, they offer a new way to think about constructing interoperable learner model data!
JSON vs XML● JSON is bandwidth-non-intensive● JSON is better adapted (than XML) to devices with limited capabilities such as
smart things● JSON possesses a very limited set of data types. Restricting itself to primitive
data types makes it deeply and immediately interoperable with pretty much any programming language that exists out there.
● JSON is a better data exchange format, XML is a better document exchange format
● JSON is a preferred format in NoSQL database - data not seamlessly conform to a columnar/relational model. Since JSON objects may be heterogeneous in terms of number and types of fields, this allows for tremendous flexibility in storing and retrieving objects as compared to relational databases.
(Should you like to know more about JSON, here is a very simple interactive tutorial. )
Bootstrapping Learner Model Data● Making data contribution as easy as possible● Bootstrapping heterogeneous learner model
data ○ the raw experience data stored in a LRS referenced by an Open Badge,
could be analyzed to perform an application’s own interpretations of that evidence, as suggested by Carmagnola, Cena, and Gena (2011)
○ Guo & Greer, 2007;
○ Tiroshi, Kuflik, Kay & Kummerfeld, 2011 ) summarized various boostrapping methods
○ ADL developed an open source project/resource called “lr-data”(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)
Sharing and Analysis of DataIn the case of sharing and making sense of learner data, a similar model can be followed where any contributing application (e.g., an ITS or game) could easily publish activity data to a LRS, Open Badges to a learner’s badge backpack, or other learner model data to their shared learner profile as JSON over a RESTful HTTP connection.
The LRS, badge backpack, or learner profile can exist anywhere (e.g., an employer organization, a commercial provider, or even self-hosted by a learner) – all the contributing applications need to know are the URLs and the learners credentials.
If a learning application wants to make sense of all this data, it too just needs the URLs and the learner’s credentials to get started. The learner could set permissions concerning what applications can access their data. (Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)
TLA Services
TLA will also include services for managing learner profiles. Open Badges can be referenced by learner profiles, which will likely contain other learner data such as goals, reflection, etc.
The TLA will also include services for creating and accessing competency definitions to serve as a common way to reference educational standards, learning objectives, and competency definitions through web APIs(Damon Regan, Elaine M. Raybourn, and Paula J. Durlach)
The whole picture =Training and Learning Architecture(TLA)
● ePortfolio● Learner modeling● Machine readable
● Competency standards
● Knowledge map● Standard
alignment
● xAPI COP● Common
vocabulary● Learning
Design
● Sharing of metadata & paradata (LR)
● Re-usability● Semantic analysis
A taxonomy of education standards
Redd, Brandt
Education Standards for Interoperability
❏ Academic standards ❏ Data Standards (consistent definition)
❏ Student data❏ Data Dictionary (data element definition)❏ Logical Data Model (entity definition)
❏ Educator data❏ Content data (metadata, paradata)
❏ xAPI vocabulary/activity profile
CEDS
Major Learner Data Categories
● Educational records● Competencies(skills, knowledge, abilities,
outcomes…) and domain learning objectives● Data in affective, motivational(disposition) and
social dimensions● Data for learning style modeling● Data for pedagogical purposes
About Content
Image credit: DARPA
Learning Registry - Social Network sharing metadata and paradata of Learning Resources
Make contributing as easy as possible !
ParadataWhile learning analytics generally refers to analysis of data about learners, paradata refers to data about learning resources.
Paradata can also record contextual information by linking resources with educational standards and curricula, pedagogic approaches and methodologies.
Paradata can be regarded as an extended and altered version of JSON ActivityStreams.
Paradata differs from ActivityStreams in that it enables complex aggregations of activities to be recorded; e.g. High school English teachers taught using this resource 15 times during the month of May 2011.
Content StrategyxAPI modernize the SCORM runtime, what about a modernized content strategy?
IEEE LTSC : the answer is HTML5 + EPUB 3
W3C standards define an Open Web Platform for application development that has the unprecedented potential to enable developers to build rich interactive experiences, powered by vast data stores, that are available on any device.
=> HTML5 is the cornerstone.
In addition to the classic “Web of documents” W3C is helping to build a technology stack to support a “Web of data,” the sort of data you find in databases. The ultimate goal of the Web of data is to enable computers to do more useful work and to develop systems that can support trusted interactions over the network. The term “Semantic Web” refers to W3C’s vision of the Web of linked data. Semantic Web technologies enable people to create data stores on the Web, build vocabularies,
and write rules for handling data. Linked data are empowered by technologies such as RDF, SPARQL, OWL, and SKOS.
RUSSEL - Re-Usability Support System for eLearning
The RUSSEL project includes:
● a novel user interface (UI), ● drag-and-drop SCORM disaggregation, ● a framework for automated metadata generation, ● a new approach to supporting instructional design best
practices while remixing existing materials, ● paradata capture from use of content in instructional
designs.
The Whole Picture
xAPI and Open Ecosystem Picture- Based on standardized tracking and communicating language - xAPI, and the standardization for learner's data
- Modularized ecosystem, new apps could always integrate in and with each other (through standard web service call), a product could have several modular components shown here
Knowledge Map / Topic graph / Competency network
Learning Objects / Resources / Tools / Widgets
Adaptive engine / Intervention engine / Recommendation algorithm
Analytics / Data mining tools
User Interface Dashboard / Reporting / Visualization
Learning Community Certifications
Next Gen Learning Environment
from ADL Andy Johnson
Building “Community of Practice(COP)” is the crucial
foundation of a standard, and the culture of using data for making instructional decisions.
xAPI is an open and flexible framework standard for tracking learning experiences, it needs further
“community-driven” rules - top-down rules won’t work - for different domains to
ensure data quality and interoperability.