the open university's institute of educational technology evaluation of mobile learning agnes...
TRANSCRIPT
The Open University's Institute of Educational Technology
Evaluation of mobile learning
Agnes Kukulska-Hulme, 4 July 2009, Brest
source: www.evaluationsupportscotland.org.uk
The Open University's Institute of Educational Technology
Session overview
1. Introduction and sources
2. Approaches to evaluation: why and how to evaluate
3. Common problems and weaknesses – drawing on experience of PhD thesis examinations and reviewing
4. Your aspirations and issues
5. Usability evaluation
6. Wrap-up/ summary of key points
The Open University's Institute of Educational Technology
Introduction and sources• Tradition of CAL/TEL evaluation at The
Open University, UK – supporting distance learners– Questionnaires– Telephone interviews– Email interviews– Focus groups– Observations captured on video– Eye tracking– Accessibility for learners
with disabilities
The Open University's Institute of Educational Technology
Mobile learning evaluation sources Traxler, J. & Kukulska-Hulme, A. (2005) Evaluating Mobile
Learning: Reflections on Current Practice. In Proceedings of mLearn 2005: The future of learning in your hands, 25-28 October 2005. http://oro.open.ac.uk/12819/
Traxler, J. & Kukulska-Hulme, A. (2006) The evaluation of next generation learning technologies: the case of mobile learning. In ALT-C 2006: The Next Generation, Research Proceedings, 5-7 Sept 2006. http://oro.open.ac.uk/12295/
Section in Big Issues in Mobile Learning, 2006 (ed Sharples): “What are appropriate methods for evaluating learning in mobile environments”? http://www.lsri.nottingham.ac.uk/msh/Papers/BIG_ISSUES_REPORT_PUBLISHED.pdf
The Open University's Institute of Educational Technology
Mobile learning evaluation sources
• Chapter on evaluation methods by Mike Sharples in the book Researching Mobile Learning, Peter Lang Verlag, 2009 (eds Vavoula, Pachler & Kukulska-Hulme)
• Recent article by Vavoula, G. & Sharples, M. (2009) Meeting the Challenges in Evaluating Mobile Learning: A 3-level Evaluation Framework. IJMBL, 1(2).
The Open University's Institute of Educational Technology
Evaluation of mobile usabilityBook chapter:
• Kukulska-Hulme, A. (2005) Mobile Usability and User Experience In: Kukulska-Hulme, Agnes and Traxler, John eds. Mobile Learning: A handbook for educators and trainers. London, Routledge, pp. 45–56.
Journal article in IRRODL: • Kukulska-Hulme, A. (2007) Mobile Usability in Educational
Contexts: What have we learnt? International Review of Research in Open and Distance Learning, 8(2), pp. 1–16. http://oro.open.ac.uk/8134/
Book chapter: • Kukulska-Hulme, A. (2008) Human Factors and Innovation with
Mobile Devices. In: Hansson, T. ed. Handbook of Research on Digital Information Technologies. IGI Global, pp. 392–403. http://oro.open.ac.uk/10670/
The Open University's Institute of Educational Technology
Meanings/purposes of ‘evaluation’
• Assessment– tests, grading achievement
• Comparison– with traditional means/methods of learning
• Development– formative evaluation, piloting
• Demonstrating the value of an innovation– Value for money, educational value
The Open University's Institute of Educational Technology
What are you going to evaluate?
“An analysis of 12 international case studies in Kukulska-Hulme & Traxler (2005) reveals that reasons given for using mobile technologies in teaching and learning relate principally to:
• improving access• exploring changes in teaching and learning• alignment with institutional or business aims”
From: Traxler, J & Kukulska-Hulme, A. (2006) The Evaluation of Next Generation Learning Technologies: the Case of Mobile Learning
The Open University's Institute of Educational Technology
Vavoula, G. & Sharples, M. (2009) Meeting the Challenges in Evaluating Mobile Learning: A 3-level Evaluation Framework. IJMBL.
• Propose a three-level framework for evaluating mobile learning:
– a micro level concerned with usability– a meso level concerned with the learning
experience– a macro level concerned with integration within
existing educational and organisational contexts.
The Open University's Institute of Educational Technology
What do you want to know? (chapter by Sharples, 2009, in Researching Mobile Learning)
• Usability: will it work?• Effectiveness: is it enhancing learning?• Satisfaction: is it liked?
Case studies: Mobile learning organiser (higher education), MyArtSpace (school/museum learning), Personal Inquiry (school/indoor and outdoor learning)
The Open University's Institute of Educational Technology
MyArtSpace case study
• Micro level: examined individual activities, such as making notes, recording audio, viewing the collection online, and producing presentations of the visit.
• Meso level: examined the learning experience as a whole, exploring whether the classroom-museum-classroom continuity worked.
• Macro level: examined the impact of MyArtSpace on educational practice for school museum visits.
For each level, the evaluation covered three stages:Stage 1: what was supposed to happen, based on pre-interviews with
stakeholders and documentation Stage 2: what actually happened, based on observer logs, focus groups, and
post-analysis of video diariesStage 3: the gaps between findings from stages 1 and 2, based on reflective
interviews with stakeholders and critical incident analysis of the findings from stages 1 and 2
The Open University's Institute of Educational Technology
What is the value of mobile learning? • How does mobility change learning?• How and why is the technology used outside of
the main activity?• How is it used in conjunction with other
technologies?• How does it fit in with the user’s work life or
lifestyle?• Does its use evolve over time?• Does it have a lasting impact?• Who is involved in the learning experience?
The Open University's Institute of Educational Technology
Taylor (2006) in Big Issues in Mobile Learning
• “Users… may find themselves fascinated by the new devices in a way which they may find interesting, and even fun, but which produces no lasting valuable impact on their work practices”
• “Our potential subjects of study may be wandering around studying things that interest them, at times that suit themselves, with little or no concern for consistency”
The Open University's Institute of Educational Technology
Taylor (2006) in Big Issues in Mobile Learning• “The current emphasis on ‘activity’ has stimulated a
great deal of evaluation effort focused around Activity Theory, where, for example, emphasis is placed on historical perspectives (learners’ past experiences) and how these relate to the technology to be evaluated”
• “The key issue is to realise that evaluation may entail the peeling back of multiple layers of activity, and the need to keep track of what has been peeled and what hasn’t is essential”
The Open University's Institute of Educational Technology
Taylor (2006) in Big Issues in Mobile Learning
• “It is essential to understand how structured learning activities blend with other more social or informal activities”
• “Traditionally, evaluators might relate the success of a design to the success with which learners can achieve pre-identified learning outcomes. The nature of learning outcomes in the mobile age needs to be adaptive. For example, they may relate to the extent to which someone has assimilated information into their own experience and development”
The Open University's Institute of Educational Technology
How will you evaluate?• Questionnaires were often used; interviews, focus
groups and observations were used less frequently. • Accounts of interviews and focus groups were brief
and suggested that the sessions were usually short and probably informal.
• Statistical analysis and system data were both used infrequently.
• In a relative small number of cases, evaluation used and combined several of these different techniques.
From: Traxler, J & Kukulska-Hulme, A. (2006) The Evaluation of Next Generation Learning Technologies: the Case of Mobile Learning
The Open University's Institute of Educational Technology
Technologies for data capture
• Audio logs• Questions by sms or email• Discussion forum• Automatic logging of interactions• Video recording• Eye tracking• Pen and paper
The Open University's Institute of Educational Technology
Participants as Researchers; ethical issues(Vavoula, G., mLearn’08 workshop)
• Automatic logging of interactions with technology– System logs enabled on personal device
• Carrying or wearing data-capture equipment– Wear mobile eye-trackers, camcorders, carry microphones
• Manually record learning experience– Keep diaries, blogging
involv
em
en
t
The Open University's Institute of Educational Technology
Mobile Learning Research Ethics (Vavoula, G., mLearn’08 workshop)
• Mobile technology translates (most often) to personal technology– Are learners willing to be monitored? How much of their privacy will
they unveil? What if they’re under-age?– Is it OK to monitor everything? How much do we really need to know?– Even if they agree, is it easy to safeguard personal data? What are
best dissemination practices?– Will users cooperate in practice? E.g. synchronise as and when
needed?• Informed consent
– Can we really inform accurately?– Can they really consent unreservedly?– How do we deal with late withdrawals?
• Ethical mobile learning practice– Participants as co-researchers
The Open University's Institute of Educational Technology
Common problems and weaknesses (based on examinations and reviews)
1. Lack of clear definitions• ‘practices’, ‘contexts’, ‘collaboration’, ‘community’…
2. Evidence from literature comes from very different cultural settings/contexts
• Japan, Finland…
3. Convenience samples used badly• Target users poorly defined; skewed samples
4. Ignoring minority views• ‘Most people were happy’
The Open University's Institute of Educational Technology
Vavoula, G. & Sharples, M. (2009) Meeting the Challenges in Evaluating Mobile Learning: A 3-level Evaluation Framework. IJMBL.
Propose six challenges in evaluating mobile learning:
1. capturing and analysing learning in context/across contexts
2. measuring mobile learning processes and outcomes
3. respecting learner/participant privacy
4. assessing mobile technology utility and usability
5. considering the wider organisational and socio-cultural context of learning
6. assessing informality
The Open University's Institute of Educational Technology
Your aspirations and issues
STEP 1
What are you intending to evaluate?
What challenges will you face? What methods will you use?
The Open University's Institute of Educational Technology
Your aspirations and issues
STEP 2
Grand Designs for Mobile Learning (pre-conference workshop at mLearn’09 in October)
Consider how your plans for introducing or extending mobile learning relate to broader social and educational agendas and global challenges, e.g.:
fostering creativity and innovation, social inclusion, cross-cultural understanding, reading literacy, lifelong learning, bridging the digital divide…
The Open University's Institute of Educational Technology
Evaluation of mobile usabilityA synthesis of usability issues across a range of mobile learning projects (Kukulska-Hulme, 2007) identified four main categories:
•• the physical attributes of mobile devices, e.g. size, weight, memory, battery life
•• content and software applications
•• network speed and reliability
•• the physical environments of use
The Open University's Institute of Educational Technology
Evaluation of mobile usability
The 4 categories relate to six aspects of mobile learning:
• the learner• other people• tasks engaged in• the device being used• networks or connectivity• the locations where learning happens
device user
network locations
people
AccessibilityFamiliarityDuration of useOwnershipPersonalisation
AccessoriesSoftwareContentApplications
Conditions of useReliabilitySpeed
MotivationsCostsDemographicsEmotions and pleasure Emergent needs
Lifestyle Social acceptabilityTravel
ContinuityLinkage across contextsPervasiveness
tasks
Technical supportCollaborationSocial networksStudy activitiesOther toolsEnhancementExtensionEmergent uses
Factors impacting on the usability of mobile devices in education (Kukulska-Hulme, 2007)
The Open University's Institute of Educational Technology
Planning your evaluationThe user (learner): How self-motivated are the learners? How
familiar are learners with all features of their device? What reward comes from mobile learning? Can the device be adapted and personalised to suit specific needs? Will it fit with lifestyle?
Other people: Who can support the learners on the go? What spontaneous or pre-planned collaboration can take place? What communities can learners be part of? What do others interacting with the mobile learner need?
Tasks engaged in: Are there tasks set by instructors/tutors or learner-generated tasks? Does speed of network access have an impact on the tasks? Are the tasks confined to the mobile device or do they connect with other environments and tools? Do interruptions matter?
The Open University's Institute of Educational Technology
Planning your evaluation
Device being used: What input devices and other accessories are available? How long will learners keep the device (if not owned)? Is there compatibility or conflict with other tools being used?
Connectivity/networks used: Are wireless networks reliable? Can learners manage to get connected? Are they dependent on connectivity? What are the costs involved, if any?
Locations of use: Are they suitable for the type of learning envisaged? Is quiet or privacy available if required? Is there continuity of use across different locations? Can location-based context be incorporated into the learning experience?
The Open University's Institute of Educational Technology
Wrap-up/ questions/summary of key points
Institute of Educational TechnologyThe Open UniversityMilton KeynesUnited Kingdom
www.open.ac.uk/iet
Thank you!