Case Study: Evaluation of a Tool for Searching inside a Collection of Multimodal e-

Lectures

Marco RonchettiAngela Fogarolli

Dipartim. Informatica e TelecomunicazioniUniversità di Trento

Italy

Logical Architecture

Synchronous (webcast)

Asynchronous (On line, or download - podcast)

Asynchronous(CD - DVD)

Acquisition setup

QuickTime™ e undecompressore TIFF (LZW)

sono necessari per visualizzare quest'immagine.

Cognitive interface

The main focus is on the projected slide, a clear voice is very important, the video carries additional information

like gestures, and can show the environment just enough

Navigation is possible It must be also possible to use the

blackboard!

LODE: the user interface

New Flash version

QuickTime™ e undecompressore TIFF (LZW)

sono necessari per visualizzare quest'immagine.

Lode - portabilitySupporto della didattica tradizionaleMac – Win - Linux

Player MP4(no slides yet)

Presently attempting a

port to mobile phones

Student feedbackEnthusiastic. We were “forced” to extend the experiment

to a second course!

students who… total 1st exam(45 stud.)

2nd exam (23 stud.)

…used the system to review at least one lecture

63% 51% 87%

…followed the course completely off-line

5% 6% 4%

75% anticipated using the system

often or very often

Advantages from student’s perspective

Better time management: ability to recover lectures lost due to forced

absence (illness, work or other time-frame incompatibility);

ability to better organize their time, deciding not to be present at some lecture (elective absence);

Better understanding review some critical point (cases of poor

understanding of a section due to concentration drop, excessive speed in an explanation or intrinsic difficulty);

ability to check the correctness of notes taken during a lecture;

Other advantages

Miscellaneous perception by the student of a better

service provided by the university; support foreign (italian) students who

might have difficulties with the Italian (English) language;

enrichment of the e-learning portfolio; possibility for the teacher to view

himself; possibility to reuse lectures (across

time, or in different contexts!)

Can we do more?

Scenario: Thousands of recorded lectures. Can we dig in them to extract

material for (less formal) learning?

Cfr. approach by Mike Wald(subtitling for accessibility)

Needle - search results

QuickTime™ e undecompressore TIFF (Non compresso)

sono necessari per visualizzare quest'immagine.

Needle - architecture

ASR

Multimodal search!

Questions

Is it useful? Is it effective?

Scenario100% accurate transcription (manually corrected)Goal: extract some info from a 2 hour lecture in half of

the time…

Assess whether the tool could reduce the learning time and help students in their learning task: we do not aim at claiming or proving that a course delivery mode is better than the other.

Comparing delivery modes (e.g. distance learning with traditional classroom-based instruction) (Sener 2005) for the purpose of establishing the superiority of one delivery mode over another is specious, irrelevant and counterproductive because there is not an uniformity of practice.

Experiment68 students, after 20 hours of a 50 hour Java

course.

Group A: LODE

Group B: NEEDLE

ExperimentPre-Test: 5 questions, 5 minutes

LODE group: 70 minutes for the lecture, plus24 minutes for the post-test (8 questions): total 94

minutes to accomplish the task

NEEDLE group: 48 minutes total for responding to the same 8 questions (50% of the time)

Pre-test results

Test results

AnalysisIn the subset of questions, which require students to have

a complete understanding of a part of the lecture both groups performed in the same way.

In answering those questions that also required some reasoning about concepts, group LODE – who had listened to the entire lecture – performed better.

In the questions where the students where asked to provide the right definition for a concept group NEEDLE performed better.

Further investigationsMultivariate regression:- no significant results

Clustering techniques- Indication that some feature seem to be present -

investigation is still on its way

Conclusion NEEDLE tool is more geared towards

information finding than towards actual learning.

A similar pattern could be reasonably expected in a similar experiment performed by reading some assigned material versus searching quick answer with Google.

However, our finding seems to show that Needle works very well as a tool for extracting information from a video source.