student research week 2006 image-based evaluation of video-acquired research skills unmil karadkar,...
Post on 25-Dec-2015
215 Views
Preview:
TRANSCRIPT
Student Research Week 2006
Image-based Evaluation of Video-acquired Research Skills
Unmil Karadkar, Marlo Nordt
Richard Furuta
Cody Lee
Christopher Quick
Texas A&M University
Center for the Study of Digital Libraries
The Department of Computer Science
Michael E. DeBakey Institute
Veterinary Physiology & Pharmacology
Student Research Week 2006
Outline
• Cardiovascular Research
• Research Skill Testing Interfaces
• Design of the Evaluation
• Results and Analysis
• Future Work
Student Research Week 2006
Outline
• Cardiovascular Research
• Research Skill Testing Interfaces
• Design of the Evaluation
• Results and Analysis
• Future Work
Student Research Week 2006
Cardiovascular Research
• Related to the blood circulation system
• Involves live bats– Non-destructive, in-vivo studies– Pallid Bats – transparent wings
• Observe bat wings under a microscope
• Use these observations for research– Normal conditions– Modified conditions (occlusion, pressure)
Student Research Week 2006
Research Skills
• Arteries and Veins
• Order of blood vessel– Vein– Venule– Capillary
• Lymphatic vessels
Student Research Week 2006
Working with videos
• Single layout of images
Montage– Facilitates comparison of
consecutive images
• Motion cues
But experienced researchers
often don’t need these cues
Student Research Week 2006
Still Images for Testing Skills
• Validated by a pilot subject
• Properties of image layoutsSpatial organization
Temporal presentation
• Constrain or facilitate contextual informationComparison of images
– Images of blood vessels at different locations– Images of a specific location over time
Student Research Week 2006
Outline
• Cardiovascular Research
• Research Skill Testing Interfaces
• Design of the Evaluation
• Results and Analysis
• Future Work
Student Research Week 2006
Montage Layout
• Spatial Dimensions (0)
• Context (user dependent)
• Temporal Dimension (used)
• Slowed down video
Student Research Week 2006
Scrolling Layout
• Spatial Dimensions (1)
• Context (4 images)
• Temporal Dimension (used)
• Smooth scrolling
Student Research Week 2006
Thumbnail Layout• Spatial Dimensions (2)
• Context (all images in set)
• Temporal Dimension (unused)
Student Research Week 2006
Outline
• Cardiovascular Research
• Research Skill Testing Interfaces
• Design of the Evaluation
• Results and Analysis
• Future Work
Student Research Week 2006
Experimental Design 1• 3 Layouts
– Thumbnail– Scrolling– Montage– 16 images in a set
A E L
Time/image 3s 4s 2s
Answers per image per image per set
Sets 2 2 3
• 3 Tasks– Artery/Vein recognition– Size Estimation– Lymphatic Vessel Wall
Identification
Student Research Week 2006
Experimental Design 2• Balanced across subjects
– Ordering of tasks (AEL, ELA, LAE)– Ordering of image layouts (TMS, MST, STM)
• Powerpoint for presenting layouts
• Feedback– Verbal– Powerpoint’s Pointer feature
• Validation of answers– Panel of experts– Accurate measurements
Student Research Week 2006
Subjects
• Trained researchers in the bat lab– About 30 subjects available world-wide
• 9 experts– Research experience of more than 2 semesters
• 6 novices– Research experience of 1 semester only
• Age group 18 to 35
• 4 female, 11 male
Student Research Week 2006
Data Collection
• Answers for the tasks
• Demographic Questionnaire– Self evaluation of expertise
• Task Questionnaire– For each task
• Most preferred layout• Perceived best performance
• Short interview
Student Research Week 2006
Outline
• Cardiovascular Research
• Research Skill Testing Interfaces
• Design of the Evaluation
• Results and Analysis
• Future Work
Student Research Week 2006
• N=16, x=9.75 σ2=9.73• No learning• Constrained by knowledge• Preferred layout T
• N=16, x=13.59 σ2=2.13• Learning (p=0.0004)• Constrained by layout• Preferred layout T
Artery/Vein RecognitionExperts
0%
20%
40%
60%
80%
100%
T M S
Layout
Per
form
ance
Round 1
Round 2
Novices
0%
20%
40%
60%
80%
100%
T M S
Layout
Pe
rfo
rma
nc
e
Round 1
Round 2
(p=0.005)89.6%
91.4
%
Validated conventional wisdom
Student Research Week 2006
Size EstimationExperts
0%
20%
40%
60%
80%
M1 M2 S1 S2 T1 T2
Round
Pe
rfo
rma
nc
e +/- 40%
+/- 30%
+/- 20%
+/- 10%
Novices
0%
10%
20%
30%
40%
M1 M2 S1 S2 T1 T2
Round
Pe
rfo
rma
nce +/- 40%
+/- 30%
+/- 20%
+/- 10%
• N=16, x=9.03 σ2=14.11• Preferred layout T
• N=16, x=4.06 σ2=12.51 • Preferred layout T
General understanding of the order of blood vessels does not translate into accurate estimation of size
(p<0.002)
(p<0.002)
Student Research Week 2006
Lymphatic Vessel Wall Identification
Experts
0%
20%
40%
60%
80%
100%
M S T
Layout
Per
form
ance
Novices
0%
20%
40%
60%
M S T
Layout
Per
form
ance
• Worked best with layout M (p=0.035)
• Preferred layout M
(p=0.03)87%
55%
Experts and Novices use different strategies for applying contextual information
• Worked best with layout T
• Preferred layout M
Student Research Week 2006
Outline
• Cardiovascular Research
• Research Skill Testing Interfaces
• Design of the Evaluation
• Results and Analysis
• Future Work
Student Research Week 2006
Future Work
• Train subjects on new interfaces– Subjects preferred familiar interfaces but
performed well with the unfamiliar ones as well
• Investigate mechanisms for conveying size of objects within images– Subjects could not accurately estimate the size
of blood vessels in spite of possessing the required knowledge
• Explore other image layouts
top related