multi-modal visualization methods

Multi-Modal Visualization Methods

The use of Haptics, Graphics, and Sonification to Better Interpret Multi-dimensional Data

By Doanna Weissgerber 7/19/01

University of California at Santa Cruz

2 of 68

Overview Goal of my research Purpose of scientific visualization Methods of visualization Why multi-modal methods? My Previous and present research

in visualization State of the art in visualization Proposed research


3 of 68

Goals

Passive haptics is an information channel which has not been researched I am most interested in passive haptic information displayed on it’s own

information channel non-redundant

I will employ usability testing to prove effectiveness of haptic mappings In combination with other modalities when appropriate

Watching for cross-modal problems and enhancements Alone when appropriate

Watching for enhancement of a performed task My starting basis will be in scientific visualization

Increasing the amount of information able to be perceived in a period of time Great possibility of moving to flight simulation if funding found


4 of 68

Purpose of scientific visualization Accurately transmit data from

computer to user Enhance user’s ability to

understand presented data Develop mental image of the data


5 of 68

Methods of visualization Does not specifically require sight

Graphical - sight Color Texture Shape

Sonification - sound Decibel level Different instruments

Haptics - touch Tactile Kinesthetic


6 of 68

Why Multi-modal methods?

"Current graphics and visualization technology cannot cope with the volume and complexity of the data produced by the simulations that will be carried out on the high-end computer platforms in the next two to four years".

They say that new techniques that merge visual, sonic, and haptic data representations need to be developed.

Department of Energy


7 of 68

Multi-modal benefits Possible to display more data at the same

time Possible to display data in a more intuitive

fashion Earthquake model

locations of the earthquakes• Graphically represented

distance traveled • Graphically represented

depth of the earthquake• Sonification using musical notes

magnitude of the quake• Haptic vibration


8 of 68

Multi-modal benefits(cont) Cross-modal stimulation can lower

detection threshold Tactile stimuli can reinforce and/or

clarify stimuli from other modalities More natural mappings

Vision and audition usually dominate haptics

Haptics dominate with texture or surface judgments


9 of 68

My previous and present visualization research ProtAlign

Protein structure prediction Graphical visualization Sonification

Passive Haptics Unit


10 of 68

Predicting Protein Structure

Why predict the structure? Protein crystallization is difficult and expensive

Why use alignments to predict structures Proteins with similar amino acid sequences will

likely have same structure and function Huge databases of protein sequences exist

Why Study Structure Gain Insight into protein functions

Guide biological experiments Discover the genetic basis of disease


11 of 68

How is protein made

DNA is comprised of Nucleic Acids

Codons are groups of three nucleic acids which code for amino acids


12 of 68

Protein Biology Protein

Polypeptide chain made up of Individual Amino Acids

20 Different Amino Acids Distinguished by the R group


13 of 68

Protein Structure Hierarchical Structure

Goal understand function of protein from primary structure

Sequence of protein is relatively easy to obtain


14 of 68

Assessing an alignment Heuristics for assessing prediction

Amino Acid similarity four basic types of amino acids

• Hydrophobic,Charged Acidic, Charged Basic, Polar (hydrophilic)

Size of amino acid Core conservation

Want the core of the protein to be most similar

Environmental preferences


15 of 68

Two Dimensional Alignment Evaluation

Belvu Possible to Edit 2-D alignment and see results Color coded to assess positions along the

alignment Must remember protein 1 letter codes Only one color mapping possible at one time

User must memorize properties of amino acids


16 of 68

Why visualize the prediction? Some heuristics lack numerical

methods Combining heuristics is difficult Sanity check


17 of 68

Other tools for evaluating structure predictions

RasMol SAE DINAMO CINEMA SwissPDB SwissMODEL


18 of 68

ProtAlign introduced

Combines features of previous tools

Adds new features


19 of 68

ProtAlign allows quick assessment Ribbon mode scoring allows quick

assessment of overall alignment

BadGood


20 of 68

ProtAlign Main Structure Visualization Modes

Backbone Ribbon Strand Streamline / rungs

Shows alignment mismatches

-Uses program MeasureShift


21 of 68

ProtAlign Main Structure Visualization Modes (cont)

Cartoon Much like ribbon Shows secondary structure

• Helices• Loop structure• Beta strands

Invisible Allows focus on specific areas Removes extraneous information


22 of 68

ProtAlign Alignment tools

Low resolution glyphs Edit alignment in three dimensions

Updates two and three dimensional display

Multi-modal visualization Environmental mapping

• Amino acid environmental preferences


23 of 68

Glyph Representation

Amino acid size Amino acid type

Hydrophobic Cylinder

Polar/Hydrophilic Square

Charged Basic Pyramid

Charged Acidic Cone


24 of 68

Alignment Representation

Evaluate positions along alignment Color

Rainbow analogy Red, yellow,

green, blue Shape Size


25 of 68

Evaluating a position along the alignment

Rasmol Histimine is polar Phenylalanine is

hydrophobic Easy to confuse as

good substitution ProtAlign

Cylinder and square peg indicates the types differ

Red indicates this is very unlikely to substituted in nature

A = histimine B=Phenylalanine

Bottom = histimineTop = phenylalanine


26 of 68

Possible to edit via 3d interface Edit alignment via 3d interface


27 of 68

ProtAlign Scoring metrics Blosum 62 matrix

Likelihood that the substitution occurs naturally in nature

Environmental Calculate environmental probability

• Exposure preferences– Buried– Partially buried– Exposed

• Preferences for neighboring amino acids• Structural preferences

– Coil– Strand– helix

Exposure None


28 of 68

Environmental Sonification added to ProtAlign Environments

Used to evaluate amino acid exposure preferences

Buried• Tend to like protein core

Partially exposed• Tend to like protein area between

core and loops

Exposed• Tend to like loop regions


29 of 68

Sonification of ProtAlign Beachhouse in Santa Cruz

Buried Wind

• In the house near the beach• Wind rattles the windows

Partially exposed Waves

• Open door walk toward beach• Hear the waves crash in the distance

Exposed Seagulls

• Next to the shore• The seagulls greet you


30 of 68

ProtAlign Usability testing Color mapping tested

Environments used to color amino acids Possible color pairings tested

• Blue/red – excellent/bad

• Blue/yellow – excellent/poor

• Blue/green – excellent/good

• Green/yellow – good/poor Users asked if they thought if substitution

was• Much worse

• A little worse

• A little better

• Much better


31 of 68

ProtAlign Usability testing Sonification mapping tested

Exposure Buried

• wind

Partially Buried• waves

Exposed• seagulls


32 of 68

ProtAlign Visual Testing Excellent/bad pairs recognized perfectly

and quickly Excellent/poor pairs moderate accuracy Good/poor pairs recognized with

surprisingly good accuracy

Blue/Green was skewed by single user allowing 45 seconds to elapse while answering


33 of 68

ProtAlign Aural Testing Exposed amino acids

More quickly recognized Accurately recognized


34 of 68

ProtAlign Aural Testing (cont) Partially exposed amino acids

Slowest recognition Moderate accuracy


35 of 68

ProtAlign Aural Testing (cont) Buried amino acids

Slow recognition Moderate accuracy


36 of 68

ProtAlign Visual Testing

Visual accuracy in multi test is fairly close to single testing


37 of 68

ProtAlign Aural Testing Time for exposed consistently faster


38 of 68

ProtAlign overall results

Faster to present all information at once Time to evaluate a position for

environment and for exposure when presented with all information less than time to present information individually

Both environmental and exposure info mapped to color


39 of 68

Passive Haptic Unit No hands required Currently controls motor filled chair

pad ACCESS RDAG-128H

Serial communication Eight DACs

Digital to Analog converters Control variable voltage items

• Motors• Fans• Heating elements


40 of 68

How haptic unit is controlled

Functions to make voltage changes to single DACS using easy function calls.

RunDAC(theDAC, voltage_level) Pulse (theDAC, voltage_level)

Causes DAC to turn on to voltage_level than turns it off

Wave (theDAC, increments, level) DAC starts at 0 and increases voltage until

levels in the number of increments• Wave(0, 2, FULL_POWER)

– causes the motor to start at power 0, go to half power, then to full power, than half power, ending with off.


41 of 68

Multiple motor control

Possible to turn motors off and on in any sequence


42 of 68

State of the Art Immersive Multi-modal Visualization

CAVE (University of Illinois) Room SGI 4 projectors (3 walls and floor) Stereographic LCD glasses

Head Mounted Display (HMD) Wearables

Belt-pack PC HMD Wireless communications hardware Input device

Touch pad Chording keyboard


43 of 68

State of the Art Non-Immersive Multi-modal Visualization

PHANToM force feedback device requires at least one

finger to operate tends to have low degrees of freedom

Fan applied to hand by Ogi et al Information given using fans on the hand Unable to use hand for anything else

Air jets Low bandwidth

Jets must be spaced to sense seperately

Tacticon 1600 electrodes on user’s fingers provide

electrical pulses


44 of 68

Past “Normal” Active Haptic Units

Require hands be used for output Haptic mice and haptic joysticks

Require user to maintain contact Input pens

Must maintain contact


45 of 68

Proposed Research

Multi-modal visualization Concentration on tactile haptics

Use of haptics Showing independent information

More information at the same time Showing redundant information

Information more quickly understood Information better understood

User testing Did the user understand the mappings? Did the haptics help understanding?


46 of 68

Proposed Research (cond) Passive haptics

Sit back and absorb the information


47 of 68

Perceptual BandwidthVisual Aural Tactile

Physical Mechanism

Light Waves Sound Waves Surface Texture

Organs Retina Ear Drum Skin sensors: pacinian corpuscles, etc.

Perceptual organization

Global, spatially mapped

Global, spatially mapped

Spatially focused, body mapped

Information bandwidth

106 bps 104 bps 101-102 bps


48 of 68

Skin Layers

Dermal papillae Interdigitations of epidermis

and dermis Fingertips Soles of feet

Dermis Papillary layer

Nerve endings Reticular layer

Bulk of dermis nerves

[36]


49 of 68

Tactile sensory receptors Thermoreceptors

Changes in skin temperature Mechanoreceptors

Pressure Vibration Slip

Nocioreceptors Pain


50 of 68

Tactile sensory receptors (cont)

Successiveness limit 5 ms to perceive as separate 20 ms to perceive order (not including time

for cortex to process the order) Adaptation

Slowly adapting Respond throughout stimulus

• Joint angle from skin stretch Rapidly adapting

Respond to start/end of stimulus• Block out extraneous signals

– wearing gloves would be constant stimulus which is adapted to


51 of 68

Haptic Perception - ReceptorsReceptor name Ordered by depth in dermis

Type Stimulus Location Shape Sensitive

Frequencies

Receptor

Field Size

Merkel

Receptor(Mostly

found in

fingertips)

Slow

Adapting

Pressure Border of

dermis and

epidermis

Disk 0-10 Hz Small

2-4 mm

Meissner

Corpuscle(mostly found on

hands and feet)

Rapid

Adapting

Taps on

the skin,

Light touch

Dermis, just

below

epidermis

Stack of flattened

cells with winding

nerve fiber

3-50 Hz 2-4 mm

Ruffini

Cylinder(not found

In glabrous/hairless skin)

Slow

Adapting

Stretching of

skin, joint

movement

Dermis Branched fibers

inside cylindrical

capsule

0-10 Hz large

Pacinian

Corpuscle

Rapid

Adapting

Rapid

vibration

Deep

dermis

Layered capsule

surrounding nerve

fiber

10-500 Hz large


52 of 68

Haptic perception: Free Nerve Endings and pain receptors

Free Nerve Endings Thermal differences

Heat receptors Cold receptors Receptor field 1-2mm

Pain receptors Painful mechanical stimulation Extreme cold and extreme heat

• < -15°C and > 45 ° C (5 ° F and 113 ° F)

• quick transmission (faster than normal response)


53 of 68

Pacinian corpuscle Pacinian corpuscle in resting

state

Skin over corpuscle touched. Corpuscle deformed.

Viscous gel between the capsules moves. Nerve ending resumes normal shape.

Pressure released, Corpuscle resumes original shape, but nerve ending deformed

Pacinian corpuscle in resting state


54 of 68

Tactile perception limitations Secular et al 1994

Sensitivity depends on area of body Fingers more sensitive than stomach or

back Females more sensitive to light touch Cooled skin less sensitive

Vibrotactile stimulation Greatest sensitivity 200 hz 10-30 hz, greatest sensitivity with

vibrating probe


55 of 68

Tactile perception limitations (cont) Two point thresholds

Location dependent Fingers 2mm Forearm 30 mm Inner thigh 34 mm Outer thigh 36mm Foot bottom 33mm Foot top 46mm Back 42-70mm

Declines with age Sensitivity and two point thresholds

correlated


56 of 68

Tactile perception limitations (cont)Karen MacLean lecture

Spatial limitations Depends on size of receptor field

Meissner corpuscle smaller receptor field than Pacinian corpuscle

Successiveness limitation 5 ms to perceieve as separate 20 ms to allow order determination

Takes longer for cortex to process Masking

Spatial or temporal stimulus interference limits maximum transmission rate


57 of 68

Multi-modal perceptionSrinivasan 1996

Crossmodal, intermodal one modality subconsciously influences

perception in another modality Multimodal

an event is perceived and integrated by multiple senses

Supramodal phenomenon that applies to all senses

Intramodal all in one sense


58 of 68

Early examples of cross-modal effectsKaren MacLean lecture

1669 Bartholmus Partially deaf people seem to hear better

in the light than the dark 1888 Urbantschitsch

Thresholds of touch and pressure affected by noise

Weak sounds lower threshold Strong sounds raise threshold

1920 Johnson Tactual discrimination 2% better in the

light than in the dark


59 of 68

Modern examples of cross-modal effects

McGurk effect McGurk and MacDonald 1976

Auditory output “bows” Vision output “goes” May perceive a synthesis of

the two Doze

What we see may influence what we hear and vice versa

http://zzyx.ucsc.edu/Psych/psych.html


60 of 68

Modern examples of cross-modal effectsDr. Dom Massaro

My dad taught me to drive See

My gag kok me koo grive

Hear My bab pop

me poo brive


61 of 68

Cross-modal factorsLecture of Karen MacLean

Stimulus intensity Low to moderate accessory

stimulus Primary stimulus facilitated

High accessory stimulus Primary stimulus inhibited

Habituation Initial effects to accessory stimulus

may differ from later effects


62 of 68

Cross-modal factorsLecture of Karen MacLean

Bimodal neurons in several areas of cortex Convergence of unimodal sensory

(afferent) inputs into single neuron Does more than sum information from

different senses Transforms information

Sensory convergence patterns in superior colliculus (monkey)

5% is trimodal 1% is auditory/somatosensory 9% is visual/somatosensory 11% is visual/auditory


63 of 68

Possible real time uses of passive haptics

“Any landing you can walk away from is a good landing. Any landing you can fly away from is a great landing.” Mark Boyd

While landing 85% of a pilots time is spent looking at the attitude indicator.

Used to determine turn angle (bank). Also used to determine whether ascending or descending.

It is possible while flying to be in a right bank and feel as though you are going left and vice versa

No frame of reference


64 of 68

Landing a plane

Pilot would ideally be looking outside at the airport and runway Airplanes (or cattle) may wander

onto runway as you approach You may be descending through a

flock of birds Haptic output could give the

information from the attitude indicator to the pilot


65 of 68

Why passive haptics?

Pilot is listening and talking while landing Sonification is not an option which the

FAA would allow Receiving instructions from tower Talking to tower Talking to other planes in the area

Pilot is already scanning many gauges Graphical/visual information is not an

option attitude indicator airspeed indicator turn indicator


66 of 68

Why passive haptics? (cont)

Pilot needs both hands and both feet to fly Active haptics would not work

Left hand on yoke Right hand on throttle Left foot on left rudder Right foot on right rudder

Passive haptics could be attached to pilot seat


67 of 68

Possible pilot passive haptic scenario Left and right bank

Two motors required If in left turn, vibrate on left If in right turn, vibrate on right

Nose pitch Three motors required

If nose down vibrate bottom motor If nose level, vibrate middle motor If node up, vibrate top motor

Possible scenario should be reversed Pilots used to stepping on the ball

If ball in turning indicator slips right, depress right rudder


68 of 68

Goals

Passive haptics is an information channel which has not been researched I am most interested in passive haptic information displayed on it’s own

information channel non-redundant

I will employ usability testing to prove effectiveness of haptic mappings In combination with other modalities when appropriate

Watching for cross-modal problems and enhancements Alone when appropriate

Watching for enhancement of a performed task My starting basis will be in scientific visualization

Increasing the amount of information able to be perceived in a period of time Great possibility of moving to flight simulation if funding found


69 of 68

Acknowledgements Alex Pang for his patience and the initial suggestion of haptics. Suresh Lodha for his direction in sonification. Chris White for the computer, but mostly for his loving support. Mark Boyd for his excellent insight into planes and helping me

conceive an exciting (possible) real-time haptic application. Mark Hansen for our years of collaborative work on ProtAlign

before sonification. Carol Mullane for her help in sorting out all of the little things

before they become big things.

multi-modal visualization methods

Documents

santa cruzof

datauniversity of california

haptics haptics

multimodal benefitspossible

passive haptic information

haptic data representations

goals passive haptics

protein crystallization