feel the music€¦ · feel the music 1experiment a formal experiment was conducted to prototype 2...

EXPLORING TACTILE MAPPINGS OF AUDITORY STIMULI AS A STEP TOWARD MAKING MUSIC MORE ACCESSIBLE FOR THOSE THAT ARE DEAF OR HEARING IMPAIRED.

Feel the Music

PROTOTYPE 2EXPERIMENT1 A FORMAL EXPERIMENT WAS CONDUCTED TO INVESTIGATE OUR EMOTIONAL RESPONSES TO TACTILE STIMULI AND IDENTIFY WHICH FEATURES OF HAPTIC SENSATIONS AFFECT THE USER'S RESPONSE.

RESULTS

STIMULI• 42 STIMULI WERE SINE AND SQUARE WAVES WITH VARYING FREQUENCY AND AMPLITUDE. • 60 STIMULI WERE GENERATED BY MULTIPLYING A 64HZ SINE WAVE WITH AN ENVELOPE WAVEFORM DEFINED BY A PA-RAMETER α. • 6 REPEATED STIMULI.

• 20 PARTICIPANTS TOOK PART. • 108 TACTILE STIMULI WERE PRESENTED.• STIMULI WERE HAPTIC SENSATIONS GENERATED BY A TACTOR PLACED ON EACH PARTICIPANT’S FINGERTIP.

PARTICIPANTS RATED THE STIMULI BY AROUSAL (INTENSITY) AND VALENCE (PLEASANTNESS). THESE MEASURES ARE BASED ON THE CIRCUMPLEX MODEL OF AFFECT, A WIDELY USED MODEL IN PSYCHOLOGY RESEARCH TO MEASURE EMOTIONAL RESPONSES.

EXAMPLE STIMULI PROGRESSION OF PROTOTYPES

TACTORS

α VALUES FOR ENVELOPESTIMULI

RESULTS

THE RESULTS LED TO THE CREATION OF A WEARABLE PROTOTYPE THAT EMPLOYED TACTILE SENSATIONS AS A MUSICAL INTERFACE. MAPPINGS FROM MUSIC SAMPLES ONTO THE DEVICE WERE DESIGNED AND TESTED IN A SECOND EXPERIMENT.

BY: ALICE HAYNESSUPERVISORS: DR J. LAWRY,

DR J. ROSSITER

EXAMPLE TACTILE MAPPING FOR

“ASHES TO ASHES’

AFTER MULTIPLE ITERATIONS AND MODIFICATIONS THE FINAL PROTO-TYPE CREATED WAS A SLEEVE CONTAINING:

• 8 VIBRATION MOTORS WITH SMOOTH CASINGS PLACED IN SEQUENCE ALONG THE FOREARM.• A SQUEEZER MODULE FORMED OF AN ENCLOSED POCKET FILLED WITH AIR VIA A VACUUM PUMP ON THE UPPER ARM.

THESE WERE CONTROLLED BY AN ARDUINO LEONARDO WITH 3 STACKED MOTOR SHIELDS AND 1 SD CARD SHIELD ONTO WHICH MAP-PINGS WERE SAVED. TACTILE MAPPINGS WERE PLAYED IN SYNC WITH MUSIC BY SENDING A START COMMAND VIA SERIAL FROM MATLAB AND PLAYING THE MUSIC SIMULTANEOUSLY.

MAPPINGSCHOSEN MUSIC SAMPLES WERE CONVERTED INTO MIDI FILES FROM WHICH THE KEY MELODIES AND RHYTHMS WERE IDENTIFIED AND TRANSLATED INTO AN ARRAY OF INTENSITIES TO CONTROL THE MOTORS AND SQUEEZER OVER TIME. 7 MUSIC SAMPLES WERE USED, INCLUDING THE JAWS THEMTUNE, ‘IS THIS LOVE’ BOB MARLEY AND ‘ASHES TO ASHES’ DAVID BOWIE.

TESTING• 20 PARTICIPANTS TOOK PART. • 28 STIMULI WERE PRESENTED.• STIMULI INCLUDED MUSIC SAMPLES, PURE TACTILE SENSATIONS AND SYNCRONISED COMBINATIONS OF THE TWO.

PARTICIPANTS RATED STIMULI BY VALENCE (PLEASANTNESS) AND SE-LECTED ADJECTIVES FROM A LIST TAKEN FROM HENVER’S ADJECTIVE CIRCLE TO DESCRIBE THE SENSATIONS.

• MEAN VALENCE RESPONSES ACROSS ALL PARTICIPANTS WERE STA-TISTICALLY THE SAME FOR MUSIC ONLY STIMULI AND COMBINATION STIMULI. (F)• MEAN VALENCE RESPONSES TO TACTILE ONLY STIMULI WERE STA-TISTICALLY LOWER THAN FOR THE STIMULI INCLUDING MUSIC.• PER INDIVIDUAL SONG SAMPLE PREFERENCE FOR MUSIC, TACTILE OR COMBINRTION PRESENTATIONS VARIED. (E) • OVER 60% OF RESPONSES PER SONG SAMPLE WERE STATISTICALLY SIMILAR FOR MUSIC ONLY AND TACTILE ONLY STIMULI SUGGESTING THAT MAPPINGS WERE EFFECTIVE.• AGE OF PARTICIPANTS DID NOT AFFECT THEIR RESPONSE TO MUSIC, BUT OLDER PARTICIPANTS GAVE STATISTICALLY LOWER VALENCE RANK-INGS TO TACTILE STIMULI THAN YOUNGER PARTICIPANTS. THIS MAY BE DUE TO CHANGE IN SKIN SENSITIVITY WITH AGE.• ADJECTIVE RESPONSES GENERALLY MATCHED FOR MUSIC AND TAC-TILE MAPPINGS BUT ALSO FOR PURPOSEFULLY NON-MATCHED STIMULI.

IN CONCLUSION, THE PROTOTYPE SUCCESSFULLY DEMON-STRATED THE POTENTIAL FOR TACTILE STIMULATION TO BE USED AS A SENSORY SUBSTITUTE FOR ACOUSTIC MUSIC.

• THE EXPERIMENT FOUND THAT THERE IS A GENERAL NEGA-TIVE TREND BETWEEN AROUSAL AND VALENCE RESPONSES AS DEMONSTRATED BY THE MEAN RESPONSES ACROSS STIMULI. (A)• FREQUENCY HAS A SIGNIFICANT POSITIVE CORRELATION WITH ELICITED AROUSAL RESPONSES. (B)• FREQUENCY IS SIGNIFICANTLY CORRELATED WITH 80% OF PARTICIPANT VALENCE RESPONSES. HOWEVER, 6 WERE POSI-TIVE AND 10 WERE NEGATIVE. (B)• AMPLITUDE IS NOT SIGNIFICANTLY CORRELATED WITH AROUSAL OR VALENCE ALTHOUGH THERE IS A GENERAL POSI-TIVE CORRELATION WITH AROUSAL AND NEGATIVE CORRELA-TION WITH VALENCE. (D)• SQUARE WAVE STIMULI MORE AROUSING THAN SINE WAVE STIMULI WHICH MAY BE DUE TO THE POSITIVE CORRELATION BE-TWEEN ENERGY IN THE STIMULUS SIGNAL AND ELICITED AROUSAL RESPONSE. • ENERGY IN A STIMULUS SIGNAL ALSO NEGATIVELY CORRE-LATED WITH VALENCE RESPONSES.• REPEATED STIMULI HAD A MEAN SPREAD OF RESPONSES ACROSS 15.5% OF THE SPACE PER PARTICIPANT. THIS INDI-CATED RELIABILITY IN PARTICIPANT RESPONSES SINCE TESTS RUN FOR MULTIPLE RANDOM RESPONSES HAD A SPREAD OF 98%. (C)

IN CONCLUSION, FREQUENCY , WAVEFORM SHAPE AND SIGNAL ENERGY HAVE A SIGNIFICANT EFFECT ON AF-FECTIVE RESPONSE. THIS CAN BE GENERALISED ACROSS PARTICIPANTS FOR AROUSAL, WHEREAS VA-LENCE IS MORE SUBJECTIVE TO THE INDIVIDUAL. PARTICIPANTS WERE RELIABLY ABLE TO DIFFERENTIATE BETWEEN STIMULI. HOWEVER, RESPONSES VARIED DRAMATICALLY BETWEEN INDIVIDUALS FOR EACH SPE-CIFIC STIMULUS.

Participant Number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Cor

rela

tion

Coe

ffici

ent

-1

-0.5

0

0.5

1Arousal vs log2(Frequency) Valence vs log2(Frequency)

Valence-1 -0.5 0 0.5 1

Arou

sal

-1

-0.5

0

0.5

1

Participant Number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Cor

rela

tion

Coe

ffici

ent

-0.5

0

0.5

Arousal vs Amplitude Valence vs Amplitude

Valence-1 -0.5 0 0.5 1

Aro

usal

-1

-0.5

0

0.5

1

Time (s)0 0.5 1 1.5

Ampl

itude

(V)

-4

-2

0

2

4

Time (s)0 0.5 1 1.5

Ampl

itude

(V)

-4

-2

0

2

4

Time (s)0 0.5 1

Am

plitu

de (V

)

-1

-0.5

0

0.5

1 = [0, -1.5, 7, 1.5]

Time (s)0 0.5 1

Am

plitu

de (V

)

-1

-0.5

0

0.5

1 = [7, -7, 1.5, 0]

CIRCUMPLEX MODEL OF AFFECT

VALENCEAR

OU

SAL

Time (s)0 0.5 1 1.5 2

Enve

lope

am

plitu

de (V

)

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1 = -7

= -1.5

= 0

= 1.5

= 7

Stimulus Number1 2 3 4 5 6 7

Mea

n re

spon

se a

cros

s pa

rtic

ipan

ts

-1

-0.5

0

0.5

1Music Tactile Combination

Participant Number1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Mea

n re

spon

se a

cros

s st

imul

i typ

es

-1

-0.5

0

0.5

1Music Tactile Combination

A

F

E B

CD

0 50 100 150 200 250 300time (sec)

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

ampl

itude

0 5 10 15 20 25 30 35 40

12345678

10

Mot

or n

umbe

r

0 5 10 15 20 25 30 35 40time (sec)40

45

50

55

60

65

70

75

80

85

note

num

ber

RAW AUDIO FILE

MIDI FILE