CRISP-SKA Cluster of Research Infrastructures for Synergies in Physics - Square Kilometre Array

Sounds like Data

Iain EmsleyOxford e-Research Centre, University of Oxfordiain.emsley@oerc.ox.ac.uk@iainemsley@minnelieder1

Before we startHow many break time conversations can you remember?How many people could you hear clearly at one time?

Overview

Experiencing the WorldHow do we hear?Text to SpeechCan objects make sound?SonificationTesting the UI3CRISP-SKA

Cluster of Research Infrastructures for Synergies in Physics - Square Kilometre Array

Rhythm

Experiencing the WorldRandy Read, https://www.flickr.com/photos/randyread/931759178/in/photolist-2qkvxq-6oc4tz-drbP4-9ZuxYb-7shE22-nyqLWX-pdaXfW-7t73gL-dnxAbd-dnrBo6-dFxs73-aszj2m-dnrBpD-d9RZ1n-dnrtjP-dnrsSt-82Eyp4-a6rcKQ-dnxoos-bwwP94-a1mCW5-a1mEHb-ckP84L-a1mERL-ar2ndA-9jpF76-gbUptq-piHBEg-5fLZNo-4GBq7K-7KWG9K-aRC11T-5xvwr-347jyH-6EH8qr-7cfrDV-5ARz3S-c3Psth-wubABS-eW8RzU-PCnK9-eXTxbX-c5AQBw-525PwA-qWn2wr-21SW1P-eLELKj-4Nsr9N-c5APq7-3LFJ4Thttps://creativecommons.org/licenses/by/2.0/

Both invasive and defensive

DefensiveHeadphonesInvasiveSound systems in carsMuzakSound Cannons

Headphones create a cocoon and give users a chance to build their own worlds and experiences. Direct interface with users: museum guided tours, listening to music on the bus.

Sound systems: think cars in spring and summer. Muzak in shops can be designed to alter emotions: supermarkets at Christmas. Music in restaurants.

How do we hear?Babies hear rhythm early, moves to toneBrain fills in gaps and patternsEarworms play on thisLinks to emotionLimbic system is associated with hearingEars filter information automaticallyrecording a room and listening backUse Doppler Labs Here products

Ears process information and quietly filter it

Babies listening to the 10 similar sounds and being upset when it changes on the 11th. Hearing tries to fill in missing gaps in sound, making up the gaps.

Can be over burdened: think of being in a room. In the atrium: how many conversations clashed?

A perceptual illusion in which our brain imposes structure and order on a sequence of sounds (Levitin 2007, p 109)

Some PrinciplesGroup sounds together; the brain can only understand a limited amount of simultaneous voicesConsider the aspect that will be usefulResearch the data and users context

Text to SpeechPhonesCarsMooseshttps://torchbox.com/blog/adios-manuel/Build onenode tts/twitter.js

Issues: The voice: its not as good as a phone or Sat nav of your choice.

If the speech is too quick, then something can be missed. Why is the final option in menus listen again?

Can objects make a sound?Pressing buttonschuck hci/keyboard.ck Device dependent

Tip jar sonification failed as device and sensor not found on this Mac.

Pressing keys on the keyboard to get a sound

SonificationWhat is sonification?Sonification is an alternative to visualization

the use of nonspeech audio to convey information. More specifically, sonification is the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation. (Kramer, 1997)

Using it for analysis11

Sonification as an alternative or complement to visualisation.

This is the standard definition of sonification. *pause* We believe that it can be extended into analytics.

Streaming from Other Sites

Listening to Wikipediahttp://listen.hatnote.com/

Julie Freeman & Zooniversehttp://blog.zooniverse.org/2014/10/09/we-need-us-online-art-powered-by-the-zooniverse/

12

In Listening..., Hatnote sonifies the size of change using a Wikipedia feed with a simple sonification.We try to move to music rather than sound.

De Roure, Blackburn sonify an artificial hyperstructure using algorithmic composition. We sonify extracted ones.

Sonifying Hamlet13Sonifying the VariantsAuditory Beacons

William Gaver 1982 paper.

Sonifying the VariantsFrom Play to SonificationUsing First Folio and Quartos dataParsing the TEI XML, converting it with rule set into numbers, sonifying the data to produce sounds

14

Sonification

----- Meeting Notes (25/10/15 20:38) -----Pipeline to transform the XML into numbers according to a simple set of rules. These numbers are then transformed into sound in the black box.

Mention the Hinman collator here and stereoscopy.

Used the First Folio Hamlet and the Quartos variants as the test data.

One streamTwo steams to create an audio version of a steroscopic illusion.

Auditory BeaconsActs and ScenesDifferent Instruments and PitchesStage DirectionsDifferent instrumentsPeriod versus Modern sounds?SpeakersIncreasing volumeStereoscopic illusion using two streams

Challenges of making the information useful for the listener.

Acts & Scenes are relatively static. We can be confident that they will be in each play.

Stage directions - use different instruments. *pause* do we use the period sounds versus modern sounds? How does this affect the listener?

Speakers - same instrument with different pitches. Also use increasing volumes as a way marker.

Hemlines and Big Macs

What do the above have in common?

https://www.flickr.com/photos/barefootadrianne/5682398457/in/photolist-9E8M6c-9EbSwo-9Eabzg-9EcP7q-9EcV5G-61b1Az-9Eacqk-9EakZP-9Ebxq7-9E8BKV-9E8M8k-9EbU8o-9Edb7U-9E8MFP-9EbMoh-faswYR-fasyKi-fasDMk-2obBgD-9E8EyM-9EcqxQ-fmXNTk-5NXQn8-9EbDCo-9E92MT-9EdgTG-9E8BhX-9EdWwJ-9E9XeP-5eFaos-fasxGD-fasDTM-6grvqH-9EdYKE-84W3Jj-9Ea7ft-9Ednjd-9Ebrg9-9Ecn6S-9E9Yw4-9E8Z5x-9EbrRf-9E9ZEp-9E8Gmc-9EagHv-9E8FVp-9EaZBt-9Ed9iL-9Ed28w-9E8vZZ Adrianne Behning Photography CC BY-NC-ND 2.0 (https://creativecommons.org/licenses/by-nc-nd/2.0/)https://www.flickr.com/photos/khawkins04/5494471976/in/photolist-9nwB6s-aTocfn-5zZ2aA-8LM2JG-aToc1g-83X5hE-atamt2-atcZL9-8o5z1v-ak2LLt-9nwB35-9MvEiU-9yWCZZ-83TW6B-atamBp-brg3Ya-atam9k-aor7zp-brg1en-bzCXQW-P8dUr-2vXUL-mhoLcx-68EUUs-a2mvS6-9AGXkj-8gycdg-brgdMg-6dv6Fq-HLc3k-brcmiH-aTockB-aToc8k-9orD3p-aToc4H-5EmCvj-9R6BEh-9nwAZb-aiviqw-4KjEze-29N7yS-9DH2uj-9Y5sxD-oXYfep-5Tnqdp-bFD6ht-3nDm1w-3GXe6-opoHC-bskYDz Ken Hawkins, CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)

Question for the audience.

Hemlines and Big MacsLight hearted views of economic data

Sound needs to be chosenAudio can have associative qualities

Actual eating sounds are not fun to listen to. Used a really cartoon version but it was too distracting: a dragon bite seemed the best way.

Hemlines: swannee whistle reminds us of 60s films such as the Carry On or Doctor films. Use the sounds chuck/mac.ck

Try changing the samplerate.

Testing the UIHead and Torso SimulatorsEars and Users

Testing audio is hard and not a solved problem.

Work in psychoacoustics and music information retrieval to simulate how the ear works.

The best way to test is still is to recruit listeners: hallways, testing groups.

Thank you for listening

Any questions?

iain.emsley@oerc.ox.ac.ukwww.austgate.co.uk@iainemsley@minnelieder19

Acknowledgements

Oxford e-Research Centre: Wes Armour, David De Roure, Rahim Lakhoo, Kevin Page, David WeiglBodleian Centre for Digital Scholarship: Pip WillcoxSoftware Acumen

Githubhttps://github.com/iaine/UXinCity

ReferencesBull, Michael, Les Back, and David Howes, eds. The auditory culture reader. Bloomsbury Publishing, 2015.Emsley, Iain and De Roure, David. It will discourse most eloquent music: Sonifying variants of Hamlet. 2015Levitin, Daniel, This is Your Brain on Music. Atlantic, 2007.Pinch, Trevor, and Karin Bijsterveld, eds. The Oxford handbook of sound studies. Oxford University Press, 2011.Sterne, Jonathan (ed),The Sound Studies Reader. Routledge, 2012.