FROM WIMP AND SILK TO SMILEExploring emerging interaction technologies

Today’s topics• Understanding the rapid change of technology

• Engaging in the Jurassic Park Test

• Exploring some of the many new (=current) ways to interact with technology

The world changes fast• Creative destruction (Schumpeter)

• One economic development arises out of the destruction of an older

• This goes for technology too• How many buy music? (CDs)• How many rent a video? (in a store)

• We are moving from “hardware” (CD) to “software” (iTunes) to “services” (Spotify for music, Netflix for movies)

• My first USB 2.0 memory stick was 128 MB• You can barely store more than the sound of “air” in good quality there

The web changes fast• From being

• World Wide Web• You sit by a home computer/laptop and browse web pages, using

your keyboard and a mouse – or through your mobile

• …we now have• A Web-Wired World

• Where you always are connected• Where you interact in many new ways• Where you (and your phone) constantly transmit information

• Fine, there are changes…• How can we exploit then for building better/more modern systems?

Before we SMILE• Let us have a look at some technology changes

• From “fatscreen” to “flatscreen” (change h/w <-> h/w)• From “local storage” to “cloud storage” (change h/w <-> service)• From “applications” (office, email) to “services (s/w <-> services)

• Office 365, Google Docs, Dropbox

• Changes come fast, and old technology is forgotten

• READY TO DO THE JURASSIC PARK TEST?

Jurassic Park test

• Which of you know what happens when pressing this?• Which of you know what this symbol represents?

• Those of you with your hands up, kindly pick up your SENIOR CITIZEN CARD when you leave this room...

• ”John”, 13”...and then you have this symbol for

saving. I don’t really know what it is but

it is sort of square, with a somewhat

bigger white area in the upper half...”

More predictions

• Remember this!• (Dialogue in 15 years)- Daddy, have you ever used

a keyboard?- Sure I have!- Why did you do that...?

• This is the next generation, after you (watch out):

• Anna, 2 years finds a glossy magazine. She immediately tries to ”change page” by swiping…without success.

• She hands over the magazine to her mother, with eyes saying ”this device is not working…”

Changes in paradigms• The 1980’s: WIMP

• Windows – Icons – Menus – Pointers (or point-and-click)• We have lived by this paradigm for 35 YEARS!!!

• The mid 1990’s: SILK (Reddy)• Envisioning what would lie ahead• Speech – Image – Language – Knowledge• We have, in part, seen S-I-L, but don’t see K (back-end)

• Now: we must include all the new ways of interaction which are possible with mobile phones, tablets, consoles• You should start to SMILE

SMILE• In an effort to cover most of the “modern” – and by all

means, also traditional – modes of interaction we coined• Speech

• Speaking to technology, and be spoken to by it

• Movement• Mouse…ok, now swiping but also hands (gestures)

• Image• Not only “photos”: “live” avatars, QR codes, face recognition

• Language• Translations, and also “emotional” language

• Environment• You, and your devices, interact…constantly

SPEECH

• Speech recognition: speak to your phone (SIRI; not very new)• Speech generation: TTS – text to speech – units speak to you• Emotions are also a language (MIT – Kismet – 1997)• http://www.youtube.com/watch?v=8KRZX5KL4fA

• WAMI – speak to the Internet• http://groups.csail.mit.edu/sls/research/vehicle.shtml • Kinect – SpeechRecognitionEngine• http://www.youtube.com/watch?v=9lePgTpjRto

• Experiments (Spring 2013): Stroop effect (WAMI och Kinect)• Classic psychology experiment: say the COLOR of the TEXT• BLACK BLUE REDBLUE RED GREEN BLACK BLUE BLUE BLACK

Motion/movement• Catch movements in 2D, 3D- Move a mouse pointer, swipe a tablet- New ways to communicate (mostly 3D)

- Sensor fixed, object moving (Kinect)- Sensor moving, object fixed (”scanning”, e.g. with Kinect)- Major movements (as GPS in mobile) is dealt with later

• Experiments (2013)• Connect the dots• Kinect Fusion

Image

• Basicly: image analysis (detect patterns)• Image filtering (blue-screen)• Combine with scanning (interpret ”the surrounding”)• Beyond ”flat” image: 3D/holograms, as here (but not really…)• DVE: http://www.youtube.com/watch?v=Wl9og1O3onQ • Miku Hatsune: http://

www.youtube.com/watch?v=dhYaX01NOfA

• Experiment (2013/14)• ”The fake postcard”

Language

• Audio analysis• As speech, but for a different purpose• Also covers intonation and ”body language”• http://www.nextup.com/acapela.html • Tomorrow’s Google translate• http://www.youtube.com/watch?v=QFL0_LsZiYQ

• For all you programmers: Google API (set language/text)• http://translate.google.com/translate_tts?tl=en&q=hello+computer+experts

• Experiments (2013/2014)• ”Oh dear” – measure pulse, comfort

Environment

• Which means ”everything around us...”• In the new WWW we are transponders which constantly

and without realising it (?) interact with our surrounding• Mobiles/GPS/Geo-location is only one, among many,

exemples• http://www.pathintelligence.com/• ”Smart” clothes is another

• Smartphones have geo-location, but not Kinect• Connecting to WiFi brings the node’s position

Experiment (2013/2014)

Live battlegrounds (3-5 meters accuracy indoors, 20 m outdoors)

http://www.navizon.com/indoors-solutions

http://www.pacmanhattan.com/

WHAT? Not the end?

• Beyond SMILE we also have AH (as in AH, SMILE!)• Data forms also change (more than e.g. audio)• We are used to graphics, sounds, printouts• A as in ARTIFACT (”things”)

• 3D printers are as low priced as 400 EUR (and up)• http://www.creativetools.se/hardvara/3d-skrivare?sort=p.price&order=ASC

• Are you cold my son? Let me print you a sweater...• We have been dating for three months. I printed these flowers to

you but they had to be blue; I ran out of red ink…

• H as in HAPTIC (touch)• Anyone who remembers FORCE FEEDBACK games?• Mobile phones that VIBRATES are also feedback

Why is this important?• Ask yourselves:

• In all the time you spend using the Internet during an average day and night, how much is spent in front of a computer screen (as compared to using mobile phones/tablets)?

• So, how do you think “tomorrow’s consumers” (you and younger generations) will interact with the WWW?

• Then why concentrate on developing traditional websites?

Class discussion• Form groups of 3-5 students• Discuss a new idea (“mental prototype”) which will build

on one or more of the SMILE technologies

• Who is if for?• Where and how will it be used?• What is the value for the user?

• You have 15 minutes

Summary• New technology is pushing, our demand pulling

• WIMP and SILK will still be around, but we (YOU!) must also be ready to AH, SMILE as means for interaction

• The WWW is turning into WWW

• Many applications will (should) be Mobile first