using vision sensors for innovative hci
TRANSCRIPT
![Page 1: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/1.jpg)
USING VISION SENSORS FORINNOVATIVE HCI
- CHALLENGES FOR THE NEXT INTERACTIVE MEDIA -
Hideki Koike Tokyo Institute of Technology
![Page 2: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/2.jpg)
OUTLINE
➤ Vision-based HCI
➤ Challenges for the next interactive media
➤ Environment: Interactive Surface on the Water
➤ Contents: Spectating Ball Sports from Ball’s POV
➤ Advertisement: Unaware Gaze Navigation
➤ Lessons learned from our prototype development
![Page 3: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/3.jpg)
VISION-BASED HCI
![Page 4: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/4.jpg)
NEW I/O DEVICES HAVE BEEN CHANGING HCI
IBM PC Macintosh DataLand
CUI GUI PUI
I: keyboard
O: character display
I: keyboard, mouse
O: bitmap display
I: gesture recognition/ speech recognition
O: large display/wearable
![Page 5: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/5.jpg)
ADVANCES IN COMPUTER VISION
➤ Hardware:
➤ Camera: smaller, higher resolution, and less expensive.
➤ GPU: faster and less expensive.
➤ Software:
➤ Open software (e.g. OpenCV, AR Toolkit)
wired unwired
![Page 6: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/6.jpg)
SPECIAL USE CAMERAS
➤ Example:
➤ far/near infrared, polarization, high-speed, depth, mocap, action cam, eye tracking, light field
➤ Can see what human cannot
➤ Enable natural and intuitive UI
far infrared camera polarization camera depth camera
![Page 7: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/7.jpg)
CHALLENGES FOR THE NEXT INTERACTIVE MEDIA
![Page 8: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/8.jpg)
AQUATOP DISPLAY: INTERACTIVE SURFACE ON THE WATER
![Page 9: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/9.jpg)
![Page 10: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/10.jpg)
![Page 11: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/11.jpg)
PROTOTYPE
![Page 12: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/12.jpg)
WATER SPECIFIC GESTURES
Poking Stroking
Scooping Hitting
![Page 13: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/13.jpg)
WATER SPECIFIC GESTURES
poking as selecting
stroking as swiping
scooping as dragging
![Page 14: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/14.jpg)
COMPARING TO TRADITIONAL UI
selection page scroll transition exit app.
PC’s GUI mouse click slider click&drag win. close
button
Touch UI touch swipe touch&drag home button
AquaTop poke stroke scoop hit
![Page 15: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/15.jpg)
IMMERSION BY FLUID SIMULATION
![Page 16: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/16.jpg)
AQUATOP DISPLAY [2012]
![Page 17: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/17.jpg)
EVALUATION: VISIBILITY
0.26 g/l
1.10 g/l
![Page 18: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/18.jpg)
EVALUATION: POINTING ACCURACY
poking touching
![Page 19: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/19.jpg)
AWARD
➤ Laval Virtual 2013: Grand Prix
➤ METI Innovative Technologies 2013: Special Award (Culture)
➤ TV Tokyo World Business Satellite 2013: Toretama Award & Kickstarter Award
➤ and more.
![Page 20: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/20.jpg)
BALLCAM!: SPECTATING BALL SPORTS FROM
BALL’S POV
![Page 21: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/21.jpg)
BALL’S POV
➤ 360 degree omnidirectional video streaming
➤ Ball’s Point of View
time
![Page 22: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/22.jpg)
PROTOTYPE
➤ What is difficult?
➤ The ball is spinning, sometimes very fast (> 600rpm)!
➤ How to stabilize the video from spinning ball?
first prototype: The iBall [2011]
second prototype
![Page 23: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/23.jpg)
BALLCAM! [2013]
![Page 24: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/24.jpg)
OVERVIEW
Image Subsampling
View Expansiontriplet image sixlet image
Motion Interpolation
Distortion Correction
![Page 25: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/25.jpg)
STEP 1: CORRECTING LENS DISTORTION AND ROLLING SHUTTER
P : back-projection
K: calibration matrixR(t): camera rotation
Gauss-Newton method
![Page 26: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/26.jpg)
STEP 2: SELECTING IMAGES USING INTENSITY
![Page 27: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/27.jpg)
STEP 3: EXTENDING VIEWS USING MULTIPLE FRAMES
frame t+1
frame t+2
triplet image N
sixlet image
triplet image N+1
frame t
![Page 28: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/28.jpg)
STEP 4: IMAGE INTERPOLATION
imageinterpolation
Mf: forward mapping
Mb: backward mapping
MI: Identity mapping
![Page 29: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/29.jpg)
BALLCAM! [2013]
![Page 30: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/30.jpg)
OUR GOAL
➤ Rugby World Cup in Japan [2019]
➤ Tokyo Olympic [2020]
➤ World Cup Soccer [2022]
20162011 2012 2013 2014 2015
![Page 31: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/31.jpg)
UNAWARE GAZE NAVIGATION
![Page 32: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/32.jpg)
![Page 33: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/33.jpg)
EXPERIMENTS
![Page 34: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/34.jpg)
EXPERIMENT 1
no blur(σ=0) σ=5
experiment image
heat map
![Page 35: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/35.jpg)
EXPERIMENT 1
no blur(σ=0) σ=5
experiment image
heat map
![Page 36: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/36.jpg)
EXPERIMENT 1
blur strength and the time to fixation
![Page 37: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/37.jpg)
EXPERIMENT 2
![Page 38: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/38.jpg)
EXPERIMENT 2
![Page 39: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/39.jpg)
USAGE SCENARIO
![Page 40: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/40.jpg)
LESSONS LEARNED FROM OUR PROTOTYPE DEVELOPMENT
![Page 41: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/41.jpg)
HOW IDEAS BECOME TO A UNIQUE PRODUCT
new(?) idea
rapid prototyping
advanced implementation
unique product
![Page 42: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/42.jpg)
NEW IDEA(?)
➤ In most cases, it is not.
➤ Survey other work carefully.
➤ OK. It seems to be a new idea!
➤ However, 10,000 people in the world are inspiring the same idea at this moment.
![Page 43: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/43.jpg)
RAPID PROTOTYPING
➤ Let’s make a prototype ASAP.
➤ The idea may be not so interesting as you thought.
➤ The idea may be more interesting than you thought.
➤ OK. It seems to be interesting!
➤ However, 100 people in the world are making the same prototype at this moment.
![Page 44: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/44.jpg)
ADVANCED IMPLEMENTATION
➤ Let’s do serious implementation.
➤ Work harder.
➤ However, you may face very difficult technical issues.
➤ Use the wisdom of predecessors.
➤ You may be only one who finally develop the system!
![Page 45: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/45.jpg)
CONCLUSION
➤ Vision-based HCI is a key to the next HCI.
➤ Effective use of special use cameras enables natural and intuitive interaction.
➤ Challenges for the next interactive media:
➤ Environment, Contents, and Advertisement.
➤ Rapid prototyping and advanced implementation make your ideas unique.
![Page 46: USING VISION SENSORS FOR INNOVATIVE HCI](https://reader031.vdocuments.net/reader031/viewer/2022021920/58ce9b601a28ab8c3b8b76b5/html5/thumbnails/46.jpg)
REFERENCES➤ AquaTop Display
➤ Y. Matoba et.al.: “AquaTop Display: A True Immersive Water Display System”, Proc. on ACM SIGGRAPH 2013, Emerging Technologies, Article 4, 2013.
➤ H. Koike et.al.: “AquaTop Display: Interactive Water Surface for Viewing and Manipulating Information in a Bathroom”, ACM Int’l Conf. on Interactive Tabletops and Surfaces (ITS’13), pp.155-164, 2013.
➤ BallCam
➤ R.Funakoshi et.al.: “Activity-Aware Video Stabilization for BallCam”, Proc. on ACM User Interface Software and Technology (UIST’16 Adjunct), pp.197-198, 2016.
➤ R. Funakoshi et.al.: “Synthesizing Pseudo Straight View form A Spinning Camera Ball”, 7th Augmented Human Int’l Conf. (AH’16), pp.30:1-30:4, 2016.
➤ K. Horita et.al.: “Experiencing the Ball’s POV for Ballistic Sports”, Proc. of the 4th Augmented Human Int’l Conf. (AH’13), pp.128-133, 2013.
➤ Unaware Blur
➤ H. Hata et.al.: “Visual Guidance with Unnoticed Blur Effect”, Int’l Working Conf. on Advanced Visual Interfaces (AVI2016), pp.28-35, 2016.