Slide 1

Slide 2

Tools for Programming in the Real World Self Introduction for MSRA Fellowship 2012 Jun Kato, The University of Tokyo 1

Slide 3

Jun Kato 2 nd - year PhD candidate Igarashi Laboratory, The University of Tokyo Concentration: Human-Computer Interaction Especially, Tools for Programming in the Real World 2

Slide 4

Agenda My motivation for the research Introduction of the projects Ideas for new projects 3

Slide 5

MY MOTIVATION FOR THE RESEARCH Tools for Programming in the Real World 4

Slide 6

CRISTAL [SIGGRAPH09 E-Tech] Tools for Programming in the Real World Multi-touch [CHI09 SRC] Roboko [ CHI13 (to be submitted)] Sharedo [Ro-MAN12 (submitted)] Phybots [DIS12] Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] DejaVu [UIST12 (to appear)] 5

Slide 7

Sharedo [Ro-MAN12 (submitted)] Roboko [ CHI13 (to be submitted)] DejaVu [UIST12 (to appear)] Phybots [DIS12] CRISTAL [SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] Phybots [DIS12] Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] CRISTAL [SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] In the Real World From Robot to Human 6

Slide 8

Sharedo [Ro-MAN12 (submitted)] Roboko [ CHI13 (to be submitted)] Phybots [DIS12] DejaVu [UIST12 (to appear)] CRISTAL [SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] Roboko [ CHI13 (to be submitted)] Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] DejaVu [UIST12 (to appear)] In the Real World From Robot to Human 7

Slide 9

Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] Roboko [ CHI13 (to be submitted)] Phybots [DIS12] DejaVu [UIST12 (to appear)] Multi-touch [CHI09 SRC] Roboko [ CHI13 (to be submitted)] Phybots [DIS12] DejaVu [UIST12 (to appear)] CRISTAL [SIGGRAPH09 E-Tech] Multi-touch [CHI09 SRC] Sharedo [Ro-MAN12 (submitted)] Tools for Programming From Keyboard to IDE 8

Slide 10

Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] Roboko [ CHI13 (to be submitted)] Phybots [DIS12] DejaVu [UIST12 (to appear)] Multi-touch [CHI09 SRC] Surfboard [UIST10 Demo] IntelliViewer [UIST10 SIC] Pressing [UIST09 SIC] CRISTAL [SIGGRAPH09 E-Tech] Sharedo [Ro-MAN12 (submitted)] Tools for Programming From Keyboard to IDE 9

Slide 11

INTRODUCTION OF THE (SELECTED) PROJECTS Tools for Programming in the Real World 10

Slide 12

Projects Phybots DejaVu Roboko 11

Slide 13

Motivation Add mobility to our daily objects 12

Slide 14

Gulf of HCI and Robotics 13 Physical UIs Toolkit for: Robots HCI researchers & Interaction designers Target users: Robotics people Prototyping Focus: Reliability Low-level&Static Software API: High-level&Extensible Small&Cheap Hardware: Phidget Kit, $200 Medium-Large&Expensive K-Junior, $938

Slide 15

Goal of Phybots Add mobility to physical objects In a cheap and easy way Through high-level and extendible API With support for the whole prototyping process 14

Slide 16

Prototyping with Phybots 15 Hardware construction Programming with built-in API Test with runtime debug tool (Extend API when needed)

Slide 17

Localization and Locomotion API Hardware setup: easy and cheap Camera PC or Mac Robotic things 16 From our user study: Miniature drive recorder

Slide 18

Localization and Locomotion API Navigation by global coordinates Move Push TracePath 17 Task move = new Move( mouseX, mouseY); move.assign(robot); move.start(); From our user study: Beach flags with obstacles Y-axis X-axis

Slide 19

Localization and Locomotion API Vector field navigation Easy design of new behavior ex) Follow another robot Combination of existing fields ex) Move + collision avoidance 18

Slide 20

Runtime Debug Tool 19 Entity MonitorService MonitorWorkflow Monitor

Slide 21

User Studies Alpha version deployment To graduate students Current version deployment To HCI students To robotics students 20

Slide 22

Lessons learned: Mere mobility was not enough. Most apps only used a single API call. Parameter configuration was painful. Alpha version deployment 21 Lessons learned: Extendibility is important. Higher-level task management is desired. Support for testing phase is needed.

Slide 23

Current version deployment 22

Slide 24

Phybots A toolkit for prototyping robotic things Localization and Locomotion API Extensible Architecture Runtime Debug Tool Open-source software available at http://phybots.comhttp://phybots.com 23

Slide 25

Projects Phybots DejaVu Roboko 24

Slide 26

Motivation Input by the users hands, body, or physical objects Captured by cameras Processed in real-time Support for computer-vision- based programming is needed Wang, 2006 Wilson, 2005 Cao, 2003 25

Slide 27

Existing approaches Hardware, Software API Becomes more and more accessible Webcam, Kinect, OpenCV, etc. Development Environments Do not provide explicit support Ex) Breakpoint-based variable monitoring OpenCV Visual Studio Eclipse 26

Slide 28

DejaVu 27

Slide 29

DejaVu Canvas Frame data visualization Freehand sketch 28

Slide 30

DejaVu Canvas Capable of showing variable values at the different positions in the code 29

Slide 31

DejaVu Timeline 30

Slide 32

Conclusion DejaVu provides integrated support for the development of interactive camera-based programs that tightly matches their nature: Visual/Continuous/Non-reproducible the workflow: Record input/Change code/Replay CanvasWorkflow 31

Slide 33

Projects Phybots DejaVu Roboko 32

Slide 34

Programming robot actions is difficult. 33 DanceClearing dishesCooking Motivation:

Slide 35

Text-based Programming joint[0].setAngle(0.6); joint[1].setAngle(0.7); joint[2].setAngle(-0.24); arm.setPosition(4.3, 2.4); robot1.setPose(pose32); 34 Problem: We cannot imagine the actual pose of the robot Typical approach:

Slide 36

Visual Editing 35 Problem: We cannot design flexible robot behavior such as repeat this for 3 times when Typical approach:

Slide 37

Overview of the IDE 36

Slide 38

Workflow of the Programmer 37 1. Capture2. Drag and Drop Camera RobotPC

Slide 39

Getting and setting pose 38 Supported APIs:

Slide 40

Execute a sequential action 39 Supported APIs:

Slide 41

Not only a robot but a human (Kinect) 40 Project outcome: new application This work is an outcome of collaboration with a MSR researcher, Xiang Cao

Slide 42

Integrating visual representations into text-based programming environment 41

Slide 43

IDEAS FOR NEW PROJECTS Tools for Programming in the Real World 42

Slide 44

Ideas for New Projects (CENSORED) 43

Slide 45

Thank you for listening Ive designed various tools for programming in the real world. I am looking forward to further collaboration with MSRA researchers. 44