1. 1. Dzmitry Tsetserukou Assistant Professor Intelligent Space Robotics Laboratory NurseSim: Virtual Reality Simulator for Nurse Training with Haptic Feedback The best way to predict the future is to invent it. Steve Jobs Dzmitry Tsetserukou, Yuuki Nakagawa and Kazuhiko Terashima
2. 2. EIIRIS: Electronics-Inspired Interdisciplinary Research Institute Professor K.Terashima Professor J. Miura M.S. Y. Nakagawa M.S. S. HosokawaPhD J. Sugiyama M.S. H. Sugiura
3. 3. Introduction Background: Serious understaffing at the care person Japan needs twice amount of care person next 20 years. Its necessary to raise international human resources having nursing care skill. Language difference causes a barrier of providing a technical assistance and training. Purpose: Develop novel nurse training system with haptic technology We develop nurse training simulator with sense of force but without a language. Simulator helps nurse training and achieve shortening of training period. Reduction personal costs by using this simulator. Haptic: Tactile feedback technology which takes advantage of the sense of touch. Care person from Indonesia
4. 4. NurseSim: Nurse Training Simulator Using this haptic interface, we achieve effective and low cost training system compared to using a human manikin or a robot. It shows an optimal motion with sense of force, effective and easy to understand teaching is realized. User wearing ExoInterface Virtual Care Receiver NurseSimNursing 3D Simulator as Education System ExoInterface: Wearable Haptic Interface Haptic Interaction Virtual weight, 3D graphics Posture information 3D Glass 3D Projector IR Emitter Xtion: Motion Capture Camera
5. 5. Wearable Haptic Interface Elbow Unit Wrist Unit Fixing Plate Flexible Belt Belt Shaft DC Motor Unit Base Timing Belt Pulley Bearing Motor Holder DC Motor Bearing Connector Unit Base Motor Shaft Elbow Unit Wrist Unit We developed wearable haptic interface for an interaction in VR environment. The flexible belt is connected from upper arm to wrist, and from wrist to hand. DC motors embedded in the interface pull flexible belts, then joint torques are generated. This device reproduces the human muscle structure by generating extensor and flexor muscle torques at the users arm joint. Doesnt limit users movement and can be used all over the place.
6. 6. 0 20 40 60 80 100 120 140 160 180 0.2 0.4 0.6 0.8 1 Elbow Angle[deg] BeltLength[m] 0 20 40 60 80 100 120 140 160 180 0 0.5 1 1.5 2 2.5 Elbow Angle[deg] JointTorque[Nm] Interface Geometrical Modeling It is necessary to prepare not only devices but also geometric models of that for generating the arbitrary joint torques in human arm. From this interface model, we calculated belt length and joint torque by MATLAB. This model helps for dynamic computation or simulation in virtual environments like bringing sense of physical interaction. e eu bul eml fal wuF uwuF sin eud Upper Motor Belt u Wrist Joint Elbow JointHand Upper Arm e Elbow Unit Geometry Model eufa bueuemfa wufaeu bl ldll Fl 2 cos sin 2222 1 Elbow Joint Torque is Belt Length and Joint Torque
7. 7. AR Application Experiment D/AA/D USB PC Sensor Circuit Motor Driver Motor Flex Sensor WireWire Camera HMD VGA Wire AR Experiment Configuration We designed an application using AR technology. The user wears a haptic interface and HMD with a camera. When camera recognizes AR marker, virtual object appears on it. When a virtual ball touches surface of the hand, the interface reproduces force feedback to elbow and wrist. AR (Augmented Reality): Technology which combines real world and virtual world, and shows its mixed image to the user.
8. 8. Dynamics Calculation Experiment Human Model Foot Lower Leg Upper Leg Waist Upper Body Hand Lower Arm Upper Arm Head To create virtual nurse training simulator, we developed the dynamics calculation space and 3D human body model with ODE. Using ODE, accurate dynamics calculation and good visualization are implemented. The model has 9 DOF in its arm joint and waist and its controlled by the users posture. Xtion ScreenOpenCV OpenNI Angle Calculation Algorithm P control RGB, Depth(m) Joint Point(x,y,z) Joint Angle(rad) Angular Velocity (rad/s) Joint Torque(Nm) Current(A) Virtual Object Load(N) Human Model ODE Space Interface Model Interface RGB ODE System Block Diagram ODE(Open Dynamics Engine): Open source physical engine which is used for simulating the dynamic interactions between bodies in space.
9. 9. Lifting Rigid Box and Multijoint Object The user can transform posture to optimal joint angle referring to ODEs visual information. We can design a care receiver body model with the same method and realize interaction between human and human.
10. 10. NurseSim Wrist Elbow Whole Body The high precision tracking by optical motion capture Nurse care scenario can be changed easily in Unity Learning by the force feedback interface Effective nurse training can be realized PC1 PC2 Camera Hub Display Motor Driver Trainer with Markers Skeleton data Voltage Current OptiTrack Camera Image Force Feedback Image Interface Motors
11. 11. Psychophysical experiment Experimental results y = d - a- d( ) 1+ x / c( ) b a = -0.0020187 b = 3.05279 c = 4.30725 d = 1.05594 a = -0.0020187 b = 3.05279 c = 4.30725 d = 1.05594 a = 0.046811 b = 4.69180 c = 4.48232 d = 0.97842 Elbow unit only Wrist unit only Both units Sigmoid Curve (4-parameter logistic curve) Elbow Unit Wrist Unit Both Units Upper DL [g] 135.66 139.26 133.92 Lower DL [g] 78.86 94.78 88.28 0 0.25 0.5 0.75 1 0 20 40 60 80 100 Rateof"heavy"response Weight [g] Rate of heavy Sigmoid Curve 20 60 100 140 180 220 0 0.25 0.5 0.75 1 0 20 40 60 80 100 Rateof"heavy"response Weight [g] Rate of heavy Sigmoid Curve 20 60 100 140 180 220 0 0.25 0.5 0.75 1 0 20 40 60 80 100 Rateof"heavy"response weight [g] Rate of heavy Sigmoid curve 20 60 100 140 180 220
12. 12. AR experiment comparing the same size ball Experiment outline Left hand weight Right hand weight Content Virtual ball bouncing to the left and right hand Question Which weight: in left hand or right hand is heavier? Answers Left weight is heavy/ Slightly heavier / Same / Right is heavy / Right is heavy Number of repetitions 25 times Simultaneous comparison Experimental conditions Haptic interfaces at wrist and elbow joints HMDwearable web camera EarplugsTo prevent noise from actuators Left and right hand are bent at 90 degree Weight of ball 5 stages(80~160g)random order
13. 13. AR: experiment comparing the same size of ball Experimental results LeftHandWeight Right Hand Weight 80g 100g 120g 140g 160g 80g 3.2 4.8 2.2 5.0 4.6 100g 2.8 4.0 4.6 4.6 5.0 120g 1.4 3.4 3.2 4.2 3.8 140g 1.4 2.6 3.4 4.0 4.0 160g 1.0 2.2 3.0 2.2 2.8 LeftHandWeight Right Hand Weight 80g 100g 120g 140g 160g 80g 0.95 0.80 0.55 1.00 0.90 100g 0.80 0.75 0.85 0.85 1.00 120g 0.85 0.65 0.95 0.95 0.95 140g 0.90 0.85 0.65 0.75 1.00 160g 1.00 0.70 0.75 0.95 0.95 LeftHandWeight Right Hand Weight 80g 100g 120g 140g 160g 80g 3 4 4 5 5 100g 2 3 4 4 5 120g 2 2 3 4 4 140g 1 2 2 3 4 160g 1 1 2 2 3 Percentage of correct answers Overall percentage of correct answers 85 Correct value, n Experimental value, m p =1- m-n 4
14. 14. NurseSim3D
15. 15. Best Student Presentation Award, Oct. 24-25, Irago 2013 Awards Student Excellence Award, March 6, 2014 Best paper Award, April 7, 2010
16. 16. LinkTouch3D: a Novel Wearable Haptic Display with Parallel Mechanism for Presentation of Three-DOF Tactile Feedback at the Fingerpad
17. 17. ! Thank you for your attention!