Human-Robot Collaboration with Robotic Assistance Confined toLocal Motion to Assure Human Safety

Shouren Huang†, Masatoshi Ishikawa†, Yuji Yamakawa‡

Abstract— Human-robot collaboration becomes more andmore promising recently. By exploring the cognitive and dex-terity capabilities of humans and the capacity of robots toproduce repetitive work and provide assistance, human-robotcollaboration combines both parties’ advantages. However, po-tential safety concerns exist in many human-robot collaborationsystems. In this report, we present our recent studies on saferobotic-assist design and development for human-robot collab-oration with the aim of extending humans’ motion accuracywhile fully preserving the cognitive and dexterity capabilitiesof humans. We focus on collaborations where human operatorwith cognitive capability implements coarse global-motion androbot with high-speed sensory feedback realizes fine localmotion. Safety for human is assured by confining the robotmotion in a local manner and reducing the physical interactionfrom robot to human.

I. INTRODUCTION

Robots are a key element to achieve manufacturing com-petitiveness, especially if they are able to collaborate withhumans in a shared workspace in the shop-floor, creating aco-working partnership. The paradigm for robot usage haschanged from an idea in which robots work with completeautonomy to a scenario in which robots collaborate withhumans. This means taking the best of each partner, byexploring the cognitive and dexterity capabilities of humansand the capacity of robots to produce repetitive work andprovide assistance.

However, when robots are acting jointly with or close tohumans, safety issue becomes a crucial aspect. In order toaddress the safety issue in human-robot collaboration, severalresearch areas are established, such as safety assessment,safety through design, and safety through planning andcontrol[1]. In this study, we focus on the safety design andimplementation as it is the most important aspect and manyefforts have been devoted in this area. For examples, in[2], safety design strategy is proposed for a production cellcovering five main safety design aspects: human area androbot area, safety robot working zones, robot safety design,operator safety monitoring system, operational sequencesafety control. Although potential collaboration risks can beeffectively reduced with the proposed safety designs, it issaid that risks are not totally being eliminated.

In this report, we present our recent studies on saferobotic-assist design and development for human-robot col-laboration. The objectives are to keep human 100% safe

† S. Huang, M. Ishikawa are with the Graduate School of InformationScience and Technology, University of Tokyo, 7-3-1 Hongo, Bunkyo-ku,Tokyo 113-8656, Japan. ‡ Y. Yamakawa is with the Institute of IndustrialScience, University of Tokyo, 4-6-1 Komaba, Meguro-ku, Tokyo 153-8505,Japan. Huang shouren@ipc.i.u-tokyo.ac.jp

even under system (e.g. sensors, controllers, mechatronics)failures and to extend humans’ motion accuracy while fullypreserving the cognitive and dexterity capabilities of humans.

II. METHODOLOGY

A. Interaction analysis and safety issue

Safety issue is of great importance for human-robot collab-oration systems. In order to address the safety issue, analysisof interaction methods in human-robot collaboration is intro-duced as shown in Fig.1. Physical interaction model (shownin Fig.1(a)) with robot physically interacting with human hasbeen adopted by the majority of commercial collaborativerobots for power-assist purpose. However, direct physicalinteraction between the robot and human poses potentialstability and safety concerns. In contrast, there is only one-way effect from human to robot for the reduced interactionmodel as shown in Fig.1(b). Human-robot interface givingfeedback to human operator is configurable and can bedesigned as non-contact. Therefore, assist behavior of robotis confined to local motion and is accordingly safe to humanoperator.

Human

Robot

Planning

Planning

Action

Action

Joint intention

robot side

human side

cognition

cognition Human

Planning Action

human side

cognition

TargetPhysical interaction

ActionRobot

robot side

Target

(a) (b)

Human-robot

interface

Fig. 1. Interaction analysis of human-robot collaboration. (a): Physicalinteraction model (e.g. collaborative industrial robots under impedancecontrol). In this model, the robot is supposed to estimate the intentionof the human operator to form a joint intention. Both human and robotrealize the global motion for a given task. However, physical interactionbetween robot and human operator is a potential risk for human safety. (b):Reduced interaction model. The robot only needs to realize local motion inan active manner and does not need the cognitive capability of understandingthe intention of human operators. Human guides the robot globally whileconsidering the limited motion range of the robot. Physical interaction fromrobot to human is reduced and thereby human safety is assured. Note thatfor both models, interactions between target and robot can be physicallycontact or non-contact according to different tasks.

(Low-bandwidth dynamics)

Coarse human motion

POSITIONING

ERROR

High-speed

sensory feedback

Compensationmodule

-

(High-bandwidth dynamics)

Fine compensation

High-speedcompensationδ

Moduleprototype

+

δ

Workpieces

・task and product variation

・easy feeding (less waiting time)

・environmental disturbance

・poor accuracy

・tremor

+

・Cognitive

Fig. 2. Proposed human-robot collaboration based on reduced interactionmodel. Human operator with cognitive capability to implement coarseglobal-motion. At the same time, fine local-motion is realized by a ready-to-use robotic module with high-speed sensory feedback.

B. Design concept of reduced interaction model (Fig.1(b))

Design concept of the reduced interaction model shownin Fig.1(b) is proposed as shown in Fig.2. The basic idea issummarized as follows:

1) Uncertainty of human motion and environmental un-certainty contribute to the accumulated positioningerror noted as δ. δ is assumed to be bounded by ∆ as|δ| ≤ ∆(∆ > 0);

2) With high-speed sensors configured in a local mannerto inspect both manipulation tool point and the ma-nipulating target at the same time, accumulated errorδ is observed and estimated as the relative error δ insensory feature space;

3) A high-speed robotic module with working rangebigger than the bound ∆ in Cartesian space realizeshigh-speed and accurate compensation with the real-time feedback information δ. Ideally, the module hasa much larger bandwidth than that of human motion.

The method above is based on a coarse-to-fine strategyinherited from previously proposed dynamic compensationapproach for industrial robots in order to realize fast andaccurate position regulation under uncertainty [3], [4]. Un-der the proposed concept, human operator is responsiblefor guiding the robot to realize coarse global motion forvarious tasks. Therefore, advantages of human cognitive areoptimally combined with the accurate motion capability ofrobots.

III. ROBOT CONFIGURATION

A. Compact high-speed vision system

A new vision chip combining high-frame-rate imagingand highly parallel signal processing with high-resolution,high-sensitivity, low-power consumption, and small chip sizeis developed [5]. Comparing with conventional high-speedvision system, the new high-speed vision system (Fig.3)based on the newly developed vision chip will greatly save

Laptop (configuration &

communication interface)

Newly developed

USB cable

high-speed visionsystem

(Processed data)

Imaging &

processingat 1000 fps

Ethernet

(to robot controller)

Microcontrollers

or

Target

Fig. 3. New high-speed vision system with high-speed imaging andprocessing integrated in the same chip.

space and energy, and is very suitable for compact usagein robotic applications. Overall latency of high-speed visualfeedback was measured to be within 3.0ms[6].

B. Robotic module for local assistance

A robotic module prototype capable of realizing finecompensation in two dimension (x, y direction) is developedas shown in Fig.2. The module is lightweight with high-speedand accurate motion ability. The module is controlled withthe feedback information from the new high-speed visionsystem, which is configured on the working table of therobotic module (see Fig.2). In order to show the advantagesof the high-speed visual feedback, the compensation mod-ule is controlled by a simple proportional-derivative (PD)method instead of developing complex control algorithms.

The module is configured at the end of a supportingmechanism formed by two frame links with three passivejoints (see Fig.4, Fig.5). Therefore, a human operator movesthe robotic module by one hand within x−y planar globallywith less physical effort.

IV. HUMAN COARSE MOTION

Since the robotic module is designed for local motionand it has limited work range, it is necessary to inform ahuman operator by the human-robot interface during his(her)implementation of coarse motion to assure a target alwaysreachable for the robotic module. In order to avoid safetyconcerns as addressed above, human-robot interface givingfeedback to human operator can be designed in a non-contact manner. Hereafter, we introduce two methods: visionperception and pneumatic haptic perception. Notice thatcontour tracing will be set as the application task hereafter.

A. Interface based on vision perception[7]

A small projector is configured to project a square area asthe indication for human operator (Fig.4(c)). Human operatormoves the robotic module along an arbitrarily placed target(here is a closed groove with 1mm width) only caring aboutthat the target is always within the projected square. Theprojected square area is aligned with the ROI region of thehigh-speed camera’s images. Therefore, reference positionsfor regulation in images are continuously calculated by the

high-speed vision, and ideally they would lie on the centerline of the groove. One sample image is shown in Fig.4(b).

Projector

Projected

Frame links

Workpiece

Passive joints

Human handModule

Electronic parts

Groove (1mm width)

x

yz

Target contour (groove)

ROI box

x

y

Moment center

Image center

(a)

(b)

(c)

12

3

100mm

160m

m

Fig. 4. Experimental setup for contour tracing by human-robot collabo-ration based on vision perception. (a): target workpiece; (b): image sampleof high-speed vision; (c): indication of projected square for human operator.

B. Interface based on pneumatic haptic perception[8]

As shown in Fig.5, for simplification, the haptic feedbackis only realized along x-direction with two nozzles (withdiameter of 1mm) that are configured around the handleto blow compressed air (0.2MPa) to the hand of humanoperator. Basically, the haptic feedback on human operatorworks as a supervisor telling the subject to guide the roboticmodule such that the target should be always kept withinits limited work range. Therefore, encoder information ofthe robot module’s joint-x can be used as the control inputfor generating the pneumatic haptic feedback. Air flow oneach nozzle is controlled with a PWM like method. Thepulse of each air blow is determined by the valve’s physicalparameters, and the air blow frequency is determined withthe encoder information of the robot module’s joint-x.

For average, tracing error under active assistance of therobotic module (with both methods) was around 3pixel inimage, which is equivalent to about 0.12mm in our exper-imental setup. Details can be referred in related works[7],[8].

V. SUMMARY

This report presents our recent studies on safe human-robot collaboration based on reduced interaction betweenrobot and human, with the aim of optimally combining thecognitive capabilities of human and accurate motion capabil-ities of robot. Under the proposed approach, human operatoris for cognitive global-motion without caring much about

Frame links

Passive joints

x

yz

1

2

3

Target

Operator

Nozzles

Valve

1000Hz vision

Actuators

Robotic module

Fig. 5. Experimental setup of the human-robot collaboration systembased on pneumatic haptic perception.

accuracy. Fine local-motion in an active manner is realizedby a robotic module based on high-speed visual feedback. Inorder to avoid safety concerns, robotic assistance is confinedto local motion and human-robot interface giving feedbackto human operator is designed in a non-contact manner.

However, the proposed method in this report is a comple-mentary to existing systems based on physical interaction,and it is not good at implementing some traditional taskssuch as power-assist applications.

REFERENCES

[1] A. Pervez and J. Ryu. Safe physical human robot interactionPast,present and future. J. Mech. Sci. Tech., 22(3):469483, 2008.

[2] J. Tan, F. Duan, R. Kato, T. Arai. Safety Strategy for HumanRobotCollaboration: Design and Development in Cellular Manufacturing.Advanced Robotics, 24:5-6:839-860, 2010.

[3] S. Huang, N. Bergstrom, Y. Yamakawa, T. Senoo, and M. Ishikawa.High-Performance Robotic contour tracing based on the DynamicCompensation. In: Proc. IEEE Int. Conf. Robotics and Automation,pp.3886-3893, 2016.

[4] S. Huang, N. Bergstrom, Y. Yamakawa, T. Senoo, and M. Ishikawa.Applying High-Speed Vision Sensing to an Industrial Robot forHigh-Performance Position Regulation under Uncertainties. Sensors,16(8:1195):1-15, 2016.

[5] T. Yamazaki, H. Katayama, S. Uehara, A. Nose, M. Kobayashi, S.Shida, M. Odahara , K. Takamiya, Y. Hisamatsu, S. Matsumoto, L.Miyashita, Y. Watanabe, T. Izawa, Y. Muramatsu, and M. Ishikawa.A 1ms High-Speed Vision Chip with 3D-Stacked 140GOPS Column-Parallel PEs for Spatio-Temporal Image Processing. In: Int. Solid-StateCircuits Conf., pp.82-83, 2017.

[6] S. Huang, K. Shinya, N. Bergstrom, Y. Yamakawa, T. Yamazaki, andM. Ishikawa. Dynamic Compensation Robot with a New High-speedVision System for Flexible Manufacturing. Int. J. Adv. Manuf. Tech.,95(9-12):4523-4533, 2018.

[7] S. Huang, M. Ishikawa, and Y. Yamakawa. Human-Robot Collabora-tion based on Dynamic Compensation: from Micro-manipulation toMacro-manipulation. In: Proc. IEEE Int. sym. on robot and humaninteractive communication, pp.603-604, 2018.

[8] S. Huang, M. Ishikawa, and Y. Yamakawa. An Active AssistantRobotic System based on High-Speed Vision and Haptic Feedbackfor Human-Robot Collaboration. To appear in The 44th AnnualConference of the IEEE Industrial Electronics Society (IECON 2018).