human haptic

8/13/2019 Human Haptic

1/13

IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 4, JULY 2013 385

Design and Evaluation of a HapticComputer-Assistant for Telemanipulation Tasks

Nikolay Stefanov, Carolina Passenberg, Angelika Peer, and Martin Buss

Abstract This paper introduces a computer-assisted teleoper-ation system, where the control over the teleoperator is sharedbetween a human operator and computer assistance in order toimprove the overall task performance. Two units, an action recog-nition and an assistance unit are introduced to provide context-specic assistance. The action recognition unit can evaluate hapticdata, handle high sampling rates, and deal with human behaviorchanges caused by the actived haptic assistance. Repairing of abroken hard drive is selected as scenario and three different task-specic assistance functions are designed. The overall computer-assisted teleoperation system is evaluated in two steps: rst, theperformance of the action recognition unit is evaluated and then,the performance of the integrated computer-assisted teleopera-tion system is compared with an unassisted system by means of auser study with 15 participants. Overall action recognition ratesof about 65% are achieved. Multivariate paired comparisons showthat the computer-assisted teleoperation system signicantly re-duces the human effort and damage possibility compared with ateleoperation system without assistance.

Index Terms HMMs, human-machine interaction, intelligentdesign assistants, telerobotics.

I. INTRODUCTION

M ULTIMODAL teleoperation systems allow a human op-erator to perform complex manipulations in a remoteen-vironment by overcoming barriers like distance and scaling. Ina multimodal teleoperation system, a human operator interactswith a humansystem interface to command a remotely locatedteleoperator that in his/her place interacts with the remote en-vironment; see Fig. 1. Visual, auditory, and haptic informationthat are captured at the remote site are fed back to the humanoperator to increase his/her feeling of telepresence. Typical ap-plication elds range from maintenance tasks in space or underwater [1], [2] to minimally invasive surgery [3].

Compared with direct manipulation, teleoperated manipula-tion tasks still take 10100 times longer and are often veryexhausting as operators are overwhelmed by the number of

Manuscript received September 14, 2011; revised October 20, 2012; ac-cepted January 12, 2013. Date of current version June 25, 2013. This work wassupported in part by the German Research Foundation (DFG) within the col-laborative research center SFB453 High-FidelityTelepresence and Teleactionand in part by the Institute of Advanced Studies of the Technische Universit atMunchen. This paper was recommended by Associate Editor Y. Lin of the for-mer IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems andHumans (2011 Impact Factor: 2.123).

The authors are with the Institute of Automatic Control Engineering,Technische Universitat Munchen, M unchen D-80290, Germany (e-mail:[email protected]; [email protected]; [email protected];[email protected]).

Color versions of one or more of the gures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identier 10.1109/TSMC.2013.2257743

Fig. 1. Teleoperation system: Solid lines show signal exchange in a classical,bilateral teleoperation system, while dashed lines show necessary extensionsto form a computer-assisted teleoperation system. Haptic signals can be eitherforce or motion depending on implemented teleoperation architecture.

provided feedback channels and devices to be operated. Whileclassical teleoperation controllers tradeoff robust stability andtransparency [4], this study focuses on improving task perfor-mance by developing a computer-assisted teleoperation sys-tem that provides task-dependent haptic assistances. The de-veloped computer-assisted teleoperation system combines anaction recognition and an assistance unit; see Fig. 1. In contrastwithstate-of-the-art implementationsour action recognitionunitis not limited to actions performed in the free space, but also al-

lows contact with the remote environment. The unit can handlehigh sampling rates as encountered when analyzing haptic dataandcandeal with human behavior changes causedby theactivedhaptic assistance. The performance of the action recognitionunit and implemented assistances are evaluated. Additionally,in order to show the benets of the overall developed computer-assisted teleoperation system, its performance is compared withunassisted teleoperation.

II. RELATED WORK

As shown in Fig. 1, a computer-assisted teleoperation sys-tem consists of two interacting units assistance and action

recognition. In the following sections, we rst provide a liter-ature overview for each unit separately, and then report aboutknown, state-of-the-art computer-assisted teleoperation systemsthat combine both units.

A. Assistance Unit

An overview of possible implementations of haptic assis-tances is given in [4] and [5] with special focus on applicationslike teleoperation and haptic humanrobot interaction. The lit-erature identies signicant performance improvements of as-sisted systems over those with xed or no assistance [6][8].In this study, we mainly focus on three state-of-the-art cate-

gories of assistance functions: variable impedance control [9],2168-2291/$31.00 2013 IEEE


2/13

386 IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, VOL. 43, NO. 4, JULY 2013

velocity, and force mappings [10][13] as well as potentialelds [14], [15].

The objective for adapting the impedance control parame-ters is to provide a high degree of delity while guaranteeingstability. This can be achieved by varying the damping in theimpedance controller depending on the encountered remote en-vironment as proposed in [9].

Velocity and force mappings between master and slave sitesare used to either improve task completion times (TCTs) [10]or to facilitate ne manipulation tasks [12], [13].

Potential elds are virtual haptic overlays that facilitate oper-ations and, thus, increase the task performance in a remote en-vironment. Attractive potential elds guide the operator towardgoal regions, while repulsive force elds hinder the operatorfrom entering the forbidden regions.

In this study, we will employ the aforementioned types of haptic assistances in a context specic manner.

B. Action Recognition Unit Applications for action recognition are manifold and range

from computer-assisted teleoperation over humanrobot inter-action to assisted-motor skill learning. Next, we provide anoverview on action recognition from motion and force data.

Hannaford and Lee [16] adopted a hidden Markov model(HMM) and analyzed force signals to determine phases of atelemanipulated peg-in-hole task. Theyadopted the well-knownViterbi algorithm to segment the task and to identify human ac-tions like moving, tapping, inserting, and extracting, but did notintegrate their algorithm into a computer-assisted teleoperationsystem. Similarly, Kulic et al. [17] employed an HMM to an-

alyze motion capture data to recognize predetermined humanactions with the nal aim of replicating them on a humanoidrobot.

Takeda et al. [18] investigated a partner dance scenario anddeveloped a human step estimator that is based on the analysisof haptic data. Features that are extracted from time series datawere used to build observation sequences that were passed toa continuous HMM for determining the currently performeddance step and predicting the next most likely one.

Computer-assisted manipulation tasks were investigated intwo similar approaches by Li and Okamura [6] and Yu et al. [7].Haptic assistances in the form of virtual xtures and potentialelds helped the operator performing a manipulation task invirtual reality.

The action recognition algorithm in [7] uses velocity signalsand distinguishes between path following, aligning target, andavoiding obstacles actions, while the algorithm in [6] uses bothforce andmotiondata to recognizepath following andavoidanceactions.

In [19], authors studied adaptive virtual xtures in a real tele-manipulation task. Motion capture data were analyzed and K-means clustering, HMMs, and support vector machines wereemployed for state generation, state sequence analysis, andstate probability estimation. In [20], the same authors extendedtheir motion intention estimation algorithm to layered hidden

Markov models. Motion data were represented on two levels of

Fig. 2. Scenario consisting of four actions: Idle, constrained transportation,constrained positioning, and carrying.

abstractiona lower layer called the gestem layer and a higherone called the task layer. The gestem layer was used to extractmotion primitives dealing with the data records directly, whilethe task layer was used to recognize the currently performedtask.

Although action and intention recognition have been studied,the application of the aforementioned algorithms to computer-assisted teleoperation involving free space and contact is notstraight-forward. In computer-assisted teleoperation, human be-havior is affected by both, interaction with the environment andthe behavior of the assistant. Most algorithms consider free-space motion or analyze motion data only. Also, none of theaforementioned studies on computer-assisted manipulation takethe inuence of theassistance algorithmon human behavior intoaccount. In contrast, this study considers both, the inuence of the environment by exploiting haptic information in the form of

force and motion data, and the inuence of the assistance alongwith the resulting changes in human behavior.

C. Computer-Assisted Teleoperation

The general idea of assisting a human operator while per-forming manipulation tasks by rst recognizing the currentlyperformed action and thenproviding task-specic assistance hasbeen investigated in [6][8], [19]. Studies are mainly based onHMMs for action recognition and use virtual xtures as hapticassistance. Constrained manipulation, however, is consideredbynone of the aforementioned implementations and most of themhave been tested in virtual environments only, see, e.g., [6][8].This not only facilitates action recognition, but also the imple-mentation of assistances. The computer-assisted teleoperationsystem that is presented in our study includes a real teleoperatorand communication channel and can handle free space as wellas constrained manipulation tasks.

III. SCENARIO

Thescenario used to evaluateour approach is the replacementof a defective hard drive. It is used as it represents the generalclass of remote maintenance tasks. The scenario requires thesequential execution of four different actions (see Fig. 2): con-

strained transportation (CT), constrained positioning, carrying


3/13

STEFANOV et al. : DESIGN AND EVALUATION OF A HAPTIC COMPUTER-ASSISTANT FOR TELEMANIPULATION TASKS 387

Fig. 3. Action recognition unit consisting of feature extraction, classication,and threshold adaptation.

(CA), and idle. These actions can be observed during typicalmaintenance tasks.

At the beginning, the operator has to pull the broken harddrive out of the rack. This is associated with CT . This actioncan be described by large-scale movements in a constrainedenvironment, which is characterized by large sliding friction.Then, the operator has to carry the broken hard drive to theexchange place, where the new hard drive is positioned. Next,(s)he has to take the new hard drive and to carry it back to the

rack. This action consisting of large-scale movements in a freespace is called CA. Finally, the operator plugs the hard drive intothe hole corresponding to another action of CT , and adjusts thehard drive on the target position, called constrained positioning .The main physical effects the operator has to deal with in thisphase are slip and stick effects that are caused by the frictionbetween the object and rack. Any state in which the object isnot grasped is considered idle.

IV. ACTION RECOGNITION UNIT

Action recognition is a process of representing signals at ahigher level of abstraction.Due to thenondeterministic behavior

of humans, stochastic classiers have proved themselves as ap-propriate tools. Many stochastic classiers can efciently dealwith time series data. In our approach, we use a discrete hiddenMarkov model (HMM), a simple belief network.

HMMs have been successfully applied to problems in do-mains such as speech recognition [21], [22], handwriting [23],and bioinformatics [24]. In robotics and especially in physicalhumanrobot interaction, HMMs are used as time-series pat-tern classiers representing different sensory information on ahigher level of abstraction. Typical scenarios are, e.g., dancingrobots [18], body motion imitation [25], and motion intentionrecognition [26].

In our study, we use discrete HMMs due to their high robust-ness against time warping of the input signal [27]. A discreteHMMuses thevector quantization-basedmethod for computingthe state probabilities [28]. Thus, the input signal needs to berepresented by a sequence of symbols, which can be evaluatedby the classier. Such a conversion is called feature extraction.Consequently, the overall process of action recognition consistsof the following two steps: feature extraction from time seriesdata and stochastic signal classication; see Fig. 3.

A. Feature Extraction From Time Series Data

The feature extraction aims at representing the input signals

in a more compact way using one symbol or a sequence of

Fig. 4. Feature concatenation.

symbols. The aim of the feature extraction algorithm is to lowerthe dimensional space and the sampling frequency of the databy representing only signicant signal developments withoutloss of information. The method we propose foresees input datadimension reduction and output data compression.

1) Signal Quantization and Dimension Reduction: Considera multidimensional record u consistingof D signals of differentnature, e.g., force versus velocity, but within the same domain(time); cf. Fig.4.Our goal is to reducethese datato a 1-D symbolsequence that represents the development of all input signalssimultaneously. For this reason, we rst split the domain of eachobserved signal ud into G subranges, which are separated fromeach other by thresholds thdg with g = 1, . . . , G 1. When thesignal passes a threshold, we emit a number that correspondsto the appropriate subrange for the signal. To avoid chattering,which occurs especially when noisy signals at low gradient areobserved, we introduce a vector of dimension D containing

hysteresis coefcients determined to be 1.5 times the varianceof the deviation of each of the signals from its average valuein the steady state. The resulting quantization vector l (t) for ameasurement at time t has dimension D and its elements ld (t)are determined by:

ld (ud , t )= g forld (t 1) = g, |ud | thdg + dld (t 1) = g + 1 , |ud | thdg d

(1)

ld (ud , t )= g + 1 forld (t 1) = g, |ud | thdg + dld (t 1) = g + 1 , |ud | thdg d

.

(2)

As we aim for a 1-D classier, which is much simpler thana classier with multiple inputs, we need to describe the be-havior of all signals with a single number. For this reason, weconcatenate all the elements of l to a large binary register L.The resulting binary number is denoted as concatenated featurenumber (CFN) , as it contains encoded features from all signals.The quantization number for each of the signals change prede-ned bits of the CFN as shown in Fig. 4. The bit length (BL) of ld is given by

BL(ld ) = log2 G . (3)

Consequently, the BL of L depends on the number of observed

signals D, and the number of subranges G for each signal:


4/13


BL(L) = D BL(ld ). When using the same number of sub-ranges for each signal it becomes BL (L) = D log2 G .Typical haptic applications require a sampling rate of about

1 ms [29], which can be assumed to be much higher than thechanging rate of human intentions and, respectively, the cur-rently executed action. Extracting a feature vector for everysampling step would produce a large amount of useless dataand in addition would make the whole algorithm very proneto time warping of the input signal. If, however, the signal issimply down-sampled by a factor k, we may lose importantinformation.

Thus, we propose an event-triggered feature compressionsimilar to the approach of Olszewski [30] who introduced itfor ECG signals. We change the CFN only when at least one of the signals passes a threshold and changes its value. We con-sider this an event that initiates a recalculation and update of our feature vector. As a result every entry of the feature vectordiffers from the previous one exactly by one value. In the veryrare case that multiple simultaneous events occur in different

channels of one sample, the algorithm is designed to generate anew CFN for every changed signal. The changes that are causedby the signals affecting the most lower bits of the CFN will ap-pear rst. From the different CFNs, we build a vector L whichis used as an observation sequence.

2) Threshold Determination of Features: A simple signalquantization suggests to divide the signal range into equal sub-ranges. If, however, we aim for describing the signal with as fewsymbols as possible, more sophisticated ways to split the signalrange need to be chosen.

A suitable way is to use a clustering algorithm. We use aK-means algorithm proposed by MacQueen [31], which allows

dening a number of clusters over a set of observations (for ex-ample records from any of the input signals ud over a given timewindow). We set this number equal to the number of subrangesG. The algorithm nds the center g of each cluster (subrange)g such that the error measure E , the sum of all squared Eu-clidean distances of an input signal from the center of a cluster,is minimized

min E = minG

g=1 u j gu j g 2 =

G

g=1

n g

j =1

u(g) j g2

(4)where ng is the number of measurements in cluster g G.Hence, u(g) j describes all measurements within one cluster g.The centers of the clusters are determined using Lloyds al-gorithm [32], [33] which also gives the distance between thecenter and each element in the cluster. Thus, also the minimalu (g)mi n and maximal u

(g)ma x elements in a cluster are given, from

which the distance between two neighboring clusters gi andgi +1 can be found. A good separation of the different clustersis realized if the thresholds th g r for a specic task r = 1, . . . , Rand two neighboring clusters are placed in the middle betweenthe clusters that lead to

thg r =

1

2(u(g i )

ma x u(g i + 1 )

min ). (5)

For each signal ud , the thresholds th g (d) are given by the meanof thresholds over all possible tasks R

thg (d) = mean R (thg r (d)) . (6)

3) Online Threshold Adaptation: Due to training effects orfatigue, human behavior changes over multiple task execu-

tions [34]. This can lead to changes in the haptic patterns andconsequently deteriorate the performance of the classier. Weobserved two tendencies of these changes: On the one hand, thepatterns can shift within the signal range but without changesin their shape. On the other hand, the patterns generated for dif-ferent tasks can change their shape. The solution for the lattertendency requires online adaptation of the classier, which isnot considered in this study. The former one can be measured,and thus, can be compensated by adapting the thresholds, whileperforming the task.

For online threshold adaptation, we observe the signal behav-ior over a longer time period. The data are recorded and sortedfor each task by using information of the classier; see Fig. 3.After collecting enough data from each action these data areused to calculate temporary thresholds t thg (d) for each signal dusing (6). As the t thg (d) are calculated using small amount of data only, we dene the threshold update n thg (d) as follows:

n thg (d) = thg (d)(1 w( )) + tthg (d)w( ). (7)

The term w [0; 1] is a weighting function which depends onthe deviation of the temporary threshold fromthe currently usedone, as given in following equation:

= t thg (d) thg (d) . (8)

Similarly to the forgetting factor that is used in the least-squares

estimation, our weighting prevents outliers to have a large im-pact on threshold adaptation and allows small changes in thethresholds only. This is in accordance with the assumption thatthe behavior of the human changes slowly with respect to thesystem dynamics. An exponential function w( ) = e c wasused as weighting function, where c < 0 determines the speedof the decay.

B. HMM-Based Stochastic Classication

HMMs represent signals at a higher level of abstraction andare often used due to their robustness and low computationalcosts. The HMM is able to represent a class of shapes of theinput signal as a specic sequence of states, whereby smallvariations in the shape of the input signal do not affect the result.According to the input data, we distinguish between discrete andcontinuous HMMs. Next, we briey discuss the notation usedherein (please refer to [21] and [35] for more details aboutHMMs).

A discrete HMM with a set of N possible hidden statesS = {s1 , . . . , s N } and a set of M possible observations V ={v1 , . . . , vM } is written as follows:

= ( AN N , B N M , N ). (9)

The relation between an observation sequence O = {o1 , . . . ,

oK } V with K discrete observations, and a corresponding


5/13


(a) (b)

Fig. 5. Schematic representation of the action recognition algorithm.

state sequence Q = {q 1 , . . . , q K } S is fully dened as fol-lows:

1) Transition probability distribution matrix A , where eachelement a ij = P (q k = s j |q k 1 = s i ) gives the probabil-ity of switching from state q k 1 to state q k at step k.

2) Emission probability distribution matrix BN M , whereeach element b j (m) = P (ot = vm |q t = s j ) with m =1, . . . , M gives the probability of emitting the symbolvm being in state q j at step k.

3) Vector of initial probability determining the probabilitydistribution of the initial observations.

In general, the HMM has a Markov Property , which meansthat the conditional probability of a hidden state q j |ok dependsonly on the previous state q i |ok 1 , but not on any older observa-tions. Given the HMM and a particular observation sequenceO, we can nd the probability of the outcome given by theHMM

P (O | ) =Q

q 1 bq 1 (o1)aq 1 q 2 aq 2 q 3 . . . aq T 1 q T bq T (10)

where Q is the state sequence corresponding to O, and Q is asum of all possible state sequences with length T . This problemstatement is known as rst canonical problem over HMMs.

In our case, we train an HMM for each of the R actions wewant to distinguish. As we are not interested in any particularmotion primitive, the states of these HMMs do not have phys-ical meaning. This allows us to generate training sequences byrecording actions as a whole. In runtime, we pass the currentlyemitted observation sequence to each of our HMMs and cal-culate their forward probability P r (O | ), a problem alreadystated in (10). This problem is solved numerically by usingthe forward algorithm, see [21]. Next, we compare all forwardprobabilities P i (O | ) witheachother. 1 The HMM with the mostlikely oberservationsequence O is considered to be thecurrentlyexecuted action A:

A = argmax( P 1 , . . . , P R ) . (11)

1) Training Procedure: During the training process of theHMMs, we record haptic data while repetitively executing eachaction. We then extract features from the haptic data and buildthe observation sequences. Finally, the HMMs are trained foreach action using the BaumWelch algorithm [21]. We use 90%from all observation sequences for training and the remaining

1The method of comparing the outcome from different classiers is also well

known as Competing Experts.

10% for testing. The validation procedure for the training pro-cess is described later.

As aforementioned, the haptic patterns depend mainly onthe performed action, but they are also affected by the appliedassistance. Thus, in order to recognize the current action in-dependently of the activated assistance, the classier needs tocope with changes in the haptic patterns that are caused by theassistance.

In [36], we adopted a method including R separate classiersfor R actions working in parallel.The singleHMMswere trainedfor the n corresponding assistances, resulting in total R ntrained HMMs; see Fig. 5(a). This approach was found to besuccessful for a rendered object in a virtual environment, butshowed poor recognition results for an increased number of tasks in a real teleoperation setting.

In this study, we recorded data for a specic action r underthe inuence of all possible assistances to train one HMM,leading to overall R HMMs. This allows the HMM to emitobservation sequencescorresponding to the currently performedaction regardless of the applied assistance; see Fig. 5(b).

2) Evaluation Problem and Observation Sequence Length:As mentioned previously, we update our observation sequenceevery time a new observation occurs. An important problem wewant to stress is how to determine the length of the observationsequence in runtime. Compared with the training phase, in run-time we do not know the exact beginning and end of an action,which leads to the following problems:

1) The forward probabilities during theevaluation are in gen-eral lower than those achieved during the validation of thetraining procedure.

2) The best recognition during the evaluation comes oftenfrom sequences with different lengths (mostly shorter)than during the training phase.

To deal with these problems,we built a vector with themediansequence lengths y andone with thestandard deviations for allactions using the existing training data and use this value for thesequence length. Next, we calculate three forward probabilitiesP i , each with a different observation sequence length: a shortone Os with y = y , a normal sequence On with y = y ,and a long sequence Ol with length y = y + and select thehighest probability

P i = argmax (P is (Os | i ), P in (On | i ), P il (Ol | i )) . (12)

3) Optimal Number of States: The states of the HMMs do

not have a physical meaning; therefore, the number of states


6/13


Fig. 6. Experimental setup.

can theoretically be chosen arbitrarily. However, the number of states plays a crucial role for the outcome P (O | ). A lowerdimensional HMM cannot t the input data and can easily beovertrained. An HMM with too many states can be overtted.

Both will affect the recognition negatively.There are several ways to optimize the number of states of

the model. Calinon and Billard [37] minimized the number of states of an HMM by weighting the likelihood of the modeloutcome with the dimension of the model given the length of the observation sequence. The solved optimization problem isknown as Bayes information criterion (BIC); see [38].

BIC can be used to optimize a single HMM with respect tocomputational effort, but does not consider how several HMMswork together. Thus, instead of optimizing theforwardprobabil-ities of each HMM by changing the number of states, we trainedk HMMs with different number of states for each action. Then,we built C = kR classiers by exploiting all possible combi-nations of HMMs for different actions with different numberof states. During the validation phase, we calculated the overallrecognition rates for each of these classiers by using prede-ned test sequences. The classier with the best result was lateron used for online evaluation.

V. COMPUTER -ASSISTED TELEOPERATION SYSTEM

A. Apparatus

The experimental setup that is used for evaluating the pro-posed approach is shown in Fig. 6. Thehyper-redundant 10 DOFViSHaRD10 robotic arm provided haptic feedback. Thearm hasa large, singularity-free workspaceanda high force output capa-bility; see [39] for a detailed performance analysis. An anthro-pomorphic 7 DOF robotic arm [40] was used at the teleoperatorsite. Both devices are equipped with JR3 6 DOF force/torquesensors that are mounted at their end-effectors. End-effector po-sitions were obtained by applying the forward kinematics to themeasured joint angles. Gravity forces of the end-effectors werecompensated in the force measurements. An aluminum bar thatis mounted at the end-effector of the haptic interface was usedas the handle for the operator. The teleoperator was addition-ally equipped with a two nger robotic gripper. To open andclose this gripper, the distance between the thumb and index

nger of the operators hand was measured using the commer-

cially available CyberGlove system by Immersion Corp. Allcontrollers ran at a sampling frequency of 1 kHz. Experimentaldata were recorded at the same frequency. A stereo video streamwas presented to the operator through a head-mounted displaywith SXGA resolution and a frame rate of 30 Hz.

B. Control

A position-based admittance controller with bilateral force-force exchange as proposed in [41] was used. Forces originatingfrom thehaptic assistance functions f a , the remoteenvironmentf e , and forces applied by the human operator f h were summedup and used as input for the virtual admittance of the localposition-based admittance controller; see Fig. 7. The desireddynamics for master and slave were as follows:

f h + f a f e = ( M n + M a )

M x d + ( D n + D a )

Dx d (13)

where M n/a , D n/a are diagonal matrices, representing vir-

tual mass and damping characteristics of the nominal controllerand assistant, respectively. The values are chosen heuristicallyensuring stability and transparency of the system. The termsx d , x d are the desired velocities and accelerations of the mas-ter and slave device. The nominal values of M n , D n were setto 12 kg and 70 Ns/m for each direction. Using the parameterspace approach as described in [17], these values were selectedfrom a set of parameters for stable interaction of the whole task.Thus, the task can be performed successfully whether or not theassistant is active.

For restricting rotational movements, a strong virtual mass-spring-damper dynamicswasusedwithout hapticfeedbackfromthe remote environment. Using a quaternion-based representa-tion, the following dynamics of the admittance is obtained:

h = M o dm + D o dm + K o m (14)

with K o = 2E T (q m )K o (15)

where q = ( , ) is a Quaternion representation of orienta-tion; see [41] for details. The values of the diagonal matricesM o , D o , K o were chosen as 0.02 kgm 2 , and 4 Nms/rad and40 Nm/rad for each direction. The desired poses for master andslave device were tracked using high-gain PD-controllers thatoperate in the joint space.

In the case of unassisted teleoperation, all assisting variables

f a , M a , D a were set to 0.

C. Assistance Functions

As the priority of objectives differ from action to action, thedesign of one universal assistance function would be unsuit-able for providing sufcient task performance. Thus, assistancefunctions are designed for each action separately. Since the sce-nario can be performed using translational movements only, theassistance functions were applied to these degrees of freedom(DOF) only. Rotational movements were constrained by a strongvirtual mass-spring-damper dynamics. The assistance functionsact as additional virtual forces or velocities in addition to the

human applied forces/velocities; see (13).


7/13


Fig. 7. Overall control structure.

1) Constrained Transportation (CT): For CT, the most im-portant objective is to ensure stability, as the motions are per-formed in a highly constrained environment, such that excessiveforces from the environment can be avoided. This is realized byintroducing additional virtual damping of dCTa = 30 Ns/m toeach direction of motion. Furthermore, as the main constraineddirection of motion is the up/down axis, forces applied by thehuman in this direction are scaled down by a factor of 90%,i.e., f CTaz = 0.9f hz resulting in an overall applied force of f hz + f CTaz = (1 0.9)f hz = 0 .1f hz .

2) Constrained Positioning (CP): CP is performed in thesame constrained environment as CT, and thus, the same assis-tance type is used. The dominating physical effects, however,are slip and stick effects. These effects complicate small-scalemotions as required for positioning resulting in long TCTs andlarge applied forces. An often used assistance function are po-tential elds that provide the operator with haptic cues to drivehim/her toward the selected points, see [14] and [42]. For ef-

ciently applying this assistance, the points of interest have tobe known exactly. In this scenario, the holes in the rack have tobe known and the relative position between the teleoperator andthe holes of the hard drive are not allowed to change. The holesin the rack can be registered when rst grasping the hard drive.However, when the new hard drive is grasped, the grasp positionmay have changed compared with the broken hard drive. With-out object recognition tools, this assistance function cannot beapplied for this task. Thus, in order to overcome the stick-slipeffects, the resolution in the lower force range is increased byamplifying the forces applied by the human in the horizontalplane by a factor 2, i.e., f hx y + f C P ax y = 2f hz . The same objectives mentioned for CT hold also here, and thus, to ensurestability of the assisted system the damping is increased andthe forces applied by the human in the vertical direction arediminished as for CT. Certainly, with this assistance, the effecton TCT and applied effort will be smaller compared with theusage of potential elds.

3) Carrying (CA): In this task, the main objective is to re-alize a high task performance, i.e., fast task execution with loweffort. CA is identied as follows: large free space motions forcarrying the hard drive and small-scale motions for positioningthe hard drive in front of the rack. As the opening of the rack isroughly 1.5 times larger than the hard drive, a potential eld isuseful to guide the operator toward the hole of the rack. Even

if the new hard drive is not grasped at the same position and

TABLE IHEURISTIC SIGNAL THRESHOLDS

the operator is, thus, guided a little bit more to the right or leftof the center of the rack opening, the hard drive can still beinserted and nding the right position for insertion is facilitated.The position for the potential eld is registered when the harddrive is grasped for the rst time and used to render a cylindri-cal attracting potential eld around this position. We selecteda cylinder with a diameter of 12 cm. A virtual spring-damperdynamics with spring constant kCA pf = 200 N/m and dampingconstant dCA pf = 25 Ns/m is xed between the center line of thecylinderand theposition of the teleoperator. In addition to facili-tate large-scale motions in a free space, the virtual mass-damperdynamics of the nominal controller is reduced on the slave siteto mCAa = 2 kg and dCAa = 50 Ns/m in each direction.

D. Classier Settings

The feature extraction algorithm and the stochastic classierran on a separate computer. For the communication, a UDPconnection was used. The feature vector u was built based onthe following four signals: human force |f h | , environmentalforce |f e |, master position |xm | , and velocity |xm | . Absolutevalues of thecorresponding signals for thethreeaxes x,y,z , e.g.,

xm = x x2m + y x

2m + z x

2m . were used. Thethresholds adoptedfor each signal are given in Table I. With the trained HMMs, we

were able to recognize four different actions, and we switchedbetween three different assistance functions.

VI. VALIDATION OF ACTION RECOGNITION UNIT

As an evaluation criterion of the proposed action recognitionalgorithm, the rate of correct recognitions measured during theexperiment is used. To verify the recognized action with theactually performed one, the operators were asked to announcethe currently performed action, which was marked by the su-pervisor using a joystick. The signals from the joystick were

then used as a reference to calculate the recognition rate of the


8/13


9/13


(a) (b) (c)

Fig. 8. Human-performed action recognition (mean and standard error). (a) Task completion time. (b) Human effort. (c) Damage possibility.

age: 26.7 standard deviation: 3.9) performed the task several

times. In the second user study evaluating the combined sys-tem of action recognition and assistance unit 15 participants(13 men, 2 women, 12 right handed, 3 left handed, mean age:27.3 std. deviation: 2.5) took part. For this second study, nineparticipants had an experience value 3 on a 5-point scale (1:no experience, 5: a lot of experience) with haptic devices andwere considered as experts.

E. Results

All results are reported as signicant at the = 0.05 level. Tocompare computer-assisted (WA) and not assisted (NA) teleop-

eration, paired Hotellings T 2

tests were conducted for the task as a whole as well as for the single actions individually. In casethe multivariate test was found to be signicant, Bonferroni-adjusted paired t-tests withadjustedsignicance level = /n(n = number of dependent variables) were performed for eachdependent variable. For the task as a whole, we evaluated vedependent variables (TCT, MFH, MFE, HS, and TF), for con-strained positioning three (TCT, MFH, and MFE), for carry-ing two (TCT and MFH) and for transportation one (MFE).All samples could assumed to be normally distributed as thecorresponding KolmogorovSmirnov tests were not signicant.Outliers having values more than 3 standard deviations awayfrom the mean were substituted with the mean of the data. Forone participant of the rst study, the trial without assistance waserroneous. Thus, the results from the second trial of the rstexperimental block were used instead.

1) Human-Performed Action Recognition: Results of therst user study with human-performed action recognition arereported in Fig. 8 and Table III. Paired Hotellings T 2 tests werefound to be signicant for CA only.

Task Performance: To understand the differences in task performance that depend on the system (assisted/unassisted),Bonferroni-adjusted paired t-tests were conducted for CA: seeTable III. No signicant differences in TCT were found.

Human Effort: To understand the differences in mean hu-

man forces, Bonferroni-adjusted paired t-tests for CA were con-

TABLE IIIHUMAN -PERFORMED ACTION RECOGNITION STUDY: STATISTICAL RESULTS OF

PAIRED HOTELLING S T2 TESTS AND BONFERRONI -ADJUSTED T -TESTSBETWEEN ASSISTED (WA) AND UNASSISTED SYSTEM (NA) FOR OBJECTIVE

AND SUBJECTIVE MEASURES

Significance is marked with *.

ducted and signicant differences were found. For the overalltask as well as for CP and CA the human effort was foundto be descriptively smaller for the assisted system than for theunassisted system.

Damage possibility: A paired t-test for constrained trans-portation was conducted and signicant differences were found.For the overall task as well as for CP and CT MFE was foundto be descriptively larger for the unassisted system than for theassisted system.

Subjective results: Subjective results were only available forthe task as a whole. As the corresponding multivariate test didnot become signicant, no paired comparisons were performedfor the items haptic support and task facilitation. As shownin Fig. 10, however, computer-assisted teleoperation is descrip-tively rated better than unassisted teleoperation.

2) Automatic Action Recognition: Resultsof theseconduserstudy combining action recognition and assistance unit are re-ported in Fig. 9. Paired Hotellings T 2 tests were found to be

signicant for the overall task as well as CP and CA.


10/13


(a) (b) (c)

Fig. 9. Automatic action recognition (mean and standard error). (a) Task completion time. (b) Human effort. (c) Damage possibility.

TABLE IVAUTOMATIC ACTION RECOGNITION STUDY: STATISTICAL RESULTS OF PAIRED

HOTELLING S T2 TESTS AND BONFERRONI -ADJUSTED T -TESTS BETWEENASSISTED (WA) AND UNASSISTED SYSTEM (NA) FOR OBJECTIVE AND

SUBJECTIVE MEASURES

Significant results are marked with *.

(a) (b)

Fig. 10. Qualitative results for both user studies (mean and standard error):(a) Human-performed action recognition. (b) Automatic action recognition.

Task Performance: No signicant differences in T CT be-tween the assisted and unassisted system were found. Yet, onaverage, TCT was descriptively smaller for computer-assistedteleoperation compared with unassisted teleoperation except forCP.

Human Effort: All three paired t-tests show signicant differ-

ences between assisted and unassisted system. The MFH values

for the investigated tasks are smaller for theassisted systemthanfor the unassisted system.

Damage Possibility : Bonferroni-adjusted paired t-tests forthe overall task, CT and CP reveal signicant differences withMFE being larger for the unassisted system than for the assistedsystem.

Subjective results: No signicant difference between the as-sisted and unassisted system could be found for both items. Asshown in Fig. 10, computer-assisted teleoperation is descrip-tively rated better than unassisted teleoperation.

F. Discussion

The purpose of these user studies was to investigate the ef-fect of computer-assisted teleoperation on objective task perfor-

mance as well as on the perceived quality.RQ1: No signicant differences were found for TCTs in caseof assisted and unassisted teleoperation. This result was ob-tained for both user studies and can thus, be clearly attributed tothe design of the assistances rather than to the performance of the action recognition unit. We assume, that reducing the admit-tance parameters is not enough to signicantly inuence TCT.We observed, however, that subjects performed the task moreprecisely with activated assistances. Human forces as well asenvironment forces for unassisted teleoperation are also higherthan for assisted teleoperation. This shows that for the task asa whole our selected assistance functions facilitated task exe-cution in terms of effort, increased precision, and reduced thedamage possibility, but did not reduce TCT.

RQ2: In order to answer these research questions, results of the rst user study are analyzed to remove effects of imperfectclassication:

a) As for the task as a whole, T CT is not signicantlysmaller for assisted teleoperation during CA. Human ap-plied forces are, however, smaller than for unassisted tele-operation. Thus, CA is performed with less effort, but notsignicantly faster.

b) The results showed a signicant decrease of the environ-ment forces in the z-direction from unassisted to assistedteleoperation. Thus, damage possibility is reduced with

the assistance function for CT.


11/13


12/13


[9] L. Love and W. Book, Force reecting teleoperation with adaptiveimpedance control, IEEE Trans. Syst., Man, Cybern. B , vol. 34, no. 1,pp. 159165, Feb. 2004.

[10] R. Dubey, S. Everett, N. Pernalete, and K. Manocha, Teleoperation as-sistance through variable velocity mapping, IEEE Trans. Robot. Autom. ,vol. 17, no. 5, pp. 761766, Oct. 2001.

[11] K. Manocha, N. Pernalete, and R. Dubey, Variable position mappingbased assistance in teleoperation for nuclear cleanup, in Proc. IEEE Int.Conf. Robot. Autom. , 2001, pp. 374379.

[12] D. Botturi, S. Galvan, and C. Secchi, A force dependent scaling for im-proving the human perception in bilateral teleoperation, in Proc. Work-shop New Vistas Challenges Telerobot. , 2008

[13] N. Stefanov, A. Peer, and M. Buss, Online intention recognition forcomputer-assisted teleoperation, in Proc. Eurohapt. , 2010, pp. 233239.

[14] P. Aigner and B. McCarragher, Human integration into robot controlutilising potential elds, in Proc. IEEE Int. Conf. Robot. Autom. , 1997,pp. 291296.

[15] N. Y. Chong, S. Kawabata, K. Ohba, T. Kotoku, K. Komoriya, K. Takase,and K. Tanie, Multioperator teleoperation of multirobot systems withtime delayPart : Aids for collision-free control, Presence: Teleoperat.Virtual Environ. , vol. 11, pp. 277291, 2002.

[16] B. Hannaford and P. Lee, Hidden Markov model analysis of force/torqueinformation in telemanipulation, Int. J. Robot. Res. , vol. 10,pp. 528539,1991.

[17] D. Kulic, W. Takano, and Y. Nakamura, Combining automated on-line

segmentation and incremental clustering for whole body motions, in Proc. Int. Conf. Robot. Autom. , 2008, pp. 25912598.[18] T. Takeda, K. Kosuge, and Y. Hirata, HMM-based dance step estimation

fordance partner robotMSDanceR, in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst. , 2005, pp. 32453250.

[19] D. Aarno, S. Ekvall, andD. Kragic, Adaptive virtual xtures formachine-assisted teleoperationtasks,in Proc.IEEE Int. Conf.Robot. Autom. , 2005,pp. 11391144.

[20] D. Aarno and D. Kragic, Motion intention recognition in robot assistedapplications, Robot. Auton. Syst. , vol. 56, pp. 692705, 2008.

[21] L. Rabiner and B. Juang, An introduction to Hidden Markov Models, IEEE ASSP Mag. , vol. 3, no. 1, pp. 416, Jan. 1986.

[22] P. Somervuo, Competing hidden Markov models on the self organizingmap, in Proc. IEEEINNSENNS Int. Joint Conf. Neural Netw. , 2000,vol. 3, p. 3169.

[23] J. Hu, M. K. Brown, and W. Turin, HMM based on-line handwritingrecognition, IEEE Trans. Pattern Anal. Mach. Intell. , vol. 18, no. 10,pp. 10391045, Oct. 1996.

[24] V. D. Fonzo, F. Aluf-Pentini, and V. Parisi, Hidden Markov Models inbioinformatics, Curr. Bioinformat. , vol. 2, pp. 4961, 2007.

[25] D. Kulic and E. Croft, Affective state estimation for human-robot inter-action, IEEE Trans. Robot. , vol. 23, no. 5, pp. 9911000, Oct. 2007.

[26] D. Aarno and D. Kragic, Layered HMM for motion intention recogni-tion, in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst. , 2006, pp. 51305135.

[27] T. Oates, L. Firoiu, and P. Cohen, Using dynamic time warping tobootstrap HMM-based clustering of time series, in Sequence Learning .vol. 1828, Berlin/Heidelberg,Germany: Springer-Verlag, 2001, pp.3552.

[28] X. Huang and M. Jack, Unied techniques for vector quantization andHidden Markov Modeling using semi-continuous models, in Proc. Int.Conf. Acoust., Speech, Signal Process. , 1989, pp. 639642.

[29] B. Siciliano and O. Khatib, Eds., Handbook of Robotics.Berlin/Heidelberg, Germany: Springer-Verlag, 2008.

[30] R. T. Olszewski, Generalized feature extraction for structural patternrecognition in time series data, Ph.D. dissertation, School Comput. Sci.,Carnegie Mellon Univ., Pittsburgh, PA, USA, 2001

[31] J. B. MacQueen, Some methods for classication and analysis of multi-variate observations, in Proc.5th BerkeleySymp. Math. Statist. Probabil. ,1967, vol. 1, pp. 281297.

[32] S. P. Lloyd, Least squaresquantizationin PCM, IEEE Trans. Inf. Theory ,vol. 28, no. 2, pp. 129137, Mar. 1982.

[33] F. Anton and C. Gold, An iterative algorithm for the determination of Voronoi vertices in polygonal and non-polygonal domains, in Proc. 9thCan. Conf. Comput. Geometry , 1997, pp. 257262.

[34] A. Hamam, N. D. Georganas, F. Alsulaiman, and A. El Saddik, Deducingusers fatigue fromhaptic data, in Proc. Int. Conf. Multimedia , New York,NY, USA, 2010, pp. 10831086.

[35] R. Duda, P. Hart, and D. Stork, Pattern Classication . New York, NY,USA: Wiley-Interscience, 2000.

[36] N. Stefanov, A. Peer, and M. Buss, Online intention recognition incomputer-assisted teleoperation systems, in Haptics: Generating and Perceiving Tangible Sensations (Lecture Notes in Computer Science vol.6191). Berlin/Heidelberg, Germany: Springer-Verlag, 2010, pp. 233239.

[37] S. Calinon and A. Billard, Learning of gestures by imitation in a hu-manoid robot, in Imitation and Social Learning in Robots, Humans and Animals: Behavioural, Social and Communicative Dimensions . Cam-bridge, U.K.: Cambridge Univ. Press, 2007, pp. 153177.

[38] N. Cruz-Ramirez, H.-G. Acosta-Mesa, R.-E. Barrientos-Martnez, andL.-A. Nava-Fernndez, How good are the Bayesian information crite-rion and the minimum description length principle for model selection?A Bayesian network analysis, in Proc. Mexican Int. Conf. Artif. Intell. ,2006, pp. 494504.

[39] M. Ueberle, N. Mock, and M. Buss, VISHARD10, a novel hyper-redundant haptic interface, in Proc. 12th Int. Symp. Hapt. Interf. Virtual Envir. Teleoperat. Syst. , 2004, pp. 5865.

[40] B. Stanczyk and M. Buss, Development of a telerobotic system for ex-plorationof hazardousenvironments, in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. , 2004, pp. 25322537.

[41] A. Peer, Design and control of admittance-type telemanipulation sys-tems, Ph.D. dissertation, Technische Universit at Munchen, Munich,Germany, 2008

[42] O. Khatib, Real-time obstacle avoidance for manipulators and mobilerobots, Int. J. Robot. Res. , vol. 5, pp. 9098, 1986.

[43] C. Passenberg, N. Stefanov, A. Peer, and M. Buss, Enhancing task clas-sication in humanmachine collaborative teleoperation systems by real-time evaluation of an agreement criterion, in Proc. World Hapt. , 2011,pp. 493498.

Nikolay Stefanov received the Masters degree incontrol system engineering and information technol-ogy from the Technical University in Soa, Soa,Bulgaria, in 2006.He is currentlyworking toward thePh.D. degree from the Institute of Automatic Con-trol Engineering, Technische Universit at Munchen,Munich, Germany.

Since 2007, he has been a Research Assistant atthe Institute of Automatic Control Engineering.

His research interests include feature extractionfrom time series data, as well as haptic signal classi-

cation and action recognition for haptic humanrobot interaction.

Carolina Passenberg received the Masters degreein electrical engineering from the Georgia Instituteof Technology, Atlanta, USA, in 2007, and the DiplomaEngineering degree in electrical engineering and in-formation technology from the Technische Univer-sitat Munchen, Munich, Germany, in 2008, whereshe is currently working toward the Ph.D. degree atthe Institute of Automatic Control Engineering.

Her research interests include the design of hap-tic assistances for shared control systems as well ascontrol approaches for teleoperation systems.


13/13


Angelika Peer received the Diploma Engineeringdegree in electrical engineering and informationtechnology, and the Doctor of Engineering degreefrom the Technische Universit at Munchen, Munich,Germany, in 2004 and 2008, respectively.

She is currently a Senior Researcher and Lec-turer with the Institute of Automatic Control En-gineering, Technische Universit at Munchen. Furthershe is a TUM-IAS Junior Fellow of the Institute of Advanced Studies, Technische Universit at Munchen.Her research interests include haptics, teleoperation,

humanhuman and humanrobot interaction, as well as human motor control.

Martin Buss received the Diploma Engineer degreein electrical engineeringfrom the Technical Univer-sity Darmstadt, Darmstadt, Germany, in 1990, andthe Doctor of Engineering degree in electrical engi-neering from the University of Tokyo, Tokyo, Japan,in 1994.

In 2000, he nished his habilitation in the De-partment of Electrical Engineering and Informa-tion Technology, Technische Universit at Munchen,Munich, Germany. From 1995 to 2000, he was a Se-nior Research Assistant and Lecturer witht the Insti-

tute of Automatic Control Engineering, Technische Universit at Munchen. Hewas a Full Professor, the Head of the control systems group, and the DeputyDirector of the Institute of Energy and Automation Technology, Technical Uni-versity, Berlin, Germany, from 20002003. Since 2003, he has been a FullProfessor (chair) with the Institute of Automatic Control Engineering, Tech-nische Universit at Munchen, Germany. His research interests include automaticcontrol,mechatronics,multimodalhumansysteminterfaces, optimization,non-linear, and hybrid discrete-continuous systems.

human haptic

Documents