IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2018 1

Object Picking through In-Hand Manipulationusing Passive End-Effectors with Zero Mobility

Caio Mucchiani, Monroe Kennedy III, Mark Yim, and Jungwon Seo

Abstract—This paper presents a new method for picking ob-jects through quasistatic in-hand manipulation with end-effectorsthat have no degrees of freedom. Our stable manipulation/graspplanning is achieved with two contacts at fixed distance and theforce of gravity, and is provably complete and correct. Practically,our robotic in-hand manipulation technique can facilitate low-cost manipulation and render simple end-effectors such as theparallel-jaw gripper more dexterous in picking, placing, andhanding-off objects in case, for example, failing actuated grippersbecome frozen and it is impossible to exert sufficiently large inter-nal forces. Our picking technique can also be applied to placingobjects. Implementation and experimentation on a custom-madedirect-drive manipulator and a conventional manipulator arepresented. Many experiments were performed for both pickingand placing and the results demonstrate usefulness of this methodfor manipulating a variety of common objects, with differentgeometries and surface conditions.

Index Terms—Grasping, manipulation planning, grippers andother end-effectors.

I. INTRODUCTION

WE are concerned with robotic object picking through in-hand manipulation using end-effectors with no internal

mobility. Fig. 1 depicts this scenario in which the robotis picking the cylindrical object with the non-functioningparallel-jaw gripper (thus no internal mobility). This lackof mobility implies that generally it can be impossible to(1) control contact positions and contact wrenches and (2)guarantee desired grasp properties such as force- or form-closure. While objects have been shown to be stable withjust two frictionless contacts [1], transitioning from graspsto release such as picking up or placing an object can becomplicated.

The proposed scenario can be practically important for low-cost robotic manipulation. Costs of robotic grippers are oftendetermined by the number of actuators. End-effectors with no

Manuscript received: September 10, 2017; Revised December 4, 2017;Accepted January 2, 2018.

This paper was recommended for publication by Editor Han Ding uponevaluation of the Associate Editor and Reviewers’ comments.

Caio Mucchiani, Monroe Kennedy, and Mark Yim are with the De-parment of Mechanical Engineering and Applied Mechanics, Universityof Pennsylvania, Philadelphia, PA, USA. caio@seas.upenn.edu,kmonroe@seas.upenn.edu, yim@seas.upenn.edu

Jungwon Seo is with the Departments of Mechanical and Aerospace/Elec-tronic and Computer Engineering, The Hong Kong University of Science andTechnology, Clear Water Bay, Hong Kong junseo@ust.hk

This letter has supplemental downloadable multimedia material availableat http://ieeexplore.ieee.org, provided by the authors. The SupplementaryMaterials contain an MP4 file showing object picking and placing using aconventional 7 DOF manipulator and a custom-made direct drive manipulator.This material is 47 MB in size

Digital Object Identifier (DOI): see top of this page.

(a) (b) (c)

(d) (e) (f)

Fig. 1. (a)–(f) The robot is picking the snack carton through tipping andregrasping with the zero mobility end-effector – a parallel-jaw gripper thatdoes not close.

internal mobility will use zero actuators (and thus no actuatorcost). The scenario is also closely related to the importantissue of robotic dexterity. For example, consider the parallel-jaw gripper, which may be by far the mostly used roboticend-effector. Although parallel-jaw gripping can be stable (itis well known that a grasp of two contacts that oppose eachother can resist any external wrench applied to the object), it isnecessary to impart sufficiently large internal forces to securethe object. The typical failure modes of parallel-jaw grippersthus include: object slippage due to insufficient internal forces,object damage from too large internal forces, and inability tomake two contacts opposing each other (for example, due tolimited range of joint motion). A more adaptive approach thatdepends less on frictional forces (and in turn, internal forces)and exact contact positioning will address those issues andrender robotic manipulation more dexterous and practical.

This paper presents a novel robotic in-hand manipulationtechnique for picking objects off a flat surface with an end-effector that can be abstracted as a collection of two pointfingers with fixed separation distance. The end-effector thuslacks internal mobility and has to be controlled passively. Theresulting operation incorporates tipping and regrasping as canbe seen in Fig. 1. It will be shown how the object can bein static equilibrium with the two end-effector contacts thatdo not necessarily oppose each other. We will also show ourtechnique can be applied to a wide range of scenarios, forexample, how it can complement the common parallel-jawgripping by addressing the issues aforementioned. Our pickingtechnique can also be directly applied to placing objects back

2 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2018

on a flat surface.The paper is organized as follows. Sec. II outlines relevant

literature. Sec. III formally states our proposed problem.Sec. IV presents our manipulation planning and discussesits properties. Sec. V presents experiments performed with acustom-made, direct-drive arm and a conventional manipula-tor. The conclusion is presented in Sec. VI.

II. RELATED WORK

A large variety of robotic object manipulation techniqueshave been developed to date. In [2], they are classified ac-cording to their models of task mechanics.

Kinematic approaches to robotic manipulation includerobotic caging, concerning how to bound the mobility ofobjects of interest without necessarily making contact. Al-gorithms for computing cages were presented in [3]. Therelationship between caging and grasping was formalized in[4] and it has been shown that caging can render roboticmanipulation more robust [5].

One fundamental form of robotic quasistatic manipulationis robotic grasping and fixturing. The closure properties ofrobotic grasps have been investigated extensively. Importantnotions of grasp stability include force-closure, form-closure,and immobilization [6]. It has been shown that the dynamicstability of a grasp is a function of local contact geometry,contact models, and material properties [7]. Model-basedalgorithms for testing and constructing stable robotic graspsbased on those notions were presented in [8], [9]. Data-drivenapproaches to grasp synthesis have also been a topic of interest[10]. Robotic pushing [11] is another practically important ca-pability that can be achieved in a quasistatic manner. Recently,robotic systems that can do grasping/manipulation with someautonomy have been presented in [12], [13].

The work presented in the paper is relevant to such qua-sistatic object handling skills, particularly for nonprehensilemanipulation. Examples in the literature include stable ro-tations of a polygonal object with two contacts [1], objecthandling with flat contact surfaces (or palm) [14], knocking anobject over with a single contact [15], and pivoting operation[16]. However, these techniques cannot be directly applied toour problem. For example, compared to stably supported rota-tions [1], our problem is formulated with a more conservativeassumption that the distance between two contacts is fixed.Our interest in keeping an object of interest within a gripperlies beyond the scope of the problem of robotic toppling [15].

Our problem is also relevant to robotic in-hand manipu-lation, which refers to the capability to move and hold anobject within a robotic hand. This is an important problemrelated to the issue of robotic dexterity. We are particularlyinterested in rendering low degrees-of-freedom (DOF) end-effectors, such as the parallel-jaw gripper [17] or single rigidbody end-effectors [18], more dexterous when it comes toobject picking and placing. Robotic in-hand manipulation ca-pabilities have been realized in the forms of robotic regrasping,dexterous manipulation, and finger gaiting, and are an activeresearch area [19]–[22]. Robotic in-hand manipulation is stilla challenging problem, with a high-dimensional search space

and limited robotic motor skills. The challenge task in thepaper is also relevant to in-hand manipulation using resourcesfrom the environment, a topic also examined in [23].

Dynamic manipulation is a relatively new area. Some earlyexamples of dynamic manipulation were demonstrated by ajuggling machine [24]. More recent examples include dynamicnonprehensile manipulation for reorienting and picking [25].

III. PROBLEM DESCRIPTION

The problem we address in the paper is concerned with theplanning and control for quasistatic object picking. We assumethat the target object is initially placed on a flat surface andthe direction of gravity is normal to the surface. Our end-effector is assumed to provide two contacts, as the commonparallel-jaw gripper. Unlike parallel-jaw gripping, however,we are interested in more conservative scenarios in which itis impossible to control the distance between the two end-effector contacts, and thus unable to squeeze objects andattain force-closure grasps. Our aim is to develop a methodfor picking objects through quasistatic in-hand manipulationwith such simple end-effectors as depicted in Fig. 1. Thiscan be practically important in case, for example, failingactuated grippers become frozen and it is impossible to exertsufficiently large internal forces (consider gripping a slippery,cone-shaped object with a parallel-jaw gripper).

(a)

C1

C2

P

g

(b)

δ

C1

C2

Pg

x

y

Fig. 2. The 3D truncated cone and the 2D trapezoidal object are incontact with three external contacts at C1, C2, and P in (a) and (b),respectively. Our end-effector is abstracted as a collection of two contacts{C1,C2|δ is constant}. The objects are supported by the ground at P. Eachcontact is shown with its friction cone.

The proposed scenario involves three contacts: two contactswith the end-effector plus one contact with the environment(Fig. 2(a)). According to screw theory [26], one approach toattaining static equilibrium with the three resultant contactwrenches and the wrench of gravity is to keep them necessarilycoplanar. Motivated by this, we shall investigate the problemof object picking on the plane as described in Fig. 2(b) inwhich the target object is a rigid polygon, moving on the plane,and placed stably on a flat surface, modeled as a sufficientlylong line segment. An end-effector is then abstracted as acollection of two contact points whose distance is fixed;thus it is unable to squeeze an object by itself. Contacts onthe object are modeled as point contacts with friction [27].We will investigate how to pick the object off the groundby manipulating it with the simple end-effector that has nomobility. We are interested in keeping the object stable withinthe end-effector, that is, between the two contacts, during the

MUCCHIANI et al.: OBJECT PICKING THROUGH IN-HAND MANIPULATION USING PASSIVE END-EFFECTORS WITH ZERO MOBILITY 3

picking operation although some strong notions such as form-closure will not be attainable due to the lack of mobilityand sufficient number of contacts. Our goal state features astable grasp under the influence of gravity. We shall alsoconsider how to generalize the resulting planar operation tothree-dimensional picking problems.

IV. TIPPING, REGRASPING, AND PICKING WITH TWOCONTACTS AT FIXED DISTANCE

This section presents our manipulation planning for theplanar picking problem described in Sec. III. Our approachfeatures three types of atomic operations. We arrange theseoperations into a feasible manipulation plan for object picking.We also discuss the properties of our manipulation planning.

A. Atomic Operations

We first present three types of atomic operations concernedwith tipping polygonal objects over under the influence ofgravity, with an end-effector making a nonantipodal contactpair at fixed distance. Without loss of generality, we assumethat the objects rotate counterclockwise (CCW).

Type I Tipping: With a nonantipodal contact pair, ourend-effector can make two types of grasps in terms of thesign of the moments it can exert (see Lemma 1). For example,as can be seen in Fig. 3(b), the moment of the contactforce couple at C1 and C2 can point in the positive (negativein Fig. 3(d)) z-direction. In Type I Tipping, the object istipped CCW by the grasp that can impart positive z-directionalmoments (Fig. 3(b)).

Type II Tipping: In Type II Tipping, the object is tippedCCW by the grasp that can impart negative z-directionalmoments (Fig. 3(d)).

Regrasping: In Regrasping, the end-effector is rotatedabout one of the contact points. As a result of the regrasping,the sign of the moment of the contact forces switches.

These operations are exhaustive in terms of the relativeposition between the object’s center of mass (CoM) and thevertex about which it rotates, as will be explained in thefollowing subsection.

B. Planning for Picking

The progress of our planning shall be explained using Fig. 3where the object rotates CCW monotonically, without loss ofgenerality.

The first step is to determine where to make contacts andhow to acquire the contacts (Fig. 3(a)). As in parallel-jawgripping, our end-effector is supposed to make two pointcontacts on an object’s edge pair where it is possible to havean antipodal grasp, whose two contacts can “see each other bya line inside both friction cones” [28]. Such an edge pair (forexample, a parallel edge pair) can be found by a combinatorialsearch in O(n2) time for an n-sided polygonal object [17]. Thetwo contact positions on each edge is selected so that theirdistance equals to δ , the distance between the two contactpoints of the end-effector. The end-effector is then position-controlled to follow a feasible path into the contact positions.

Next, the atomic operations are performed sequentiallyto roll the object CCW until the inward pointing contactnormal at the contact C1 is antiparallel to the force of gravity(Fig. 3(e)). The position of the object’s CoM relative tothe contact normal at the ground contact determines whichaction to take. If the CoM is on the right (left) half planedelimited by the contact normal, as illustrated in Fig. 3(b)(Fig. 3(d)), the end-effector is controlled to perform TypeI Tipping (Type II Tipping). If the CoM is on the line ofthe contact normal (Fig. 3(c)), the end-effector is controlledto perform Regrasping from grasp {C1,C2} to {C1,C′2}. Ifother edge of the object hits the ground in the meantime, themotion proceeds with the new pivot (see the pivot changesfrom P to Q between Fig. 3(c) and (d)). Thus, during thewhole manipulation process the pivot can switch at most ntimes for an n-sided polygonal object. Finally, at most oneRegrasping may be needed in order to properly counteractthe moment of gravity before lifting the object: for example,between Fig. 3(d) and (e), the grasp has changed from {C1,C′2}to {C1,C2} before the lifting-up happens.

During the tipping and regrasping operations and upon thetermination of the process, the contact forces can properlycounteract gravity; thus the object remains in static equilibrium(see Lemma 1). The manipulation process should terminateand can be applied to any polygonal shape that admits an-tipodal grasps (see Lemma 2). Because two distinct poles ofrotation (one for the tipping, the other for the regrasping)suffice for the operations, it is possible to implement thepicking process using a position-controlled robot arm with twoDOF. Two types of implementation can be considered: reactivecontrol based on force feedback and deliberative control byplanning in advance (see Sec. V).

C. Discussion

The following lemma shows that the object remains in staticequilibrium during the manipulation; thus it is possible toexecute the picking operation in a quasistatic manner.

Lemma 1. During the manipulation process stated inSec. IV-B, an object can be in static equilibrium with two pointcontacts on an edge pair that can accommodate antipodalgrasps.

Proof. First, we will check for static equilibrium during thetipping operation. If the two end-effector contacts are antipo-dal, they form a force-closure [27] grasp that can be in staticequilibrium. Otherwise (with two nonantipodal contacts), thetwo contacts alone cannot be in force-closure because theyfail to be in torque-closure [29], that is, the moments of thetwo contact forces are necessarily unilateral and the sign isdetermined by their relative position. Still, the two contactsare in force-direction closure [29], where the contact wrenchesspan the space of all forces. No matter where the contacts arelocated on the edges, the condition of force-direction closureis invariant. According to the description of Type I/II Tipping,the signs of (1) the moment of gravity with respect to thepivot and (2) the moment of the contact force couple arealways opposite. Therefore, the system of all wrenches can

4 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2018

(a)

δ

C1

C2

y

x

(b)

y

x P

C1

C2

+

(c)

y

x P

+

C1

C2

C′2

(d)

y

xC1

C′2

Q

−

(e)

y

x

C1

C2

Fig. 3. Picking the polygonal object with two point contacts at fixed distance (δ remains constant). The blue (red) arrows represent the frictionless contactwrenches (gravity). (a) Initially on the object, there are two point contacts, C1 and C2, with the end-effector and one line contact with the ground. (b) Theobject is tipping CCW. The object is in stable equilibrium even with frictionless contact wrenches: according to moment labeling [2], the composite wrenchcone of the contact wrenches (the + labeled gray set) can resist gravity. (c) When the CoM is right above P, the object is marginally stable in that gravity canbe balanced by a wrench on the boundary of the composite wrench cone. (d) The object is tipping CCW about the next pivot Q with the new grasp {C1,C′2}.This motion terminates when C1 hits the ground. Static equilibrium is maintained. (e) The object can be lifted up with the frictional grasp {C1,C2}. In (b),(c), and (d), the object can be in static equilibrium even with the frictionless contact wrenches.

be in torque-closure and this in turn implies force-closure andpossibility of static equilibrium.

Second, during the regrasping, the ground contact and oneof the two end-effector contacts are able to exert wrenchescounteracting the wrench of gravity. In Fig. 4(a), the frictioncones define a band (see the orange set) between the twosupporting hyperplanes (parallel to the direction of gravity) ofthe two convex regions that are the composite wrench cone.As long as the line of the force of gravity remains in the band,the object can be in static equilibrium. The finite size of theband implies the regrasping operation can be performed withsome robustness in the actual orientation of the object.

Third, at the terminating condition, the line of the contactnormal at C1 is supporting the composite wrench cone be-cause it is delimited by the edges of the friction cone at C1(Fig. 4(b)). At the same time, the contact normal is parallelto the wrench of gravity. Therefore the wrench of gravity canbe spanned by the composite wrench cone.

(a)

Px

y

C1

+ −

(b)

C1

C2

+

Fig. 4. The +/− labeled gray sets represent the composite wrench cone ofthe contact wrenches with Coulomb friction according to the moment labelingtechnique when (a) there is one contact with the ground at P and one contactwith the end-effector at C1; and (b) the tipping operation terminates.

By taking advantage of frictional contact forces, the wholeprocess can actually be terminated before the terminatingcondition is satisfied exactly. This is because the compositewrench cone is pointed at C1 as can be seen in Fig. 4(b).Therefore the contact normal at C1 does not have to benecessarily parallel to the force of gravity. During the tippingoperations, if the center of rotation of the end-effector is on the

pivot (P or Q in Fig. 3), then it is possible to tip over the objectwith no sliding. If the pole is off the pivot, the object will slideboth on the ground and the end-effector while tipping over;contact mode analysis shows that it is feasible without losingstatic equilibrium. While it is possible to tip over an objectwith only one point contact, with two contacts it is possibleto rely less on frictional forces and exact contact positioning asdiscussed in the proof of Lemma 1. In contrast, in one-contactobject tipping, friction and contact positioning is critical [15].In our two-contact manipulation, it is also possible to retainthe object within the end-effector, which is impossible withonly one contact.

The following lemma shows that the picking operation canbe applied to a wide range of shapes.

Lemma 2. The manipulation process stated in Sec. IV-Bterminates for any polygonal shape that admits antipodalgrasps.

Proof. A polygonal object is in contact with a sufficientlylarge flat surface only at the vertices on its convex hull.Therefore the object keeps turning CCW by the manipulationprocess of Sec. IV-B because it is impossible for concavevertices to make contact with the surface. The angle betweenthe contact normal at C1 and the force of gravity changesmonotonically and thus the motion terminates.

Departing from the two-dimensional polygonal objectmodel, our technique can be applied to three-dimensionalobjects by effectively suppressing the motions off the plane,for example, using curved fingers as investigated in our pre-vious work [18]. This point shall be demonstrated in the nextsection with experiments performed with essentially prismsand objects with rotational symmetry, a class of object shapeswhich may be of industrial importance.

Our object picking technique can also be applied to objectplacing, by reversing the picking motion in a quasistaticmanner. In the next section, we will demonstrate not onlypicking but also placing experiments.

The lemmas above imply that our object picking techniquewith the passive, zero mobility end-effector model can beas complete as two-dimensional parallel-jaw gripping in the

MUCCHIANI et al.: OBJECT PICKING THROUGH IN-HAND MANIPULATION USING PASSIVE END-EFFECTORS WITH ZERO MOBILITY 5

sense that it can be executed by two contacts on an edge pairaccommodating parallel-jaw grasps (that is, antipodal grasps).Our method can also generalize the way we use the commonparallel-jaw gripper by (1) not depending on antipodal contacts(thus no need to search for feasible antipodal grasps and lesssensitive to sensing/positioning accuracy), (2) not squeezingthe objects (thus no need to control contact/friction forces),and (3) adapting the shape of the gripper fingers. Our approachcan thus be applied even to a failing parallel-jaw gripper,which is not able to squeeze objects, to do successful objectpicking. But, the object may roll on its convex hull untilthe tipping operation terminates and the regrasping actionswill need sufficiently large free space, which can render end-effector accessibility/reachability check harder than parallel-jaw gripping.

V. IMPLEMENTATION AND EXPERIMENTATION

We demonstrate the effectiveness of our technique using acustom-made direct-drive arm capable of joint torque sensingand our state machine software built upon the model ofmechanics discussed in Sec. IV. Sec. V-A introduces ourhardware and software system and Sec. V-B presents ourexperiments for object picking and placing performed with thedirect-drive arm. The state machine software enabled the armto transition between Type I/II Tipping and Regrasping actionsin an autonomous manner through joint torque feedback.We also present experiments with a conventional 7 DOFmanipulator in Sec. V-C.

A. Direct-Drive Arm: Hardware and Software

(a) (b)Motor M1

Motor M2

Fig. 5. (a) CAD exploded view showing two motors, a parallelogram linkage(the black links), and two white rigid body fingers constituting a zero DOFend-effector. The motor axes are collinear. (b) Completed hardware setup.

We developed a two DOF arm using a parallelogram five-bar linkage with a closed-loop kinematic chain (Fig. 5); similarapproaches can be seen in [30]. Its two DOF motions arecomposed of one DOF internal deformation and the rigid bodyrotation of the parallelogram itself. The links are laser-cut ABSplastic. 3D-printed, single rigid body fingers can be attached tothe arm linkage using mechanical adapters. We employ direct-drive actuation. Two iPower GBM6212H-150T brushless DCmotors (torque limit: 0.5Nm) are directly coupled to twolinks of the parallelogram linkage, with no gearbox. If thetwo motors, denoted M1 and M2 (Fig. 5(a)), spin with thesame angular velocity, the parallelogram linkage performsrigid body rotation. If M2 spins solely, the parallelogram is

deformed. The motors are controlled by IQinetics IQ-MC-17-15-24C-H motor controllers1 with a high resolution positionsensing capability—12 bits or 4096 counts per revolution. Ourdirect-drive arm is capable of reliable joint torque sensingwith no explicit torque sensor [31], and thus enables low-cost,autonomous manipulation in a reactive manner. The torqueon the motors is calculated from their current readings, with±20% estimated errors due to noise and friction at the joints.Serial communication with the motors is realized using FTDIUSB to serial UART.

We developed MATLAB software for communicating withthe motors and implementing functions for not only low-leveltorque calculation but also in-hand manipulation capabilities.The operation of our software can be explained using theabstraction of a finite state machine, also can be seen in [32]–[34], with the following four states:

Regrasp CCW: The arm rotates the end-effector CCWabout one of the two fingers.

Regrasp CW: The arm rotates the end-effector CW aboutone of the two fingers.

Tip: The arm itself rotates CCW.Lift: The arm translates the end-effector upward by

deforming the parallelogram linkage.The organization of these states can be customized to suit

the purpose of experiments (that is, object picking or placing),as will be elaborated in the following subsection.

B. Experiments with the Direct-Drive Arm

Regrasp CCWstart Tip

Regrasp CWLift

A0

B0

C0B1

D0

A1

Fig. 6. Finite state machine for object picking. Its operation terminates in“Lift” state.

1) Object Picking: Fig. 6 represents our software in theobject picking mode, implementing the planning in Sec. IV-B.We need a precondition regarding the initial placement of thefingers that one of the two fingers is already in contact withthe object. This is because our 2 DOF arm lacks sufficientmobility to address general path planning/trajectory followingscenarios. This can be relaxed by using, for example, aconventional 6 DOF industrial arm. Given the precondition,the finite state machine of Fig. 6 is executed. First, in RegraspCCW state the end-effector is rotated until the condition ofedge label A0 is satisfied (this is the moment when the otherfinger also makes contact with the object). Then in Tip state thearm rotates CCW until B0 is satisfied (at this moment the CoMis on the line of the contact normal at the ground contact).Next, after the regrasping is done (C0) in Regrasp CW state,the arm rotates CCW again in Tip state. If B1 is satisfied, we

1http://iqinetics.com/

6 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2018

enter Lift state. If the lifting operation is unsuccessful (D0),the state is transitioned to Regrasp CCW before attempting tolift again.

Those edge labels (A0, B0, and so on), called the stateguard conditions, are formulated using (1) τM1 (τM2 ), thetorque sensed on motor M1 (M2) in Nm; (2) τM , the sumof no load torques on both motors; and (3) θM1 (θM2 ), theabsolute angle of motor M1 (M2) in radians. Each conditionwas obtained empirically. For example, B0 (Eq. 3) concerns themoment when the center of mass is right above the pivot. Atthe instant when the event happens, in principle τM1+τM1=τM .But, in practice a small positive margin is needed to makethe condition more robust. We chose 0.04 Nm in Eq. 3,which resulted in a local maximum in terms of the experimentsuccess rates. Other conditions were determined in a similarmanner and are as follows:

A0 :τM2 > 0.12, and θM1 = 0 (1)A1 :τM2 > 0.12, and θM1 > 0 (2)B0 :τM1 + τM2 − τM < 0.04 (3)B1 :τM1 + τM2 − τM > 0.35, and θM1 < π/2 (4)C0 :τM2 > 0.085 (5)D0 :τM2 < 0.06 (6)

Fig. 7 represents two of our experiments done with thedirect-drive arm running the state machine in Fig. 6. Dueto its reactive nature, the picking operation is performedautonomously with no information on object mass propertiesand frictional characteristics of the object, fingers, and tabletop. For planning purposes, the shapes of the objects in Fig. 7can be projected into a trapezoid and a triangle, respectively.When it comes to executing the resulting manipulation plan,the combination of the semicircular and straight fingers helpstabilize the three-dimensional object by suppressing the mo-tions off the plane.

Such experiments were repeated with a range of objects andsurface conditions. See Table I. In total, 165 out of 210 trials(78%) were successful. Recall that objects are allowed to slideon the ground and on the end-effector (recall the discussionin Sec. IV-C). To check the effect of the sliding on stablepicking, we changed the distance between the pivot of tippingand the motor axis from 5 to 35mm for object #7. Here 27 outof 30 trials (90%) were successful. In unsuccessful trials, theempirically determined guard conditions did not seem to work.The possible ±20% errors in torque estimation may also haveaccounted for the failed trials. Specifically, the direct-drive armmade a wrong estimate of the position of the CoM in case ofthin, lightweight objects. This caused unwanted sliding duringthe regrasping operations.

2) Object Placing: Picked objects can be placed again byreversing the picking motion in a quasistatic manner. See Fig.8 for the state machine in the placing mode. We need a newguard condition for detecting the event of object touchdown:D0 : τM1 + τM2 − τM < 0.57. Starting from Touchdown state,the operation of the state machine can be explained similarlyto the picking mode. Experiments with objects #6 and #7 wereperformed and out of 30 trials, 20 (67%) were successful

(Table I). Unsuccessful trials occurred especially when therobot made a wrong estimate of the object state in Touchdown.

C. Experiments with a Conventional Manipulator

We also tested object picking and placing using a conven-tional manipulator, Rethink Robotics’ Sawyer arm (Fig. 9(a)).The robot has a parallel-jaw gripper (Fig. 9(b)) equipped onthe wrist of a 7 DOF arm. For the experiments here, therobot’s parallel-jaw gripper is not actuated. Our 3D printed,curved finger can be used with the default, straight finger whenhandling curved objects (Fig. 9(c)).

Given an object of interest, we obtain the gripper trajec-tory for picking using our direct-drive arm and the Sawyerrobot is then position controlled to follow the trajectory ina feedforward, open-loop manner due to its lack of torquesensing capability. Object placing is done in a similar manner,with the placing trajectory prescribed by again our direct-drivearm. We performed a set of experiments with six differentobjects and the overall success rate was 73% (see Table I).Initial positioning errors and external disturbances (such as thevibration of the manipulator) may account for the unsuccessfultrials. Fig. 10 presents picking and placing experiments withthe cone-shaped object.

VI. CONCLUSION

We presented a robotic in-hand manipulation technique forpicking and placing objects that takes advantage of two non-antipodal contacts at fixed distance and gravity. Our manipu-lation technique can be applied to a wide range of objects androbot platforms and is less dependent on exact contact posi-tioning and sufficiently large internal forces. Experiments witha custom-made autonomous manipulator and a conventionalmanipulator had high success rates for picking and placingseveral objects on different surface conditions.

While in principle this approach can work on a large varietyof object shapes, rigorously determining the space of allobject geometries that will practically work (such as thoseelongated shapes addressed in our experiments) is left forfuture work. We are planning to generalize our techniquewith additional sensing modalities, more dexterous in-handmanipulation skills, and more expressive contact models.

REFERENCES

[1] T. Abell and M. Erdmann, “Stably supported rotations of a planar poly-gon with two frictionless contacts,” in Intelligent Robots and Systems95.’Human Robot Interaction and Cooperative Robots’, Proceedings.1995 IEEE/RSJ International Conference on, vol. 3, pp. 411–418, IEEE,1995.

[2] M. T. Mason, Mechanics of robotic manipulation. MIT press, 2001.[3] M. Vahedi and A. van der Stappen, “Caging polygons with two and

three fingers,” The International Journal of Robotics Research, vol. 27,no. 11-12, pp. 1308–1324, 2008.

[4] A. Rodriguez, M. Mason, and S. Ferry, “From caging to grasping,” TheInternational Journal of Robotics Research, vol. 31, no. 7, pp. 886–900,2012.

[5] M. Dogar and S. Srinivasa, “A framework for push-grasping in clutter,”Robotics: Science and Systems VII, 2011.

[6] B. Siciliano and O. Khatib, Springer handbook of robotics. Springer,2016.

[7] W. S. Howard and V. Kumar, “On the stability of grasped objects,”Robotics and Automation, IEEE Transactions on, vol. 12, no. 6, pp. 904–917, 1996.

MUCCHIANI et al.: OBJECT PICKING THROUGH IN-HAND MANIPULATION USING PASSIVE END-EFFECTORS WITH ZERO MOBILITY 7

(a) (b) (c) (d) (e) (f) (g) (h)

Fig. 7. Two picking experiments with the beverage container (upper) and the cone-shaped object (lower). In (a), the two fingers (one curved, one straight) arein contact with the objects at the instant guard condition A0 (Eq. 1) is satisfied. The instant is marked with the red arrow in (b) where M2 angle is increasingwith time. In (c), the CoM is right above the pivot at the instant guard condition B0 (Eq. 3) is satisfied. The instant is marked with the red arrow in (d) whereM1 angle is increasing with time. (e) show the configurations after regrasping at the instant guard condition C0 (Eq. 5) is satisfied. The instant is marked withthe red arrow in (f) where M2 angle is decreasing with time. After further rotation (g), the objects are lifted up (h).

Regrasp CW Tip

Regrasp CCWTouchdownstart

A0

B0

C0D0

Fig. 8. Finite state machine for object placing.

(a) (b) (c)

Fig. 9. (a) Rethink Robotics’s Sawyer. (b) Original end-effector. (c) Adaptedend-effector with a curved finger.

[8] J. C. Trinkle, “On the stability and instantaneous velocity of graspedfrictionless objects,” Robotics and Automation, IEEE Transactions on,vol. 8, no. 5, pp. 560–572, 1992.

[9] A. F. Van der Stappen, C. Wentink, and M. H. Overmars, “Comput-ing form-closure configurations,” in Robotics and Automation, 1999.Proceedings. 1999 IEEE International Conference on, vol. 3, pp. 1837–1842, IEEE, 1999.

[10] N. S. Pollard, “Closure and quality equivalence for efficient synthesis ofgrasps from examples,” The International Journal of Robotics Research,vol. 23, no. 6, pp. 595–613, 2004.

[11] K. M. Lynch and M. T. Mason, “Stable pushing: Mechanics, controlla-bility, and planning,” The International Journal of Robotics Research,vol. 15, no. 6, pp. 533–556, 1996.

[12] A. Saxena, J. Driemeyer, and A. Y. Ng, “Robotic grasping of novelobjects using vision,” The International Journal of Robotics Research,vol. 27, no. 2, pp. 157–173, 2008.

[13] N. Hudson, T. Howard, J. Ma, A. Jain, M. Bajracharya, S. Myint,C. Kuo, L. Matthies, P. Backes, P. Hebert, et al., “End-to-end dexterousmanipulation with deliberate interactive estimation,” in Robotics and

(a)

(b)

Fig. 10. Implementation on the Sawyer robot showing (a) object picking(clockwise from the top leftmost panel) and (b) placing (left to right).

Fig. 11. Objects used for the experiments in Table I.

Automation (ICRA), 2012 IEEE International Conference on, pp. 2371–2378, IEEE, 2012.

[14] W. H. Huang and G. F. Holden, “Nonprehensile palmar manipulationwith a mobile robot,” in Intelligent Robots and Systems, 2001. Proceed-ings. 2001 IEEE/RSJ International Conference on, vol. 1, pp. 114–119,IEEE, 2001.

[15] K. M. Lynch, “Toppling manipulation,” in Robotics and Automation,IEEE International Conference on, vol. 4, pp. 2551–2557, IEEE, 1999.

[16] Y. Aiyama, M. Inaba, and H. Inoue, “Pivoting: A new method of grasp-less manipulation of object by robot fingers,” in Intelligent Robots andSystems’ 93, IROS’93. Proceedings of the 1993 IEEE/RSJ International

8 IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JANUARY, 2018

TABLE IEXPERIMENTAL RESULTS FOR OBJECT PICKING AND PLACING. SEE FIG. 11 FOR THE OBJECTS USED HERE.

ObjectDescription Mass (g) Task Platform Surface Number of Trials Number of Successes

Cylinder (#1, Fig. 11) 140 Picking DD Arm Rubber 5 5Aluminium 8020 (#2) 180 Picking DD Arm Rubber 5 5

Metal bar (#3) 265 Picking DD Arm Rubber 5 5Bottle brush (#4) 125 Picking DD Arm Rubber 5 2

Oil can (#5) 205 Picking DD Arm Rubber 5 1Potato chip carton (#6) 185 Picking DD Arm Rubber 5 3Potato chip carton (#6) 185 Picking Sawyer Carpet 5 3Potato chip carton (#6) 185 Placing DD Arm Rubber 5 3Potato chip carton (#6) 185 Placing DD Arm MDF 5 2Beverage container (#7) 190 Picking DD Arm Rubber 30 27Beverage container (#7) 215 (25ml water) Picking DD Arm Rubber 10 10Beverage container (#7) 240 (50ml water) Picking DD Arm Rubber 10 8Beverage container (#7) 190 Picking DD Arm ABS plastic 10 7Beverage container (#7) 190 Picking DD Arm MDF 10 8Beverage container (#7) 190 Picking DD Arm Paper 10 8Beverage container (#7) 190 Picking DD Arm Carpet 50 41Beverage container (#7) 190 Picking DD Arm Masking Tape 10 10Beverage container (#7) 190 Placing DD Arm Rubber 5 5Beverage container (#7) 190 Placing DD Arm MDF 5 4Beverage container (#7) 215 (25ml water) Placing DD Arm MDF 5 3Beverage container (#7) 240 (50ml water) Placing DD Arm MDF 5 3

DVD (#8) 75 Picking DD Arm Rubber 5 3Pencil case (#9) 90 Picking DD Arm Rubber 5 2Sip cup (#10) 120 Picking DD Arm Rubber 5 3

Index cards (#11) 120 Picking DD Arm Rubber 5 2Wool spool (#12) 74 Picking DD Arm Rubber 5 5Milk carton (#13) 80 Picking DD Arm Rubber 5 1Water bottle (#14) 190 Picking Sawyer Carpet 5 4

Conic glass container (#15) 275 Picking Sawyer Carpet 5 4ABS cone (#16) 100 Picking DD Arm Carpet 5 3ABS cone (#16) 100 Picking Sawyer Carpet 5 4

Pasta carton (#17) 125 Picking Sawyer Carpet 5 4Soap container (#18) 125 Picking Sawyer Carpet 5 4

Aluminum foil carton (#19) 125 Picking Sawyer Carpet 5 2

Conference on, vol. 1, pp. 136–143, IEEE, 1993.[17] G. Smith, E. Lee, K. Goldberg, K. Bohringer, and J. Craig, “Computing

parallel-jaw grips,” in Robotics and Automation, IEEE InternationalConference on, vol. 3, pp. 1897–1903, IEEE, 1999.

[18] J. Seo, M. Yim, and V. Kumar, “A theory on grasping objects usingeffectors with curved contact surfaces and its application to whole-armgrasping,” The International Journal of Robotics Research, vol. 35, no. 9,pp. 1080–1102, 2016.

[19] P. Tournassoud, T. Lozano-Perez, and E. Mazer, “Regrasping,” inRobotics and Automation. Proceedings. 1987 IEEE International Con-ference on, vol. 4, pp. 1924–1928, IEEE, 1987.

[20] J. K. Salisbury and J. J. Craig, “Articulated hands: Force control andkinematic issues,” The International journal of Robotics research, vol. 1,no. 1, pp. 4–17, 1982.

[21] L. Han and J. C. Trinkle, “Dextrous manipulation by rolling and fingergaiting,” in Robotics and Automation, 1998. Proceedings. 1998 IEEEInternational Conference on, vol. 1, pp. 730–735, IEEE, 1998.

[22] A. Bicchi and A. Marigo, “Dexterous grippers: Putting nonholonomyto work for fine manipulation,” The International Journal of RoboticsResearch, vol. 21, no. 5-6, pp. 427–442, 2002.

[23] N. C. Dafle, A. Rodriguez, R. Paolini, B. Tang, S. S. Srinivasa, M. Erd-mann, M. T. Mason, I. Lundberg, H. Staab, and T. Fuhlbrigge, “Extrinsicdexterity: In-hand manipulation with external forces,” in Robotics andAutomation (ICRA), 2014 IEEE International Conference on, pp. 1578–1585, IEEE, 2014.

[24] A. A. Rizzi and D. E. Koditschek, “Progress in spatial robot juggling,” inRobotics and Automation, 1992. Proceedings., 1992 IEEE InternationalConference on, pp. 775–780, IEEE, 1992.

[25] K. M. Lynch and M. T. Mason, “Dynamic nonprehensile manipulation:Controllability, planning, and experiments,” The International Journalof Robotics Research, vol. 18, no. 1, pp. 64–92, 1999.

[26] K. H. Hunt, Kinematic geometry of mechanisms, vol. 7. OxfordUniversity Press, USA, 1978.

[27] R. M. Murray, Z. Li, and S. S. Sastry, A mathematical introduction torobotic manipulation. CRC, 1994.

[28] K. Lynch and F. Park, Modern Robotics. Cambridge University Press,2017.

[29] V.-D. Nguyen, “Constructing force- closure grasps,” The InternationalJournal of Robotics Research, vol. 7, no. 3, pp. 3–16, 1988.

[30] H. Asada and K. Youcef-Toumi, Direct-drive robots: theory and prac-tice. MIT press, 1987.

[31] M. Piccoli and M. Yim, “Anticogging: Torque ripple suppression, mod-eling, and parameter selection,” The International Journal of RoboticsResearch, vol. 35, no. 1-3, pp. 148–160, 2016.

[32] M. Kennedy, K. Queen, D. Thakur, K. Daniilidis, and V. Kumar,“Precise Dispensing of Liquids Using Visual Feedback,” IEEE/RSJ 2017International Conference on Intelligent Robots and Systems, IROS 2017- Conference Proceedings, 2017.

[33] J. Lygeros, C. Tomlin, and S. Sastry, “Hybrid systems: modeling,analysis and control,” preprint, 1999.

[34] J. Felip, J. Bernabe, and A. Morales, “Contact-based blind grasping ofunknown objects,” in Humanoid Robots (Humanoids), 2012 12th IEEE-RAS International Conference on, pp. 396–401, IEEE, 2012.