ram_20120901_sep_2012

Contents | Zoom in | Zoom out Search Issue | Next PageFor navigation instructions please click here

Contents | Zoom in | Zoom out Search Issue | Next PageFor navigation instructions please click here

____________________

http://www.qmags.com/clickthrough.asp?url=http://www.ieee-ras.org/ram&id=17487&adid=PCOVER 1E1

Previous Page | Contents | Zoom in | Zoom out | Front Cover | Search Issue | Next PageIEEE

M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM

Qmags®THE WORLD’S NEWSSTAND

qqM

Mq

qM

MqM


_____________________

________________

______________________

________________

http://www.qmags.com/clickthrough.asp?url=www.mobilerobots.com&id=17487&adid=PCOVER 2A1



mailto:[email protected]

http://www.qmags.com/clickthrough.asp?url=www.ieee-ras.org/ram&id=17487&adid=logo

http://www.qmags.com/clickthrough.asp?url=www.qmags.com&id=17487&adid=logo



Vol. 19, No. 3 September 2012ISSN 1070-9932http://www.ieee-ras.org/ram

FEATURES •20 Multirotor Aerial Vehicles

Modeling, Estimation, and Control of QuadrotorBy Robert Mahony, Vijay Kumar, and Peter Corke

33 Build Your OwnQuadrotorOpen-Source Projects on Unmanned Aerial VehiclesBy Hyon Lim, Jaemann Park, Daewon Lee, and H.J. Kim

46 Toward a Fully Autonomous UAVResearch Platform for Indoor and OutdoorUrban Search and RescueBy Teodor Tomi�c, Korbinian Schmid, Philipp Lutz,Andreas D€omel, Michael Kassecker, Elmar Mair,Iris Lynne Grixa, Felix Ruess, Michael Suppa, and Darius Burschka

57 Shared ControlBalancing Autonomy and Human Assistancewith a Group of Quadrotor UAVsBy Antonio Franchi, Cristian Secchi, Markus Ryll,Heinrich H. B€ulthoff, and Paolo Robuffo Giordano

69 Agile Load TransportationSafe and Efficient Load Manipulation with Aerial RobotsBy Ivana Palunko, Patricio Cruz, and Rafael Fierro

80 Tutorial: Point Cloud LibraryThree-Dimensional Object Recognitionand 6 DoF Pose EstimationBy Aitor Aldoma, Zoltan-Csaba Marton, Federico Tombari,Walter Wohlkinger, Christian Potthast, Bernhard Zeisl,Radu Bogdan Rusu, Suat Gedikli, and Markus Vincze

92 Guidance of Unmanned Surface VehiclesExperiments in Vehicle FollowingBy Marco Bibuli, Massimo Caccia, Lionel Lapierre, and Gabriele Bruzzone

SEPTEMBER 2012 • IEEE ROBOTICS & AUTOMATIONMAGAZINE • 1

ON THE COVERThe cover shows examples ofquadrotor aerial vehicles, whichare one of the most flexible andadaptable platforms for under-taking aerial research.

COVER IMAGE: ©ISTOCK PHOTO.COM/ANDREJS ZAVADSKIS

Digital Object Identifier 10.1109/MRA.2012.2205613

©ISTOCKPHOTO.COM/©

IAKOV

FILIM

ONOV


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________

_________

http://www.qmags.com/clickthrough.asp?url=http://www.ieee-ras.org/ram&id=17487&adid=P1E1

http://www.qmags.com/clickthrough.asp?url=http://ISTOCKPHOTO.COM/&id=17487&adid=P1E2






COLUMNS&DEPARTMENTS •4 FROM THE EDITOR’S DESK

6 PRESIDENT’SMESSAGE

9 NEWS AND VIEWS

12 COMPETITIONS

14 ROS TOPICS

16 TC SPOTLIGHT

19 FROM THE GUEST EDITORS

104 STUDENT’S CORNER

106 SOCIETY NEWS

114 HISTORY

120 WOMEN IN ENGINEERING

126 CALENDAR

128 TURNING POINT

A Publication of the IEEE ROBOTICSANDAUTOMATION SOCIETY

Volume 19, No. 3 September 2012 ISSN 1070-9932 http://www.ieee-ras.org/ram

EDITORIAL BOARDEditor-in-ChiefPeter Corke ([email protected])School of Electrical Engineering andComputer ScienceQueensland University of TechnologyBrisbane, Australia

Associate EditorsRaffaella CarloniUniversity of Twente(The Netherlands)

You-Fu LiCity University of Hong Kong

Srikanth SaripalliArizona State University (USA)

Yu SunUniversity of South Florida

Bram VanderborghtVrije Universitaet Brussel (Belgium)

Loredana ZolloUniversit�a Campus Bio-Medico, Roma

Past Editor-in-ChiefStefano StramigioliUniversity of Twente (The Netherlands)

Industry EditorRaj MadhavanUniversity of Maryland College Park

Video EditorJonathan RobertsCSIRO (Australia)

Web EditorBram VanderborghtVrije Universitaet Brussel (Belgium)

COLUMNSResearch/Industry News: Jeanne DietschAdept Mobile Robots, Inc. (USA)

Competitions:William Smart([email protected])Washington University

From the Editor’s Desk: Peter Corke

ROS Topics: Steve CousinsWillow Garage (USA)

On the Shelf: Alex SimpkinsRDP Robotics (USA)

Education: Andreas BirkJacobs University (Germany)

Student Corner: Laura MargheriScuola Superiore Sant’Anna

IFRR (International Foundation ofRobotics Research): Oussama KhatibStanford University

Turning Point: Peter CorkeEditor-in-Chief

Vice President,Publication ActivitiesAlessandro De LucaDip. di Ing. Inform., Autom., Gest.Univ. of Roma “La Sapienza”

RAM homepage:http://www.ieee-ras.org/ram

Robotics and AutomationEditorial AssistantRachel O. [email protected]

Advertising SalesSusan SchneidermanBusiness Development ManagerTel: +1 732 562 3946Fax: +1 732 981 [email protected]

IEEE PeriodicalsMagazines DepartmentDebby NowickiManaging Editor([email protected])

Janet DudarSenior Art Director

Gail A. SchnitzerAssistant Art Director

Theresa L. SmithProduction Coordinator

Felicia SpagnoliAdvertising Production Manager

Peter M. TuohyProduction Director

Dawn M. MelleyEditorial Director

Fran ZappullaStaff Director,Publishing Operations

IEEE-RAS Membershipand Subscription Information:+1 800 678 IEEE (4333)Fax: +1 732 463 3657http://www.ieee.org/membership_services/membership/societies/ras.html

IEEE prohibits discrimination, harassment, and bullying. For more information, visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html.

©ISTOCK

PHOTO.COM/©

ANDREJS

ZAVADSKIS

IEEE Robotics & Automation Magazine (ISSN 1070-9932) (IRAMEB) is published quarterly bythe Institute of Electrical and Electronics Engineers, Inc. Headquarters: 3 Park Avenue, 17th Floor,New York, NY 10016-5997 USA, Telephone: +1 212 419 7900. Responsibility for the content restsupon the authors and not upon the IEEE, the Society or its members. IEEE Service Center (fororders, subscriptions, address changes): 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ 08855 USA.Telephone: +1 732 981 0060. Individual copies: IEEE members $20.00 (first copy only), non-mem-bers $104.00 per copy. Subscription rates: Annual subscription rates included in IEEE Robotics andAutomation Society member dues. Subscription rates available on request. Copyright and reprintpermission: Abstracting is permitted with credit to the source. Libraries are permitted to photocopy

beyond the limits of U.S. Copyright law for the private use of patrons 1) those post-1977 articlesthat carry a code at the bottom of the first page, provided the per-copy fee indicated in the code ispaid through the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923 USA; 2)pre-1978 articles without a fee. For other copying, reprint, or republication permission, write Copy-rights and Permissions Department, IEEE Service Center, 445 Hoes Lane, Piscataway, NJ 08854.Copyright @ 2012 by the Institute of Electrical and Electronics Engineers Inc. All rights reserved.Periodicals postage paid at New York and additional mailing offices. Postmaster: Send addresschanges to IEEE Robotics & Automation Magazine, IEEE, 445 Hoes Lane, Piscataway, NJ 08854USA. Canadian GST #125634188 PRINTED IN THE U.S.A.

•


2 • IEEE ROBOTICS & AUTOMATIONMAGAZINE • SEPTEMBER 2012


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


______________________

__________________

___

_____

____

____

____

____

__

______

__________

__________

_________

_________

http://www.qmags.com/clickthrough.asp?url=http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html&id=17487&adid=P2E4

http://www.qmags.com/clickthrough.asp?url=http://www.ieee.org/membership_services/membership/societies/ras.html&id=17487&adid=P2E3



http://www.qmags.com/clickthrough.asp?url=http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html&id=17487&adid=P2E4

http://www.qmags.com/clickthrough.asp?url=www.sfiprogram.org&id=17487&adid=P2E6












M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________________________

http://www.qmags.com/clickthrough.asp?url=www.youbot-store.com&id=17487&adid=P3A1





FROM THE EDITOR’S DESK •

Farewells andWelcomesBy Peter Corke

Welcome to the Septemberissue. In the last issue, Ibid farewell to Roz Snyder,and in this issue, I would

like to welcome Rachel O. Warnick,who is now the magazine’s editorialassistant as well as IEEE Robotics andAutomation Society (RAS) societyadministrator. Rachel’s second day onthe job was the magazine’s editorialboard meeting at the IEEE Interna-tional Conference on Robotics andAutomation (ICRA), something of abaptism by fire. Rachel, with a littleguidance from Roz, has helped buildthis issue, and together we will work toimprove and evolve the magazine.

There are a few other changes tothe magazine around columns. JeanneDietsch has moved on to new things,and we will miss her insightful com-mentary on the robotics industry. BillSmart, who launched our competitionscolumn, is also moving on and will bereplaced by Steve Balakirsky from theNational Institute of Standards andTechnology (who chairs the RAS Com-petitions and Challenges committee).

It is now that time of year when weseek new associate editors to join themagazine team. Themagazine has a rel-atively small board, just six associateeditors, while our big brothers, IEEETransactions on Robotics and IEEETransactions on Automation Scienceand Engineering, have a multilevelboard with editors and associate editors.There are numerous benefits of servingon an editorial board. First, you get tohave some real influence over the direc-tion of the publication through deci-

sions on special issues andby recruiting articles for thepublication. This can anddoes have a direct effect onthe number of people whoread and cite the publication.Second, and particularly useful ifyou are an academic, you get to see firsthand how the review process reallyworks. You need to find and remindreviewers to assess the manuscript andultimately make a crisp recommenda-tion to the editor or editor-in-chiefabout the manuscript. For the maga-zine, a significant number of articles fallinto the “nearly but not quite” category,more formally “conditional accept,”which can lead to several iterations ofthe article going back and forth betweenthe authors and the associate editor,and their joint effort leads to higherquality articles. For the magazine, anassociate editor might handle 12 articlesper year, generally within the technicalareas that they nominate. If you likereading the magazine and want to helpit in its mission of providing substan-tive, readable technical articles aboutthe practice of robotics and automationthen please apply. The appointment isfor a three-year period (2013–2015)with an option of extension, and thedetails are on page 110.

In recent editorials, Ihave remarked on the pass-ing of several pioneers inour field and about our Soci-ety’s robotics history project.

I am very happy to includethe first fruit of that activity

with an article in this issue on the lifeand achievements of George Devol.

The topic of this special issue isaerial robotics and the quadrotor plat-form. This is really a hot topic inrobotics at the moment, and we haveseen some amazing results over thelast few years. Robert Mahony andVijay Kumar have assembled a greatcollection of articles from the largenumber that were submitted, and thesenicely cover the underlying theory aswell as a diverse range of applications.

Finally, I would like to reiterate afew points. First, it is important tokeep articles short, as our goal is eightmagazine pages, and this is spelled outin detail in our online author guidelines(http://ieee-ras.org/ram/forauthors.html). Shorter articles are quicker towrite, and I can publish more of themper issue, and that helps to reduce timefrom submission to print. For all yourprojects, think about writing a shortmagazine article as well as your journalarticles. Second, articles are now goingonline on IEEE Xplore well before printpublication: see the Early Access link onthe magazine’s (IEEE Xplore page http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=100). Third, I wouldreally like to receive feedback, so ifthere is anything you like about theissue or anything that would like to seeimproved or change, please write andlet me know ([email protected]).

Enjoy the issue.Digital Object Identifier 10.1109/MRA.2012.2205616

Date of publication: 10 September 2012 Rachel and Peter at ICRA 2012.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___

______________

____

______________

http://www.qmags.com/clickthrough.asp?url=http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=100&id=17487&adid=P4E2

http://www.qmags.com/clickthrough.asp?url=http://ieee-ras.org/ram/forauthors.html&id=17487&adid=P4E1







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________________________

http://www.qmags.com/clickthrough.asp?url=www.cyberbotics.com&id=17487&adid=P5A1






PRESIDENT’S MESSAGE •

Global PerspectiveBy David E. Orin

The 2012 IEEE InternationalConference on Robotics andAutomation (ICRA) was held14–18 May in Saint Paul, Min-

nesota. It was a wonderful opportunityto be together in the capital of Minne-sota, known as the land of 10,000 lakes.

The technical and social programswere excellent, and the conferenceattracted one of the largest numbers ofattendees for ICRA throughout its morethan 25-year history. The social pro-gram was highlighted by the Past Presi-dents’ Celebration, in which all livingpast presidents of the IEEE Roboticsand Automation Society (RAS) andAntal (Tony) Bejczy from the Councilwere present (see the report later in the“Society News” column in this issue).

I would like to thank all of the con-ference organizers, especially Prof.Nikos Papanikolopoulos, general chair,Prof. Paul Oh, program chair, andProf. Lynne Parker, editor-in-chief ofthe Society’s Conference EditorialBoard, for their tireless and dedicatedefforts on behalf of the conference.

Regretfully, ICRA 2012 marked thetime when most of us said good-bye toRosalyn Snyder. She is retiring as thenewsletter editor, the editorial assist-ant for the magazine, and administra-tor for RAS. In many ways, as oneofficer remarked, Roz (as we call her)was the face of the Society. In my ownview, I believe that Roz may knowmore members of the Society thanalmost any other person in RAS.

At ICRA in St. Paul, we had anopportunity to celebrate with Roz onthe occasion of her retirement. Figure 1

shows Roz along with the last two presi-dents that she has worked with. We willsurely miss Roz for her warm personal-ity and upbeat attitude.We really appre-ciate all of her many contributions tothe Society over our 25 years together.

ICRA 2017 Site SelectionAnAdministrative Committee (AdCom)meeting was held after the conference.One of the major items of businesswas to select the site for the ICRA2017 conference in the Asia andPacific region. Two excellent siteswere proposed, and Singapore waschosen. The general chair of the con-ference will be Prof. I-Ming Chen ofNanyang Technological University,Singapore, and the program chair willbe Prof. Yoshihiko Nakamura of theUniversity of Tokyo, Japan. The siteand organizers of the ICRA are nowset through 2017 and are given inTable 1.

Global PerspectiveOne of the key principles in our ICRAsite selection these days as a Society isthe rotation of the conference amongthe three major geographical areas ofthe world:l the Americas (IEEE Regions 1–7, 9)l Europe, the Middle East, and Africa

(IEEE Region 8)l Asia and Pacific (IEEE Region 10).

This aligns with one of our corevalues as a Society—global perspec-tive: welcoming membership andleadership from all regions of theworld. This core value was articulatedby the Long-Range Planning Com-mittee during the past two years,along with other core values upheldby the IEEE: inclusiveness, network-ing, professionalism, technical excel-lence, and volunteerism.

By rotating ICRA among the regionsof the world, we believe that this bettersupports our global membership as a


Date of publication: 10 September 2012

Figure 1. From left: Kazuhiro Kosuge, past president; David Orin, president; andRosalyn Snyder celebrating her retirement at ICRA 2012.


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_____________________________________

__________________

http://www.qmags.com/clickthrough.asp?url=www.robotis.com&id=17487&adid=P7A1








Society. For ICRA 2012, more than onehalf of the papers presented were fromregions outside of North America. Inparticular, about one third were fromcountries in Europe, the Middle East,and Africa, and nearly one fourth werefromAsia and Pacific.

We also apply the same rotationprinciple to our largest cosponsoredconference IEEE/Robotics Society ofJapan (RSJ) International Conference onIntelligent Robots and Systems (IROS).The conference is cosponsored with theIEEE Industrial Electronics Society, RSJ,the Society of Instrument and ControlEngineers (SICE), New TechnologyFoundation, and Institute of Control,Robotics, and Systems in Japan. Thelocations selected by the IROS SteeringCommittee for the six years up to andincluding 2016 are given in Table 2. In

addition, our flagship conference in theautomation area—IEEE InternationalConference on Automation Science andEngineering (CASE)—also rotates outof North America for two of three years,as shown in Table 3 for CASE.

In addition to rotating our majorconferences around the regions of theworld, 12 of our 18 elected AdCommembers are set to equally representthe three geographical areas. Thisautumn (in the Northern hemi-sphere), four of the six members whoare elected to the AdCom for athree-year term (2013–2015) will bedesignated for specific regions, whilethe other two elected will be at large.Also, since Toshio Fukuda becamethe first president elected from out-side the United States, we have rotatedour presidents among the three areas:

Toshio Fukuda (1998–1999, Japan),T.C. Steve Hsia (2000–2001, USA),Paolo Dario (2002–2003, Italy),Kazuo Tanie (2004–2005, Japan),Richard Volz (2006–2007, USA),Bruno Siciliano (2008–2009, Italy),Kazuhiro Kosuge (2010–2011, Japan),David Orin (2012–2013, USA), andRaja Chatila (2014–2015, France).

All of these measures have helpedus to achieve a better global per-spective to support our membershipworldwide. As I mentioned in a previ-ous message, the international charac-ter of the Society has really enrichedboth my professional and personallife, and I believe that it has for manyof you as well. As we move forwardtogether, I hope that we would alluphold this core value in everythingthat we do as a Society.

•Table 2. Sites and organizers of the IROS conferences.

IROS Location General Chair Program Chair

2011 San Francisco, California, USA Oussama Khatib Gaurav Sukhatme

2012 Vilamoura, Algarve, Portugal Anibal T. de Almeida/Urbano Nunes Eugenio Guglielmelli

2013 Tokyo, Japan Shigeki Sugano Makoto Kaneko

2014 Chicago, Illinois, USA Kevin Lynch Lynne Parker

2015 Hamburg, Germany Jianwei Zhang Wolfram Burgard

2016 Daejeon, Korea Il-Hong Suh Dong-Soo Kwon

•Table 3. Sites and organizers of the CASE conferences.

CASE Location General Chair Program Chair

2009 Bangalore, India Yadati Narahari Spyros Reveliotis

2010 Toronto, Canada JohnWen Yu Sun

2011 Trieste, Italy Maria Pia Fanti Alessandro Giua

2012 Seoul, Korea Hyouk Ryeol Choi Nak Young Chong

2013 Madison, Wisconsin, USA Leyuan Shi Lawrence Holloway

2014 Taipei, Taiwan Ren C. Luo Fan-Tien Cheng

•Table 1. Sites and organizers of ICRAs.

ICRA Location General Chair Program Chair

2012 St. Paul, Minnesota, USA Nikos Papanikolopoulos Paul Oh

2013 Karlsruhe, Germany R€udiger Dillmann Markus Vincze

2014 Hong Kong, China Ning Xi William Hamel

2015 Seattle, Washington, USA Lynne Parker Nancy Amato

2016 Stockholm, Sweden Danica Kragic Antonio Bicchi

2017 Singapore I-Ming Chen Yoshihiko Nakamura



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






NEWS AND VIEWS•

DARPAEnticesRoboticists to Take theNext StepBy Jeanne Dietsch

After three years, this is my last“News and Views” column.When I began writing in2009, the Defense Advanced

Research Projects Agency (DARPA)Grand Challenges were history. Tech-niques they showcased were beginningto creep into the designs of commercialvehicles. Now, Mercedes is offering anautonomously driving car, and DARPAhas issued a new challenge. DARPA isoffering US$2 million to creators of arobot that can drive an open-frameutility vehicle to a building in need ofrepairs, find a leaking pipe, and fix it, asshown in Figure 1. This is not a pipedream. Even under supervised auto-nomy, it illustrates the incredibleprogress of robotics since the firstGrand Challenge was announced adecade ago. DARPA has budgetedUS$34million to the new Rescue Chal-lenge, including government furnishedequipment (GFE). What follows is thespeculation about what might help towin the Rescue Challenge, intertwinedwith other robotics news.

One of the greatest difficulties forrobots operating under supervisedautonomy is to maintain communica-tion with the base station. UltraElectronics Maritime Systems Inc.may have the long-sought answerwith its MAGneto-InductivE commu-nications link. Maggie, as the PioneerAT robot and its payload are called(Figure 2), was able to drive in andsend video from a cave 30 m belowground to a remote station above.This technology may be part of theDARPA Rescue Challenge winner.

Which mechatronics will be chosenfor GFE torso and legs? BostonDynamics announced a successor toPetman named Atlas (Figure 3). Atlasshould have the compact energysource and necessary range of motion.The earlier android was cabled, butAtlas is designed to be independent.

Could Atlas carry theDARPA auton-omous robotic manipulation (ARM)

system designed by Dean Kamen andteam (Figure 4)? ARM might be strongenough to serve as a pipe wrench itself,thanks to the advantage of a rotatingwrist. Osaka University has beenexperimenting with other advantagesof rotating appendages. Their omni-finger is a spiral-wrapped finger thatmoves a spherical object sideways asthe finger rotates in either direction.




Figure 1. DARPA envisions robots responding to minimize disasters such asFukushima. (Photo courtesy of DARPA.)

Figure 2. Maggie, MAGneto InductivE communications link lets Pioneer AT driveunderground. (Photo courtesy of Ultra Electronics Maritime Systems.)


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Another new approach to grippingis being investigated by SRI Inter-national spin-off Grabit. Grabit’spatented electrostatic flexible padstemporarily adhere with many mate-rials. Like OctoMotion graspers,which imitate octopus and squidmanipulation, Grabit is trying to solvethe problem of efficiently manipulat-ing unpredictably shaped objects,such as fruit. It will be interesting tosee whether the winner of DARPA’s

Rescue Challenge copies biologicalforms or breaks the mold to ac-complish the same result in a moremechanically unique manner.

Ken Schweller of the Great ApeTrust in Des Moines, Iowa, has takenspecies imitation to a new level withhis RoboBonobo remote-handling bot(Figure 5). Schweller plans to useRoboBonobo for apes to find and com-municate with their keepers and tospray people with water cannons as

part of an interspeciesempathy experiment.

Whether such workleads to a Planet ofthe Apes or to an IRobot scenario awaitsto be seen. Still, incountries like theUnited States, wherelabor costs are high,androids are about tomake an appearanceat least from the waistup in factories. Heart-land Robotics hasreserved a booth thesame size as Mitsu-bishi Electric at Auto-matica 2013, a factoryautomation trade show.There they will dem-onstrate and, hope-fully, offer for salethe intelligent torsosdesigned to worksafely side by sidewith people withoutthe cage now requiredaround industrial ro-botic arms. Safe armslike these will also

be needed in the DARPA RescueChallenge.

Is there a demand for robotic armswith the added cost of compliance andmore sensing outside military research?Meka Robotics, Willow Garage, and SRIInternational are betting on it. The threehave combined forces under the nameRedwood Robotics to design a safe, inex-pensive, and coworker robot. However,Redwood might focus less on the intelli-gent learning andmore on an easy-to-useinterface for people training the robot.

Intelligent robotics has alreadyproven to be reliable, cost-cutting in-dustrial partners. The flexible mobile-robotic storage systems made by Kivahave transformed warehouse-pickingsystems so that online shopping giantAmazon picked up the entire com-pany for US$775 million.

With DARPA leading the way, therobotics industry is striding toward abrave new world of software-driven elec-tromechanical agents. The results of suchprojects are not predictable as the Stux-net/Flame debacle recently demonstrated.

Over the last four years, I haveexhorted readers to think of the impactsand implications of their work withinthe background of the Internet andpowerful actors on the world stage. Ihave asked you to help nontechnicaldecision makers to understand theseeffects and possibilities. Please remem-ber always: they cannot make it happenwithout you. I hope you will be braveenough and apply enough foresight topressure the system toward good.

Best wishes to all my readers andcolleagues at IEEE Robotics & Automa-tionMagazine! I will miss writing for you.

•

Figure 3. Boston Dynamics is building anupgrade of Petman without cables.(Photo courtesy of Boston Dynamics.)

Figure 4. DARPA’s ARM combines powerwith compliance. (Photo courtesy ofDARPA.)

Figure 5. RoboBonobo will let apesexercise remote control over their habitats.(Photo courtesy of Great Ape Trust.)



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________________

http://www.qmags.com/clickthrough.asp?url=www.reflexxes.com&id=17487&adid=P10A1






M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


____________________

http://www.qmags.com/clickthrough.asp?url=www.ieee.org/tryieeexplore&id=17487&adid=P11A1





COMPETITIONS •

ICRARobot ChallengeBy Bill Smart

In the last “Competitions” column, Iposed a question: Should we, as afield and as individual roboticists,be focusing on competitions, chal-

lenges, or journal papers? Thanks toeveryone who sent in their thoughts.The overwhelming opinion that Ireceived was that competitions andchallenges were a good thing for thefield and gave us the chance (and theobligation) to demonstrate our workas part of a complete integrated sys-tem, but that there were still someproblems with incentives. While com-petitions might be a good thing, theyare not often well rewarded, at least inthe traditional academic sense. Forexample, they generally count littlefor promotion and tenure at a univer-sity. However, it was pointed out thatthese events are great showcases forstudents and their work, giving thema venue to impress future employersand colleagues. The key question formany seemed to be how we can makeparticipation in these events moreattractive and valuable (from a careerstandpoint) for people in the roboticscommunity.

Despite these open questions,competitions and challenges seem tobe thriving at our annual conferences.The IEEE International Conferenceon Robotics and Automation (ICRA)Robot Challenge 2012 held in St. Paulfeatured five individual events,28 teams, and approximately 100 par-ticipants. There will be two com-petitions at the IEEE InternationalConference on Intelligent Robots and

Systems (IROS) in Vilamoura thisOctober. One involves the Tower ofHanoi problem, famous from artificialintelligence textbooks. Robots willcompete to solve a real, physical ver-sion of this problem, dealing autono-mously with all the sensing, planning,manipulation, and navigation prob-lems they encounter. The secondevent at IROS will be the new Robo-Cup@Work competition. This eventfocuses on using robots for tasks inthe workplace, such as loading andunloading containers, delivery, coop-erative assembly and transportation,and the operation of other machines(by pressing buttons, pulling levers,and using other controls designed forhumans). These new events lookexciting, and it is particularly nice tosee RoboCup making an appearanceat IROS. More details on the upcom-ing IROS competitions can be foundat the IROS 2012 Web site: www.iros2012.org/.

The IEEE Robotics and Automa-tion Society (RAS) Competitions andChallenges committee met at ICRA2012. The discussion of future plansincluded the creation of a “bestpractices” document to make it easierfor event chairs (and the overall chal-lenge chair) to get an event up andrunning, generate publicity, recruitparticipants, and understand how thechallenge interacts with the rest ofICRA. This will make it easier fornew chairs to create and run newchallenges within the ICRA RobotChallenge framework and will helpto ensure the long-term sustainabilityof the challenge. If you have anidea for a new challenge, either atICRA or elsewhere, I encourage you

to get in touch with the competi-tions and challenges chair, StephenBalakirsky ([email protected]), to talkabout it.

Finally, this will be the last “Com-petitions” column with me as the edi-tor. Starting in the next issue, StephenBalakirsky will take over as editor ofthe column. Steve is the chair of theRAS Committee of Competitions andChallenges and a past chair of theIRCA Robot Challenge, and I knowI leave the column in good hands.I have enjoyed helping to get thiscolumn started and want to thankeveryone who graciously agreed towrite pieces for it or who sentfeedback.

2012 Invention andEntrepreneurship in Roboticsand AutomationThe RAS and the InternationalFederation of Robotics (IFR) jointlysponsor the Invention and Entrepre-neurship Award, which highlightsand the honors the achievementsof the inventors with value-creatingideas and entrepreneurs who pro-pel those ideas into world-classproducts.

The cowinners of the eighth an-nual Invention and EntrepreneurshipAward were Gino De-Gol of Robo-coaster Ltd. and Esben H. Østergaardof Universal Robots. The award in-cluded a US$2,000 prize and a plaqueshared by the winners and waspresented during the IERA Network-ing Dinner at the AUTOMATICA inMunich.

“The two winners are excellentexamples of the exciting pioneeringwork that is being done in advanced





M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________

____

____________

http://www.qmags.com/clickthrough.asp?url=www.iros2012.org/&id=17487&adid=P12E1






robotics and is now being commer-cialized into real-world applicationsin emerging markets,” explained JohnDulchinos, Adept Technology andmember of the Awards Committee.“The important advancement in saferobots for the entertainment industryby RoboCoaster and the developmentof a new generation of simple robotsfor SME’s by Universal Robots notonly represent positive steps forwardfor the robotics industry but alsoreflect businesses that have success-fully translated technology intocommercial success.”

Raj Madhavan, vice president ofRAS Industrial Activities Board andchair of the Awards Committeeadded, “Esben Østergaard, UniversalRobots, presented a flexible, light, andeasy-to-use six-axis industrial robotarm that will enable small andmedium enterprises to compete effi-ciently in the marketplace. GinoDe-Gol received the award for pavingthe ground of using industrial robotsystems in entertainment, motionsimulation, and medical applications,in particular for establishing newbusiness opportunities for industrialrobots by developing RoboCoastersystems for various uses in the enter-tainment industry and theme parkswith a perfect safety record.”

“I was very pleased to have beenselected as a finalist for the IERAAward, especially as the other finalists’

submissions were so strong. It reallyunderpins how this prestigious awardhas the global power to draw excitingapplications,” stated Gino De-Gol.

“Finally winning the IERA Awardwas an incredibly proud moment forme, and I share it with all my col-leagues and associates who havesupported both mine and RoboCoas-ter’s journey.”

“We are very happy about the IERAAward. This award recognizes thecommon achievements of all employ-ees of Universal Robots as of our global75 distributers. Moreover, the awarddemonstrates that our vision of a flexi-ble, user-friendly, and secure robotbecame true: The UR5 is commerciallysuccessful and optimizes the produc-tions, especially of smaller andmedium-sized companies, for which the heavyand expensive automation technol-ogy was not in question so far. Thismakes me proud—also as scientistin the field of robotics,” explainedEsben Østergaard.

•

From left: Shinsuke Sakakibara (IFR president), Gino De-Gol (RoboCoaster Ltd),Esben Østergaard (Universal Robots Aps), and Raj Madhavan (vice president ofIEEE RAS Industrial Activities Board) at the IERA award presentation.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


____________

http://www.qmags.com/clickthrough.asp?url=www.ati-ia.com/ras&id=17487&adid=P13A1





ROS TOPICS •

Milestones: First ROSconandOSRFBy Steve Cousins and Brian Gerkey

In May of this year, there were twomajor milestones in the evolutionand growth of the robot operatingsystem (ROS): the first ROS confer-

ence (ROScon) and the formation ofthe Open Source Robotics Foundation(OSRF). In ten years, we will lookback fondly at ROScon 2012 as thefirst time the ROS community reallygelled in person. And, I predict, tenyears from now, we will all rely onOSRF to support the core robot soft-ware in the growing number of robotsin the world. Let us look at these twomilestones in detail.

ROSconThe ROS community has grown largeenough to have its own conference.Following the IEEE International Con-ference on Robotics and Automation(ICRA) in St. Paul this in May 2012,more than 200 ROS developers andusers gathered to get to know oneanother, to share tips and techniques,and to learn about the past, present,and future of ROS. ROScon was a sin-gle-track conference, with an adjacentspace for demonstrations and hacking,which featured various ROS-controlledrobots, including many variations onTurtleBot (Figure 1). The audience wasbalanced between academics and ROSdevelopers from industry (Figure 2).

All of the talks from ROScon arefreely available online at http://roscon.ros.org and collectively provide a newway to learn about and get started withROS. Morgan Quigley kicked off theconference with a retrospective onwhere ROS came from and where it

could be going. He introduced theterm wild code to refer to code that isnot and may never be released. It isgood to make the wild code availableas open source, he argued, even if it isnot supported or documented, so thatfuture developers can learn from pastwork. Later in the conference, Ben Pitzertook up the theme of code quality, intro-ducing a set of automatic metrics thatcan be used to judge and manage thequality of the released code base in ROS.

A series of longer talks introducedcore ROS concepts. ROS can be usedon different robots in part because ofthe unified robot description format(URDF), and David Lu’s talk is anexcellent introduction to URDF. IoanSucan and Sachin Chitta introducedthe ROS motion planning stacks anddiscussed the motivation for the migra-tion of the Arm Navigation stacks to

MoveIt. John Hsu and Nate Koenigintroduced the Gazebo Simulator(more on that in the OSRF section).And finally, Tully Foote talked aboutthe ROS transform system.

One of the best things about ROS isthe community, which is coordinatedthrough the ROS wiki at www.ros.org.Melonee Wise reminded the audienceof the value of the wiki and encouragedeveryone in the community to contrib-ute to it directly. Bill Smart talkedabout his experience using ROS inteaching a mobile robotics course,using TurtleBots as the primary plat-form, but with some students using aPR2 for the exercises.

ROS runs on more and more robotsall the time. Chad Rockey gave a talk onwriting hardware drivers, the low-levelsoftware that connects new devices toROS. One of the most popular recentdevices for robots is the Primesense 3-D vision system (found in theMicrosoftKinect and the Asus Xtion devices),and Patrick Mihelich described thestate of ROS drivers for those devices.Michael Carroll described the opensound control system for hardware andsoftware, which he used to demonstrateteleoperation of a ROS-controlled robotfrom an iPhone.

ROS is being ported to a variety ofplatforms. Daniel Stonier talked aboutwhat currently works on Windowsand described the work that is goingon to make that port more complete.Sarah Osentoski’s talk, given by BenPitzer, described tools for using theWeb to interact with robots that arerunning ROS. With Android devicesin the mix, rosjava, presented byDamon Kohler, is allowing the usersto run more and more on tablets




Figure 1. TurtleBots in the exhibit area ofROScon.


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


http://www.qmags.com/clickthrough.asp?url=http://roscon.ros.org&id=17487&adid=P14E2

http://www.qmags.com/clickthrough.asp?url=www.ros.org&id=17487&adid=P14E1





[even laser-based simultaneous local-ization and mapping (SLAM)].

Although ROS was developed foruse on one particular personal robot, itis being used broadly in the roboticscommunity. Jeff Rousseau describedvarious techniques for using ROS onmultirobot teams. Jonathan Gammelland Chi Hay Tong shared their experi-ences using ROS on field robots indemanding environments. ArminHornung described the adaptationsneeded to use ROS with bipedalhumanoid robots. A particularly inter-esting humanoid robot, Robonaut 2, iscurrently in the field onboard theinternational space station, and thesecond day’s keynote by Stephen Hartmotivated the shift NASA is making toROS and Orocos on that robot. Finally,William Woodall described Moe, anautonomous lawnmower that can helptake care of the field.

ROScon was a rousing success. Theturnout was larger than the organizersdared to hope. More important, thetalks will have value to a much largeraudience than the attendees andprovide a reference and valuable intro-duction to ROS that remains usefullong after the conference attendeeshave gone home. At the conference,people were focused on the technicalelements of ROS, but the question oneveryone’s mind was “what is this newopen source robotics foundation?”

OSRFOSRF is a nonprofit entity set up tosupport the development, distribution,and adoption of open-source softwarefor use in robotics research, educa-tion, and product development. It was

created to be the neutral steward ofthe community to make it clear thatROS is neither owned nor controlledby Willow Garage or any othercompany but rather is a communityartifact that we can all contribute toand take advantage of. Companiesand governments can contribute tothe organization OSRF to help main-tain and improve software in the ROS

ecosystem. Willow Garage with itshistory of helping to create ROS andmaking significant investments in it isa founding sponsor of OSRF.

OSRF was founded with a grantfromWillow Garage but will grow andthrive over the next few years, thanksto U.S. government funding that willsupport improvement of core ROScomponents, starting with the Gazebosimulator. With a larger team workingon Gazebo, we look forward to morepowerful and accessible simulationcapabilities for the entire community.

The Foundation’s office opened inJuly 2012 in Mountain View, Califor-nia, and is led by Brian Gerkey. Lookfor them online at www.osrfoundatio-n.org or see their work, of course, atwww.ros.org. If you are interested incollaborating with or supportingOSRF, e-mail [email protected].

•

Figure 2. ROScon participants.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________

_____________

____

___________________________

http://www.qmags.com/clickthrough.asp?url=www.ros.org&id=17487&adid=P15E2

http://www.qmags.com/clickthrough.asp?url=www.osrfoundation.org&id=17487&adid=P15E1


http://www.qmags.com/clickthrough.asp?url=http://butterflyhaptics.com&id=17487&adid=P15A1





© EYEWIRE

TC SPOTLIGHT •

RecentActivities of the Safety, Security,andRescueRobotics Technical Committee

By SSRR TC Members

What Is SSRR?The IEEE Robotics and AutomationSociety (RAS) Technical Committee(TC) on Safety, Security, and RescueRobotics (SSRR), was established inFebruary 2002, shortly after thedeployment of robots at the 9/11World Trade Center collapse and sub-sequent accelerated adoption ofrobots for homeland security andpublic safety. The primary outreachactivity for the TC is to emergencyresponders, federal and local govern-ment agencies, and nongovernmentalorganizations for training and acqui-sition guidance. This is an ongoingactivity with numerous events thatinclude the annual IEEE InternationalSymposium on SSRR and the relateddemonstrations.

Symposium on SSRRThe IEEE International Symposiumon SSRR is held annually and encom-passes the design and implementationof robotics, automation, intelligent

machines, systems, anddevices that can con-tribute to civilian safety.Applications includesearch and rescue; rapidscreening of travelers andcargo; hazardous materialhandling; humanitarian demining;fire fighting; law enforcement detec-tion of chemical, biological, radiolog-ical, nuclear, or explosive risks; andresponse to natural disasters. Whilethis symposium is conceptuallycentered on a set of applicationdomains, the focus is on qualityresearch and fundamental principlesthat led to useful robots and automa-tion in all domains. The symposiumhas continuously grown over thelast four years. Table 1 presents anoverview of the statistics related tothe meeting. A unique aspect ofthis meeting is the ability forresearchers to demonstrate theirtechnology at test sites colocated withthe symposium.

RescueRobotics CampThe Rescue RoboticsCamps aim to bring to-gether graduate students

and responders to educateall regarding the current state of

the art in the field of SSRR. Severalcamps have been held, focusing onscenarios such as hurricane response,fire fighting response, tornado re-sponse, and general urban search andrescue (Figure 1).

The National Science Foundation(NSF)–Japan Science and Technol-ogy Agency (JST)–National Instituteof Standards and Technology(NIST) Workshop on Rescue Ro-botics: Rescue Robotics Camp heldin March 2010 in Disaster City,Texas, was organized by RobinMurphy. The 50 workshop par-ticipants represented 16 univer-sities from the United States, Japan,and China. Over one dozen land,marine, and aerial robots were tested




•Table 1. SSRR Symposium statistics for the last four years.

SSRR 2011 SSRR 2010 SSRR 2009 SSRR 2008

Date 1–5 November 26–30 July 3–5 November 21–24 October

Location Kyoto, Japan Bremen, Germany Denver, Colorado Sendai, Japan

General chair Fumitoshi Matsuno Andreas Birk Richard Voyles Satoshi Tadokoro

Submitted papers 80 46 42 36

Accepted papers 59 29 30 26

Tutorials 2 0 0 0

Invited talks 7 3 4 5

Panels 1 0 0 0

Participants 84 31 47 40


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






using a robot response exercise and otherscenarios.

The 2007 Rescue Robotics Camp 2007 washeld in September 2007 at Istituto SuperioreAntincendi in Rome, Italy. Daniele Nardi,Adam Jacoff, and Satoshi Tadokoro organizedthis camp. Eight teams demonstrated tech-nology, and a total of 75 people attended thecamp, including Italian firemen and localinstitutions.

Rescue Robotics ExercisesRescue Robotics Exercises bring togetherresearchers and responders in field exercises totest equipment, strategies, and techniques in amutually supportive environment that pro-vide real stressors and environmental condi-tions. The exercises have grown to includefield trials at real disasters, such as hurricanesKatrina and Ike, mudslides, building collapsesin the United States and Europe, mine collap-ses, and the earthquake and tsunami thatrocked Japan.

The 2007 Rescue Robotics Exercise was organ-ized by Daniele Nardi and held at Istituto Superi-ore Antincendi in Rome, Italy, with the RescueRobotics Camp.

Activities for Disaster Responseand Recovery in Great EasternJapan EarthquakeOn 11 March 2011, the Great Eastern JapanEarthquake occurred in the Tohoku area ofJapan, causing extensive damage, but more sig-nificantly, the resulting tsunami and radioac-tive contamination by the nuclear plant’sdisaster that followed. As emergency respond-ers to this disaster, some TC members contrib-uted to the rescue and recovery operationsusing their own robotic systems. This activityincluded:l a monitoring operation in Fukushima-Daiichi

nuclear power plant by Crawler type robots(Figure 2) deployed by Satoshi Tadokoro’sresearch group (see “ICRA 2012 Symposiumon Robotic Solutions Toward NuclearDecommission”)

l inquiry in the sea near the disaster siteby marine robots deployed by FumitoshiMatsuno and Robin Murphy’s researchgroups

l surveillance in collapsed buildings by UGVsand UAVs

l information exchange between internationalresearchers, professionals, media, local govern-ments, governmental organizations, public


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_____________

____________________________

____________________________

________________

http://www.qmags.com/clickthrough.asp?url=http://www.youtube.com/user/motionlabLLC&id=17487&adid=P17A1

http://www.qmags.com/clickthrough.asp?url=www.motionlab.com&id=17487&adid=P17A2

http://www.qmags.com/clickthrough.asp?url=www.vehiclemotion.net&id=17487&adid=P17A3


http://www.qmags.com/clickthrough.asp?url=www.vehiclemotion.net&id=17487&adid=P17A4





sectors, and private companies asan international hub

l promotion of cooperative actions(response, recovery, research anddevelopment, etc.) including NSFRAPID Program.

Goals for Next Three YearsGoals for next three years are asfollows:l Further extend and promote the

annual SSRRmeeting as a core scien-tific event for the SSRR community.

l Continue and also increase andstrengthen networking and out-reach activity to all stakeholdersincluding responders, corpo-rations, and decision-makersas follow up from the SSRRsymposium.

l Maintain and expand the TC’s aca-demic activities, including presencein journal special issues, work-shops, and special sessions at IEEEconferences.

l Increase the outreach and educa-tion activities through trainingcamps and summer schools.Formore details on the SSRRTC visit

http://tab.ieee-ras.org/committeeinfo.php?tcid=21.

•

•

ICRA 2012 Symposium on Robotic Solutions Toward Nuclear DecommissionBy Satoshi TadokoroThe work toward decommissioning of Fukushima-DaiichiNuclear Power Plant Unit 1-4 is in progress. Various robot tech-nologies are essential for success. Public offerings of applicabletechnologies as a technical catalog were open by the Ministryof Economy, Trade and Industry of Japan in March, and the sec-ond offering was closed on 6 April.On the basis of their results, this workshop was organized by

IEEE Robotics and Automation Society (RAS) technical commit-tee (TC) on Robotics and Automation in Nuclear Facilities, TCon Safety, Security and Rescue Robotics, the International Res-cue System Institute, the Center of Robotics for Extreme andUncertain Environments, and the International Research Insti-tute of Disaster Science of Tohoku University.At first, engineers of Tokyo Electric Power Company (TEPCO),

Toshiba, and Hitachi GE Nuclear Energy introduced the road-map of technical development toward the decommissioning,the necessary robotic tasks, and various robot technologiesapplied to these offerings.At the panel, the technologies necessary for the future work

were actively discussed. During the Q&A portion, the audienceand panelists discussed the current situation of the plant includ-ing radiation, required robots and equipment, design issues of

robots and power plants their robustness, and wireless commu-nication in the reactor buildings. The group concluded that RASwill support the decommissioning scientifically and technicallyand share the information worldwide.The presentation materials are available at http://www.rm.

is.tohoku.ac.jp/~tadokoro/120518Fukushima/

(a) (b)

(e) (c)

(g)(f)

(d)

(a) (b)

(e) (c)

(g)(f)

(d)

Figure 1. Examples of Rescue Robotics Camps. (a) Marine robot by Texas A&MUniversity, (b) underground inquiry by robots in IRS, (c) and (d) mine inquiry robot byTexas A&M University, and (e)–(g) 3-D mapping by robots in Jacobs University.

Figure 2. Crawler type robot introducedfor the inspection in Fukushima-Daiichinuclear power plant.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________

____________________________

http://www.qmags.com/clickthrough.asp?url=http://www.rm.is.tohoku.ac.jp/~tadokoro/120518Fukushima/&id=17487&adid=P18E2

http://www.qmags.com/clickthrough.asp?url=http://tab.ieee-ras.org/committeeinfo.php?tcid=21&id=17487&adid=P18E1





FROM THE GUEST EDITORS•

Aerial Robotics and theQuadrotorBy Robert Mahony and Vijay Kumar

Aerial robotics is a growing fieldwith tremendous civil andmilitary applications. Poten-tial applications include sur-

veying and maintenance tasks, aerialtransportation and manipulation,search and rescue, and surveillance.The challenges associated with tacklingrobotics tasks in complex, three-dimensional, indoor and outdoorenvironments bring into focus someof the limitations of accepted solu-tions to classical robotics problems insensing, planning, localization, andmapping. Moreover, the fundamentalweight and size limitations of flyingvehicles pose challenges in engineer-ing design as well as efficiency of sens-ing paradigms and control andestimation algorithms. Quadrotoraerial vehicles are one of the mostflexible and adaptable platforms forundertaking aerial research. In thesame way that the wheeled mobilerobots was the testing ground of muchof the fundamental work in roboticvehicle mobility throughout the1990s, the quadrotor platform isemerging as the fundamental researchplatform of choice for aerial roboticsresearch to investigate research prob-lems related to three-dimensionalmobility and perception. This specialissue consists of articles describingresearch on component technologiesand articles addressing systems design

and technological challenges in aerialrobotics and with respect to the quad-rotor platform in particular. This spe-cial issue brings together articles fromexperts in the field to address thetheory and practice underlying quad-rotor robots.

The first article by Mahony et al. isa tutorial on quadrotors. The authorspresent a comprehensive treatment ofthe rigid body dynamics and the aero-dynamics for these vehicles and adiscussion of algorithms required forestimating the six-dimensional poseand velocity. The article also includesa discussion of the control and plan-ning algorithms required to plan andcontrol three-dimensional motions.

The next article provides valuableinformation for educational and re-search institutions starting new quad-rotor projects. Lim et al. surveyopen-source projects on quadrotors,comparing them in terms of imple-mented control algorithms, electroniccomponents, and embedded software.Detailed comparisons between theopen-source projects are provided.

Tomi�c et al. discuss the deploy-ment of quadrotors in urban searchand rescue missions in which roboticsystems must operate autonomously.With no external infrastructure fornavigation and communication beingavailable, robotic systems are able tooperate autonomously. They discussthe flight performance, sensors, andprocessors for quadrotors constrainedin size, weight, and power.

Franchi et al. discuss the challengeswith teleoperating quadrotors. Theydescribe the modeling, control, andexperimentation with human opera-tors interacting with a remote fleet ofsemiautonomous unmanned aerialvehicles thatmain-tain formationconstraints andavoid collisionswith obstacleswhile collectivelyfollowing the hu-man commands.

Palunko et al.address applica-tions related totransportation.Specifically, theydevelop control-lers and trajec-tory planners thatenable a quadro-tor to transportsuspended pay-loads while adapt-ing to changes inthe mass and inertia, minimizingthe swinging of the payload. Thiswork has application in emergenceresponse, search and rescue missions,and construction operations.

We hope that you enjoy this specialissue dedicated to quadrotors. We aregrateful to the authors, anonymousreviewers, and editor-in-chief for allthe support in making this issuepossible.




•The quadrotor

platform is emerging

as the fundamental

research platform of

choice for aerial

robotics research to

investigate research

problems related to

three-dimensional

mobility and

perception.

•


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






•By Robert Mahony,Vijay Kumar,and Peter Corke

20 • IEEE ROBOTICS & AUTOMATIONMAGAZINE • SEPTEMBER 2012 1070-9932/12/$31.00ª2012IEEE


Date of publication: 27 August 2012

This article provides a tutorialintroduction to modeling, es-timation, and control formulti-rotor aerial vehicles that includesthe common four-rotor or quad-rotor case.

Aerial robotics is a fast-growingfield of robotics and multirotor air-craft, such as the quadrotor (Fig-ure 1), are rapidly growing inpopularity. In fact, quadrotor aerialrobotic vehicles have become astandard platform for roboticsresearch worldwide. They alreadyhave sufficient payload and flightendurance to support a number ofindoor and outdoor applications,and the improvements of batteryand other technology is rapidlyincreasing the scope for commercialopportunities. They are highly ma-

neuverable and enable safe andlow-cost experimentation in mapping,

navigation, and control strategies forrobots that move in three-dimensional

(3-D) space. This ability to move in 3-Dspace brings new research challenges com-

pared with the wheeled mobile robots thathave driven mobile robotics research over the

last decade. Small quadrotors have been demon-strated for exploring and mapping 3-D environ-

ments; transporting, manipulating, and assemblingobjects; and acrobatic tricks such as juggling, balancing,

and flips. Additional rotors can be added, leading to general-izedN-rotor vehicles, to improve payload and reliability.

Modeling, Estimation,

and Control of Quadrotor

©ISTOCK PHOTO.COM/© ANDREJS ZAVADSKIS


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________





http://www.qmags.com/clickthrough.asp?url=www.istockphoto.com&id=17487&adid=P20E1

This tutorial describes the fundamentals of the dynamics,estimation, and control for this class of vehicle, with a biastoward electrically powered micro (less than 1 kg)-scalevehicles. The word helicopter is derived from the Greekwords for spiral (screw) and wing. From a linguistic perspec-tive, since the prefix quad is Latin, the term quadrotor ismore correct than quadcopter and more common than tet-racopter; hence, we use the term quadrotor throughout.

Modeling of Multirotor VehiclesThe most common multirotor aerial platform, the quadro-tor vehicle, is a very simple machine. It consists of fourindividual rotors attached to a rigid cross airframe, asshown in Figure 1. Control of a quadrotor is achieved bydifferential control of the thrust generated by each rotor.Pitch, roll, and heave (total thrust) control is straightfor-ward to conceptualize. As shown in Figure 2, rotor i rotatesanticlockwise (positive about the z axis) if i is even andclockwise if i is odd. Yaw control is obtained by adjustingthe average speed of the clockwise and anticlockwise rotat-ing rotors. The system is underactuated, and the remainingdegrees of freedom (DoF) corresponding to the transla-tional velocity in the x-y plane must be controlled throughthe system dynamics.

Rigid-Body Dynamics of the AirframeLet f~x,~y,~zg be the three coordinate axis unit vectorswithout a frame of reference. Let {A} denote a right-handinertial frame with unit vectors along the axes denotedby f~a1,~a2,~a3g expressed in {A}. One has algebraicallythat~a1 ¼~x,~a2 ¼~y,~a3 ¼~z in {A}. The vector r ¼ (x, y, z) 2fAg denotes the position of the center of mass of the vehicle.Let {B} be a (right-hand) body fixed frame for the airframewith unit vectors f~b1,~b2,~b3g, where these vectors are theaxes of frame {B} with respect to frame {A}. The orientationof the rigid body is given by a rotation matrix ARB ¼R ¼ ½~b1,~b2,~b3� 2 SO(3) in the special orthogonal group.One has~b1 ¼ R~x, ~b2 ¼ R~y, ~b3 ¼ R~z by construction.

We will use Z-X-Y Euler angles to model this rotation,as shown in Figure 3. To get from {A} to {B}, we first rotateabout a3 by the the yaw angle, w, and we will call this inter-mediary frame {E} with a basis f~e1,~e2,~e3g where ~ei isexpressed with respect to frame {A}. This is followed by arotation about the x axis in the rotated frame through theroll angle, /, followed by a third pitch rotation about thenew y axis through the pitch angle h that results in the

body-fixed triad f~b1,~b2,~b3g

R ¼cwch� s/swsh �c/sw cwshþ chs/swchswþ cws/sh c/cw swsh� cwchs/

�c/sh s/ c/ch

0@

1A,

where c and s are shorthand forms for cosine and sine,respectively.

Let v 2 fAg denote the linear velocity of {B} withrespect to {A} expressed in {A}. Let X 2 fBg denote the

angular velocity of {B} with respect to {A}; this timeexpressed in {B}. Let m denote the mass of the rigid object,and I 2 R33 3 denote the constant inertia matrix (expressedin the body fixed frame {B}). The rigid body equations ofmotion of the airframe are [2] and [3]

_n ¼ v, (1a)

m _v ¼ mg~a3 þ RF, (1b)

_R ¼ RX3 , (1c)

I _X ¼ �X3 IXþ s: (1d)

The notationX3 denotes the skew-symmetric matrix, suchthat X3 v ¼ X3 v for the vector cross product 3 and anyvector v 2 R3. The vectors F, s 2 fBg combine the princi-pal nonconservative forces and moments applied to thequadrotor airframe by the aerodynamics of the rotors.

Dominant AerodynamicsThe aerodynamics of rotors was extensively studied duringthe mid 1900s with the development of manned helicop-ters, and detailed models of rotor aerodynamics are avail-able in the literature [4], [5]. Much of the detail about theseaerodynamic models is useful for the design of rotorsystems, where the whole range of parameters (rotor

x

yz

{B}

T1

T2

T3

T4d

Front

Φi

Figure 2. Notation for quadrotor equations of motion. N ¼ 4;Ui

is a multiple of p=4 (adapted with permission from [1]).

Figure 1. A quadrotor made by Ascending Technologies withVICON markers for state estimation.

MICHAELSHOMIN,CMU



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






geometry, profile, hinge mechanism, and much more) arefundamental to the design problem. For a typical roboticquadrotor vehicle, the rotor design is a question for choos-ing one among five or six available rotors from the hobbyshop, and most of the complexity of aerodynamic model-ing is best ignored. Nevertheless, a basic level of aerody-namic modeling is required.

The steady-state thrust generated by a hovering rotor(i.e., a rotor that is not translating horizontally or verti-cally) in free air may be modeled using momentum theory[5, Sec. 2.26] as

Ti :¼ CTqAri r2i -

2i , (2)

where, for rotor i, Ari is the rotor disk area, ri is the radius,-i is the angular velocity, CT is the thrust coefficient thatdepends on rotor geometry and profile, and q is the densityof air. In practice, a simple lumped parameter model

Ti ¼ cT-2i (3)

is used, where cT > 0 is modeled as a constant that can beeasily determined from static thrust tests. Identifying thethrust constant experimentally has the advantage that itwill also naturally incorporate the effect of drag on the air-frame induced by the rotor flow.

The reaction torque (due to rotor drag) acting on theairframe generated by a hovering rotor in free air may bemodeled as [5, Sec. 2.30]

Qi :¼ cQ-2i , (4)

where the coefficient cQ (which also depends on Ari , ri, andq) can be determined by static thrust tests.

As a first approximation, assume that each rotor thrustis oriented in the z axis of the vehicle, although we notethat this assumption does not exactly hold once the rotorbegins to rotate and translate through the air, an effect that

is discussed in “Rotor Flapping.” For an N-rotor airframe,we label the rotors i 2 f1 � � �Ng in an anticlockwise direc-tion with rotor 1 lying on the positive x axis of the vehicle(the front), as shown in Figure 2. Each rotor has associatedan angle Ui between its airframe support arm and thebody-fixed frame x axis, and it is the distance d from thecentral axis of the vehicle. In addition, ri 2 f�1,þ1gdenotes the direction of rotation of the ith rotor:þ1 corre-sponding to clockwise and �1 to anticlockwise. The sim-plest configuration is for N even and the rotors distributedsymmetrically around the vehicle axis with adjacent rotorscounter rotating.

The total thrust at hover (TR) applied to the airframe isthe sum of the thrusts from each individual rotor

TR ¼XNi¼1

jTij ¼ cTXNi¼1

-2i

!: (5)

The hover thrust is the primary component of the exoge-nous force

F ¼ TR~zþ D (6)

in (1b), where D comprises secondary aerodynamic forcesthat are induced when the assumption that the rotor is inhover is violated. Since F is defined in {B}, the direction ofapplication is written ~z, although in the frame {A} thisdirection is~b3 ¼ R~z.

The net moment arising from the aerodynamics (thecombination of the produced rotor forces and air resistan-ces) applied to the N-rotor vehicle use are s ¼ (s1, s2, s3).

s1 ¼ cTXNi¼1

di sin (Ui)-2i ,

s2 ¼ �cTXNi¼1

di cos (Ui)-2i ,

s3 ¼ cQXNi¼1

ri-2i : (7)

For a quadrotor, we can write this in matrix form

TR

s1

s2

s3

0BBBB@

1CCCCA ¼

cT cT cT cT

0 dcT 0 �dcT

�dcT 0 dcT 0

�cQ cQ �cQ cQ

0BBBB@

1CCCCA

|fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl}C

-21

-22

-23

-24

0BBBB@

1CCCCA, (8)

and given the desired thrust and moments, we can solvefor the required rotor speeds using the inverse of the con-stant matrix C. In order for the vehicle to hover, one mustchoose suitable -i by inverting C, such that s ¼ 0 andTR ¼ mg.

x, b1

y, b2

e2

e1X, a1

Y, a2

Z, a3

Z, b3

C

Oψ

ψ

ξ

{A}

{B}{E}

Figure 3. The vehicle model. The position and orientation ofthe robot in the global frame are denoted by n and R,respectively.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Blade Flapping and Induced DragThere are many aerodynamic and gyroscopic effects asso-ciated with any rotor craft that modify the simple forcemodel introduced above. Most of these effects cause onlyminor perturbations and do not warrant consideration fora robotic system, although they are important for thedesign of a full-sized rotor craft. Blade flapping andinduced drag, however, are fundamental effects that are ofsignificant importance in understanding the natural stabil-ity of quadrotors and how state observers operate. Theseeffects are particularly relevant since they induce forces inthe x-y rotor plane of the quadrotor, the underactuateddirections in the dynamics, that cannot be easily dominatedby high gain control. In this section, we consider a singlerotor and we will drop the subscript i used in the “DominantAerodynamics” section to refer to particular rotors.

Quadrotor vehicles are typically equipped with light-weight, fixed-pitch plastic rotors. Such rotors are not rigid,and the aerodynamic and inertial forces applied to a rotorduring flight are quite significant and can cause the rotorto flex. In fact, allowing the rotor to bend is an importantproperty of the mechanical design of a quadrotor and fit-ting rotors that are too rigid can lead to transmission ofthese aerodynamic forces directly through to the rotor huband may result in a mechanical failure of the motormounting or the airframe itself. Having said this, rotors onsmall vehicles are significantly more rigid relative to theapplied aerodynamic forces than rotors on a full-scalerotor craft. Blade-flapping effects are due to the flexing ofrotors, while induced drag is associated primarily with therigidity of the rotor, and a typical quadrotor will experi-ence both. Luckily, their mathematical expression isequivalent and a single term is sufficient to represent botheffects in a lumped parameter dynamic model.

When a rotor translates laterally through the air it dis-plays an effect known as rotor flapping (see “RotorFlapping”). A detailed derivation of rotor flapping involvesa mechanical model of the bending of the rotor subject toaerodynamic and centripetal forces as it is swept through afull rotation [5, Sec. 4.5]. The resulting equations of motionare a nonlinear second-order dynamical system with adominant highly damped oscillatory response at the forcedfrequency corresponding to the angular velocity of therotor. For a typical rotor, the flapping dynamics convergeto steady state with one cycle of the rotor [5, p. 137], andfor the purposes of modeling, only the steady-stateresponse of the flapping dynamics need be considered.

Assuming that the velocity of the vehicle is directlyaligned with the X axis in the inertial frame, v ¼ (vx, 0, 0),a simplified solution is given by

b :¼ � lA1c

1� 12l

2� � , b? :¼ � lA1s

1þ 12 l

2� � (9)

for positive constants A1c and A1c, and where l :¼ jvxj=-ris the advance ratio, i.e., the ratio of magnitude of

the horizontal velocity of the rotor to the linear velocityof rotor tip. The flapping angle b is the steady-state tilt ofthe rotor away from the incoming apparent wind and b? isthe tilt orthogonal to the incident wind. Here, we use equa-tions (4.46) and (4.47) from [5, p. 138], noting that addingthe effects of a virtual rotor hinge model [5, Sec. 4.7] resultsin additional phase lag between the sine and cosine com-ponents of the flapping angles [5, Question 4.7, p. 157] thatare absorbed into the constants A1c and A1s in (9).

Rotor flapping is important because the thrust gener-ated by the rotor is perpendicular to the rotor plane andnot to the hub of the rotor. Thus, when the rotor disk tiltsthe rotor thrust is inclined with respect to the airframe andcontains a component in the x and y directions of thebody-fixed frame.

In practice the rotors are stiff and oppose the aerody-namic force which is lifting the advancing blade so that itsincreased thrust due to tip velocity is not fully counteractedby a lower angle of attack and lower lift coefficient—thethrust is increased. Conversely for the retreating blade thethrust is reduced. For any airfoil that generates lift (in ourcase the rotor blade) there is an associated induced drag

•Rotor FlappingWhen a rotor translates horizontally through the air, theadvancing blade has a higher absolute tip velocity and willgenerate more lift than the retreating blade. Thinking of therotor as a spinning disk, the mismatch in lift generates anoverall moment on the rotor disk in the direction of theapparent wind (Figure S1). The high angular momentum ofthe rotor disk makes it act like a gyroscope, which causesthe rotor disk to tilt around the axis given by the crossproduct of rotor hub axis and the torque axis, i.e., an axisthat is offset from the apparent wind by 90� in the horizon-tal plane of the rotor. Since the motor shaft is vertical, theblade flaps up as it advances into the wind and back downagain as it retreats from the apparent wind. Equilibrium isestablished because the advancing blade rises anddecreases its angle of attack, which reduces its lift coeffi-cient, thereby countering the additional lift that would havebeen generated due to its increased tip velocity. Converselyfor the retreating blade, the reduced lift due to decreased tipvelocity is countered by the increased angle of attack andincreased thrust coefficient. In this state, the rotor will havea stable constant tilt away from the apparent wind causedby a translational motion of the rotor. This effect is known asrotor flapping and is ubiquitous in rotor vehicles [6].

Inclined Lift

Vehicle Velocity

Flapping Angle

Ti

TiAflapRTv

Apparent Wind

β

si

Figure S1



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






due to the backward inclination of aerodynamic forcewith respect to the airfoil motion. The induced drag isproportional to the lift generated by the airfoil. In normalhover conditions for a rotor, this force is equally distrib-uted in all directions around the circumference of therotor and is responsible for the torque Q (4). However,when there is a thrust imbalance, then the sector of therotor travel with high thrust (for the advancing rotor) willgenerate more induced drag than the sector where therotor generates less thrust (for the retreating blade). Thenet result will be an induced drag that opposes the direc-tion of apparent wind as seen by the rotor, and that isproportional to the velocity of the apparent wind. Thiseffect is often negligible for full scale rotor craft, however,it may be quite significant for small quadrotor vehicleswith relatively rigid blades. The consequence of bladeflapping and induced drag taken together ensures thatthere is always a noticeable horizontal drag experiencedby a quadrotor even when maneuvering at relativelyslow speeds.

We will now use the insight from the discussion aboveto develop a lumped parameter model for exogenous forcegeneration (6). We assume that all four rotors are identicaland rotate at similar speeds so that, at least to a firstapproximation, the flapping responses of the rotors andthe unbalanced aerodynamic forces are the same. It followsthat the reactive torques on the airframe transmitted bythe rotor masts due to rotor stiffness cancel. For generalmotion of the vehicle, the apparent wind results in theadvance ratio

l ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiv0x

2 þ v0y2

q=-r,

where v0 ¼ R>v is the linear velocity of the vehicleexpressed in the body-fixed frame, with v0x and v

0y being the

components in the body-fixed x-y plane. Define

Aflap ¼ 1-R

A1c �A1s 0A1s A1c 00 0 0

0@

1A,

where - is the set point for the rotor angular velocity.This matrix describes the sensitivities of the flapping angleto the apparent wind in the body-fixed frame, given that lis small and l2 is negligible in the denominators of (9).The first row encodes (9) for the velocity along the body-fixed frame x axis. The second row of Aflap is a p=2 rotationof this response to account for the case where a componentof the wind is incoming from the y axis, while thethird row projects out velocity in the z axis of the body-fixed frame.

We model the stiffness of the rotor as a simple torsionalspring so that the induced drag is directly proportional tothis angle and is scaled by the total thrust. The flappingangle is negligible with regard to the orientation of the

induced drag, and in the body-fixed frame the induceddrag is

Dind:v0 � diag (dx , dy, 0)v

0,

where dx ¼ dy is the induced drag coefficient.The exogenous force applied to the rotor can now be

modeled by

F :¼ TR~z� TRDv0, (10)

where D ¼ Aflap þ diag (dx , dy, 0), and TR is the nominalthrust (5).

An important consequence of blade flapping andinduced drag is a natural stability of the horizontal dynam-ics of the quadrotor [7]. Define

Ph :¼ 1 0 00 1 0

� �(11)

to be the projection matrix onto the x-y plane. The hori-zontal component of a velocity expressed in {A} is

vh :¼ Phv ¼ (vx, vy)> 2 R2: (12)

Recalling (1b) and projecting onto the horizontal compo-nent of velocity, one has

m _vh ¼ �TRPh ~zþ RDv0ð Þ:

If the vehicle is flying horizontally, i.e., vz ¼ 0, thenv ¼ P>

h vh and one can write

m _vh ¼ �TRPh~z� PhRDR>P>

h vh, (13)

where the last term introduces damping since, for a typicalsystem, the matrix D is a positive semidefinite.

A detailed dynamic model of the quadrotor, includingflapping and induced drag, is included in the robotics tool-box for MATLAB [8]. This is provided in the form ofSimulink library blocks along with a set of inertial andaerodynamic parameters for a particular quadrotor. Thegraphical output of the animation block is shown in Fig-ure 4. Simulink models, based on these blocks, that illus-trate path following and vision-based stabilization aredescribed in detail in [1].

The discussion provided above does not considerseveral additional aerodynamic effects that are impor-tant for high-speed and highly dynamic maneuvers for aquadrotor. In particular, we do not consider transla-tional lift and drag that will effect thrust generationat high speed, axial flow modeling and vortex statesthat may effect thrust during axial motion, and groundeffect that will affect a vehicle flying close to theground. It should be noted that high gain control candominate all secondary aerodynamic effects, and high



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






performance control of quadrotor vehicles has beendemonstrated using the simple static thrust model [23],[24]. The detailed modeling of the blade flapping andinduced drag is provided due to its importance in under-standing the state estimation algorithms introducedlater the tutorial.

Size, Weight, and Power (SWAP)Constraints and Scaling LawsReducing the scale of the quadrotor has an interestingeffect on the inertia, payload, and ultimately the maximumachievable angular and linear acceleration. To gain insightinto scaling, it is useful to develop a simple physics modelto analyze a quadrotor’s ability to produce linear and angu-lar accelerations from a hover state.

If the characteristic length is d, the rotor radius r scaleslinearly with d. The mass scales as d3 and the moments ofinertia as d5. On the other hand, from (3) and (4), it isclear that the lift or thrust, T, and drag, Q, from the rotorsscale with the square of the rotor speed, -2. In otherwords, T � -2d4 and Q � -2d4, the linear accelerationa ¼ _v, which depends on the thrust and mass, andthe angular acceleration a ¼ _X, which depends onthrust, drag, the moment arm, and the moments of iner-tia, scale as

a � -2d4

d3¼ -2d, a � -2d5

d5¼ -2:

To explore the scaling of rotor speed with length, it isuseful to adopt the two commonly accepted approachesto study scaling in aerial vehicles [9]. Mach scaling isused for compressible flows and essentially assumes thatthe blade tip speed, vb, is a constant leading to- � (1=r). Froude scaling is used for incompressibleflows and assumes that, for similar aircraft con-figurations, the Froude number, (v2b=dg) ¼ (-2r2=dg),is constant. Here, g is the acceleration due to gravity.

Assuming r � d, we get - � (1=ffiffir

p). Thus, Mach

scaling predicts

a � 1d, a � 1

d2, (14)

while Froude scaling leads to the conclusion

a � 1, a � 1d: (15)

Of course, Froude or Mach number similitudes takeneither motor characteristics nor battery properties intoaccount. While motor torque increases with length, theoperating speed for the rotors is determined by matchingthe torque–speed characteristics of the motor to the dragversus speed characteristics of the rotors. Further, themotor torque depends on the ability of the battery tosource the required current. All these variables are tightlycoupled for smaller designs since there are fewer choicesavailable at smaller length scales. Finally, as discussed inthe previous subsection, the assumption that rotor bladesare rigid may be wrong. Further, the aerodynamics of theblades may be different for blade designs optimized forsmaller helicopters and the quadratic scaling of the lift withspeed may not be accurate.

In spite of the simplifications in the above similitudeanalysis, the key insight from both Froude and Mach num-ber similitudes is that smaller quadrotors can producefaster angular accelerations while the linear acceleration isat worst unaffected by scaling. Thus, smaller quadrotorsare more agile, a fact that is easily validated from experi-ments conducted with the Ascending Technologies Pelicanquadrotor [10] (approximately 2 kg gross weight whenequipped with sensors, 0.75 m diameter, and 5,400 r/minnominal rotor speed at hover), the Ascending Technolo-gies Hummingbird quadrotor [11] (approximately 500 ggross weight, 0.5 m diameter, and 5,000 r/min nominalrotor speed at hover), and laboratory experimental proto-types developed at GRASP laboratory at the University ofPennsylvania (approx. 75 g gross weight, 0.21 m diameter,and approximately 9,000 r/min nominal rotor speed).

Estimating the Vehicle StateThe key state estimates required for the control of a quad-rotor are its height, attitude, angular velocity, and linearvelocity. Of these states, the attitude and angular velocityare the most important as they are the primary variablesused in attitude control of the vehicle. The most basicinstrumentation carried by any quadrotor is an inertialmeasurement unit (IMU) often augmented by some formof height measurement, either acoustic, infrared, baromet-ric, or laser based. Many robotics applications requiremore sophisticated sensor suites such as VICON systems,global positioning system (GPS), camera, Kinect, or scan-ning laser rangefinder.

1

0.8

0.6

0.4

0.2

01

0.50

–0.5–1 –1

–0.50

0.51

xy

z (H

eigh

t Abo

ve G

roun

d)

Figure 4. Frame from the Simulink animation of quadrotordynamics.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Estimating AttitudeA typical IMU includes a three-axis rate gyro, three-axisaccelerometer, and three-axis magnetometer. The rate gyromeasures the angular velocity of {B} relative to {A}expressed in the body-fixed frame of reference {B}

XIMU ¼ Xþ bX þ g 2 fBg,

where g denotes the additive measurement noise and bXdenotes a constant (or slowly time-varying) gyro bias. Gen-erally, the gyroscopes installed on quadrotor vehicles arelightweight microelectromechanical systems (MEMS) devi-ces that are reasonably robust to noise and quite reliable.

The accelerometers (in a strap down IMU configura-tion) measure the instantaneous linear acceleration of {B}due to exogenous force

aIMU ¼ R>( _v � g~z )þ ba þ ga 2 fBg,

where ba is a bias term, ga denotes additive measurementnoise, and _v is in the inertial frame. Here, we use the nota-tion ~z ¼~a3 since we will need to deal with the algebraicexpressions of the coordinate axes throughout this section.Accelerometers are highly susceptible to vibration and,mounted on a quadrotor, they require significant low-passmechanical and/or electrical filtering to be usable. Mostquadrotor avionics will incorporate an analogue anti-aliasing filter on a MEMS accelerometer before the signalis sampled.

A commonly used technique to estimate the bias bXand ba is to average the output of these sensors for a fewseconds while the quadrotor is on the ground and themotors are not yet active. The bias is then assumed con-stant for the duration of the flight.

The magnetometers provide measurements of theambient magnetic field

mIMU ¼ R>Amþ Bm þ gb 2 fBg,

where Am is the Earth’s magnetic field vector (expressed inthe inertial frame), Bm is a body-fixed frame expression forthe local magnetic disturbance, and gb denotes themeasurement noise. The noise gb is usually low for magne-tometer readings; however, the local magnetic disturbanceBm can be significant, especially if the sensor is placed nearthe power wires to the motors.

The accelerometers and magnetometers can be used toprovide absolute attitude information on the vehicle whilethe rate gyroscope provides complementary angular veloc-ity measurements. The attitude information in the magne-tometer signal is straightforward to understand; in theabsence of noise and bias, mIMU provides a body-fixedframe measurement of R>Am and, consequently, con-strains two DoF in the rotation R.

The case for using the accelerometer signal for attitudeestimation is far more subtle. Using the simplest model (6)

with D � 0, aIMU ¼ R>( _v � g~z) ¼ (TR=m)~z � g~z. Thisshows that the measured acceleration, for this simplemodel, would always point in the body-fixed frame direc-tion~z and provides no attitude information. In practice, itis the blade-flapping component of the thrust that contrib-utes attitude information to the accelerometer signal [7].Recalling (10) and ignoring bias and noise terms, themodel for aIMU can be written as

aIMU ¼ �TR

m~z� TR

mDR>v: (16)

As we show later in the section, only the low-frequencyinformation from the accelerometer signal will be used inthe observer construction. Thus, it is only the low-frequencyor approximate steady-state response �v of the velocity v thatwe need to estimate to build a model for the low-frequencycomponent of aIMU. Setting _v ¼ 0 in (1b), substituting forforce (10), and rearranging, we obtain an estimate of thelow-frequency component of the velocity signal

DR>�v � R>~z�~z:

Substituting DR>�v for DR>v in (16), we obtain

�aIMU � �TR

mR>~z, (17)

where �aIMU denotes the low-frequency component of theaccelerometer signal. That is, the low-frequency content ofaIMU when the vehicle is near hover is the body-fixed frameexpression for the supporting force that is the negativegravitational vector expressed in the body-fixed frame.Most robotics applications involve a quadrotor spendingsignificant periods of time in hover, or slow forward flight,with _v � 0, and using the accelerometer reading as an atti-tude reference during this flight regime has been shown towork well in practice.

The attitude kinematics of the quadrotor are given by(1c). Let R denote an estimate for attitude R of the quadrotorvehicle. The following observer [12] fuses accelerometer,magnetometer, and gyroscope data as well as other directattitude estimates RE (such as provided by a VICON or otherexternal measurement system) should they be available:

_R :¼ R XIMU � b�

3� a,

_b :¼ kba,

a :¼ kag2

((R>~z)3 �aIMU)þ km

jAmj2 ((R>Am)3mIMU)

!3

þ kEPso(3) RR>E

� �, (18)

where ka, km, kE, and kb are arbitrary nonnegative observergains and Pso(3)(M) ¼ (M �M>)=2 is the Euclidean



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






matrix projection onto skew-symmetric matrices. If anyone of the measurements in the innovation a are not avail-able or unreliable, then the corresponding gain should beset to zero in the observer. Note that both the attitude Rand the bias corrected angular velocity X ¼ XIMU � b areestimated by this observer. The observer (18) has beenextensively studied in the literature [12], [13] and shownto converge exponentially (both theoretically and experi-mentally) to the desired attitude estimate of attitude with bconverging to the gyroscope bias b. The filter has a com-plementary nature, using the high-frequency part of thegyroscope signal and the low-frequency parts of themagnetometer, accelerometer, and external attitudemeasurements [12]. The roll-off frequencies associated witheach of these signals is given by the gains ka, km, and kE inrad.s�1, and good performance of the observer depends onhow these gains are tuned. In particular, the accelerometergains must be tuned to a frequency below the normal band-width of the vehicle motion, less than 5 rad.s�1 for a typicalquadrotor. The magnetometer gain and external gain canbe tuned for a higher roll-off frequency depending on thereliability of the signals. The bias gain kb is typically chosenan order of magnitude slower than the innovation gainskb < ka=10, leading to a rise time of the bias estimate asslow as 30 s or more. This dynamic response is necessary totrack slowly varying bias and decouples the bias estimatefrom the attitude response; however, it is necessary to initi-alize the observer with a bias estimate at take off to avoid along transient in the filter response.

A particular advantage of this observer formulation isthat the gains can be adjusted in real time as long as care istaken that the bias gain is small. Adjusting the gains in realtime allows one to use the accelerometer during a periodwhen the vehicle is in hover and then set the gain ka ¼ 0during acrobatic maneuvers when the low-frequencyassumptions on �aIMU no longer hold. The nonlinearrobustness, guaranteed asymptotic stability, and flexibilityin gain tuning make this observer a preferred candidate forquadrotor attitude estimation compared with classical fil-ters such as the extended Kalman filter (EKF), multiplica-tive EKF, or more sophisticated stochastic filters.

Estimating Translational VelocityThe blade-flapping response provides a way to build anobserver for the horizontal velocity of the vehicle based onthe IMU sensors [7], at least while the vehicle is flying inthe horizontal plane. Assume that a good estimate of thevehicle attitude R is available and that the vehicle is flyingat constant height.

Recalling the projector (11), the horizontal componentof the inertial acceleration can be measured by

Aah :¼ PhAa ¼ PhRa � PhRa, (19)

where the signals a and R are available. Since we assumethat the vehicle is flying at a constant height, one has

vz � 0, and recalling (12), P>h vh � v. Further, the thrust

TR � mg must compensate the weight of the vehicle.Recalling (16) and taking the horizontal component,one has

Aah � �gPhR~z� gPhRDR>P>

h vh: (20)

Assuming that the attitude filter estimate is good, i.e.,R ¼ R, then (19) and (20) can be solved for an estimateof vh

vh � � 1g

PhRDR>P>

h

h i�1Aah þ gPhR~z� �

: (21)

This estimate of vh will be well defined as long as the 23 2matrix PhRDR>P>

h is invertible, a condition that will holdas long as the vehicle pitches or rolls by less than 90� dur-ing flight.

Equation (21) provides a measurement of the horizon-tal velocity; however, since it directly incorporates theunfiltered accelerometer readings, it is generally too noisyto be of much use. Its low-frequency content can, however,be used to drive a velocity complementary observer thatuses the attitude estimate and the system model (1b) alongwith the thrust model (10) for its high-frequency compo-nent. Let vh be an estimate of the horizontal component ofthe inertial velocity of the vehicle. Recalling (1b), we pro-pose the following observer

_vh ¼ �gP>h R~zþ RDR

>P>

h vh�

� kw(vh � vh), (22)

where vh is given by (21). The gain kw > 0 provides a tun-ing parameter that adjusts the roll-off frequency for theinformation from vh that is used in the filter. It also uses anestimated velocity vh to provide an approximation of themore correct RDR>P>

h vh term in the feedforward velocityestimate; however, since the underlying dynamics associ-ated with this term are stable, the observer is stable evenwith this approximation.

Estimating PositionThe final part of state that must be estimated is position,which is typically considered separately as position in theplane and height. Considering the height first, there are infact two separate heights that are of importance: the first isthe absolute height of the vehicle and the second is the rela-tive height over the terrain at a given time. Unfortunately,there is no effective way to use the IMU to estimate abso-lute height; at best, some low-frequency information fromthe z axis of the accelerometer provides limited informa-tion about vertical motion. Most quadrotors include abarometric sensor that can resolve absolute height to a fewcentimeters. Absolute height can also be estimated usingGPS, VICON, or a full SLAM system. Relative height canbe estimated using acoustic, laser-ranging or infrared



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






sensors. Once a sufficiently accurate height measurementis available, it is better to use this directly in the controlthan add additional levels of complexity in designing aheight observer, especially since, for a typical system, theonly feedforward information available is the noisy accel-erometer readings.

Position in the plane can also be determined in a rela-tive or absolute way. Absolute position can be obtainedfrom a GPS (few-centimeter accuracy at up to 10 Hz [6])or an external localization device such as a VICONmotion capture system (50 lm accuracy at 375 Hz). How-ever, a GPS does not work indoors and motion-capturesystems are expensive, and their sensor array has alimited spatial extent that is impractical to scale up forlarge indoor environments.

Relative position can be estimated by measuring the dis-tance to objects in the environment from onboard sensors,typically small onboard laser range finders (LRFs) orRGBD camera systems such as the Kinect. Well-knownSLAM techniques, borrowing LRF-based techniquessimilar to those developed for mobile ground robots overthe last decade, have been applied to quadrotors [14].However, LRFs provide only a cross section of the 3-Denvironment and this scan plane tilts as the vehicle maneu-vers, resulting in apparent changes to the distance of walls,and, in extreme cases, the scan plane can intersect the flooror ceiling. LRFs are heavy and power hungry, which pre-vents their application to the next generation of muchsmaller quadcopters.

Vision has the advantage that the sensor is small, light-weight, and low power, which will become increasinglyimportant as the size of aerial vehicles decreases. Visioncan provide essential navigational competencies such asodometry, attitude estimation, mapping, place and objectrecognition, and collision detection. There is a long historyof applying vision to aerial robotic systems [15]–[19] forindoor and outdoor environments, and the well-knownParrot AR.Drone game device makes strong use of visionfor attitude and odometry [20]. Vision can also be used forobject recognition based on color, texture, and shape, aswell as collision avoidance.

Vision is not without its challenges. First, vision is com-putationally intense and can result in a low sample rate.Since onboard computational power is limited (by SWAPconsumption), most reported systems transmit the imageswirelessly to a ground station, which increases system

complexity, control latency, and the susceptibility to inter-ference and dropouts. However, processor speed continuesto improve, and we can also utilize the vision and controltechniques used by flying insects that perform complextasks with very limited sensing and neural capability [21].Second, there is an ambiguity between certain rotationaland translational motions, particularly, when a narrowfield of view perspective camera is used. Third, the under-actuated quadrotor uses the roll and pitch DoF to point thethrust vector in the direction of the desired translationalmotion. For a camera that is rigidly attached to the quadro-tor, this attitude control motion induces a large apparentmotion in the image. It is therefore necessary to estimatevehicle attitude at the instant the image was captured bythe sensor to eliminate this effect. Biological systems facesimilar problems, and interestingly, mammals and insectshave developed similar solutions: gyroscopic sensors(the vestibular sensors of the inner ear and the halteres,respectively) [22]. Finally, there exists a problem withrecovering motion scale when using a single camera. Stereois possible, but the baseline is constrained, particularly asvehicles get smaller.

ControlThe control problem, to track smooth trajectoriesR�(t), n�(t)ð Þ 2 SE(3), is challenging for several reasons.First, the system is underactuated: there are four inputsu ¼ (TR, s>)>, while SE(3) is six dimensional. Second, theaerodynamic model described above is only approximate.Finally, the inputs are themselves idealized. In practice, themotor controllers must overcome the drag moments togenerate the required speeds and realize the input thrust(TR) and moments (s). The dynamics of the motors andtheir interactions with the drag forces on the propellerscan be difficult to model, although first-order linear mod-els are a useful approximation.

A hierarchical control approach is common for quadro-tors. The lowest level, the highest bandwidth, is in controlof the rotor rotational speed. The next level is in control ofvehicle attitude, and the top level is in control of positionalong a trajectory. These levels form nested feedback loops,as shown in Figure 5.

Controlling the MotorsRotor speed drives the dynamic model of the vehicleaccording to (8), so high-quality control of the motor

speed is fundamentally importantfor overall control of the vehicle;high bandwidth control of thethrust TR, denoted by u1, and thetorques (sx, sy, sz), denoted by u2,lead to high performance attitudeand position control. Most quadro-tor vehicles are equipped withbrushless dc motors that use backelectromotive force (EMF) sensing

Rigid BodyDynamics

MotorControllerAttitude

Controller

PositionController

TrajectoryPlanner R*

R,Ω

u1

u2Attitude

Planner ξ,v

ξ*

ψ*

Figure 5. The innermost motor control loop, the intermediate attitude control loop, andthe outer position control loop.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






for rotor commutation and high-frequency pulsewidthmodulation (PWM) to control motor voltage. Thesimplest systems generally use a direct voltage control ofthe motors since steady-state motor speed is propor-tional to voltage; however; the dynamic response issecond-order due to the mechanical and electricaldynamics. Improved performance is obtained by in-corporating single-input single-output control at themotor/rotor level

Vi ¼ k(-�i � -i)þ Vff (-

�i ), (23)

where Vi is the applied motor voltage, -�i is the desired

speed, and the actual motor speed -i can be measuredfrom the electronic commutation in the embeddedspeed controller. This can help to overcome a commonproblem where the rotor speed for a given PWM com-mand setting will decrease as the battery voltagereduces during flight. The significant load torque due toaerodynamic drag will lead to a tracking error that canbe minimized by high proportional gain (k) and/or afeedforward term. A positive benefit of the dragtorque is that the system is heavily damped, whichprecludes the need for derivative control. The feed-forward term Vff (-�

i ) compensates for the steady-statePWM associated with a given velocity set point byincorporating the best available thrust model deter-mined using static thrust tests and possibly includingbattery voltage.

The performance of the motor controllers is ultimatelylimited by the current that can be supplied from the bat-teries. This may be a significant limiting factor for smallervehicles. Overly aggressive tuning and extreme maneuversmay cause the voltage bus to drop excessively, reducingthe thrust from other rotors and, in extreme cases, causingthe onboard electronics to brownout. For this reason, itis common to introduce a saturation, although thisdestroys the linearity of the motor/rotor response duringaggressive maneuvers.

Attitude ControlWe first consider the design of an exponentially converg-ing controller in SO(3). Given a desired airframe attitudeR?, we want to first develop a measure of the error in rota-tions. We choose the measure

eR3¼ 1

2(R�)TR� RTR��

, (24)

which yields a skew-symmetric matrix representing theaxis of rotation required to go from R to R� and whosemagnitude is equal to the sine of the angle of rotation.

To derive linear controllers, we linearize the dynamicsabout the nominal hover position at which the roll (/) andpitch (h) are close to zero and the angular velocities areclose to zero. If we write R ¼ ARB as a product of the yaw

rotation ARE(w) and ERB(/, h), which is a composition ofthe roll and pitch, we can linearize the rotation about(w,/, h) ¼ (w0, 0, 0)

ARB ¼ ARE(w0 þ Dw) ERB(D/,Dh)

¼

cosw � sinw Dh coswþ D/ sinw

sinw cosw Dh sinw� D/ cosw

�Dh D/ 1

0BBB@

1CCCA,

where w ¼ w0 þ Dw. If R? ¼ ARB(w0 þ Dw,D/,Dh) andR ¼ ARB(w0, 0, 0), (24) gives

eR3¼

0 Dw �Dh

�Dw 0 D/

Dh �D/ 0

0BB@

1CCA, (25)

which, as we expect, corresponds to the error vector

eR ¼ (D/,Dh,Dw)>,

with components in the body-fixed frame. If the desiredangular velocity vector is zero, we can compute theproportional and derivative error to obtain the PD con-trol law

u2 ¼ �kReR � kXeX, (26)

where kR and kX are positive definite gain matrices. Thiscontroller guarantees stability for small deviations fromthe hover position.

To obtain convergence for larger deviations fromthe hover position, it is necessary to revert back to (24)without linearization. This allows us to directly computethe error on SO(3). By compensating for the nonlinearinertial terms and by including the correct error term,we obtain

u2¼J(�kReR�kXeX)þX3 JX�J(X3RTR?X?�RTR? _X?):

(27)

This controller is guaranteed to be exponentiallystable for almost any rotation [23]. From a practicalstandpoint, it is possible to neglect the last three termsin the controller and achieve satisfactory performance,but the correct calculation of the error term is impor-tant [24].

Trajectory ControlWe now turn our attention to the control of the trajec-tory along a specified trajectory n?(t). As before, wefirst consider linear controllers by linearizing the dy-namics about n ¼ n?(t), h ¼ / ¼ 0,w ¼ w?(t), _n ¼ 0, and



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






_/ ¼ _h ¼ _w ¼ 0, with the nominal input given byu1 ¼ mg , u2 ¼ 0. Linearizing (1a), we get

€n1 ¼ g(Dh cosw? þ D/ sinw?),

€n2 ¼ g(Dh sinw? � D/ cosw?),

€n3 ¼1mu1 � g: (28)

To exponentially drive all three components of error, wewant to command the acceleration vector €ncom to satisfy

(€n�(t)� €ncom)þ Kd( _n�(t)� _n)þ Kp(n

�(t)� n) ¼ 0:

From (28), we can immediately write

u1 ¼ m g þ €n�3 þ kd, z( _n�3 � _n3)þ kp, z(n

�3 � n3)

� , (29)

to guarantee (n3(t)� n?3(t)) ! 0. Similarly, for the othertwo components, we choose to command the appropriateh? and /? to guarantee exponential convergence

/? ¼ 1g(€ncom1 sinw?(t)� €ncom2 cosw?(t)), (30a)

h? ¼ 1g(€ncom1 cosw?(t)þ €ncom2 sinw?(t)), (30b)

where the above equations are obtained by replacing Dh byh? and D/ by /? in (28). Finally, (w?,/?, h?) are provided asset points to the attitude controller discussed in the previoussection. Thus, as shown in Figure 5, the control problem isaddressed by decoupling the position control and attitudecontrol subproblems, and the position control loop providesthe attitude set points for the attitude controller.

The position controller can also be obtained withoutlinearization. This is done by projecting the position error(and its derivatives) along b3 and applying the input u1that cancels the gravitational force and provides the appro-priate proportional plus derivative feedback

u1 ¼ m~bT3€n� þ Kd( _n

� � _n)þ Kp(n� � n)þ g~a3

� : (31)

Note that the projection operation is a nonlinear functionof the roll and pitch angles, and, thus, this is a nonlinearcontroller. In [23], it is shown that the two nonlinear con-trollers (27) and (31) result in exponential stability andallow the robot to track trajectories in SE(3).

Trajectory PlanningThe quadrotor is underactuated, and this makes it difficult toplan trajectories in 12-dimensional state space (6 DoF positionand velocity). However, the problem is considerably simplifiedif we use the fact that the quadrotor dynamics are differentiallyflat [25]. To see this, we consider the output position n and theyaw angle w. We show that we can write all state variables and

inputs as functions of the outputs (n,w) and their derivatives.Derivatives of n yield the velocity v and the acceleration,

_v ¼ 1mu1~b3 þ g~a3:

From Figure 3 we see that

~e1 ¼ cosw, sinw, 0½ �T,and the unit vectors for the body-fixed frame can be writ-ten in terms of the variables w and _v as

~b3 ¼ _v � g~a3_v � g~a3k k ,

~b2 ¼~b3 3~e1~b3 3~e1 , ~b1 ¼~b2 3~b3

provided~b3 3~e1 6¼ 0. This defines the rotation matrix ARB asa function of _v (the second derivative of n) and w. In this way,we write the angular velocity and the four inputs as functionsof position, velocity, acceleration, jerk (c), and snap, or thederivative of jerk (r). From these equations, it is possible to ver-ify that there is a diffeomorphism between the 183 1 vector

nT, vT, aT, cT, rT,wT, _wT, €w

� Tand

R3 nT, _nT,XT, u1, _u1, €u1, u

T2

� T:

This property of differential flatness makes it easy to designtrajectories that respect the dynamics of the underactuatedsystem. Any four-times-differentiable trajectory in the spaceof flat outputs, (n>(t),w(t))>, corresponds to a feasible trajec-tory—one that satisfies the equations of motion. All inequalityconstraints of states and inputs can be expressed as functionsof the flat outputs and their derivatives. This mapping to thespace of flat outputs can be used to generate trajectories thatminimize a cost functional formed by a weighted combinationof the different flat outputs and their derivatives:

minn(t),w(t)

Z T

0L(n, _n, €n, n

��, n��w, _w, €w)dt,

g(n(t),w(t)) 0: (32)

In [24], minimum snap trajectories were generated byminimizing a cost functional derived from the snap andthe angular yaw acceleration with

L(n, _n, €n, n��, n��w, _w, €w) ¼ (1� c)( n

��)4 þ c( €w)2:

By suitable parameterizing trajectories with basis functions inthe flat space and by considering linear inequalities in the flat



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






space to model constraints on states and inputs (e.g., u1 0),it is possible to turn this optimization into a quadratic pro-gram that can be solved in real time for planning.

Finally, as shown in [11], it is possible to combine thiscontroller with attitude-only controllers to fly throughvertical windows or land on inclined perches with closeto zero normal velocity. A trajectory controller is used bythe robot to build up momentum, while the attitude con-troller enables reorientation while coasting with the gener-ated momentum.

Vision-Based Perception and ControlThere are two approaches to the question of controlling anaerial vehicle based on visual information. The first is to useclassical robotic SLAM techniques, although with thecaveat that the environment and state estimation are inher-ently 3-D. There are many researchers currently workingon this problem, and we will not attempt to discuss thisapproach further, except to say that should a good-qualityenvironmental estimation and localization algorithm bedeveloped, the control techniques discussed above can beapplied. The second approach is direct sensor-based con-trol [26], the most commonly referred to case, being that ofimage-based visual servo control [27]–[29].

The motion of a point in an image is a function of itscoordinate (u, v) and the camera motion

_u_v

� �¼ J(u, v, Z)m, (33)

where Z is the point depth, m ¼ (vx, vy, vz , xx, xy, xz)> is

the spatial velocity of the camera (and vehicle), and J( � ) isthe visual Jacobian or interaction matrix. J can be formu-lated for a perspective camera [30], where (u, v) are pixelcoordinates; or a spherical camera [31] where (u, v) are lat-itude and longitude angles.

The pitch and roll motion of the vehicle are controlledby the attitude subsystem to maintain a position or to fol-low a path in space, and this causes image motion. We par-tition the equations as

_u_v

� �¼ J1(u, v)(vx, vy, vz , xz)

> þ J2(u, v)xx

xy

� �, (34)

where the right-most term describes the image motion dueto the exogenous roll and pitch motion. Rearranging wecan write

_u0

_v0

� �¼ _u

_v

� �>� J2(u, v)

xx

xy

� �(35)

¼ J1(u, v)(vx, vy , vz , xz)>, (36)

where (u0, v0) represent image points for which the roll andpitch motion has been removed based on the knowledge ofxx andxy, which can be obtained from gyroscopes.

Now consider a point in the image (u0i, v0i) and its

desired location in the image (u�i , v�i ). This desired position

might come from a snapshot of the scene taken when thevehicle was at the desired pose that we wish to returnto. The desired image motion is therefore ( _u�i , _v

�i ) ¼

k(u�i � u0i, v�i � v0i), where the operator � represents the

difference on image plane or sphere. For N points, wecan write

k

_u�1_v�1

..

.

_u�N_v�N

0BBBBBBBBB@

1CCCCCCCCCA

�J1(u1, v1)

..

.

J1(uN , vN)

0BBB@

1CCCA xx

xy

!0BBBBBBBBB@

1CCCCCCCCCA

¼J2(u1, v1)

..

.

J2(uN , vN)

0BBB@

1CCCA

|fflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflffl}B

vx

vy

vz

xz

0BBBBB@

1CCCCCA: (37)

If N > 2 and the matrix B is nonsingular, we can solve forthe required translational and yaw velocity to move thevehicle to a pose where the feature points have the desiredimage coordinates (u�i , v

�i ). The desired velocity is input to

a control system as discussed earlier. This is an example ofimage-based visual servoing for an underactuated vehicle,and the technique can be applied to a wider variety ofproblems, such as holding station, path following, obstacleavoidance, and landing.

ConclusionsIn this article, we have provided a tutorial introduction tomodeling, estimation, and control for multirotor aerialvehicles, with a particular focus on the most commonform—the quadrotor. The dynamic model includes therigid body motion of the vehicle in SE(3), the simple aero-dynamics associated with hover, and the extension to thecase of forward motion where blade flapping becomesimportant. State estimation based on accelerometers, gyro-scopes, and magnetometers was discussed for attitude andtranslational velocity, and GPS, motion-capture systems,and cameras for position estimation. A hierarchy of con-trol techniques was discussed, from the individual rotorsthrough attitude control, aggressive trajectory following,and image-based visual control. The future possibilities ofhighly agile small-scale vehicles were laid with a discussionon dimensional scaling for which vision will be an impor-tant sensing modality.

AcknowledgmentThis research was partly supported by the AustralianResearch Council through Future Fellowship FT0991771,



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Foundations of Vision Based Control of Robotic Vehicles,the U.S. Army Research Laboratory Grant W911NF-08-2-0004, and the U.S. Office of Naval Research GrantsN00014-07-1-0829, N00014-09-1-1051, and N00014-08-1-0696.

References[1] P. I. Corke, Robotics, Vision & Control: Fundamental Algorithms in

MATLAB. Berlin: Springer-Verlag, 2011.

[2] T. Hamel, R. Mahony, R. Lozano, and J. Ostrowski, “Dynamic model-

ling and configuration stabilization for an X4-flyer,” in Proc. Int. Federa-

tion of Automatic Control Symp. (IFAC), 2002, p. 6.

[3] S. Bouabdallah, P. Murrieri, and R. Siegwart, “Design and control of

an indoor micro quadrotor,” in Proc. IEEE Int. Conf. Robotics and Auto-

mation (ICRA), Apr. 26–May 1, 2004, vol. 5, pp. 4393–4398.

[4] R. W. Prouty, Helicopter Performance, Stability and Control. Mel-

bourne, FL: Krieger, 1995 (reprint with additions, original edition 1986).

[5] J. Leishman. (2000). Principles of Helicopter Aerodynamics (Cambridge

Aerospace Series). Cambridge, MA: Cambridge Univ. Press. [Online].

Available: http://books.google.com.au/books?id=nMV-TkaX-9cC

[6] H. Huang, G. Hoffmann, S. Waslander, and C. Tomlin,

“Aerodynamics and control of autonomous quadrotor helicopters in

aggressive maneuvering,” in Proc. IEEE Int. Conf. Robotics and Automa-

tion (ICRA), May 2009, pp. 3277–3282.

[7] P. Martin and E. Salaun, “The true role of accelerometer feedback in

quadrotor control,” in Proc. IEEE Int. Conf. Robotics and Automation,

Anchorage, AK, May 2010, pp. 1623–1629.

[8] P. Corke. Robotics toolbox for MATLAB. (2012). [Online]. Available:

http://www.petercorke.com/robot.

[9] C. H. Wolowicz, J. S. Bowman, andW. P. Gilbert, “Similitude require-

ments and scaling relationships as applied to model testing,” NASA,

Tech. Rep. 1435, Aug. 1979.

[10] S. Shen, N. Michael, and V. Kumar, “3D estimation and control for

autonomous flight with constrained computation,” in Proc. IEEE Int.

Conf. Robotics and Automation, Shanghai, China, May 2011, p. 6.

[11] D. Mellinger, N. Michael, and V. Kumar, “Trajectory generation and

control for precise aggressive maneuvers with quadrotors,” Int. J. Robot.

Res., vol. 31, no. 5, pp. 664–674, Apr. 2012.

[12] R. Mahony, T. Hamel, and J.-M. Pflimlin, “Non-linear complemen-

tary filters on the special orthogonal group,” IEEE Trans. Automat.

Contr., vol. 53, no. 5, pp. 1203–1218, June 2008.

[13] S. Bonnabel, P. Martin, and P. Rouchon, “Non-linear symmetry-pre-

serving observers on lie groups,” IEEE Trans. Automat. Contr., vol. 54,

no. 7, pp. 1709–1713, 2009.

[14] A. Bachrach, S. Prentice, R. He, and N. Roy, “Range-robust autono-

mous navigation in GPS-denied environments,” J. Field Robot., vol. 28,

no. 5, pp. 644–666, 2011.

[15] O. Amidi, T. Kanade, and R. Miller, “Vision-based autonomous heli-

copter research at Carnegie Mellon Robotics Institute,” in Proc. Heli

Japan, 1998, vol. 98, p. 6.

[16] P. Corke, “An inertial and visual sensing system for a small

autonomous helicopter,” J. Robot. Syst., vol. 21, no. 2, pp. 43–51, Feb. 2004.

[17] L. Meier, P. Tanskanen, F. Fraundorfer, and M. Pollefeys, “Pixhawk:

A system for autonomous flight using onboard computer,” in Proc. ICRA,

2011, p. 6.

[18] C. Kemp, “Visual control of a miniature quad rotor helicopter,”

Ph.D. dissertation, Univ. Cambridge, Cambridge, U.K., 2006.

[19] M. Achtelik, A. Bachrach, R. He, S. Prentice, and N. Roy, “Stereo

vision and laser odometry for autonomous helicopters in GPS-denied

indoor environments,” in Proc. SPIE Unmanned Systems Technology XI.

Orlando, FL, SPIE, 2009, vol. 7332, p. 10.

[20] P. Bristeau, F. Callou, D. Vissi�ere, and N. Petit, “The navigation and

control technology inside the AR. drone micro UAV,” in Proc. World

Congress, 2011, vol. 18, no. 1, pp. 1477–1484.

[21] V. Srinivasan and S. Venkatesh, From Living Eyes to Seeing

Machines. London, U.K.: Oxford Univ. Press, 1997.

[22] P. Corke, J. Lobo, and J. Dias, “An introduction to inertial and

visual sensing,” Int. J. Robot. Res., vol. 26, no. 6, pp. 519–536, June

2007.

[23] T. Lee, M. Leok, and N. McClamroch, “Geometric tracking control

of a quadrotor UAV on SE(3),” in Proc. IEEE Conf. Decision and Control,

2010, p. 6.

[24] D. Mellinger and V. Kumar, “Minimum snap trajectory generation

and control for quadrotors,” in Proc. Int. Conf. Robotics and Automation

(ICRA), Shanghai, China, May 2011, p. 6.

[25] M. J. V. Nieuwstadt and R. M. Murray, “Real-time trajectory genera-

tion for differentially flat systems,” Int. J. Robust and Nonlinear Control,

vol. 8, no. 11, pp. 995–1020, 1998.

[26] C. Samson, M. Le Borgne, and B. Espiau, Robot Control: The Task

Function Approach (The Oxford Engineering Science Series). Oxford,

U.K.: Oxford Univ. Press, 1991.

[27] T. Hamel and R. Mahony, “Visual servoing of an under-actuated

dynamic rigid-body system: An image based approach,” IEEE Trans.

Robot. Automat., vol. 18, no. 2, pp. 187–198, Apr. 2002.

[28] N. Guenard, T. Hamel, and R. Mahony, “A practical visual servo

control for a unmanned aerial vehicle,” IEEE Trans. Robot., vol. 24, no. 2,

pp. 331–341, Apr. 2008.

[29] R. Mahony, P. Corke, and T. Hamel, “Dynamic image-based visual

servo control using centroid and optic flow features,” J. Dynamic. Syst.,

Meas. Contr., vol. 130, no. 1, p. 12, Jan. 2008.

[30] S. Hutchinson, G. Hager, and P. Corke, “A tutorial on visual servo

control,” IEEE Trans. Robot. Autom., vol. 12, no. 5, pp. 651–670, Oct.

1996.

[31] P. I. Corke, “Spherical image-based visual servo and structure

estimation,” in Proc. IEEE Int. Conf. Robotics and Automation, Anchor-

age, AK, May 2010, pp. 5550–5555.

Robert Mahony, Research School of Engineering, Austra-lian National University, Canberra 0200, Australia. E-mail:[email protected].

Vijay Kumar, Department of Mechanical Engineeringand Applied Mechanics, GRASP Laboratory, Universityof Pennsylvania, Philadelphia, USA. E-mail: [email protected].

Peter Corke, School of Electrical Engineering andComputer Science, Queensland University of Technology,Australia. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________

____________________

_________

___________

http://www.qmags.com/clickthrough.asp?url=http://www.petercorke.com/robot&id=17487&adid=P32E2

http://www.qmags.com/clickthrough.asp?url=http://books.google.com.au/books?id=nMV-TkaX-9cC&id=17487&adid=P32E1








Open-Source

Projects on

Unmanned

Aerial Vehicles

•By Hyon Lim, Jaemann Park,Daewon Lee, and H.J. Kim

1070-9932/12/$31.00ª2012IEEE SEPTEMBER 2012 • IEEE ROBOTICS & AUTOMATIONMAGAZINE • 33



This article presents a survey on publicly available open-source proj-ects (OSPs) on quadrotor unmanned aerial vehicles (UAVs).

Recently, there has been increasing interest in quadrotor UAVs.Exciting videos have been published on the Internet by manyresearch groups and have attracted much attention from the public

[1]–[7]. Relatively simple structures of quadrotors has promoted interest fromacademia, UAV industries, and radio-control (RC) hobbyists alike. Unlikeconventional helicopters, swashplates, which are prone to failure without con-stant maintenance, are not required. Furthermore, the diameter of individualrotors can be reduced as a result of the presence of four actuators [8].

Many research groups or institutions have constructed their own quadro-tors to suit specific purposes. Successes have been reported from academia,such as the X4-flyer [9], OS4 [10], STARMAC [11], and Pixhawk [12] to men-tion a few. To the commercial market, the Draganflyer X4, Asctec Humming-bird, Gaui Quad flyer, Parrot ARDrone, and DJI Wookong have beenintroduced. At the same time, a number of OSPs for quadrotors have emergedas shown in Figure 1, with contributions from RC hobbyists, universities [12],[13], and corporations.


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Quadrotor OSPs use community-hosting sites (e.g.,Google code and Github) to create code, blueprints, orschematics, which are freely available under open-sourcelicenses such as the general public license (GPL) [14].These tools help talented independent developers to join

OSPs freely from all over the world. This setting allowsvery fast development processes because new features canbe tested not only by the developer but also by otherpeople in the community. Feedback is given in real timefrom various conditions and configurations, which makeopen-source software more robust in a relatively shortperiod of time.

OSPs have been successful in many disciplines andremain competitive with commercial alternatives withLinux being the most famous operating system. In therobotics area, more than 2,000 projects have been estab-lished based on the robot operating system [15] with itswell-organized framework that encourages open-sourcedevelopment.

In the case of quadrotor OSPs, one of the main reasonsto use them is flexibility in both hardware and software,which makes modification easier to meet the specificrequirements of a user. In addition, OSPs allow researchersto replicate and extend the results of others and provide abaseline for comparison among various approaches.

In this article, we introduce eight quadrotor OSPs andcompare them in terms of hardware and software toprovide a compact overview for use in a variety of areas aswell as in academic research.

Open-Source Projects for Quadrotor UAVsIn this section, we introduce the quadrotor OSPs that arelisted in Table 1. These projects have been selected basedon the user volume, activity, and project maintenancestatus. All of these projects are still in development.Therefore, readers should note that the informationdescribed in this article is as of May 2012. We use the termOSP to refer to code, electronics, or auxiliary softwaresuch as the ground-control software (GCS), depending onthe context.

ArducopterArducopter is a quadrotor autopilot project based on theArduino framework developed by individual engineersworldwide, which is described earlier [Figure 1(a)]. Agraphical-user-interface (GUI)-based software GCS isprovided to tune control gains and display flight in-formation [Figure 2(a)]. This project shares the sameavionics platform with Ardupilot, which is a fixed-wingaircraft autopilot OSP. A helicopter autopilot is alsosupported. There are more than 30 contributors onthe project Web site and it uses the GNU Lesser GPL(LGPL) [16].

OpenpilotOpenpilot is an OSP led by RC hobbyists [Figure 1(b)]using GPL [14]. This project features a real-time operat-ing system modified from FreeRTOS, which is anopen-source operating system. Openpilot supportsfixed-wing aircraft and helicopters with the same auto-pilot avionics. A GUI-based GCS is provided to tune

(a) (b)

(c) (d)

(e) (f)

(h)(g)

©3D

RO

BO

TIC

S

©O

PE

NP

ILO

T C

C B

Y-S

A V

3©

PIX

HA

WK

TE

AM

CC

BY

-SA

V3

©FL

YC

AM

Figure 1. Open-source quadrotor autopilots. (a) Arducopter,(b) Openpilot, (c) Paparazzi, (d) Pixhawk, (e) Mikrokopter(photo courtesy of Inkyu-Sa), (f) KKmulticopter, (g) Multiwii(photo courtesy of Alexandre Dubus), and (h) Aeroquad (photocourtesy of Ian Johnston).

•Table 1. OSPs on quadrotor autopilot.

Project Name Web site URL

Arducopter http://code.google.com/p/arducopter

Openpilot http://www.openpilot.org

Paparazzi http://paparazzi.enac.fr

Pixhawk http://pixhawk.ethz.ch

Mikrokopter http://www.mikrokopter.de

KKmulticopter http://www.kkmulticopter.com

Multiwii http://www.multiwii.com

Aeroquad http://www.aeroquad.com



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


http://www.qmags.com/clickthrough.asp?url=http://www.aeroquad.com&id=17487&adid=P34E8

http://www.qmags.com/clickthrough.asp?url=http://www.multiwii.com&id=17487&adid=P34E7

http://www.qmags.com/clickthrough.asp?url=http://www.kkmulticopter.com&id=17487&adid=P34E6

http://www.qmags.com/clickthrough.asp?url=http://www.mikrokopter.de&id=17487&adid=P34E5

http://www.qmags.com/clickthrough.asp?url=http://pixhawk.ethz.ch&id=17487&adid=P34E4

http://www.qmags.com/clickthrough.asp?url=http://paparazzi.enac.fr&id=17487&adid=P34E3

http://www.qmags.com/clickthrough.asp?url=http://www.openpilot.org&id=17487&adid=P34E2

http://www.qmags.com/clickthrough.asp?url=http://code.google.com/p/arducopter&id=17487&adid=P34E1





gains and receive flight data [Figure 2(b)]. On the Website of this project, various videos are available describ-ing gain tuning, assembly processes, and flight princi-ples to help users.

PaparazziPaparazzi is an autopilot system oriented toward inexpen-sive autonomous aircraft of all types [Figure 1(c)]. It hasbeen in development since 2003 [13]. Originally a fixed-wing autopilot, it now supportsquadrotor configurations by modify-ing the control mixing rule. Ninedifferent autopilot hardware systemsare developed under the lead of thePaparazzi team at ENAC University.Paparazzi provides GUI-basedGCS with flight-path scripting thatmakes mission planning in out-doors convenient [Figure 2(c)].This project uses GPL for hardwareand software.

PixhawkPixhawk [12] uses onboard com-puter-vision algorithms developedby ETHZ Computer Vision Group[Figure 1(e)]. Among the projectsintroduced here, only the Pixhawkproject has computer vision equip-ment which has been used in severalpapers [17]–[19]. It also providesGUI-based GCS [Figure 2(d)] calledQgroundcontrol, which is a separateOSP in collaboration with theMAVlink protocol project. ThePixhawk project is available underthe GPL.

MikrokopterMikrokopter is a quadrotor autopilot system devel-oped by a subsidiary of HiSystems GmbH in 2006[Figure 1(e)]. GUI-based software for gain tuning andhealth monitoring is provided as shown in Figure 2(e).Mikrokopter is operated in well-organized Internetshops for their autopilot boards. In 2010, the Universityof Tasmania and the Australian Antarctic Division madeuse of Mikrokopter to monitor moss beds in Antarctica.

(a) (b) (c)

(g)(f)(e)(d)

Figure 2. Screenshots of open-source GCS. (a) Arducopter, (b) Openpilot, (c) Paparazzi, (d) Pixhawk, (e) Mikrokopter, (f) Multiwii,and (g) Aeroquad.

Ground Control Systems

Actuators Motor ControllersMotor

Counterclockwise Propeller

Clockwise Propeller

× 4

× 2

× 2

ESC

RS-232 Serial

Flight Controller

RC Receiver

Flight Avionics

12C, PPM

I/O Pins

Sensors

RS

-232 Serial

PP

M, S

eria

l

Wireless

Figure 3. Configuration of a typical quadrotor UAV system.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






The source code is available for noncommercial pur-poses only.

KKmulticopterKKmulticopter is contributed by 20 people around theworld [Figure 1(f)]. This project has targeted hobbyistswho want to capture aerial photographs using quadrotors.The autopilot hardware of this project is the most basicamong the projects described in this article. It is equippedonly with a triaxis gyroscope for inertial measurement andan 8-b microcontroller for control. No GCS is providedand gains are tuned by variable resistors on board.

MultiwiiMultiwii is a quadrotor autopilot system developed byRC hobbyists [Figure 1(g)]. This project uses an Ardu-ino board as a main processor while the sensor systemcan vary. This project aims to make the fabrication ofelectronics easy. It uses gyroscopes and accelerometersof the commercial off-the-shelf Wii motion controllerfrom Nintendo, which needs less soldering. GUI-basedGCS is provided as shown in Figure 2(f). GPL is used forthis project.

AeroquadAeroquad is a quadrotor autopilot based on Arduino[Figure 1(h)]. Similar to Multiwii, it uses a standard Ardu-ino board instead of making its own fully fledged singleboard. Aeroquad also provides GUI-based GCS softwareas shown in Figure 2(g). Arducopter was separated fromthis project in May 2010. GPL is used by Aeroquad.

Arduino PlatformAlthough the Arduino platform is not a quadrotor auto-pilot, we introduce it here because the Arducopter, Mul-tiwii, and Aeroquad projects all use it. Arduino is thename of both the open-source single-board microcon-troller circuit and the integrated development environ-ment (IDE). Arduino has a well-organized device driverlibrary for different sensors and actuators. It is fre-quently used for rapid prototyping because of the fol-lowing advantages:l Simple setup: Arduino’s IDE is easy to install, and

firmware can be easily downloaded via USB or RS-232without an expensive JTAG interface.

l Rich device drivers: There are more than 100 librariesrelated to hardware peripherals and signal analysis onthe Arduino platform.

l Operating systems supported: The Arduino developmentIDE is ported to Mac OS X, Windows, and Linux.

Components of Open-Source Projectsfor Quadrotor UAVsFigure 3 shows overall configuration of a typical quadrotorUAV, which consists of flight avionics, sensor systems,radio transmitters, receivers, and communication systems.

•Ta

ble

2.Main

components

oftheOSPquadro

tors.

Description

Arduco

pter

Openpilot

Paparazzi(Lisa/M

)Pixhawk

Mikrokopter

KKmulticopter

Multiw

iiAeroquad

Dim

ension(m

m)

66

340.5

36

336

51

325

40

330.2

44.6

350

49

349

N/A

aN/A

a

Weight(g)

23

8.5

10.8

835

11.7

N/A

aN/A

a

Processor

ATm

ega2560

STM

32F1

03CB

STM

32F1

05RCT6

LPC2148

ATm

ega644

ATm

ega168

Arduinob

Arduinob

Processorfrequency

(MHz)

16

72

60

60

20

20

8–20

8–20

Gyroscope

Motionprocessor

unit-6000

(MPU-6000)

ISZ/IDC-500

MPU-6000

ISZ/IDC-500

ADXRS610

ENC-03

ISZ/IDC-650

ITG3200

Accelerometer

MPU-6000

ADX330

MPU-6000

SCA3100-D

04

LIS344ALH

—LIS3L0

2AL

ADXL3

45

Magnetometer

HMC5843

HMC5843

HMC5843

HMC5843

KMZ51

—HMC5883L

HMC5883L

Barometer

MS5611

BMP085

MS5611

BMP085

MPX4115A

—BMP085

BMP085

aNotavaila

ble.B

ecause

theMultiw

iiandAeroquadsu

pportdyn

amichardware

configuration,thesize

dependsonco

nfiguration.

bTh

eprojectisbasedontheArduinoboard,sotheactualp

rocessorvaries.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Flight AvionicsFlight avionics for the various projects mentioned in thesection “Open-Source Projects for Quadrotor UAVs” areshown in Figure 1. Most of the introduced projects provideelectronic schematics for self-production. Typically, flightavionics consists of a processor, input/output (I/O) pins,and sensors. The I/O pins connect an off-the-shelf

electronic speed controller (ESC) and RC receiver to theflight controller. The sensor suite consists of a gyroscope,accelerometer, barometer, magnetometer, and global posi-tioning system (GPS).

Table 2 describes flight avionics composition. Mostflight avionics are full-fledged with six degrees of freedom(6DoF) inertial measurement unit (IMU), magnetometer,

•Table 3. Specifications of accelerometers.

Chip Name Outputs AxisSupply(V)

Power(mA)

MeasurementRange (g)

Bandwidth(kHz)

Nonlinearity(%) Dimension (mm)

ADXL330 Voltage Three axes 1.8–3.6 0.32 �3 1.6 (XY),0.55 (Z)

�0.3 4.0 3 4.0 3 1.45

SCA3100-D04 SPI Three axes 3.0–3.6 3 �2 2 �2 7.6 3 3.3 3 8.6

LIS344ALH Voltage Three axes 2.4–3.6 0.68 �6 1.8 �0.5 4.0 3 4.0 3 1.5

MPU-6000 I2C Three axes 2.375–3.46 0.5 �16 1 �0.5 4.0 3 4.0 3 0.9

LIS3L02AL Voltage Three axes 3.4–3.6 0.85 �2 1.5 �0.3 (XY),�0.5 (Z)

5.0 3 5.0 3 1.52

•Table 4. Specifications of gyroscopes.

Chip Name Outputs AxisSupply(V)

Power(mA)

Range(degree/s)

Response(Hz max.) Dimension (mm)

ENC-03 Voltage One axis 2.7–5.5 5 �300 50 15.5 3 8.0 3 4.3

IDG-500 Voltage Two axes (XY) 2.7–3.3 7 �500 140 4.5 3 5.0 3 1.2

ISG-500 Voltage One axis (Z) 2.7–3.3 4.5 �500 140 4.0 3 5.0 3 1.15

IDC-650 Voltage Two axes (XY) 2.7–3.3 7 �2,000 140 4.0 3 5.0 3 1.15

ISG-650 Voltage One axis (Z) 2.7–3.3 4.5 �2,000 140 4.0 3 5.0 3 1.15

ADXRS610 Voltage One axis 4.75–5.25 3.5 �300 2,500 7.0 3 7.0 3 3.0

MPU-6000 I2C Three axes 2.375–3.46 3.6 �2,000 256 5.0 3 5.0 3 1.52

•Table 5. Specifications of magnetometers.

Chip Name Outputs Axis Supply (V) Power (Ma) Range (G) Rate (Hz) Dimension (mm)

HMC5843 I2C Three axes 2.5–3.3 0.8 �4 116 4.0 3 4.0 3 1.3

HMC5883 I2C Three axes 1.6–3.3 0.64 �8 116 3.0 3 3.0 3 0.9

KMZ51 Voltage One axis 5.0–8.0 — �2.5 — 5.0 3 4.0 3 1.75

•Table 6. Specifications of barometers.

Chip Name Interface Supply VoltagePowerConsumption Range

Response Time(ms)

Dimension(mm)

MPX4115A Analog(voltage)

4.85–5.35 0.1mA (max) 150–1150 hPa 1 11.38 3 10.54 3 12.7

BMP085 I2C 1.8–3.6 5 lA 300–1100 hPa 7.5 5.0 3 5.0 3 1.2

MS5611 I2C 1.8–3.6 0.9–12.5 lA 450–1100mBar 0.5–8.22 3.0 3 5.0 3 1.7



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






and barometer. However, KKmulticopter has only threegyroscopes, because it is devoted to manual flight. KKmul-ticopter implements a stability augmented system (SAS),which will be discussed in the section “Open-SourceProjects Internals.” Most flight controllers implement pro-portional-integral-derivative (PID) control for stabiliza-tion of the quadrotor, although the structure of the PIDcontrollers between the projects varies slightly. This willalso be discussed in detail in the section “Open-SourceProjects Internals.”

SensorsDetailed specification of the sensors used in the OSPsis given in Tables 3–6. For the accelerometer, five differ-ent chips are used in the OSPs, which are shown inTable 3. For the gyroscope, there are seven differentchips used as listed in Table 4. Magnetometers are usedto correct attitude information and estimate drift ofgyroscopes. There are three different magnetometersused in the OSPs as shown in Table 5. Three typesof barometers used to measure altitude are shownin Table 6.

Radio Transmitters and ReceiversRecently, some groups have modified off-the-shelf RCtransmitters to fit their requirement such as complexcontrol mixing or curve shaping of a stick. As a result, cus-tom firmwares for a few RC transmitters have beenreleased as open source. In addition, open-source RCtransmitters and receivers have been emerging [Figure 4(a)and (b)]. The OpenLRS project was initiated for open-source RC radio transmitter and receiver development.The OSRC project has developed not only a radio part butalso controller hardware as shown in Figure 4(c). Theseprojects are useful when a flight-avionics package needs tobe more compact without additional hardware such as anRC receiver.

Communication SystemsXBee is a popular communication system because of itssimple setup, low cost, and reasonable communicationrange when compared with its size. All the projects ad-dressed here use Xbee.

Arducopter and Pixhawk imple-ment the MAVLink protocol forground control. One advantage ofthe MAVLink protocol is that onecan use Qgroundcontrol without aneed to develop separate GCS.

Open-Source Projects Internals

Attitude EstimationBecause a sensor suite is typicallycomposed of a three-axis gyroscopeand a three-axis accelerometer, which

provide linear accelerations and angular rates only, a properattitude estimation algorithm should be employed.

Extended Kalman FilterThe Openpilot and Pixhawk projects have designed anattitude estimation algorithm based on the extendedKalman filter (EKF). Here, we provide only an over-view of the EKF-based attitude estimation of the Open-pilot project, and the complete EKF algorithm can befound in [20].

Let p and v be three-dimensional (3-D) position andvelocity in earth-fixed frame, q the quaternion, and b thegyro bias. Let Reb(q) and X(q) be rotation matrix that con-verts body-fixed frame to earth-fixed frame and quater-nion rates matrix, respectively, as a function of the unitquaternion. Let a denotes linear acceleration in body-fixed frame and x the angular velocity in body-fixedframe. Then, the state equation in discrete time can bewritten as

xk ¼pkvkqkbk

2664

3775 ¼

vk�1

Reb(qk�1) � ak�112X(qk�1) � xk�1

wb, k�1

2664

3775: (1)

In (1), the gyro bias b is modeled with noise wb. The sys-tem input u consists of measurements of angular velocityxm and linear acceleration am:

uk ¼ xm, k

am, k

� �¼ xk � wx, k þ bk

ak � wa, k � RTeb(qk)½0 0g�T

� �, (2)

where wx and wa represent noise and g is gravitationalacceleration. Substitution of (2) into (1) yields the follow-ing nonlinear model:

xk ¼ f (xk�1, uk�1)þ wk�1

¼

vk�1

Reb(qk�1)(am, k�1 þ wa, k�1)þ ½0 0g�T12X(qk�1)(xm, k�1 þ wx, k�1 � bk�1)

wb, k�1

26664

37775, (3)

(a) (b) (c)

©F

LYT

RO

N

©F

LYT

RO

N

Figure 4. Open-source RC transmitters and receivers. (a) OpenLRS transmitter, (b) openLRSreceiver, and (c) OSRC transmitter.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






where wk ¼ ½wx, k,wa, k,wb, k�T is process noise. The non-linear measurement model is (we omit time index k fornotational simplicity)

zk ¼ h(xk)þ vk ¼pvmb

hb

2664

3775 ¼

pv

RTeb(q)me

�Pz

2664

3775, (4)

where mb is the measurement of the magnetic field of theearthme in body frame, hb is the heightmeasured by the baro-metric sensor reading Pz , and vk is the measurement noise.

The states are estimated by the standard EKF algorithmand measurements from accelerometers, gyroscopes,magnetometers, GPS, and barometer are fused to estimatethe states.

Linear Complementary FilterThe Mikrokopter project implements the linear comple-mentary filter (LCF) and is shown in Figure 5 on each axisof the accelerometer and gyroscope. It is designed to fusemultiple independent noisy measurements of the samesignal that have complementary spectral characteristics.The details of complementary filters can be found in [21]and [22].

Let yu be the rate measurement of the angle h and yx theangle measured by accelerometer. The complementary fil-ter to estimate the angle h is given by

_h ¼ yu þ kp(yx � h), (5)

where h denotes the estimate of h and kp is a gainthat determines crossover frequency. The complementaryfilter described in (5) assumes that there is no steady-stateestimation error. However, in practice, the gyro bias variesover time. To compensate for this, an integrator[Figure 5(b)] is added to obtain the following:

_h ¼ yu � bþ kp(yx � h), (6)

_b ¼ �kI(yx � h): (7)

Nonlinear Complementary Filters on the SO(3) GroupAn LCF is extended to the nonlinear SO(3) group [22] (Fig-ure 6). The final form of the filter with bias estimate is given by

_R ¼ R(Xy � bþ k)3 (8)

_b ¼ �kIk, b(0) ¼ b0 (9)

k ¼ vex(pa(~R)), ~R ¼ RTRy, (10)

where pa(~R) ¼ 1=2(~R� ~RT) andR, ~R 2 SO(3) are attitude estimateand estimate error, respectively, andthe vex operator is the inverse oper-ation of a skew-symmetric matrix.

Ry is the rotation matrix reconstructed using roll and pitchmeasured from the accelerometer. Xy is the measurementfrom the three-axis gyroscope. Because R has to satisfy theconstraint RTR ¼ I, the computation load becomes anissue in implementing this on an embedded system. Forthis, in [22], the filter based on quaternion is provided.Arducopter, Multiwii, and Aeroquad have implementedthis algorithm with the rotation matrix representation, andthe Paparazzi project provides both rotation matrix andquaternion representations.

We evaluated the nonlinear complementary filters (NCFs)on the SO(3) group using a Vicon motion capture system,which gives accurate ground-truth measurements. The atti-tude computed by the flight controller was sent to the GCSby an XBee 2.4 GHz transceiver. Attitude estimates of thequadrotor are shown in Figure 7, which suggests that the atti-tude computed by the algorithm is accurate and reliable.

ControllersIt is well known that the open-loop rotational dynamics ofa quadrotor are unstable as studied in [23]. The identifiedmodel reveals that poles are located in the right-half plane

Ryk

Ωy

R = RAR

RT

^

^

^ ^^

RTRy

–kpπ(R)∼

π(R)∼R

∼

Figure 6. NCF on SO(3) group [22].

Accelerometer

Accelerometer

Gyroscope

Gyroscope– +

+

C(s)

C(s)

Kp

Kp

Ki

yu

yu

yx

yx

Angle

Angle

1s

1s

1s

+

+ –

–

–

θ∧

b∧

b∧

θ∧

θ∧

θ∧

θ∧

Figure 5. Complementary filter blends two different sensors thathave different frequency responses [21]. (a) Complementary filterwithout bias compensation and (b) complementary filter withbias compensation.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






of the real-imaginary axis and damping ratio is negative.Therefore, it needs to be stabilized by a feedback controlalgorithm, for example, SASs [24].

SAS, which makes the aircraft stable via the ratemeasurement in the feedback loop, is popular in aircraftcontrol [25]. SAS is shown in Figure 8 with a dotted-linebox. It consists of rate feedback with gain. If SAS is appliedto a quadrotor, damping is increased. As a result, the quad-rotor becomes controllable by a user.

The KKmulticopter project implements its SAS exactlyas shown in Figure 9(f). Consequently, it has onlythree gyroscopes. Because the SAS only provides rateregulation, an autopilot is required to maintain the atti-tude of a quadrotor. We describe different autopilotstructures. We begin with a proportional-derivative con-troller to illustrate how different architectures producedifferent characteristics.

The Pixhawk project implements a single-feedback loopas shown in Figure 10(a). In this case, the controller is

Gc(s) ¼ KP þ KDs: (11)

Then, the unity-feedback closed-loop transfer function is

YR¼ (KP þ KDs)GP

1þ (KP þ KDs)GP: (12)

Desirable closed-loop poles can beachieved by adjusting KP and KD. Inaddition to the noise problem dueto differentiation, there is now a

SAS

Controller

Gc(s)r

–

+

Plant

yActuator Quadrotor

RateGyro

Kd

Figure 8. General attitude autopilot configuration [24]. Vehicle stability is enhanced bythe SAS and the vehicle attitude is controlled by the outer loop with an integrator.

5 10 15 20 25−5

0

5

Rol

l Ang

le (

°)

5 10 15 20 25−5

0

5

Pitc

h A

ngle

(°) True (Vicon) IMU

True (Vicon) IMU

(a)

(b)

Figure 7. Attitude estimates obtained from the Vicon systemplotted in black dashed curves, and the NCF on the SO(3) groupplotted in red solid curves.

(a) (b)

(c) (d)

(e) (f)

P

P s

+

+–

–

–

–

–

–

–

–

–

++

+

+

+ +

+

+

+

+

+

I 1

s1

s1

s1

sθ

θ

θ

θ

θ

θ

1s1

s1

s1

s1

s1

s1

s1PI

PI

PID

Pilot

P

PI

PI

P

θ

θ

θ

θ

θ

θ

θd

θd

θd

θd

θd

θd

Figure 9. Various PID control structures of the OSPs. (a) Arducopter, (b) Openpilot, (c) Paparazzi, Multiwii (d) Pixhawk, Aeroquad,(e) Mikrokopter, and (f) KKMulticopter.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






closed-loop zero at s ¼ �KP=KD, and this zero can cause alarge overshoot in case of a step disturbance unless theplant poles are heavily damped.

To resolve the above problem, consider the inner-loopconfiguration of Figure 10(b),

Gc ¼ KP,Hi ¼ KDs: (13)

Then, the close-loop transfer function is

YR¼ KPGP

1þ (KP þ KDs)GP: (14)

Therefore, with rate feedback, we can achieve the sameclosed-loop poles, but without an undesirable zero. Gyro-scopes directly provide the rate information of the aircraftattitude, so this particular compensator structure is useful.All the other OSPs employ inner-loop configurations,while Pixhawk uses the single-loop configuration.

The Arducopter project implements a controllerbased on an inner-/outer-loop structure as shown inFigure 9(a). To mitigate steady-state error, an integratoris added in the forward path. The controller of the Open-pilot project [Figure 9(b)] closely resembles Figure 8,which is proposed in [24]. The Paparazzi project also hasthe same structure, but the derivative term in the forwardpath is removed as shown in Figure 9(c). Therefore,Paparazzi controller is exactly the same as Figure 8. TheMultiwii project has the same control configuration asPaparazzi. The Pixhawk project implemented a standardPID controller as shown in Figure 9(d). The rate measure-ment is not used in this controller. Only the derivative ofthe error signal is used to provide control inputs to theplant. As mentioned earlier, this controller may not besuitable for dynamic maneuvering, because a step changein the reference input will cause an undesirable initialspike in the control signal. However, Pixhawk is designedto support indoor navigation with relatively slow motion,

so it is suitable for this purpose. Mikrokopter has only aproportional-integral (PI) controller in the forward pathas shown in Figure 9(e).

As mentioned earlier, the KKmulticopter project onlyimplements SAS that controls the body rate as shown inFigure 9(f). This controller is particularly useful for systemidentification experiments. This controller only depends onthe gyroscope output: u ¼ �Ky. In this case, we can per-form closed-loop system identification to obtain an open-loop systemmodel. Consider the following linear system:

_x ¼ Ax þ Bu, y ¼ Cx, u ¼ �Ky: (15)

Then, the closed-loop system becomes _x ¼ Aclx, whereAcl ¼ A� BKC. Therefore, once we perform closed-loopidentification to obtain Acl, the open-loop A matrix canalso be obtained. This is only possible when there is no

•Table 7. Summarized control structure and thenumber of gains to be tuned.

Project NameNumber of Gainson Each Axis

ControllerConfiguration

Arducopter 3 + 1 (antiwindup) PI + P

Openpilot 4 + 2 (antiwindup) PI + PI

Paparazzi 3 + 1 (antiwindup) PI + P

Pixhawk 3 + 1 (antiwindup) PID

Mikrokopter 2 + 1 (antiwindup) PI

KKmulticopter 1 P

Multiwii 3 + 1 (antiwindup) PI + P

Aeroquad 3 + 1 (antiwindup) PID

20 22 24 26 28 30 32 34 36 38 40−3−2−1

0123

Time (s)(a)

(b)

20 22 24 26 28 30 32 34 36 38 40Time (s)

Rol

l Ang

le (

°) RollReference

−4

−2

0

2

4

6

Pitc

h A

ngle

(°) Theta Reference

Figure 11. Attitude tracking result of the quadrotor. Arducopterautopilot is tested with ground truth. The delay between referenceand roll angle is due to communication delay.

(a)

(b)

GC

GC Gp

Gpy

y

Hi

r +

r + +

–

–

–

Figure 10. Feedback control with two configurations. Gp denotesthe plant and Hi is the inner-loop controller. (a) Feedback controlwith a single-loop configuration. (b) Feedback control with aninner-loop configuration.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






integrator involved and K is fully known, which is notalways the case for typical quadrotors in the market.

Controller ParametersControl structure and the number of gains to be tunedin each project are shown in Table 7. KKmulticopter isthe simplest one, which has only one gain for tuning.Among many controller configurations, PI+P is domi-nant. P is for the inner loop (rate feedback), and PI isfor the forward attitude error compensation.

Controller EvaluationWe have constructed quadrotors using five different auto-pilots among mentioned OSPs: the Arducopter, Paparazzi,Mikrokopter, KKmulticopter, and Multiwii. Among these

projects, Arducopter, Paparazzi, and Multiwii share thesame controller composition as shown in Table 7. Forqualitative evaluation, we mount markers on a quadrotorto acquire ground-truth data from the Vicon system. Thedesired angle is transmitted to the Arducopter-based quad-rotor while quadrotor attitude from the Vicon and thetransmitted commands are recorded simultaneously. Thesatisfactory attitude tracking result is shown in Figure 11.The delay is due to RC signal processing.

Selection GuidelinesWe have analyzed eight OSPs with attitude estimationalgorithm, control configuration, electronic components,and features. A comparison of features between OSPs isgiven in Table 8.

Availability of Flight AvionicsAll the projects we described provideelectronic schematic and bill of mate-rials to reproduce their flight avion-ics. However, it takes high initial costto manufacture electronics individu-ally. Only five projects among themare available for purchase now:Arducopter, Paparazzi, Mikrokopter,KKmulticopter, and Aeroquad. It isrecommended to start with theseprojects if a reader prefers to avoidelectronics fabrication.

•Table 8. Comparison of functionality between the OSPs.

Arducopter Openpilot Paparazzi Pixhawk MikrokopterKKmulti-copter Multiwii Aeroquad

Attitude estimationalgorithm

NCF EKF NCF EKF LCF — LCF NCF

GPS-based waypointnavigation

pD

p— D D Da —

Altitude holdp

Dp p p

— —p

Hardware in the loopsimulation

p p p— — — — —

Support of othermultirotor airframes

p p p—

p p—

p

Support of computervision

— — —p

— — — —

GCS providedp p p p p

—p p

Airframe designprovidedb

p p— — — — —

p

Camera stabilizationp p p

— D —p p

Availabilitycp

—p

—p p

—p

Open-source license LGPL GPL GPL GPL Dd — GPL GPL

Used by [26], [27] — — [28], [12] [29], [30] — — —

p: supported, D: partially supported (e.g., additional navigation electronics),—: not supported.

aOnly GPS-based homing is supported.bThe project provides a quadrotor airframe design in computer-aided design files.cThe project avionics on sale.dOnly noncommercial purposes.

(a) (b)

©P

IXH

AW

K T

EA

M C

C B

Y-S

A V

3

©P

IXH

AW

K T

EA

M C

C B

Y-S

A V

3

Figure 12. Pixhawk-based quadrotor platform with a camera and onboard computer[31]. (a) Pixhawk quadrotor platform and (b) flight environment of the Pixhawkquadrotor with ARToolkit markerboard on the floor.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Attitude Estimation Algorithm DevelopmentFor attitude estimation tests, Arducopter and Paparazziwill be a good choice. The other projects are equipped withtwo or more gyro chips, which are hard to be calibrated foralignment. Only Arducopter and Paparazzi are equippedwith 6-DoF IMU in a single chip: MPU 6000. The dynamicrange is the best among the accelerometers and gyroscopesas described in Tables 3–6.

Minimalistic ConfigurationAs studied in [23], an open-loop model can be easily iden-tified when control input is fully known and no integratorsexist in a controller as mentioned in the section “AttitudeEstimation Algorithm Development.” Because SAS isimplemented in [23] to identify the open-loop dynamics,KKmulticopter is a good choice to this end. The system issimple to understand and modify because a source codefor attitude control is less than 500 lines in C.

GPS-Based NavigationFor GPS-based outdoor missions (e.g., waypoint naviga-tion and hovering): Arducopter, Openpilot, Paparazzi, orMikrokopter will be a good choice. Only these projectssupport GPS-based navigation. Although Multiwii hasGPS, it only supports a homing capability to move a quad-rotor back to the initial position.

Vision-Based NavigationOnly the Pixhawk project supports vision-based naviga-tion capability. It can synchronize an IMU and a camera inhardware level, which allows tight integration of IMUmeasurements into the computer vision pipeline.

Open-Source Projects in Research

Vision-Based NavigationThe Pixhawk UAV is designed to be a research platformfor computer-vision-based autonomous flight [28]. ThePixhawk team has constructed a localization test setupusing augmented reality ToolKit+(ARToolKitþ). They successfully per-formed waypoint navigation using acamera on the localization test bedas shown in Figure 12.

In [26], adaptive image-basedvisual serving (IBVS) was integratedwith adaptive sliding mode controlbased on Arducopter. Figure 13 showsthe experiment in process where theinset picture is the image obtainedfrom the onboard camera. The fidu-cial marker and its tracking resultare shown.

Real-time vision-based localizationwas performed on a quadrotor systembased on Arducopter [Figure 14(a)]

[27]. This quadrotor is equipped with a frontal-view gray-scale USB2.0 camera with 640 3 480 pixel resolution.Image data from the camera are transferred to a single-board computer and processed in a real time [Figure 14(b)]to obtain the vehicle location based on a map createdin advance.

Indoor FlightA Mikrokopter-based quadrotor flew autonomously usinga laser range finder (LRF) [29]. Equipped with LRF, Gum-stix, and external IMU, it successfully performed autono-mous indoor navigation without external localizationsensors. Indoor position control based on an onboard LRFwas performed on the Mikrokopter-based quadrotor plat-form shown in Figure 15 [32]. An autoregressive movingaverage with exogenous terms model of the stabilized Mik-rokopter was identified in [30]. Recently, the quadrotorplatform with shared autonomy was investigated for infra-structure inspection [33].

Multiagent-related research can be easily performedon the indoor quadrotor flight system. Especially, as thecommunication topology between agents can be user-defined within the GCS, various settings and algorithmscan be exploited. Figure 16 shows three quadrotors in flightwhere an auction algorithm is being tested for online task

(a) (b)

Figure 14. Arduino-based quadrotor equipped with a camera, single-board computer, andexternal IMU synchronized with a camera [27]. (a) Experimental quadrotor and (b) real-timevision-based localization is running on the single-board computer on a quadrotor.

Figure 13. Arduino-based quadrotor platform with a camera,designed to perform IBVS [26].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






assignment. As described in Figure 17, each quadrotor isequipped with an onboard controller to track input com-mands sent by the GCS that collects position and/or atti-tude data of the quadrotors from the Vicon motioncapture system. Data from the onboard vision sensors aresent to the GCS using a dedicated communication link.

ConclusionsThis article has presented eight quadrotor OSPs withdescriptions of their avionics, sensor composition, analysisof attitude estimation and control algorithms, and compar-ison of additional features. Several research projects thatuse OSPs as a main flight controller are described.

Among the eight OSPs summarized in this article, wehave implemented five by utilizing the benefit of OSPs thatallow to build own systems at a low cost with less effort. Tobring out continued improvements based on communities’work, objective evaluations of OSPs remain an importantopen problem.

The meaning of OSP had been more about software,but it is expanding to hardware and even products. Thereis already a project that has open hardware blueprints anda 3-D model of the quadrotor airframe that can be orderedfrom 3-D printing services. Sharing the same platform willbecome easier with such services. We expect that moreOSPs for UAV will be initiated in the future.

Figure 16. Three quadrotors in an experiment for an onlinetask assignment algorithm.

Figure 15. Mikrokopter-based quadrotor equipped withGumstix and LRF [32].

Vicon Motion Capture System

Communication Electronics

Ground Control Station

ViconTracker Software

ControllerPCTx PPMGenerator

OnboardAttitude Controller

OnboardVision Sensor

Individual Quadrotors

IMU

2.4 GHz RadioTransmitter

2.4 GHz RadioReceiver

2.4 GHz VideoReceiver

2.4 GHz VideoTransmitter

Display Command

USBFrame Grabber

Operator UserInterface

Camera/MotionCapture Hardware

Figure 17. Overall hardware architecture of the indoor quadrotor flight system for Figure 16. Multiple layers indicate one for eachquadrotor.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






AcknowledgmentsThis work was supported in part by the National ResearchFoundation of Korea (NRF) grant funded by the Korea gov-ernment (MEST) (nos. 20120000921 and 2012014219), andby the New and Renewable Energy Program of the KoreaInstitute of Energy Technology Evaluation and Planning(KETEP) grant funded by the Korea government Ministry ofKnowledge Economy (no. 20104010100490).

References[1] M. Muller, S. Lupashin, and R. D’Andrea, “Quadrocopter ball

juggling,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems

(IROS), Sept. 2011, pp. 5113–5120.

[2] M. Hehn and R. D’Andrea, “A flying inverted pendulum,” in Proc.

IEEE Int. Conf. Robotics and Automation (ICRA), May 2011, pp. 763–770.

[3] V. Kumar. (2012, Feb.). Robots that fly and cooperate, in Proc. TED

Conf. [Online]. Available: http://www.ted.com/talks/lang/en/vijay_

kumar_robots_that_fly_and ,


control for precise aggressive maneuvers with quadrotors,” in Proc. Int.

Symp. Experimental Robotics, Dec. 2010.

[5] D. Mellinger, M. Shomin, and V. Kumar, “Control of quadrotors for

robust perching and landing,” in Proc. Int. Powered Lift Conf., Oct. 2010.

[6] S. Lupashin and R. D’Andrea, “Adaptive open-loop aerobatic maneu-

vers for quadrocopters,” in Proc. Int. Federation of Automatic Control

World Cong. (IFAC), 2011, pp. 2600–2606.

[7] Q. Lindsey, D. Mellinger, and V. Kumar, “Construction of cubic structures

with quadrotor teams,” in Proc. Robotics: Science and Systems, June 2011.

[8] P. E. I. Pounds, “Design, construction and control of a large quadrotor

micro air vehicle,” Ph.D. dissertation, Australian National Univ., 2007.

[9] N. Guenard, T. Hamel, and R. Mahony, “A practical visual servo con-

trol for an unmanned aerial vehicle,” IEEE Trans. Robot., vol. 24, no. 2,

pp. 331–340, Apr. 2008.

[10] S. Bouabdallah and R. Siegwart, “Towards intelligent miniature fly-

ing robots,” in Proc. Field and Service Robotics, 2006, pp. 429–440.

[11] G. Hoffmann, D. Rajnarayan, S. Waslander, D. Dostal, J. Jang,

and C. Tomlin, “The stanford testbed of autonomous rotorcraft for

multi agent control (STARMAC),” in Proc. Digital Avionics Systems

Conf., 2004, vol. 2, pp. 12.E.4–121.10.

[12] L. Meier, P. Tanskanen, F. Fraundorfer, and M. Pollefeys,

“PIXHAWK: A system for autonomous flight using onboard computer

vision,” in Proc. ICRA, May 2011, pp. 2992–2997.

[13] P. Brisset, A. Drouin, M. Gorraz, P. Huard, and J. Tyler, “The papar-

azzi solution,” in Proc. Micro Aerial Vehicle, Sandestin, Florida, 2006.

[14] Free Software Foundation, Inc. (2007, June 29). GNU general public

license [Online]. Available: http://www.gnu.org/copyleft/gpl.html.

[15] S. Cousins, B. Gerkey, K. Conley, and W. Garage, “Sharing software

with ROS,” IEEE Robot. Automat. Mag., vol. 17, no. 2, pp. 12–14, June 2010.

[16] Free Software Foundation, Inc. (2007, June 29). GNU lesser GPL

[Online]. Available: http://www.gnu.org/licenses/lgpl.html.

[17] L. Heng, L. Meier, P. Tanskanen, F. Fraundorfer, and M. Pollefeys,

“Autonomous obstacle avoidance and maneuvering on a vision-guided

MAV using on-board processing,” in Proc. ICRA, May 2011, pp. 2472–2477.

[18] G. Lee, M. Achtelik, F. Fraundorfer, M. Pollefeys, and R. Siegwart,

“Benchmarking tool for mav visual pose estimation,” in Proc. Int. Conf.

Control, Automation, Robotics and Vision, 2010, pp. 1541–1546.

[19] G. H. Lee, F. Fraundorfer, and M. Pollefeys, “MAV visual SLAM

with plane constraint,” in Proc. ICRA, May 2011, pp. 3139–3144.

[20] D. Simon, Optimal State Estimation: Kalman, H [infinity] and Non-

linear Approaches. Hoboken, NJ: Wiley, 2006.

[21] A. Pascoal, I. Kaminer, and P. Oliveira, “Navigation system design

using time-varying complementary filters,” IEEE Trans. Aerosp. Electron.

Syst., vol. 36, no. 4, pp. 1099–1114, 2000.

[22] R. Mahony, T. Hamel, and J.-M. Pflimlin, “Nonlinear complemen-

tary filters on the special orthogonal group,” IEEE Trans. Automat.

Contr., vol. 53, no. 5, pp. 1203–1218, June 2008.

[23] D. S. Miller, “Open loop system identification of a micro quadrotor

helicopter from closed loop data,” Master’s thesis, Univ. Maryland, Col-

lege Park, 2011.

[24] F. Lewis and B. Stevens, Aircraft Control and Simulation. Hoboken,

NJ: Wiley, 2003.

[25] R. Nelson, Flight Stability and Automatic Control. New York:

McGraw-Hill, 1989.

[26] D. Lee, H. Lim, H. J. Kim, and Y. Kim, “Adaptive image-based visual

servoing for an under-actuated quadrotor system,” AIAA J. Guid. Control

Dyn., vol. 35, no. 4, pp. 1335–1353.

[27] H. Lim, S. N. Sinha, M. Cohen, and M. Uyttendaele, “Real-time

image-based 6-dof localization in large-scale environments,” in Proc.

IEEE Conf. Computer Vision and Pattern Recognition (CVPR), June 2012,

pp. 1043–1050.

[28] L. Meier, P. Tanskanen, L. Heng, G. Lee, F. Fraundorfer, and M. Pol-

lefeys, “PIXHAWK: A micro aerial vehicle design for autonomous flight

using onboard computer vision,” Auton. Robots, vol. 33, no. 1, pp. 21–39,

2012.

[29] S. Grzonka, G. Grisetti, and W. Burgard, “A fully autonomous

indoor quadrotor,” IEEE Trans. Robot., vol. 28, no. 99, pp. 1–11, 2012.

[30] I. Sa and P. Corke, “System identification, estimation and control for

a cost effective open-source quadcopter,” in Proc. IEEE Int. Conf. Robotics

and Automation (ICRA), 2012, pp. 2202–2209.

[31] ETHZ PixhawkMAV [Online]. Available: https://pixhawk.ethz.ch.

[32] I. Sa and P. Corke, “Estimation and control for an open-source

quadcopter,” in Proc. Australian Conf. Robotics and Automation, 2011.

[33] I. Sa and P. Corke, “Vertical infrastructure inspection using a quad-

copter and shared autonomy control,” in Proc. Int. Conf. Field and Service

Robotics, 2012.

Hyon Lim, Department of Mechanical and AerospaceEngineering, Seoul National University, Seoul 151-742,South Korea. E-mail: [email protected].

Jaemann Park, Department of Mechanical and AerospaceEngineering, Seoul National University, Seoul 151-742,South Korea. E-mail: [email protected].

Daewon Lee, Department of Mechanical and AerospaceEngineering, Seoul National University, Seoul 151-742,South Korea. E-mail: [email protected].

H.J. Kim, Department of Mechanical and Aerospace Engi-neering, Seoul National University, Seoul 151-742, SouthKorea. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_______________

_____________

_____________

_____________

_____________

_____________

http://www.qmags.com/clickthrough.asp?url=http://www.gnu.org/licenses/lgpl.html&id=17487&adid=P45E3

http://www.qmags.com/clickthrough.asp?url=http://www.gnu.org/copyleft/gpl.html&id=17487&adid=P45E2

http://www.qmags.com/clickthrough.asp?url=www.ted.com/talks/vijay_kumar_robots_that_fly_and_cooperate.html&id=17487&adid=P45E1

http://www.qmags.com/clickthrough.asp?url=https://pixhawk.ethz.ch&id=17487&adid=P45E4









•By Teodor Tomi�c, Korbinian Schmid, Philipp Lutz, Andreas D€omel, Michael Kassecker,

Elmar Mair, Iris Lynne Grixa, Felix Ruess, Michael Suppa, and Darius Burschka




Urban search and rescue missions raise special requirements on roboticsystems. Small aerial systems provide essential support to human taskforces in situation assessment and surveillance. As external infrastruc-ture for navigation and communication is usually not available, roboticsystems must be able to operate autonomously. A limited payload of

small aerial systems poses a great challenge to the system design. The optimal trade-off between flight performance, sensors, and computing resources has to be found.Communication to external computers cannot be guaranteed; therefore, all

Research Platform for Indoor and Outdoor

Urban Search and Rescue

©PHILIPP

LUTZ


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






processing and decision making has to be done on board.In this article, we present an unmanned aircraft systemdesign fulfilling these requirements. The components ofour system are structured into groups to encapsulate theirfunctionality and interfaces. We use both laser and stereovision odometry to enable seamless indoor and outdoornavigation. The odometry is fused with an inertialmeasurement unit in an extended Kalman filter. Naviga-tion is supported by a module that recognizes knownobjects in the environment. A distributed computationapproach is adopted to address the computational require-ments of the used algorithms. The capabilities of the sys-tem are validated in flight experiments, using a quadrotor.

Civil and commercially oriented unmanned aerial vehi-cle (UAV) missions range from rather modestly structuredtasks, such as remote sensing (e.g., wild-fire detection), tohighly complex problems including common security jobsor search and rescue (SAR) missions. Disaster search andurban rescue-related missions are still a fairly demandingchallenge because of their exceedingly variable nature. Themission planning has to take a multitude of scenarios intoaccount, considering arbitrary, unknown environmentsand weather conditions. It becomes apparent that it is alsonot feasible to preconceive the large number of unforesee-able events possibly occurring during the mission. Inresearch, certain types of SAR, in particular, wildernessSAR [1], [2], have already been successfully mastered usingUAV systems. However, to date, the persistent perform-ance of robotic systems operating in an urban environmentis very challenging, even with a low degree of human inter-vention [3]. A stable broadband radio link cannot be guar-anteed in such environments because it requires a highlevel of autonomy of the systems. The limited availabilityof computing resources and low-weight sensors operatingin harsh environments for mobile systems pose a greatchallenge in achieving autonomy.

Our research goal is to develop robotic systems capableof accomplishing a variety of mixed-initiative missions fullyautonomously. Completely autonomous execution of urbanSAR (USAR) missions poses requirements on the roboticsystems operating therein. Various such missions requirethe robots to be modular and flexible in terms of sensor andplanning capabilities. The robots have to operate in unstruc-tured indoor and outdoor environments, such as collapsedbuildings or gorges. Navigation systems therefore have towork without external aids, such as global positioning sys-tems (GPS), since their availability cannot be guaranteed.Flying systems additionally have to provide robust flightcapabilities because of the changing local wind conditionsin such environments. A key feature to achieving fullautonomy in urban disaster areas is on-board processingand decision making. Search assignments also require mis-sion-specific recognition capabilities on the robots.

As a first step, we have developed a modular and extensi-ble software framework for autonomous UAVs operating inUSAR missions. The framework enables a parallel and

independent development of modules that address individ-ual challenges of such missions. It features reliable flight andnavigational behavior in outdoor and indoor environments,and permits execution of higher level functions such as theperception of objects and persons, failsafe operation, andonline mission planning. The framework is implementedand tested on a commercial quadrotor platform. A quadro-tor has been chosen because of its favorable rotor size andsafety when compared to a conventional small-size helicop-ter. The platform has been extended in terms of sensor,computer, and communication hardware (Figure 1).

Similar platforms have already been developed by otherresearchers. The platforms are tailored to solve a simulta-neous localization and mapping (SLAM) problem. Thisproblem requires a lot of computational power, which isnot readily available on flying systems. Therefore, theauthors in [4] and [5] chose to send laser scanner data to aground station for processing. Pose estimation and high-level tasks are done on the ground station, whereas controland stabilization of the platform are done on the quadro-tor. More recently, through the optimization of algorithmsand faster processors, pose estimation and planning havebeen done onboard. Notable implementations are laserbased [6] for indoor environments and monocular visualSLAM [7] for both indoor and outdoor environments. Thepose estimate of the SLAM solution is commonly fusedwith inertial measurement unit (IMU) measurements inan extended Kalman filter (EKF) to obtain a full-state esti-mate, which is then used for control.

Our approach differs from the previous work in threemajor ways. First, instead of one sensor, we rely on twocomplementary exteroceptive sensors. This enables flight inboth indoor and outdoor environments. As in state-of-the-art systems, the respective odometry is fused with the IMUusing an EKF. Second, no geometric map is built. Instead, wecorrect for drift errors by recognizing known landmarks in

3

1 2

5 6

4

7

Figure 1. An experimental platform based on the AscendingTechnologies Pelican quadrotor, showing 1) laser scanner, 2)mirrors, 3) stereo cameras, 4) a modular computation stack, 5)wired Ethernet connection, 6) XBee modem, and 7) WLAN stick.One of the propellers is pointing downward to improve the viewof a front-facing camera (not depicted).

PHILIPPLUTZ



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






the environment. This lends itself to navigation in largerenvironments, as memory requirements are smaller whencompared to SLAM. To guarantee our robots’ autonomy, allprocessing is done onboard, akin to most recent systems.There are many computationally intensive tasks, which needto run simultaneously—stereo processing, visual odometry,laser odometry, and computer vision. In contrast to the state-of-the-art approaches, we adopt a distributed computationplatform consisting of several onboard boards instead of one.

The components of our framework are inspired by theinternational micro air vehicle (IMAV) [8] indoor explora-tion challenge. The objective of the challenge is to fly into ahouse of known shape and dimensions, detect objects, andreturn outside to land on a defined pattern. Several prob-lems found in USAR missions have to be addressed. TheUAV system must first find the house, as it starts behind awall without direct sight of the house. Once found, precisedetection of and navigation through the door, window, orchimney of the house is required. Pattern recognition isused for object detection and landing zone identification.External aids, such as GPS or a motion tracking system,are not available. Adding artificial features to the environ-ment is penalized. Although not all difficulties of a USARmission are addressed, the navigation, autonomous deci-sion making, and object recognition challenges are present.The size of our 70-cm-wide platform in relation to the 1-m-wide passages into the house poses an additional chal-lenge. Our system’s architecture is explained in terms ofthe aforementioned mission.

Current approaches, which try to tackle this kind of chal-lenge, use a laser scanner [5] or monocular vision [9]. The

processing in these approaches was done offboard. For thevision system, artificial features had to be added to the indoorenvironment. Our system will use the best odometry sensorin a given situation. Systems have autonomously flown intothe house through the window and doors; however, no sys-tem has yet flown the complete mission autonomously.

Software FrameworkOur modular framework consists of intercommunicatingcomponents, enabling the easy exchange of task-relatedfunctionality and exchangeability of components. Tofurther define their scope, the system components are sub-divided considering their degree of autonomy and cogni-tive functionality, as depicted in Figure 2.

Concerning the level of autonomy, the system is structuredinto low- and high-level components. The low-level compo-nents are responsible for the data fusion and flight control ofthe quadrotor. They allow for reliable autonomous flight andnavigation, shared through an abstract and unified interfacebetween humans and high-level components. As the stabilityof the system depends on these components, a hard real-timesystem with a high execution rate is required. The high-levelcomponents provide situational awareness and mission plan-ning functionality with a representation of the environment.The status of the system, the mission, and the environmentare monitored, and commands are issued accordingly. Theytake over tasks usually done by a human operator.

Furthermore, the components are grouped into percep-tion, cognition, and action layers [10], [11] as depicted inFigure 2. The perception layer includes all tasks concerningacquisition and processing of sensor data. Therein, the data

IMU Cam Left Cam Right Laser Cam Up Cam Front

StereoProcessing

Laser Transform

Data Fusion Odometry Recognition

Environment Path Planning Mission Control

Controller

Autopilot

Perception

Cognition

Action

Sensing

Low Level High Level

sync

Atom Gumstix 1 Gumstix 2 Gumstix RT Peripheral Real Time

Figure 2. System architecture and software deployment of components. In addition to being organized into cognitive layers, thesystem components are partitioned according to their autonomy level as well as cognitive functionality.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






fusion component fuses the proprioceptive and exterocep-tive sensor data. Mission-dependent recognition of worldfeatures, such as persons or interesting objects, is done inthe recognition module. World representation, as well asplanning and decision-making functionalities, is realized inthe cognition layer. Lastly, the action layer is involved in sta-bilizing and moving the UAV in the desired manner. Suchcategorization allows for a clear definition of the interfacesbetween components and the minimum required set ofcomponent functionalities. The realization of this structurein our experimental system can be presented more clearlywhen grouped by the layer subdivision.

PerceptionThe UAV should be able to fly in structured indoor environ-ments as well as outdoors. The indoor environments consistof clearly defined vertical structures (walls) that can bedetected by a laser scanner. However, poor lighting conditionsand a low number of environment features make indoor envi-ronments unsuitable for a camera-based odometry system.Conversely, the outdoor environments lack clear structures.Sunlit environments contain light in the part of the spectrumthat coincides with that used by infrared laser scanners, dis-turbing the measurements. This makes low-powered light-weight laser scanners, which are commonly employed onflying systems, unsuitable for such environments. Outdoorenvironments have many natural features and good lightingconditions, which makes them well suited for visual odometrysystems. In such environments, previous camera images can beeasily recognized, so the camera can be used for loop closure.

In our approach, we use both laser and stereo odometryfor pose estimation. The combination of two odometryapproaches allows compensating drawbacks of a single sen-sor. Moreover, the estimation of all six degrees of freedom(6 DoF) states can be done using only one filter. This differsfrom other approaches, where either laser odometry [6] ormonocular visual odometry [7] is used for pose estimation.

The stereo camera in our system points downward notonly to ensure that the odometry is available in outdoor areasbut also to enable detection of a target from above. Drifterrors can be compensated by using key frames in the visualodometry system, as well as recognition of known landmarksin a topological map. For the indoor exploration mission, themap is fixed and predefined, as known landmarks include thewindow, door, and chimney. Their exact position is knownwith respect to the house, so they can be used to correct drifterrors. These are detected and tracked using front-facing andupward-facing cameras (not shown in Figure 1), respectively.Two separate cameras provide more stable tracking than apan-tilt unit with one camera of the same weight.

Odometry

Laser OdometryThe laser odometry system is based on Censi’s canonicalscan matcher [12]. The laser scan is projected to the ground

plane in the laser transform component, using attitude infor-mation from the data fusion component (Figure 2). The pro-jected data are only valid for scan matching if the scannedenvironment objects contain vertical planes. This assump-tion is valid for most indoor environments. The algorithmuses an iterative closest point (ICP) variant to computethree-dimensional (3-D) delta movement information[change in (x, y) position and yaw angle] between two pointsin time and the corresponding measurement covariance.

Visual OdometryA correlation-based algorithm [13], [14] is used to obtain adisparity image from two time-synchronized cameraimages in the stereo-processing component. Based on this3-D information, the six-dimensional delta position andorientation between two points in time as well as the corre-sponding measurement covariance are calculated [15]. Thealgorithm supports a key frame buffer so that the deltameasurement refers not just to the last acquired image butalso to the image in the buffer that gives the delta measure-ment with the smallest absolute covariance.

As shown in the “Experimental Results” section, theestimated variances for laser and camera odometry are agood indicator to classify the environment into indoor andoutdoor. In the variance calculation for each sensor, it isassumed that there are no outliers in the measurement.During the experiments, we have found that, under badsensor conditions, outliers in the measurements occurred.These could not be detected by an outlier rejection mecha-nism using Mahalanobis distance. Therefore, the measure-ment variance is invalid. Fusing these measurementswould lead to unpredictable behavior of the filter. Becauseof this, we switch to the sensor that works well in a specificenvironment. We assume that the sensor with the smallestmeasurement variance is best suited in the current envi-ronment and is therefore used for fusion.

Data FusionThe proprioceptive sensor information from the IMU andthe exteroceptive odometry information have to be fusedto get the current system state estimate. There are twomain challenges.

First, the odometry data give only relative position andorientation information. Second, the odometry data are timedelayed because of measurement and data processing time.Precise times of measurement are obtained through hard-ware synchronization triggers. The total delay of the laserodometry in the experimental system is about 100 ms withan update frequency of 10 Hz, and for the visual odometry,the delay is more than 300 ms, with a frequency of 3 Hz.Therefore, the measurement refers to a state in the past. Asthe estimate is used to control the UAV, and the quadrotordynamics are fast compared to the measurement delays, thelatter have to be considered in the data fusion algorithm.This is realized using an indirect feedback Kalman filter withstate augmentation [16] using two state vectors.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






The direct state vector includes position, velocity, andattitude (as quaternion) in the world frame and the biasesof the acceleration and gyro sensors in the body frame2 R16. Quaternions are used to circumvent the gimbal lockproblem that might occur when using minimal attituderepresentations. The direct state is calculated by the strap-down algorithm.

The main filter state vector includes the errors in posi-tion, velocity, and orientation in the world frame and theIMU acceleration and gyro bias errors in the system bodyframe 2 R15. Since we assume small errors in the filter, thesmall angle approximation is employed to efficiently repre-sent the attitude. Hence, the scalar part of the quaternioncan be omitted, as it is implicitly defined to one. This alsosimplifies modeling of the attitude sensor noise. A hard-ware synchronization signal of the laser and the camerasystem signaling the start of a data acquisition sequence isdirectly registered by the real-time system running the fil-ter algorithm. At every synchronization trigger, a substateincluding position and orientation of the current directstate is saved and augmented to the main filter state. Thedelayed delta measurement includes two time stamps foreach measurement. These time stamps are used to find thecorresponding states within the state vector and construct asuitable measurement matrix referencing the selected states.

The absolute position and orientation of the system areunobservable with only delta measurements, which arereflected in an unbounded covariance for these states.Therefore, further absolute measurements are included:l The height to ground is measured by laser beams

reflected to the ground. Height jumps caused by objectslying on the ground are detected and compensated.

l Measurement of the gravity vector is used as pseudoab-solute measurement for roll and pitch.

l Measurements with respect to known landmarks, ifavailable, are used to correct position drift errors.

If absolute position measurements arrive only in the rangeof minutes, there might be small jumps in the position esti-mate. Nevertheless, these jumps do not cause jumps in thevelocity estimate as its covariance is bounded by the regulardelta position measurements. This is an important featurefor the underlying UAV controller, as jumps in the velocityprediction can significantly degrade flight performance.

RecognitionIdentifying and locating persons, animals, or objects (e.g.,landmarks, signs, or a landing zone) is a central issue inUSAR missions. The conceptual idea behind the recogni-tion module is to offer related object detection and recog-nition services. The module acts as an interface betweenthe mission planner, environment (cognition), and thesensors. Triggered by the mission planner, it interpretssensory information and returns semantic and locationinformation, respectively. The recognition module sup-ports absolute localization in the sense that it detectsknown objects and estimates their relative positions and

heading with respect to the UAV frame. It leverages typicalobject recognition techniques in computer vision and 3-Dpoint-cloud processing. Three demonstrator recognizersare currently implemented: a pattern recognizer for two-dimensional images, a house detector based on stereovision, and a laser object detector.

The pattern recognizer is typically used when searchinga marked landing zone. The pattern is a gray-value imageor drawing of the landing zone. Together with a descrip-tion of its size, the pattern matcher checks for similaroccurrences in camera images in a more efficient way thana common template matching approach. The first steps tryto reduce the problem size by segmenting the image data.A corner detector is applied to the template image toobtain interesting points. Small patches are then generatedaround these points and stored in a database. A simpleoperation to calculate a descriptor of the patches based onorientation histograms is used [17]. Subsequently, thedescriptors are compared pairwise using normalized cross-correlation, sorted in an arbitrary number of classes, and aBayes classifier is learned. This results in a sequence ofdescriptors, which in turn define the pattern.

Specifically for the IMAV challenge, we have developeda house detector and laser object detector. The housedetector uses disparity images. It is used to detect the housefrom above, since the cameras are pointing downward. Acombination of principal component analysis and an algo-rithm similar to the ICP algorithm is used to fit the shapeof a model of the house into the point cloud. In addition,parts of the house (e.g., chimney) are identified in amonocular image of the stereo pair and fused with resultsof the point cloud fitting. The laser object detector is ableto detect corners in a room, walls, and windows.

ActionThe controller component implements a position control-ler running at 100 Hz on the real-time system. Controlinputs are attitude commands that are sent to the autopi-lot, which implements a PD attitude controller in a 1-kHzcontrol loop. The purpose of the position controller is tofollow a reference position, velocity, and acceleration,using the data fusion’s pose estimate. The position control-ler is a full-state feedback controller that uses a combina-tion of integral sliding mode [18] and time-delaydisturbance estimation [19]. The integral action provides azero steady-state error, whereas the disturbance estimatoruses accelerometer measurements and previous controlinputs to respond to disturbances faster. This combinationprovides sufficient robustness to fly in indoor and outdoorenvironments and through narrow passages.

In-flight switching and configuration of position con-troller implementations simplify their testing. The positionbetween two waypoints is interpolated as a straight line inCartesian space using a constant velocity. This interpolatedposition is run through a linear filter that represents thequadrotor’s translational dynamics to generate smooth



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






reference trajectories. Using thismethod, it is easy to configure thetransient behavior of the vehicle’sposition by setting the interpolationvelocity and filter parameters. Suchconfigurations are stored as flightmodes (e.g., fast, careful, accurate bydecreasing velocity). A unified inter-face allows flight modes to be setfor each path segment individually.High-level components can set theflight mode according to the currentmission task.

A state machine in the low-levelsystem is implemented in the con-troller as depicted in Figure 3 anddefines the activation of compo-nents based on the readiness level ofthe system, as summarized in Table 1.It signals the availability of low-levelabilities to high-level components.The system starts in the preparationstate, during which data fusion and position control aredisabled, so the quadrotor can be moved freely to a startingposition. This is useful for initialization of the systembefore a mission, as the movement will not affect the stateestimate. It is assumed that the quadrotor is stationary inthe powered off state, so the data fusion is initialized andstate estimation starts. Upon engaging the motors, the sys-tem is in the powered on state and the sensor data fusionassumes that the quadrotor is moving. The initial statetherein is landed, which assumes that the quadrotor isstill on the ground. The take-off command activates theposition control feedback loop, while commandingthe quadrotor to hover at a predefined height above thestarting location. During the ascend, the system is in thetransitional taking off state, which can be canceled with theland command. Once the hover point is reached withdefined accuracy, the system is automatically transitionedinto the nominal fault-free flying state, and paths can onlybe flown in this state. Landing occurs through the transi-tional landing state analogously to taking off. The transi-tional states ensure that corresponding physical changeshave been safely completed before allowing any otheractions to be taken. This abstraction greatly simplifiesexperiments, since components are activated and initial-ized as needed.

To ensure a fall-back strategy in case of fatal errors duringflight, a fail-safe state is implemented in which the systemperforms an emergency landing routine. For example, thesystem enters this state automatically if a data fusion diver-gence has been detected. This shall protect the system andminimize the possibility of harm to humans or to the plat-form itself. In such an event, fused data are not used for posi-tion control—instead, only the raw altitude measurementfrom the laser scanner is used for descending, while the

vertical velocity is obtained by using an a–b filter of themeasured altitude. Altitude stabilization is active through theautopilot and does not depend on the fusion information. Inthis state, the quadrotor’s horizontal position will drift, butmore importantly, the quadrotor will not crash to the ground,and it is easier for a safety pilot to take over. This method ofdescent is safer than simply reducing the thrust or turningthe motors off. It has proven to be useful during experimentsand when testing new components.

CognitionCompletely autonomous execution of USAR missionsrequires interpretation of the robot’s environment and theperforming of actions accordingly. To achieve that, therobot requires a representation of the world, as well as path-planning and decision-making capability. In our frame-work, these are implemented in the cognition layer. Itsmodularity allows for implementation of different algo-rithms through the use of the available interfaces and choos-ing the combination best suited for a particular mission.

Preparation Powered Off

Landing

LandedEmergency Landing

Taking Off

Flying

Powered On

Entry Point DonePrepare

Engage Motors

Take OffLand

Finished

Land

Finished

Finished Disengage Motors

Exception

Exception

Exception

Figure 3. Low-level state machine implemented in the controller component.

•Table 1. Activation of system componentsdepending on low-level system state.

System State Fusion Control Waypoints

Preparation — — —

Powered off — —

Landed — —

Taking off —

Flying Landing —

Emergency landing — —



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






For the exploration challenge, the current implementa-tion of the environment component contains a worldmodel and objects therein as a topological map. If a knownobject is detected with a high confidence by the recognitioncomponent, drift errors can be corrected by sending therelative location of the object to the data fusion. The mis-sion control component provides autonomous missionexecution through a hierarchical state machine composedof tasks. The abilities are atomic states of each task, andthey represent basic functionalities or actions provided byother system components and can be invoked withparameters.

For example, the FlyTo ability invokes the path plan-ning component, which uses environment information tofind a list of waypoints from the current position to thedesired topological pose while avoiding obstacles anddangerous zones. Together with the specified flight mode,the determined path is then sent to the controller for exe-cution, i.e., the plan is static while the ability is active. Oncethe last waypoint is reached with sufficient accuracy, atransition is triggered.

We will illustrate this by the leavehouse by window task, depicted inFigure 4. The quadrotor first flies toa window position that is stored in amap. Once this position is reached, awindow detector tries to determinethe window position more preciselyusing vision and thereafter slowlyapproaches it. If successful, the UAVflies to the determined position.

At this point, the visual servoingstarts. The position determined bythe window tracker is continuouslysent to the controller. The UAVslowly flies through the windowmidpoint while the window is visi-ble. Once the other side of the win-dow is reached, the task is finished.Window detection and trackingservices are provided by the recogni-tion module. If the window cannotbe detected precisely, the FindPosi-tionIndoor fall-back task is invoked.This determines the quadrotor posi-tion in relation to the house. In thecase of too many detection or track-ing failures, the task exits to a fall-back task, like leaving the housethrough the door.

Hardware and InfrastructureBecause of high payload capacity,the Ascending Technologies Pelicanquadrotor was chosen as the flightplatform, which is shown in Figure 1.

With a total weight of 2.05 kg, our system hovers atapproximately 70% of the quadrotor’s maximum thrust,leaving a control reserve that is only sufficient for relativelyslow maneuvers. Maximum flight time is approximately 10min with one accumulator. The used hardware compo-nents are listed in Table 2. The most notable difference tosimilar systems is the time-synchronized modular compu-tation stack, connected through Ethernet.

A Hokuyo UTM-30LX laser scanner and PointGreyFirefly cameras are used as exteroceptive sensors via USB,connected to different computers to parallelize the dataacquisition process.

The onboard computational hardware consists of oneCoreExpress Atom board, three Gumstix Overo Tideboards, and an Ethernet switch, as shown in Figure 5. Theatom board is used for stereo processing because ofhigh computational requirements. Image processing andcognition tasks are executed on dedicated Gumstix boards.If more computational power is required, computerscan be added to the system without changes in thesystem architecture.

environment = indoor,position = known

entry point

FlyTo: windowExitPosition, accurate

Detect: window

FlyTo: windowExitPosition, accurate

Track: window

FlyTo: windowTrackingPosition, careful

FindPositionIndoor

(exit) fall back

exit pointenvironment = outdoor,

position = known

positionReached

windowPositionEvaluated detectingWindowFailedNTimesen

able

Tra

ckin

gFai

led

positionReached

trackingEnabled

trackingWindowFailedNTimes

positionReached

detectingWindowFailed

actual Position Found

Ability:parameters

Task

condition

Figure 4. Example of a mission control task–leave house by window. The task is a statemachine composed of atomic states called abilities. Tasks are fixed and mission specific.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






The time-critical strapdown, data fusion, and controltasks run on the real-time system. It is an Ubuntu Linuxwith an RT-patched kernel and is connected to the autopi-lot, which also provides the IMU. A high IMU poll rate isrequired for the strapdown algorithm; therefore, I2C com-munication with the autopilot runs at 400 kHz. The result-ing bandwidth allows IMU data polling at 200 Hz andattitude command sending from the position controller at100 Hz. All remaining Gumstix computers are runninglatency-tolerant high-level tasks such as image processingandmission control on an Ubuntu Linux operating system.

An eight-port fast-Ethernet switch is used for high-bandwidth onboard communication between the comput-ing hardware, shown in Figure 6. Ethernet has been chosenbecause of well-established standard protocols, low-latencycommunication, and readily available middleware.Screwable GIGCON connectors provide vibration-resist-ant connections of the Ethernet cables. An external dataconnection to the system is possible through wired Ether-net, WLAN, XBee modem, and USB. WLAN is madepossible through a tiny USB stick connected to the Atomcomputer with a maximum bandwidth of 150 Mb/s. TheUbuntu Linux on the Atom processor runs a softwarebridge where incoming connections from WLAN arerouted to the internal onboard network. A slower andmore reliable connection is provided by the XBee modemconnected to a serial port of the Atom computer.Lastly, each of the Gumstix computers provides a serialterminal interface over USB, used only when the system isnot flying.

The distributed approach of splitting tasks amongmulti-ple computers requires a suitable middleware to enablecommunication between them. Our software frameworkposes the following requirements: scalability and supportfor distributed nodes; clock synchronization; flexible dataformats and API; small footprint (usable for embedded sys-tems); suitability for robotic applications and software. As aresult of the evaluation of differentframeworks and middleware, robotoperating system (ROS) was chosenas most suitable, although it lacksclock synchronization and real-timecommunication because of its design.

All real-time critical tasks run asthreads in a single process (nodelet),so they communicate throughshared memory. A good example isthe data flow from IMU to datafusion to controller to autopilot ascan be seen in Figure 2. This zerocopy transport approach is also usedto reduce communication overheadwhere large amounts of sensor dataare shared among software modules.

An open-source implementationof PTPd2 (precision time protocol

daemon) is used for time synchronization between thecomputers. Low data bandwidth and a synchronizationrate of 4 Hz are sufficient to maintain an average deviationof system clocks well below 500 ls. The Atom board servesas master clock, and all other computers are configured asslave clocks. On all computers the PTPd daemon runs witha high real-time priority to keep operating system schedul-ing latency as short as possible.

Deployment of compiled nodes and configuration filesis done from external development computers using anenhanced ROS build workflow, which invokes rsyncprogram for fast data synchronization to the research plat-form over Ethernet or WLAN. ROS launch files are used torun and configure nodes across all computers with onlyone command.

Atom

Gumstix RTLinux

Gumstix Linux 1

Gumstix Linux 2

Laser

Cam L

Cam R

Autopilot

Cam Up

Cam Front

Switch

WLAN XBee

eth

eth

eth

eth

USB UART

sync

I 2C

USB

USB

USB

USB

USB

Figure 5. Onboard distributed computation architecture and sensor communication.sync: synchronization; eth: Ethernet.

•Table 2. Hardware components of theexperimental system.

Component Hardware

Quadrotor AscTec Pelican

Atom board 1.6 GHz Intel Atom, 1 GBDDR2

Gumstix boards ARM Cortex-A8, 720 MHz,512MB RAM

IMU, accelerometer Memsic MXR9500M

IMU, gyroscope Analog Devices ADXRS610

Laser Hokuyo UTM-30LX, 30m,270� scanning range

Cameras PointGrey Firefly FMVU-03MTM/C-CS

XBee XBee 2.4GHz radio modem

WLAN Lightweight USBmodule,max 150Mb/s

Switch 100Mb, eight port switch



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Experimental ResultsA flight experiment was conducted to show the effectivenessof using two sensing paradigms. Figure 7 shows the esti-mated path flown inside and outside the experimental facil-ity (depicted in Figure 8). Figure 9 shows the reference andestimated system states as well as absolute covariances ofthe two odometries. Time instants when the system switchesbetween visual and laser odometry are marked on the timeaxis. Because of practical difficulties in outdoor measure-ments, no ground truth is provided. A rectangular patharound the house was chosen because it was clear of groundobstacles. The quadrotor has been yawing during the flightso that the laser scan points toward the house.

The quadrotor starts inside the house using laserodometry. Visual odometry is not available becausethe cameras are too close to the ground, and so there isnot enough overlapping in the images. At 7.5 s, duringautonomous take-off, when the quadrotor is at 68 cmaltitude, the depth image becomes available, andthe covariance of the visual odometry becomes smaller

than that of laser odometry. There-fore, the system switches to visualodometry.

Shortly afterward, the system iscommanded to fly outside. At 21 s,visual odometry becomes unavailableas indicated on the out-of-axis covari-ance in Figure 9. This is caused at firstby motion blur when the quadrotorstarts moving in the weak lighting con-ditions in the house. The system auto-matically switches to laser odometry.During flight through the 1-m-widewindow, a jump in the raw laser height

measurement can be seen due to flying above the 20-cm-widewall. The jump is detected by the data fusion and a constant alti-tude is kept. The vertical velocity is also unaffected, so the vehiclepasses smoothly through the window. The visual odometry isstill unavailable as the window pane is too close to the cameras.

When the quadrotor is outside, the cameras need toadjust their exposure time, so visual odometry is againavailable at approximately 1 m behind the window. It isclearly visible that the covariance of the laser odometryoutdoors is very large compared to indoors, due to lessvalid laser measurements. Therefore, only visual odometryis used for the outdoor flight.

During autonomous landing, the disparity imagebecomes unavailable under 60 cm of altitude. This is indi-cated by the high covariance of the visual odometry, andthe system switches to laser odometry.

The reference velocity and position are tracked accord-ing to the estimated values. The position control error withrespect to the estimated states is under 20 cm in bothindoor and outdoor environments.

(a) (b)

Figure 6. Details of avionics hardware components. (a) Modular computation stack.(b) An eight-port fast Ethernet switch.

PHILIPPLUTZ

−4 −3 −2 −1 0 1 2

−1

0

1

2

3

4

5

VisualLaserVisual

Laser

x (m)

(a) (b)

–y (

m)

−4−3

–2–1

01

−10

12

34

0

1

2

VisualLaser

Visual

Laser

x (m)–y (m)

–z (

m)

Figure 7. Flown path estimated in the experiment. The dashed red line shows the reference trajectory, while the solid line shows theestimated path. The house outline is shown in gray. Locations where switching between visual and laser odometry occurs are alsoindicated. (a) Estimated flown path in the horizontal (x, y) plane. (b) Three-dimensional view of the desired and estimated position(x, y, z) of the quadrotor during the experiment.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Conclusion and Future WorkWe introduced a modular and extensible software and hard-ware framework for the autonomous execution of USARmissions using aerial robots. An implementation of theframework on an experimental quadrotor system has beenpresented. The implemented computation and communica-tion hardware enables the simultaneous execution of severalcomputationally demanding tasks, including navigation andcomputer vision. Furthermore, the hardware can be easilyexpanded to provide more on-board computation power ifrequired. Our data fusion enables the seamless use of differ-ent sensing paradigms with delayed information on a highlydynamic quadrotor vehicle. Its effectiveness is shown by anautonomous flight from an indoor to an outdoor environ-ment through a 1-m-wide window, motivated by an explo-ration mission to enter and leave a building.

Currently, the system cannot automatically avoidobstacles. Therefore, reactive collision avoidance schemeswill be implemented on the low-level system. This requiresfurther development of object recognition and scene inter-pretation on resource-limited systems. As resources arelimited, merely a subset of tasks can be fulfilled; therefore,we will focus on the elaboration of these tasks.

Our future work also includes miniaturization of thesystem, mainly through weight reduction of the sensingequipment. For this reason, using other sensors such as

−4

−3

−2

−1

0

Pos

ition

(m

)

Position Estimates,Velocity Estimates, and Odometry Covariance During an Autonomous Flight

xd yd zdx y zzm

−0.75−0.5

−0.250

0.250.5

0.75

Vel

ocity

(m

/s)

xd yd zd

x y z

0 5 10 15 20 25 30 35 40 45 50 55 60

10−5

V L V L

0 5 10 15 20 25 30 35 40 45 50 55 60V L V L

0 5 10 15 20 25 30 35 40 45 50 55 60V L V L

2 · 10−6

5 · 10−6

2 · 10−5

Time (s)

Cov

aria

nce

Laser OdometryVisual Odometry

(a)

(b)

(c)

Figure 9. Flight from inside the house out through the window, located at x ¼ �2m. (a) The reference position (xd ; yd ; zd),estimated position (x; y; z), and raw laser height measurement (zm). (b) The reference velocity _xd ; _yd ; _zd and estimated velocity ( _x; _y; _z).(c) The magnitude of laser and visual odometry covariances. Also shown on the time axis are indicators when the system has switchedto visual (V) or laser (L) odometry. The covariance plot goes out of scope when a particular odometry is unavailable or too imprecise.

Figure 8. The house used for the experiments, located the DLRoutdoor testbed for mobile robots. It corresponds in shape anddimensions to the house used for the IMAV explorationchallenge. Provided are both an indoor environment, suitable fornavigation using a laser scanner, and an outdoor environment,which is suitable for vision-based navigation.

PHILIPPLUTZ



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






omnidirectional cameras and sonar will be investigated.Instead of stereo cameras, the Microsoft Kinect is also aviable sensor for obtaining depth images. However,because it uses artificial infrared lighting, it is only suitablefor indoor applications.

In USAR missions, it might be necessary to fly to globallydefined positions. This capability will be achieved by inte-grating GPS into the data fusion. We will also address thecooperation with multiple mobile and ground robots, as wellas human interfaces to a team of robots. Currently, ourfusion system is based on local navigation, yet the navigationwill be improved by having higher level position informa-tion, e.g., by using a topological map. In this way, the systemis enabled to navigate through large environments on astrongly hardware-limited system. Supplementary informa-tion about the DLRmulticopters can be found online [20].

References[1] L. Lin, M. Roscheck, M. Goodrich, and B. Morse, “Supporting wilder-

ness search and rescue with integrated intelligence: autonomy and infor-

mation at the right time and the right place,” in Proc. 24th AAAI Conf.

Artificial Intelligence, Atlanta, GA, 2010, pp. 1542–1547.

[2] M.A. Goodrich, J.L. Cooper, J.A. Adams, C. Humphrey, R. Zeeman, and

B.G. Buss, “Using a mini-UAV to support wilderness search and rescue:

Practices for human–robot teaming,” in Proc. IEEE Int. Workshop Safety,

Security and Rescue Robotics (SSRR), Rome, Italy, Sept. 2007, pp. 1–9.

[3] J. Casper and R.R. Murphy, “Human–robot interactions during the

robot-assisted urban search and rescue response at the World Trade Center,”

IEEE Trans. Syst. Man, Cybern. B: Cybernet., vol. 33, no. 3, pp. 367–385, 2003.

[4] S. Grzonka, G. Grisetti, and W. Burgard, “Towards a navigation sys-

tem for autonomous indoor flying,” in Proc. IEEE ICRA, Kobe, Japan,

May 2009, pp. 2878–2883.

[5] A. Bachrach, R. He, and N. Roy, “Autonomous flight in unstructured

and unknown indoor environments,” in Proc. European Micro Aerial

Vehicle Conf (EMAV), Delft, The Netherlands, 2009, pp. 1–8.

[6] S. Shen, N. Michael, and V. Kumar, “Autonomous multi-floor indoor

navigation with a computationally constrained MAV,” in Proc. ICRA,

Shanghai, China, May 2011, pp. 20–25.

[7] M. Achtelik, S. Weiss, and R. Siegwart, “Onboard IMU and monocu-

lar vision based control for MAVs in unknown in- and outdoor environ-

ments,” in Proc. IEEE ICRA, Shanghai, China, 2011, pp. 3056–3063.

[8] IMAV2011—International micro aerial vehicle competition. (2011).

Online. Available: http://www.imav2011.org/.

[9] M. Bl€osch, S. Weiss, D. Scaramuzza, and R. Siegwart, “Vision based

MAV navigation in unknown and unstructured environments,” in Proc.

IEEE ICRA, Anchorage, AK, 2010, pp. 21–28.

[10] N. L. Cassimatis, J. G. Trafton, M. D. Bugajska, and A. C. Schultz,

“Integrating cognition, perception and action through mental simulation

in robots,” Robot. Auton. Syst., vol. 49, no. 1–2, pp. 13–23, 2004.

[11] M. A. Goodale and G. K. Humphrey, “The objects of action and

perception,” Cognition, vol. 67, no. 1–2, pp. 181–207, 1998.

[12] A. Censi, “An ICP variant using a point-to-line metric,” in Proc.

IEEE ICRA, 2008, pp. 19–25.

[13] H. Hirschm€uller, P. R. Innocent, and J. M. Garibaldi, “Real-time cor-

relation-based stereo vision with reduced border errors,” Int. J. Comput.

Vis., vol. 47, no. 1–3, pp. 229–246, 2002.

[14] H. Hirschm€uller, “Stereo vision based mapping and immediate

virtual walkthroughs,” Ph.D. dissertation, School of Computing, De Mon-

tfort University, Leicester, U.K., 2003.

[15] H. Hirschm€uller, P. R. Innocent, and J. M. Garibaldi, “Fast, uncon-

strained camera motion estimation from stereo without tracking and

robust statistics,” in Proc. 7th Int. Conf. Control, Automation, Robotics

and Vision (ICARCV), Singapore, Dec. 2002, pp. 1099–1104.

[16] S. Roumeliotis and J. Burdick, “Stochastic cloning: A generalized

framework for processing relative state measurements,” in Proc. IEEE

ICRA, Washington, DC, 2002, vol. 2, pp. 1788–1795.

[17] W. T. Freeman andM. Roth, “Orientation histograms for hand gesture

recognition,” Mitsubishi Electric Research Labs., 201, Tech. Rep., 1995.

[18] V. Utkin and J. Shi, “Integral sliding mode in systems operating

under uncertainty conditions,” in Proc. 35th IEEE Decision and Control,

Kobe, Japan, 1996, vol. 4, pp. 4591–4596.

[19] K. Youcef-Toumi and O. Ito, “A time delay controller for systems with

unknown dynamics,” in Proc. Am. Control Conf., Atlanta, GA, 1988, pp. 904–913.

[20] “GermanAerospaceCenter (DLR) Institute of Robotics andMechatronics.”

[Online]. Available: http://www.dlr.de/rm/desktopdefault.aspx/tabid-7903

Teodor Tomi�c, German Aerospace Center (DLR),Muenchner Strasse 20, 82234 Wessling, Germany. E-mail:[email protected].

Korbinian Schmid, German Aerospace Center (DLR),Muenchner Strasse 20, 82234 Wessling, Germany. E-mail:[email protected].

Philipp Lutz, German Aerospace Center (DLR), MuenchnerStrasse 20, 82234Wessling, Germany. E-mail: [email protected].

Andreas D€omel, German Aerospace Center (DLR),Muenchner Strasse 20, 82234 Wessling, Germany. E-mail:[email protected].

Michael Kassecker, German Aerospace Center (DLR),Muenchner Strasse 20, 82234 Wessling, Germany. E-mail:[email protected].

Elmar Mair, German Aerospace Center (DLR), MuenchnerStrasse 20, 82234Wessling, Germany. E-mail: [email protected].

Iris Lynne Grixa, German Aerospace Center (DLR),Muenchner Strasse 20, 82234 Wessling, Germany. E-mail:[email protected].

Felix Ruess, German Aerospace Center (DLR), MuenchnerStrasse 20, 82234Wessling, Germany. E-mail: [email protected].

Michael Suppa, German Aerospace Center (DLR),Muenchner Strasse 20, 82234 Wessling, Germany. E-mail:[email protected].

Darius Burschka, Technische Universitaet Muenchen,Boltzmannstr. 3, 85748 Garching bei Muenchen, Ger-many. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___________

___________

___________

_______________

_______________

_________________

________________

____________

_________________

______________

http://www.qmags.com/clickthrough.asp?url=http://www.imav2011.org/&id=17487&adid=P56E2

http://www.qmags.com/clickthrough.asp?url=http://www.dlr.de/rm/desktopdefault.aspx/tabid-7903&id=17487&adid=P56E1















•By Antonio Franchi, Cristian Secchi,Markus Ryll, Heinrich H. B€ulthoff, andPaolo Robuffo Giordano

Robustness and flexibility con-stitute the main advantages ofmultiple-robot systems withrespect to single-robot ones asper the recent literature. The

use of multiple unmanned aerial vehicles(UAVs) combines these benefits with theagility and pervasiveness of aerial plat-forms [1], [2]. The degree of autonomy ofthe multi-UAV system should be tunedaccording to the specificities of the situa-tion under consideration. For regular mis-sions, fully autonomous UAV systems areoften appropriate, but, in general, the use ofsemiautonomous groups of UAVs, super-vised or partially controlled by one or morehuman operators, is the only viable solutionto deal with the complexity and unpredict-ability of real-world scenarios as in, e.g., thecase of search and rescue missions or explora-tion of large/cluttered environments [3]. Inaddition, the human presence is also manda-tory for taking the responsibility of criticaldecisions in high-risk situations [4].

In this article, we describe a unified frame-work that allows 1) letting the group of UAVsautonomously control its topology in a safe andstable manner and 2) suitable incorporation ofsome skilled human operators in the controlloop. This way, the human’s superior cognitivecapabilities and precise manual skills can beexploited as a valid support for the typicalautonomy of a group of UAVs. In fact, drawing




Balancing Autonomy and Human Assistance

with a Group of Quadrotor UAVs

©ISTOCK PHOTO.COM/© ANDREJS ZAVADSKIS


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________

http://www.qmags.com/clickthrough.asp?url=http://www.istockphoto.com&id=17487&adid=P57E1





from the well-established field of bilateral teleoperation[5], our framework includes the possibility of providingthe operators with haptic cues informative of the UAVgroup and environmental state, as in [6] for the single-UAV case. This will typically increase the situationalawareness of humans, execution performance, and theability to make correct decisions, as often demonstrated inbilateral teleoperation systems [7].

As for the UAV group behavior, rather than focusingon a particular task to be executed (e.g., coverage, map-ping, or surveillance), we address the general aspects typi-cally shared in any operational scenario, i.e., UAV sensing/planning/motion control, collective behavior during navi-gation, and human interfacing. The term shared controlrefers to both sharing the autonomy between humans andUAVs and sharing the competences between two or morehuman operators assisting the group of UAVs from differ-ent perspectives.

The feasibility of our framework is demonstrated byillustrating several experiments run in our test bed, whichincludes a group of quadrotors (QRs) and two haptic inter-faces integrated in a flexible software framework. Theexplicit integrated possibility of both autonomous QRflight behavior and bilateral interaction between humanoperators and a UAV group can be considered as a novelfeature compared with the existing (and successful) testbeds for multiple QRs [8]–[10].

UAV Model and Overall ArchitectureWe consider a group of N UAVs modeled as rigid bodiesin R3. The configuration of the ith UAV is representedby its position pBi

2 R3 and rotation matrix RBi 2 SO(3)with respect to a common inertial frame. The rotation mayalso be described by the usual yaw wBi

, pitch hBi , and roll/Bi

angles.With special focus on the QR platform, we assume

that the UAV is able to track a smooth reference trajectory(pi(t), wi(t)) in the four-dimensional space R3 3S1.This is the case for helicopter and QR UAVs [11] as well asfor any other UAV whose position and yaw anglepBi

, wBi

� �are flat outputs [12], i.e., algebraically defining,

with their derivatives, the state and control inputs ofthe UAV.

Architecture of the Bilateral Shared ControlFigure 1 illustrates the proposed system architecture withsome details on the single-UAV module. One or morehuman assistants are in charge of defining the current taskfor the group of UAVs by means of suitable task interfaces(e.g., a touch-user interface) [13]. A task is a long-/medium-term activity that the group of UAVs is asked tocarry out autonomously, e.g., covering or exploring anarea, navigating between via points, surveilling a perimeter,or cooperatively transporting a load. Our goal is to proposea framework that complements a particular task algorithmwith some general supportive features of common utility

in any supervised/shared operation, such as obstacle avoid-ance, interagent behavior, human assistance, and humantelepresence.

A core component is the flat-output trajectory planner(FOTP) that provides the reference flat outputs (pi, wi),and their derivatives, to the flight controller (FC). TheFC of each UAV acts on the UAV physical control inputs(e.g., the propeller speeds) to let the UAV outputs(pBi

,wBi) track the desired ones (pi,wi). The FOTP of

the ith UAV coordinates with the FOTPs of the otherUAVs using a communication interface. This can either bethe same used to communicate with the human assistantsor a dedicated (local) one. The FOTP is designed so asto generate the quantities (pi(t), wi(t)) as the time evolu-tion of two virtual systems (henceforth unified under thename agent): one system for the desired yaw wi (the yawagent) and one system for the desired position pi (the posi-tion agent).

In this article, we only consider kinematic yaw agents,as this is, in practice, an acceptable assumption for manyQR-like UAVs. This then results in

_wi ¼ wi, (1)

where wi 2 R is the yaw-rate input. On the other hand, weconsider to steer the position agent either at the kinematic(first-order) level, i.e., by commanding a linear velocityui 2 R3,

_pi ¼ ui, (2)

or at the dynamic (second-order) level, i.e., interpretingthe command ui 2 R3 as a force,

_pi ¼ vi, (3)

Mi _vi ¼ ui � Bivi: (4)

Here, vi 2 R3 and Mi 2 R33 3 are the velocity and the(symmetric positive definite) inertia matrix of agent i,respectively, and Bi 2 R33 3 is a positive definite matrixrepresenting an artificial damping added to asymptoticallystabilize the behavior of the agent and also take intoaccount the typical physical phenomena such as wind/atmosphere drag. The meaning of ui (either velocity orforce input) will be clear from the context.

Modeling UAVs as kinematic agents is a commonassumption in the multirobot literature (e.g., similarassumptions have been made in [1] and [2]). Due to theirhigher complexity, dynamic agents are less commonlyadopted. Nevertheless, dynamic agents provide a betterapproximation of the actual UAV dynamics, and therefore,are more appropriate whenever the dynamic properties ofthe UAVs are more stressed, as in the case of intermittentinteractions (see the variable topology cases in this article).The interested reader can also find in [14] a comparison ofstability and performance issues for a network of kinematicand dynamic agents.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






The inputs (ui,wi) will depend on the contribution offour terms: (uti ,w

ti ), (u

soi , 0), (u

sti ,w

sti ), and (uhi ,w

hi ). These

terms are generated by four individual subsystems, whosemeaning and purpose are explained in the following sec-tions. See Figure 1 for a visual reference.

Task ControllerBy means of a task interface, a human assistant is given thepossibility to select an algorithm, generically referred to astask controller (TC), for executing a particular task. Theinternal description of specific TCs is out of the scope of

SecondHuman

Assistant BilateralControlDevice

TopologicalController

TC

Agent

Dynam

ics

FCUAV

Dynamics

Sensing

FOTP

ObstacleController

TaskInterface

FirstHuman

Assistant

1st UAV

j th UAV

N th UAV

BilateralControlDevice

TaskInterface

Com

munication Interface

Inter-UAVCommunicationInfrastructure

Environm

ent

(ui , ψi ) t t

(ui , ψi ) st st

(ui , 0)so

(ui , ψi ) h h

Communication

Communication i th UAV

i th UAV

Figure 1. The overall system architecture as seen from the point of view of the generic ith UAV. The blocks in charge of thesupportive features are the bilateral control device, topological controller, obstacle controller, and agent dynamics.

MAXPLANCKINSTITUTEFORBIO

LOGICALCYBERNETICS



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






this article. However, any TC will eventually generate aninput (uti ,w

ti ) for the agent based on the sensorial feedback

and inter-UAV communication. Apart from being respon-sible for FOTP generation, the TC may also be in controlof additional specific task requirements, such as control-ling the end effector of a robotic arm attached to the UAV,or regulating the tension of a rope used for transportation.Finally, the TC is in charge of discriminating which part ofthe environment should be considered as an obstacle bythe obstacle controller (e.g., walls) and which part shouldbe considered as safe (e.g., an object to be grasped orpushed). At the same time, the TC can also communicatethe desired inter-UAV behavior (e.g., desired formation)to the topological controller.

Obstacle ControllerThe obstacle controller generates an input (usoi , 0) as thegradient of an obstacle-avoidance artificial potential toensure collision-free navigation with respect to theobstacles identified by the TC. We do not consider anobstacle-avoidance action on the yaw-rate input, as weassume that it is always acceptable to virtually augment theUAV size up to a yaw-symmetric shape (e.g., an ellipsoid).

Topological ControllerFormation control, requiring that the UAVs maintain somepreferred interdistances or mutual poses, is a basic featureneeded in almost all multiaerial tasks. The formation con-straints can be more strict or more relaxed depending on thetask. For example, in case of a simple coverage, the con-straints can be relaxed, while in the case of precise sensorfusion (e.g., aerial measurements and interferometry), a par-ticular fixed formation must be kept with high accuracy. Thetopological controller is meant to implement the desired taskformation specifications by generating the inputs (usti ,w

sti ).

The “Topological Controller” section provides a descriptionof this controller for some relevant scenarios.

Human AssistanceIn any complex situation, human assistance cannot belimited to a generic supervision as in the case of taskassignment. In the real world, complex tasks are made of acollection of activities, and some of them, usually the morechallenging ones, need direct human control. To this aim,the command (uhi ,w

hi ), generated by the bilateral control

device, allows the human to precisely act on partial aspectsof the task, e.g., by steering the centroid of the formation(global intervention) or by precisely maneuvering only oneparticular UAV (local intervention). This aspect is detailedin the “Human Assistance and Telepresence” section.

Human TelepresenceThe nature of the task interface and the bilateral controldevice is meant to provide a suitable feedback to the humanoperator(s) to increase his (their) telepresence. In particu-lar, the task interface can also be used to interactively

monitor the state of the task, e.g., by providing augmentedvideo streams from the remote site. In fact, to obtain anacceptable performance in the direct-assistance case, thehuman operator needs to be constantly informed about theUAV/environment state with a good perceptive quality.Because of this, in our framework, we decided to adopt thetypical paradigm of visual/force feedback for human–robotshared control tasks, a paradigm known in the literaturealso as bilateral control. The “Telepresence” section isdevoted to the description of this component.

Topological ControllerThe topological controller generates an input (usti ,w

sti ) to

implement a desired mutual interaction behavior, e.g., toreach a prescribed set of precise mutual configurations orto approximately keep a given separation distance amongthe UAVs. The collection of all the mutual interactions canbe described by an interaction-topology graph G ¼ (V, E),where the vertices V represent the UAVs and the weightededges E � V 3V represent the intensity of the interactionbetween two UAVs. A nonzero weight models the presenceof interaction while a zero weight means no interactionand is equivalent to the absence of that edge. We considerthree possible classes of interaction graphs in detail (seeFigure 2): 1) constant topology, when the task requires thatthe interaction pairs are always the same; 2) unconstrainedtopology, when the topology can freely change over time,allowing the group to even disconnect into several sub-groups; 3) connected topology, when the interaction graphcan freely change but with the constraint of remainingconnected at all times, i.e., to ensure group cohesion.

Constant TopologyThe need for constant-topology (i.e., fixed) UAV forma-tions naturally arises from the specifications of many UAVapplications, e.g., for interferometry and transportation[2], for guaranteeing inter-UAV visibility or environmen-tal coverage [1], [3]. The desired formation may be an out-put of the TC or directly specified by human operators.

Assuming that an interaction-topology graph is chosen, aformation is then commonly described by assigning thedesired relative behavior between every UAV pair (i, j) 2 E.In these notes, we consider two typical cases: assigning eitherthe relative bearings bij or the interdistances �dij 8(i, j) 2 E.

A formation controller only constraining the relativebearings can be implemented by relying on sole relativebearings as measurements [15]. This is an interesting fea-ture since relative bearings can be obtained using onboardmonocular cameras, i.e., lightweight, low-energy, andcheap sensors providing spatial information on the neigh-bors. Furthermore, when constraining the relative bear-ings, the UAV formation still possesses five degrees offreedom (5 DoF), such as translation of the centroid, syn-chronized rotation about the vertical axis, and expansion[15]. These DoFs can then be used to give motion com-mands to the whole formation.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






On the contrary, the regulation ofthe relative interdistances cannot beachieved using interdistances, butone also needs knowledge of the rela-tive bearings [16]. In this case, theformation can be shown to still pos-sess 6 DoF: translation of the centroidand rotation about any axis in space.

As for formation control, thetypical approaches are based onartificial potentials that generate(usti ,w

sti ) as the gradients of energy-

like functions having a minimum atthe desired interagent value (see [16]for one of these cases). However, thedrawback of artificial potential ap-proaches is the presence of local min-ima. In this regard, the work in [15]presents an interesting differentstrategy not based on artificial poten-tials and almost globally convergent.

Unconstrained TopologySome tasks require loose inter-UAVcouplings, for example, the possibil-ity of creating and destroying pair-wise interactions (splits and joins) atany time. Typical applications in thiscontext are as follows: 1) navigationin cluttered environments, where anunconstrained interaction topologyis much more adaptable than a fixedone and 2) multirobot exploration,where UAVs frequently need to divide or gather them-selves into smaller or larger groups.

To model this flexibility [17], [18], consider the pres-ence of a switching signal for every pair of UAVsrij(t) 2 f0, 1g meant to represent the status of the inter-action among agents i and j (with rij ¼ 1 indicatingpresence of the interaction, and rij ¼ 0 otherwise). Thetime behavior of rij(t) can not only model the effectof limited-sensing capabilities of the UAVs (e.g., maxi-mum sensing/communication range and occlusions ofthe line of sight) but also are triggered by the TC toaccount for any additional task (e.g., to split or join dif-ferent subgroups).

In the unconstrained topology case, the topologicalcontroller should only ensure some form of loose aggrega-tion. Therefore, the control term usti is designed as the sumof local interactions over all the neighbor UAVs, i.e., onlythose jth UAVs such that rij(t) ¼ 1. Each interactionforce mimics the nonlinear spring behavior depicted in thepotential plot of Figure 3, i.e., a repulsive action ifdij < d0, an attractive action if d0 < dij D, and a nullaction if dij > D, where d0 < D is a neutral interdistancebetween agents.

Connected TopologyAs a variation of the previous case, we also considered thepossibility of allowing for time-varying topologies butunder the constraint of maintaining connectivity of thegraph G, despite the creation or disconnection of individ-ual links. In fact, while flexibility of the formation topologyis a desirable feature, connectedness of the underlyinggraph is often a prerequisite for implementing distributedcontrol/sensing algorithms.

The work in [19] shows a topological controller able toaccount for connectivity maintenance by exploiting adecentralized estimation k2 of k2, the second smallesteigenvalue of the Laplacian matrix L associated with thegraph G. In fact, it is well known that the graph G resultsconnected if and only if k2 > 0. Exploiting the decentral-ized estimate k2, in [19] it is shown how to implement adecentralized gradient-based controller that allows fora time-varying topology due to loss/gain of visibilityor maximum range exceedance between agents whileensuring connectivity maintenance at all times. As an addi-tional feature, this same controller is also proven to implic-itly guarantee interrobot and obstacle-collision avoidance:this is achieved by suitably shaping the weights on the

Con

stan

tU

ncon

stra

ined

Con

nect

ed

(a)

(b)

(c)

Figure 2. The three topological behaviors: (a) the interaction graph always remainsconstant regardless of the interaction with the environment, (b) the graph isunconstrained, thereby changing due to the environmental interaction and eventuallybecome disconnected, and (c) the graph still changes but under the constraint ofremaining connected.


LOGICALCYBERNETICS



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






edges of neighboring robots as functions of interrobot androbot–obstacle relative distances.

Ensuring a Stable Behavior of the Agent GroupGuaranteeing passivity of the agent group is a sufficientcondition for guaranteeing a stable behavior in free motionand when interacting with unknown (passive) environ-ments. In both the cases of unconstrained and connectedtopologies, the controlled dynamic agents in (3) and (4) areequivalent to floating masses with damping so that theirinterconnection can be considered as a mechanical elasticelement. Thus, the agents intuitively behave in a stable way,being essentially a network of springs and (damped) masses.This can be formally proven by showing that the system ispassive, such as the energy exchanged with the externalworld (i.e., humans and environment) is either stored in theform of (virtual) kinetic or potential energy or dissipatedand that no destabilizing regenerative effects are present.

However, the need for flexibility and connectivity mainte-nance of the approaches reported in the “UnconstrainedTopology” and “Connected Topology” sections can possiblythreaten passivity. In fact, both split and join events, as illus-trated in Figure 3, and the use of an estimation of the con-nectivity eigenvalue k2 in place of the real value k2 can createextra energy in the agent group; see [18] and [19]. On theother hand, passivity is preserved if and only if the amountof energy dissipated by the agent group is higher than theamount of energy produced. Thus, the amount of dissipatedenergy is a good measure of the current passivity margin.

To understand whether an energy-producing actioncan be implemented without violating the overall passivity,

i.e., whether it is within the margin, the dynamics of eachagent is augmented as follows:

_pi ¼ viMi _vi ¼ ui � Bivi_ti ¼ 1

ti(vTi Bivi)þ si

8<: , (5)

where ti 2 R is the state of an energy-storing element,called tank, characterized by the energy functionTi(ti) ¼ (1=2)t2i . It is easy to see that _Ti ¼ (vTi Bivi)þ sTi ti.Thus, all the energy dissipated by the UAV is stored in thetank, and one can still inject/extract energy from the tankusing the control input si. For each passivity-threateningaction, every UAV computes the amount of energy Ei thatwould be produced by its actual implementation. If Ei < 0,the action is dissipative, and the agent refills its tank bysuitably acting on si to inject back the amount�Ei. On theother hand, if Ei > 0, the agent can implement the actiononly if Ti(ti) > Ei. If this is the case, the action is imple-mented, and si is exploited to extract Ei from the tank. IfTi(ti) < Ei, the agent can still increase the amount ofenergy in its tank (e.g., by artificially increasing its damp-ing) until the action can be implemented. Using thisstrategy, flexibility, connectivity, and passivity can easilyand elegantly coexist. A more formal and detailed illustra-tion can be found in [18] and [19].

Human Assistance and TelepresenceWe consider M (usually � N) bilateral devices (i.e., withforce-feedback capabilities) as human–robot interfaces forallowing a human operator to intervene on some aspects of

01

23

45

67

80 10 20 30 40 50 60 70 80 90

d0

D

Obstacle

UAV

UAV

Split Join Esplit Ejoin > EsplitNo Interaction

Nonlinear S

pring Potential

Obstacle

(b)(a)

Figure 3. (a) The plot of the nonlinear spring potential modeling the agent interaction. (b) When the UAVs split, the energy Esplit isstored in the spring, while when they join the energy Ejoin > Esplit is needed to implement the new desired coupling. In this case,without proper strategies, an amount, Ejoin � Esplit > 0, of energy would be introduced into the system, thus violating passivity.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






the UAV motion while receiving suitable haptic cues. Thedevices are modeled as generic mechanical systems

Mi(xi)€xi þ Ci(xi, _xi) _xi ¼ sci þ shi , i ¼ 1 . . .M, (6)

where xi 2 Rdi is the configuration vector of the ith inter-face, Mi(xi) 2 Rdi 3 di its positive-definite and symmetricinertia matrix, Ci(xi, _xi) 2 Rdi 3 di represents Coriolis andcentrifugal terms, and (sci , s

hi ) 2 Rdi 3Rdi are the control/

human forces acting on the device, respectively. As usuallydone, we also assume that gravity effects are locally com-pensated. In the following, we describe two different waysfor interfacing a human operator with the UAV group dy-namics, such as a local and a global intervention modality.

Operator Intervention

Local InterventionIn the local intervention case, each haptic interface influen-ces the motion of a single UAV. A straightforward applica-tion of this case is the leader–follower modality: a group offollower UAVs can be guided through an environment by aleader UAV, which in turn is controlled by a human opera-tor. Another possibility is to exploit this modality to let askilled operator temporarily help a UAV to solve conflictingsituations. A final possibility (multileader/multifollower) isto cooperatively transport an object with a team of UAVs, ofwhich only a small subset is driven by human operators.

We assume that di ¼ 3 for every i ¼ 1 . . .M and con-sider an injective map a : f1, . . . ,Mg ! f1, . . . ,Ng fromthe set of force-feedback devices to the set of UAVs. Theposition of the ith haptic device will be treated as a (scaled)velocity reference for the a(i)th position agent. In the case ofa kinematic position agent, this results in either uha(i) ¼ kixior uhj ¼ 0 for all the jth position agents not in Im(a). In thecase of a dynamic position agent, the local interventionresults in the following proportional controller:

uha(i) ¼ Bh(kixi � va(i)), (7)

where Bh 2 R33 3 is a positive definite damping matrix.Similarly, we also have uhj ¼ 0 8j 62 Im(a).

In the case, not considered here, of di ¼ 4, one couldextend the same concept to also control the yaw of theUAVs with an additional DoF of the interface.

Global InterventionGlobal intervention allows the operator to control some gen-eralized velocities of the whole group of UAVs by acting onthe configuration of the available haptic interfaces. Amongthe many possible choices, we consider here the followingcases: cohesive motion of the whole group, synchronized rota-tion about any axis passing through the kth UAV, and theexpansion/contraction of the whole formation. xt , xx 2 R3,and xs 2 R denote the configurations of the devices in controlof the desired group velocity, angular velocity, and expansion

rate, respectively. In case of kinematic position agents, thedesired global intervention action is implemented by setting

uhi ¼ ktJtixt þ kxJxixx þ ksJsixs, 8i ¼ 1 . . .N , (8)

where Jti, Jxi, and Jsi are the ith components of the mapbetween the desired global velocities (generating transla-tion, rotation, and dilation, respectively) and the velocityof the ith agent, and kt > 0, kx > 0, and ks > 0 are suita-ble scaling factors from the haptic device configurations(xt , xx, xs) to the desired global velocities.

If a constant-topology controller is also acting on theUAVs, the global velocities should be orthogonal to theinterrobot constraints enforced by the topological control-ler to preserve the current interdistances and/or relativebearings. The relative bearings are preserved when all theUAVs translate, dilate, or cohesively rotate around thevertical axis, i.e., when xx ¼ 0 0 zxð Þ and wh

i ¼ zx,with zx 2 R. The relative interdistances are preserved ifthe group does not dilate, i.e., when xs ¼ 0.

Global intervention in the dynamic case is implementedby using an approach similar to (7) to track the desiredagent velocity given by (8).

TelepresenceThe force feedback on the bilateral devices is designed toprovide haptic cues informative of how well the real UAVsare executing the desired human commands and to feel thedisturbances to which the UAVs are subject to, such as tur-bulences and wind gusts. Recalling the “UAV Model andOverall Architecture” section, we let _pBi

2 R3 be the body-frame velocity vector of the ith UAV, and _wBi

2 R its yawrate. We stress, again, that these represent real (measured)UAV quantities and not the reference (virtual) velocities ofthe agent tracked by the UAVs.

In case of local intervention, we consider the mismatchbetween the velocity commanded through the ith interfaceand its actual execution by the a(i)th UAV. Therefore,we set sci ¼ �Bh(kxi � _pBa(i)

). In case of global intervention,we consider the mismatch between the commanded gener-alized velocities and their average execution by all UAVs.

Depending on the actual situation, the resulting forces canprovide different cues to the human operator. In case ofUAVs possessing large inertia (e.g., when the UAVs are coop-eratively transporting a load), the operator would feel highcounteracting forces when asking for rapid changes of thegroup velocity. In case of significant air viscosity, the forcewould be proportional to the UAV velocity, thus giving theimpression of pulling a load with some friction. Finally, if theoperator tries to push the UAVs against an obstacle as, e.g., awall, the UAVs would still remain without following the oper-ator commands. Therefore, the force felt would be similar toa springlike obstacle repulsion. Along these lines, we also referthe interested reader to some preliminary psychophysicalstudies addressing the problem of human perception and tel-epresence in such novel bilateral control scenarios [20], [21].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Stabilization of the Teleoperation SystemIn both the cases of local and global intervention, it is nec-essary to consider the destabilizing effects (e.g., communi-cation delay, packet loss, and sample and hold) arisingfrom the interconnection between the bilateral controldevices and the agent group.

The passive-set position modulation (PSPM) frame-work [22] can be used as a general tool for guaranteeingmaster passivity and, therefore, stability of the closed-loopteleoperation system when dealing with the aforemen-tioned destabilizing effects by acting on the signalsexchanged over the communication channel. We omitfurther details here and refer the interested reader to [16]and [22] and references therein for a complete treatmentand formal proofs of these statements.

Another possibility is to use the two-layer approach[23] that, enforcing passivity, guarantees a stable behavior.Broadly speaking, this approach separates the exchange ofinformation between bilateral control devices and agentsinto two layers. The passivity layer is controlled in such away that a passive energetic interconnection is establishedbetween the bilateral control device and agents. In thisway, if the bilateral control device and agents are passive, astable behavior of the teleoperation system is guaranteedindependently of sampling, variable delays, and loss of

communication. The transparency layer determines theinputs to be implemented on the bilateral control deviceand agents to obtain the desired performance [e.g., (7)].These desired inputs are then elaborated by the passivitylayer that, exploiting the energy exchanged between localand remote sites, is able to implement them in a passivity-preserving way. For further details, the interested readercan refer to [23] and [24].

QR-Based Experimental Test Bed

Hardware Setup

UAV SetupWe used QRs as UAVs because of their versatility, robust-ness, and construction simplicity. Furthermore, the QRmodel perfectly matches our assumptions of the “UAVModel and Overall Architecture” section, i.e., the ability totrack the reference quantities (pi, wi) generated by theFOTP owing to the flatness of the QR outputs (pBi

, wBi).

Our QR setup is a customized version of the MK-Quadro (http://www.mikrokopter.de) open-source plat-form [see Figure 4(a)]. The QR frame spans 0:5 m, weighs0:12 kg, and is made of four aluminum rods joinedtogether in a cross shape by two plastic center plates. Four

brushless Roxxy 2827-35 motors aremounted at the end of each rod.Each motor is driven via a pulse-width modulation (PWM) signal bya BL-Ctrl V2.0 brushless controller(BC) that can withstand an averagepower consumption of 160 W and apeak current of 40 A. A propeller of0:254 m diameter is attached to eachmotor. By means of a custom-mademeasure test bed using a Nano17force/torque sensor (http://www.ati-ia.com/) [see Figure 4(b)], we foundthat the maximum force and torqueare 9:0 N and 0:141 N�m, respec-tively, and that the motor dynamicscan be approximated by a first-orderlinear system with a time constant of0:047 s. For control design purposes,this can be in first approximationneglected with respect to the mechan-ical dynamics of QR.

A 50-mm-square electronic boardis mounted in the middle of theframe and hosts a 8-b Atmega1284pmicrocontroller (lC) clocked by a20-MHz quartz. The lC can send thedesired motor speeds to the BCs bymeans of an I2C bus. The small boardalso hosts 1) a three-dimensional (3-D) LIS344alh accelerometer with a

Reflective Marker

Motor

μC Board

BC

Modular Frame

LipoBattery

Q7 Board

Power Supply Board

Colored Marker

Force/Torque Sensor

Monocular Camera

(a)

(b)

Figure 4. (a) The QR setup with its parts and (b) the test bed is used for theidentification of the motor/propeller dynamics.


LOGICALCYBERNETICS



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


__________

_____

http://www.qmags.com/clickthrough.asp?url=http://www.ati-ia.com/&id=17487&adid=P64E2

http://www.qmags.com/clickthrough.asp?url=http://www.mikrokopter.de&id=17487&adid=P64E1





resolution of 0:0039g0 m/s2 and range of �2g0 m/s2 and 2)three ADXRS610 gyros with a resolution of 0:586�=s andrange of�300�=s. These are all accessible by the lC.

A 723 100 mm Q7 electronic board (http://www.seco.it/en/, http://www.qseven-standard.org/) is mounted under-neath the frame. The board hosts a Z530 Intel Atom proces-sor, 1-GB DDR2 533-MHz RAM, an 8-GB onboard flashdisk, and a wireless fidelity (Wi-Fi) card. The power con-sumption of this board is 10 W. The Q7 board communi-cates with the lC through a serial (RS 232) cable with a baudrate up to 115, 200 b/s. During the debugging phase, the Q7board can also be dismounted from the QR and operated asa desktop computer. In this case, the cable is replaced by awireless serial connection XBee-PRO 802.15.4.

An adapted low-cost monocular camera is mounted ontop of the lC board. This is connected to the Q7 boardthrough a USB and has a horizontal/vertical field of viewof about 88�=60� and a weight of less than 50 g. A set ofreflective markers is used by an external tracking system toretrieve the position and orientation of the QR. A single-colored ball is instead tracked by the cameras of the QRs tomeasure their relative bearing bij.

The QR carries a four-cell 2, 600-mAh LiPo batteryunderneath the Q7 board, which powers the whole systemby means of a custom power-supply board, allowing to han-dle the diversity of supplied components. The autonomyprovided by the battery in full configuration is about 10 min.All the electronic devices can also be supplied by an ac/dcsocket adapter, e.g., while the battery is replaced.

Haptic InterfacesThe bilateral devices used in our test bed are Omega.3 andOmega.6 (http://www.forcedimension.com), as shown inFigure 5. The Omega.3 has three fully actuated DoF, whileOmega.6 is a 6-DoF device with three actuated translationaland three passive rotational DoF. Each device is connectedto a mini PC by means of a USB connection and can be con-trolled at 2:5 kHz. The workspace of the devices is approxi-mately a cube with an edge of 0:12 m, and the maximumprovided force is about 10 N.

Additional ComponentsOur hardware setup also includes additional componentsthat, however, are not described in detail. These consist ofthe network infrastructure, the motion capture (MOCAP)system (http://www.vicon.com/), and other human inter-faces (e.g., joypads and screens).

Software SetupThe distributed software implementation of the whole sys-tem involves several processes interconnected through cus-tom-developed interfaces, see Figure 6. A custom C++algorithmic library provides the control and signalprocessing functionalities needed by each process, such asforce-feedback algorithms, topological controllers, obstacleavoidance techniques, FCs, signal processors, and filters.

UAV SoftwareThe Q7 board runs a GNU-LINUX OS and hosts thehigh-level UAV controller (HLUC) process that imple-ments the FOTP and part of the FC. The lC board runs asingle process, the low-level UAV controller (LLUC),which implements the remaining parts of the FC and inter-faces directly with the inertial measurement unit (IMU)and the four motor controllers through the I2C bus. TheFC is a standard cascaded controller similar to the oneused in [9].

The HLUC can use Wi-Fi to communicate via socketinterprocess communication (IPC) with several otherprocesses hosted by different machines, such as the HLUCof other UAVs, haptic-device controllers, a MATLABinstance, a MOCAP system (using the VRPN protocol),sensor modules (which may also be hosted on the Q7board), and input–output interfaces [e.g., using the TUIO(http://www.tuio.org/) protocol, a touchpad or a smartphone]. A similar interface also allows direct communica-tion between haptic-device controllers and a MATLABinstance if required. The communication frequencychanges for each interface (e.g., it is 1204240 Hz for theMOCAP system and 25 Hz for the camera-sensor module).

The communication between the HLUC and the LLUCis mediated by the safe module (SM) process, a very com-pact and well-tested program whose role is to check theinputs generated by the HLUC and take full control ofthe UAV in case of detection of some inconsistencies inthe input signals (e.g., erroneous frequencies and excessivejittering). In addition to SM, in emergency cases, a humanoperator may also bypass the Q7 board and manually con-trol the UAV using a radio remote controller.

The LLUC provides an estimate of roll and pitch angles(/Bi

, hBi ) by fusing the IMU readings with a complemen-tary filter. The yaw wBi

can be either measured with anonboard compass or retrieved by the MOCAP system. Asensor module processing the onboard camera signal isused to obtain the relative bearings by detecting thespheres placed on top of the QRs. Finally, every BC runs aP-controller to regulate the motor speeds.

Control PCs

ThreeMotors

ThreeMotors 3 DoF

Haptic Device6 DoF

Haptic Device

Handle 3 DoF Handle

Figure 5. Haptic interfaces with the corresponding control PCsused in the experimental test bed.


LOGICALCYBERNETICS



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___

http://www.qmags.com/clickthrough.asp?url=http://www.vicon.com/&id=17487&adid=P65E5

http://www.qmags.com/clickthrough.asp?url=http://www.forcedimension.com&id=17487&adid=P65E4

http://www.qmags.com/clickthrough.asp?url=http://www.tuio.org/&id=17487&adid=P65E3

http://www.qmags.com/clickthrough.asp?url=http://www.qseven-standard.org/&id=17487&adid=P65E2

http://www.qmags.com/clickthrough.asp?url=http://www.seco.it/en/&id=17487&adid=P65E1

http://www.qmags.com/clickthrough.asp?url=http://www.seco.it/en/&id=17487&adid=P65E1





Other Software ComponentsThe haptic device controller implements the local controlloop computing the forces sc at a frequency of 2:5 kHz,sends the needed quantities (e.g., x) to the HLUC, andreceives the UAV measurements needed to implement sc.An instance of MATLAB can also be interfaced withHLUC for debugging and testing purposes (e.g., offlinepostprocessing) or for fast prototyping of complexcontrol algorithms.

Illustrative ExperimentsFigures 7 and 8(a) show two experimental sequences whereour bilateral shared control framework is exploited to assistthe flight of four QRs using one of the 3 DoF haptic interfa-ces described earlier.

In the sequence of Figure 7, we used the constant topol-ogy controller based on artificial potentials described in[16]. This controller keeps the desired interdistancesconstant and leaves free 6 DoF of the UAV group (three

HapticInterface

(IPC Socket)

Input/OutputInterfaces

(IPC Socket,TUIO)

MATLABInterface

(IPC Socket)M

C Interface (S

erial)

BC

Interface (I2C)

IMU

RemoteController

TouchpadSmartphone

MATLABS-Function

Joystick

HapticDevice

HapticDevice

Controller

ScreenKeyboard

RC Interface(Radio)

LLUC

Motors

SensorInterface

(IPC Socket,VRPN)

OnboardCamera

MoCap System

SensorModule

Filters/SignalProcessing

TopologicalControl

ObstacleAvoidance

ForceFeedback

FCAlgorithmic Library

Haptic Interface (IPC Socket)

HLUC

SM

Interface (IPC

Pipe)

SM

Q7 Board

MotorControllers

Inter-UAVInterface

(IPC Socket)Other UAVs

μC Board BCs

Figure 6. The software setup.


LOGICALCYBERNETICS

Force Feedback Device

Force Feedback

Force Feedback Device Force Feedback Device

Force Feedback

Top Camera Top Camera Top Camera

Figure 7. An experiment employing constant topology controller and global intervention.


LOGICALCYBERNETICS



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






for rotations and three for translation). Note how the inter-action links (in yellow) are not changing over time becauseof the chosen fixed topology. The resulting desired forma-tion is a tetrahedron. The human global intervention isapplied only to the translational DoFs, and the human tele-presence is realized by applying a force to the haptic deviceproportional to the mismatch between the desired and cur-rent centroid velocity. From the reported sequence, onecan appreciate how the actions of the human commandand obstacle avoidance produce a rotation of the wholeformation, thus allowing to overcome the obstacles.

In the sequence of Figure 8(a), we used the unconstrainedtopology controller described in [18] and [25]. The control-ler in this case does not overly constrain the topology: this isclearly visible in the sequence since the interaction links arechanging over time. The human intervention is local and islimited to the UAV highlighted with the red ball. Themotion of the UAVs leads to several split and rejoins, trig-gering in some cases the tank/spring energy exchangeneeded to preserve passivity of the slave side described in the“Ensuring a Stable Behavior of the Agent Group” section.The force cue sci displayed to the human operator during theexperiment is shown in Figure 8(b): the peaks of sci occurduring the transient discrepancies between uhi and _pBa(i)

.These inform the human operator about the lag between theconnected UAV and its velocity command. Figure 8(c)shows the evolution of the six interagent potentials (links)over time. At the beginning of the motion, three links startnot connected with their potential at the infinity value 0:5 Jwhile, as time goes on, new links are created/destroyed ascan be seen from the various jumps in the interagent

potentials. Figure 8(d) shows the superimposition of theexternal energy supplied to the slave system (blue solid line)and the variation of the internal UAV-group energy (reddashed line). From this plot, it is possible to verify that thepassivity condition for the group of UAVs is always met.

We finally encourage the interested reader towatch the videosof these and other experiments based on the presented frame-work at http://www.youtube.com/user/MPIRobotics/videos.

Conclusions and Future WorkIn this article, we presented a control framework and its associ-ated experimental test bed for the bilateral shared control of agroup of QRUAVs. The control architecture allows the integra-tion of a topological motion controller, a human assistancemodule, and a force-feedback possibility to increase the telepre-sence of human assistants. The versatility of the proposed frame-work has been demonstrated by means of experiments using ahardware and software architecture based onQRUAVs.

In future, we aim at extending our test bed to outdoorscenarios, thus replacing the MOCAP system with GPSand/or other localization algorithms. We are also consider-ing the possibility of combining bilateral shared controlwith UAVs autonomously performing complex tasks, suchas cooperative transportation or exploration. Finally, weare also running extensive user studies to better assess theimmersiveness and quality of feedback provided to humanassistants during the operation of UAVs.

AcknowledgmentsThis research was partly supported by World Class Univer-sity (WCU) program funded by the Ministry of Education,

0 20 40 60 80 100–2

–1.5–1

–0.50

0.51

1.52

2.5

Time (s) Time (s) Time (s)(b) (c) (d)

Fm

(N)

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

V(J

)

0 20 40 60 80 100

0–1

1234567

H(J

), E

ext(J

)

(a)

Figure 8. (a) An experiment with unconstrained topology controller and local intervention, (b) force displayed to the humanoperator on the bilateral control device, (c) behavior of the six link potentials over time, and (d) behavior of the external energysupplied to the UAV-group system (solid blue line) and the internal UAV-group energy (dashed red line).


LOGICALCYBERNETICS



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


http://www.qmags.com/clickthrough.asp?url=http://www.youtube.com/user/MPIRobotics/videos&id=17487&adid=P67E1





Science, and Technology through the National ResearchFoundation of Korea (R31-10008). The authors also thankDr. Hyoung Il Son, CarloMasone, and Volker Grabe for theiruseful help in the development and implementation of someof the presented controllers and the QR hardware setup.

References[1] M. Schwager, B. Julian, M. Angermann, and D. Rus, “Eyes in the sky:

Decentralized control for the deployment of robotic camera networks,”

Proc. IEEE, vol. 99, no. 9, pp. 1541–1561, 2011.

[2] J. Fink, N. Michael, S. Kim, and V. Kumar, “Planning and control for

cooperative manipulation and transportation with aerial robots,” Int. J.

Robot. Res., vol. 30, no. 3, pp. 324–334, 2010.

[3] X. C. Ding, M. Powers, M. Egerstedt, R. Young, and T. Balch,

“Executive decision support: Single-agent control of multiple UAVs,”

IEEE Robot. Automat. Mag., vol. 16, no. 2, pp. 73–81, 2009.

[4] R.Murphy, J. Kravitz, S. Stover, andR. Shoureshi, “Mobile robots inmine res-

cue and recovery,” IEEERobot. Automat.Mag., vol. 16, no. 2, pp. 91–103, 2009.

[5] P. F. Hokayem and M. W. Spong, “Bilateral teleoperation: An histori-

cal survey,” Automatica, vol. 42, no. 12, pp. 2035–2057, 2006.

[6] S. Stramigioli, R. Mahony, and P. Corke, “A novel approach to haptic

tele-operation of aerial robot vehicles,” in Proc. IEEE Int. Conf. Robotics

and Automation, Anchorage, AK, May 2010, pp. 5302–5308.

[7] T. M. Lam, H. W. Boschloo, M. Mulder, and M. M. V. Paassen,

“Artificial force field for haptic feedback in UAV teleoperation,” IEEE T.

Syst. Man CY A, vol. 39, no. 6, pp. 1316–1330, 2009.

[8] M. Valenti, B. Bethke, D. Dale, A. Frank, J. McGrew, S. Ahrens, J. P. How,

and J. Vian, “The MIT indoor multi-vehicle flight testbed,” in Proc. IEEE Int.

Conf. Robotics and Automation, Rome, Italy, Apr. 2007, pp. 2758–2759.

[9] N.Michael, D. Mellinger, Q. Lindsey, and V. Kumar, “The GRASPmultiple

micro-UAV testbed,” IEEE Robot. Automat.Mag., vol. 17, no. 3, pp. 56–65, 2010.

[10] S. Lupashin, A. Sch€ollig, M. Hehn, and R. D’Andrea, “The flying

machine arena as of 2010,” in Proc. IEEE Int. Conf. Robotics and Automa-

tion, Anchorage, AK, May 2010, pp. 2970–2971.

[11] V. Mistler, A. Benallegue, and N. K. M’Sirdi, “Exact linearization

and noninteracting control of a 4 rotors helicopter via dynamic feedback,”

in Proc. 10th IEEE Int. Symp. Robots and Human Interactive Communica-

tions, Bordeaux, Paris, France, Sept. 2001, pp. 586–593.

[12] M. Fliess, J. L�evine, P. Martin, and P. Rouchon, “Flatness and defect

of nonlinear systems: Introductory theory and examples,” Int. J. Control,

vol. 61, no. 6, pp. 1327–1361, 1995.

[13] B. Bethke, M. Valenti, and J. P. How, “UAV task assignment,” IEEE

Robot. Automat. Mag., vol. 15, no. 1, pp. 39–44, 2008.

[14] M. Schwager, N. Michael, V. Kumar, and D. Rus, “Time scales and

stability in networked multi-robot systems,” in Proc. IEEE Int. Conf.

Robotics and Automation, Shanghai, China, May 2011, pp. 3855–3862.

[15] A. Franchi, C. Masone, H. H. B€ulthoff, and P. R. Giordano, “Bilateral

teleoperation of multiple UAVs with decentralized bearing-only forma-

tion control,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems,

San Francisco, CA, Sept. 2011, pp. 2215–2222.

[16] D. Lee, A. Franchi, P. Robuffo Giordano, H. I. Son, and H. H.

B€ulthoff, “Haptic teleoperation of multiple unmanned aerial vehicles over

the internet,” in Proc. IEEE Int. Conf. Robotics and Automation, Shanghai,

China, May 2011, pp. 1341–1347.

[17] A. Franchi, P. Robuffo Giordano, C. Secchi, H. I. Son, and H. H.

B€ulthoff, “A passivity-based decentralized approach for the bilateral

teleoperation of a group of UAVs with switching topology,” in Proc. IEEE Int.

Conf. Robotics and Automation, Shanghai, China, May 2011, pp. 898–905.

[18] A. Franchi, C. Secchi, H. I. Son, H. H. B€ulthoff, and P. Robuffo Gior-

dano. Bilateral teleoperation of groups of mobile robots with time-varying

topology, IEEE Trans. Robot., to be published [Online]. Available: http://

ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6199993.

[19] P. Robuffo Giordano, A. Franchi, C. Secchi, and H. H. B€ulthoff,

“Passivity-based decentralized connectivity maintenance in the bilateral tel-

eoperation of multiple UAVs,” in Proc. Robotics: Science and Systems, Los

Angeles, CA, June 2011.

[20] H. I. Son, J. Kim, L. Chuang, A. Franchi, P. Robuffo Giordano, D.

Lee, and H. H. B€ulthoff, “An evaluation of haptic cues on the tele-opera-

tor’s perceptual awareness of multiple UAVs’ environments,” in Proc.

IEEEWorld Haptics Conf., Istanbul, Turkey, June 2011, pp. 149–154.

[21] H. I. Son, L. L. Chuang, A. Franchi, J. Kim, D. J. Lee, S. W. Lee, H. H.

B€ulthoff, and P. Robuffo Giordano, “Measuring an operator’s maneuver-

ability performance in the haptic teleoperation of multiple robots,” in

Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, San Francisco,

CA, Sept. 2011, pp. 3039–3046.

[22] D. J. Lee and K. Huang, “Passive-set-position-modulation frame-

work for interactive robotic systems,” IEEE Trans. Robot., vol. 26, no. 2,

pp. 354–369, 2010.

[23] M. Franken, S. Stramigioli, S. Misra, C. Secchi, and A. Macchelli, “Bilateral

telemanipulation with time delays: A two-layer approach combining passivity

and transparency,” IEEE Trans. Robot., vol. 27, no. 4, pp. 741–756, 2011.

[24] C. Secchi, A. Franchi, H. H. B€ulthoff, and P. Robuffo Giordano,

“Bilateral teleoperation of a group of UAVs with communication delays

and switching topology,” in Proc. IEEE Int. Conf. Robotics and Automa-

tion, St. Paul, MN, May 2012, pp. 4307–4314.

[25] P. Robuffo Giordano, A. Franchi, C. Secchi, and H. H. B€ulthoff,

“Experiments of passivity-based bilateral aerial teleoperation of a group of UAVs

with decentralized velocity synchronization,” in Proc. IEEE/RSJ Int. Conf. Intelli-

gent Robots and Systems, San Francisco, CA, Sept. 2011, pp. 163–170.

Antonio Franchi, Max Planck Institute for BiologicalCybernetics, Spemannstraße 38, 72076 T€ubingen, Ger-many. E-mail: [email protected].

Cristian Secchi, Department of Science and Methods ofEngineering, University of Modena and Reggio Emilia, viaG. Amendola 2, Morselli Building, 42122 Reggio Emilia,Italy. E-mail: [email protected].

Markus Ryll, Max Planck Institute for Biological Cyber-netics, Spemannstraße 38, 72076 T€ubingen, Germany.E-mail: [email protected].

Heinrich H. B€ulthoff, Max Planck Institute for BiologicalCybernetics, Spemannstraße 38, 72076 T€ubingen, Ger-many, and Department of Brain and Cognitive Engineer-ing, Korea University, Anam-dong, Seongbuk-gu, Seoul136 713, Korea. E-mail: [email protected].

Paolo Robuffo Giordano, Max Planck Institute for Biolog-ical Cybernetics, Spemannstraße 38, 72076 T€ubingen, Ger-many. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________________________

___

________________

________________

_____________________

__________________

________________________

http://www.qmags.com/clickthrough.asp?url=http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6199993&id=17487&adid=P68E1










•By Ivana Palunko, Patricio Cruz, and Rafael Fierro

In the past few decades, unmanned aerial vehicles (UAVs) have become promisingmobile platforms capable of navigating semiautonomously or autonomously in uncer-tain environments. The level of autonomy and the flexible technology of these flyingrobots have rapidly evolved, making it possible to coordinate teams of UAVs in a widespectrum of tasks. These applications include search and rescue missions; disaster relief

operations, such as forest fires [1]; and environmental monitoring and surveillance. Insome of these tasks, UAVs work in coordination with other robots, as in robot-assistedinspection at sea [2]. Recently, radio-controlled UAVs carrying radiation sensors and videocameras were used to monitor, diagnose, and evaluate the situation at Japan’s FukushimaDaiichi nuclear plant facility [3].

One specific type of aerial vehicle, the quadrotor, has the capability of not only taking offand landing in rough or inaccessible areas but also carrying more weight than other aerialplatforms due to its four propellers. Several research groups have developed notable applica-tions and experiments using multiple quadrotor UAVs as part of their robotic platforms.Existing results range from basic hovering [4] and trajectory tracking [5], to formation con-trol, [6] surveillance [7], aggressive maneuvering [8], and aerobatic flips [9].

Once the functionality of the UAVs advances from simple environmental sensing tomodification or manipulation of their external environment, a wide and novel set of practi-cal applications and challenges will appear in the aerial robotics research field. In fact, for




Safe and Efficient Load Manipulation with Aerial Robots

©ISTOCK PHOTO.COM/©IAKOV FILIMONOV


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________

http://www.qmags.com/clickthrough.asp?url=www.istockphoto.com&id=17487&adid=P69E1





small-size UAVs, a variety of examples of interacting withexternal objects have been recently introduced. Individualor cooperative transport of suspended load [10]–[12], grasp-ing and manipulating [13], [14], applying force to a wall[15], and building structures [16] are examples where aerialrobots interact with their surroundings. An interestingapproach that brings together mobile manipulation using aUAV equipped with dexterous arms is introduced in [17].Finally, an innovative idea of using a network of quadrotorsto deliver medicines to remote villages is presented in [18].

In this article, we are concerned with the challenging prob-lem of using quadrotors to transport and manipulate loads

safely and efficiently. Aerial manipulation is extremely impor-tant in emergency rescue missions as well as in military andindustrial applications. For example, safe aerial transport of avictim from a dangerous area is vital in an emergencyresponse. In addition, delivering equipment and supplies toinaccessible places is commonly achieved using aerialtransportation. Another application is landmine detectionwith a sensor suspended from a cable. Flying with a suspendedload is a challenging and, sometimes, hazardous task since theload significantly changes the flight characteristics of the aerialvehicle. In addition, the stability of the vehicle-load systemmust be preserved for safe operation. Therefore, it is essential

that the flying robot has the ability toadapt to changes in the systemdynamics and reduce the swing of theload during assigned maneuvers. Inthis article, we describe and summa-rize two possible approaches that ena-ble agile and safe load transportationusing a single quadrotor UAV:1) an adaptive controller consider-

ing changes in the center ofgravity (CoG) [19]

2) an optimal trajectory generationbased on dynamic programmingfor swing-free maneuvering [20].Simulation results verify the va-

lidity of the proposed algorithms.Moreover, we present experimentalresults of the proposed optimalswing-free trajectory tracking.

Problem Statementand Motivation

Adaptive ControlThe design of an adaptive controlleris motivated by focusing on the prob-lems that can alter the CoG of thequadrotor. The coordinates of theCoG of the quadrotor are denoted bythe vector r ¼ xG yG zG½ �T measuredin fBg, which represents the distancefrom the origin of fBg to the quadro-tor’s CoG, as shown in Figure 1.Usually, when modeling an aerialvehicle, the vector r ¼ 0. This meansthat the vehicle is balanced, i.e., theCoG coincides with the origin of themoving aircraft-fixed coordinatesystem fBg. However, there are atleast two cases in which this assump-tion fails, and we address them inthis article:1) The change in the CoG of the load

attached to the quadrotor by a

{B}

ρH

rG

ZB

XB

YB

YH

ZH

XH

{A}

ZA

YA

XA

Position Vector of the {A}Interial Frame with Respect

to the Aircraft CoG

rG

{H }

ΘL ϕL

{B}

{L}

{A}

ρL

rG

ZB

XB

YB

ZA

YA

XA

YL

ZLXL

(a)

(b)

{B}

{L}

ρL

Position Vector of the HookFrame {H } with Respect to the

Aircraft CoG

Figure 1. The coordinate frames used. A quadrotor grasping (a) a load and (b) asuspended load.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






rigid link [Figure 1(a)]: By a rigid link we mean anytype of gripper firmly attached to the body of a quadro-tor. The change in the CoG can occur if the load is notgripped at the point of CoG. This can happen if thegripper missed the exact gripping point of the load orthe load slides due to flight conditions.

2) The change in the position of suspension point of the sus-pended load [Figure 1(b)]: The change in the positionof the suspension point of the load qH can occur when,e.g., one or more of the suspension cables snap in thecase of multipoint suspension load.Since, in both cases, the change in r, qL, and qH

produces additional forces and torques acting on the quad-rotor, we can observe these two cases and treat them as achange in a single vector r. In this article, we propose amethod based on adaptive control to solve this problem.

Dynamic ProgrammingFrom this point forward, we focus on the case of a quadro-tor carrying a single-point suspended load. The suspendedload exerts forces and torques on the quadrotor that aregiven by (8). FH and TH are functions of the load–displace-ment angles gL ¼ /L hL½ �T shown in Figure 1(b). In thesecond part of this article, we present a method based ondynamic programming, which ensures swing-free maneu-vers of the suspended load.

Definition 1Given a small positive constant � > 0 and the timerequired to accomplish the desired trajectory tF > 0, wesay the quadrotor carrying a suspended load performs aswing-free maneuver when /L tð Þj jj j � and hL tð Þj jj j �for all t tF .

By minimizing the angles in gL, we are minimizing theforce FH and the torque TH that represent external disturb-ance forces and torques for the quadrotor baseline attitudecontroller, respectively. By combining adaptive controland trajectory generation using dynamic programming, wedevelop a hierarchical control architecture that ensuresrobustness of the quadrotor baseline attitude controllerwith respect to model uncertainties (change in the CoG ofthe quadrotor) and to external forces FH and torques TH

exerted by the suspended load. Besides that, the system isable to perform agile swing-free maneuvers that can beuseful in a wide range of applications.

PreliminariesFirst, we present the model of a quadrotor carrying a sus-pended load. While modeling, we consider the quadrotor-load system as a system of two separate rigid bodies con-nected by the suspension cable. The interaction betweentwo rigid bodies is described through force and torqueequations presented in the following subsections. Next, wedescribe the structure of a baseline attitude controller. Adetailed derivation of the quadrotor model and stabilityproperties of the baseline controller can be found in [19].

Quadrotor ModelWe begin bymodeling a quadrotor as a rigid body. To analyzethe motion of a rigid body through 6 degrees of freedom(DoF), we define two coordinate frames as indicated in Fig-ure 1. The moving coordinate frame fBg is fixed to the quad-rotor and is called the aircraft-fixed reference frame. Theorigin of the aircraft-fixed frame is chosen to coincide with theCoG in the case when the vehicle is balanced. A ground-fixedreference frame fAg is considered to be the inertial frame.The position and orientation of the vehicle are described rela-tive to the inertial reference frame fAg, while the linear andangular velocities of the vehicle are expressed in the aircraft-fixed coordinate frame fBg. As in [21], the following variablesare used to describe the quadrotor kinematics and dynamics:n, the position of the origin of fBg measured in fAg;g2 ¼ / h w½ �T , the angles of roll /, pitch h, and yaw wthat parameterize locally the orientation of fBg with respectto fAg; v, the linear velocity of the origin of fBg relative tofAg expressed in fBg (i.e., body-fixed linear velocity);X, theangular velocity of fBg relative to fAg expressed in fBg (i.e.,body-fixed angular velocity), and r ¼ xG yG zG½ �T , thedistance from the origin of fBg to the quadrotor’s CoG. Thequadrotor’s 6 DoF nonlinear dynamic equations of motioncan be expressed in a compact form as

M _m þ C mð Þm þDm þ G gð Þ ¼ sþ sL, (1)

where g ¼ ½n g2�T is the vector of position and orientation,and m ¼ ½v X�T is the vector of linear and angular veloc-ities. M is the mass and inertia matrix of the quadrotor andmatrix C mð Þ consists of Coriolis and centripetal terms. Usingthe results from [22], we obtain a parameterization such thatC mð Þ is skew symmetric. Decomposing the vectors of externalforces acting on a rigid body, we define four distinct vectorsDm, G gð Þ, s, and sL. The dissipative force and torque vectoris given by Dm, where D is the damping matrix. Also, withG gð Þ, we denote the vector of gravitational forces andtorques. The control inputs are given as a vector s

fs g2ð Þ ¼ AR�1B (g2)

00U1

24

35, s g2,Uð Þ ¼

fs g2ð ÞU2

U3

U4

2664

3775,

where U1, U2, U3, and U4 represent control forces gener-ated by four rotors [4], and fs is the vector of forces that aregenerated by decomposing the total thrust U1.sL ¼ FH TH½ �T represents the vector of forces and tor-ques (8) that the load exerts on the quadrotor. A compactrepresentation of system kinematics is

_n_g2

� �¼

ARB(g2) 00 Q(g2)

� �vX

� �, (2)

where ARB g2ð Þ and Q g2ð Þ represent the transformationmatrices. The Lagrangian form of the quadrotor dynamics



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






(1) is obtained by transforming the dynamics derived fromfirst principles based on Newton–Euler axioms.

Suspended Load ModelVery thoroughmodels of single- and multipoint suspensionas well as multilift variations of external suspended loadsystems can be found in [23]. In more recent publications[24], [25], the single-point suspension type models foundin [23] have been implemented and simulated. Consideringthe models presented in the literature, we outline the modelof the single-point suspended load in this subsection.

The suspended load is modeled as a point mass spheri-cal pendulum suspended from a single point. The coordi-nate systems we use are shown in Figure 1(b). The motionof the load is described in polar coordinates using angles/L and hL, where /L and hL are measured from the ZH axisin direction of XH and YH , respectively. Therefore, theposition vector qL of the load with respect to the suspen-sion point is given by

qL ¼ RYH hLð ÞRXH /Lð Þ00lL

24

35, (3)

where RYH hLð Þ and RXH /Lð Þ are rotational matrices, and lLis the length of the cable. The position vector qH of Hf gwith respect to the quadrotor CoG is given by qH ¼xH yH zH½ �T . The absolute velocity mL of the load isgiven by

mL ¼ v þ _qþX3 q, (4)

where v is the linear velocity of the quadrotor, q ¼qL þ qH is the position vector of the load with respect tothe CoG of the quadrotor, and X is the angular velocity ofthe quadrotor. The absolute acceleration _mL of the loadbecomes

_mL ¼ _v þ €qþ _X3 qþ 2X3 _qþX3 (X3 q), (5)

where _v is the linear acceleration of the quadrotor. Thevector given by GL represents the vector of gravitationalforces and torques

GL ¼ RXB /ð Þ�1RYB hð Þ�100

mLg

24

35, (6)

where / and h are, respectively, the roll and pitch anglesof the quadrotor, andmL is the mass of the load. By enforcingtorque equilibrium about the suspension point, we obtain

fsL /L, hL, m, gð Þ ¼ �qL 3 �mL _mL þ GLð Þ ¼ 0, (7)

where m and g are vectors of quadrotor states. In anexpanded form, (7) is a system of three second-order

equations in Cartesian coordinates in the Hf g frame. Aftersolving these three equations for €/L and €hL, we obtain theequations of motion for the given system in polar coordi-nates. The symbolic computation for obtaining these equa-tions is performed using Mathematica, and because of thelength of the equations, is omitted. The suspended loadintroduces additional terms denoted by sL in the equationsof motion of the quadrotor. The force FH that the load exertson the vehicle and the torque TH are, respectively, given by

FH ¼ �mL _mL þ GL, TH ¼ qH 3FH : (8)

Both FH and TH are functions of /L and hL as well as ofthe quadrotor states m and g.

Quadrotor Baseline Attitude ControllerThe baseline controller presented in this article consists ofa feedback-linearizing controller and a linear control algo-rithm that stabilizes the 6 DoF of the quadrotor, namely,orientation g2 and position n. Conventional control tech-niques such as pole placement are applied to the linearizedsystem to design the lead–lag controller used for stabiliza-tion and trajectory tracking. In robotics, feedback lineari-zation is commonly used for computed torque control.The control objective is to transform the quadrotordynamics (1) into a linear system _m ¼ #, where # can beinterpreted as a commanded acceleration vector. There-fore, the control algorithm can be written as follows:

s ¼ C mð Þm þDm þ G gð Þ þM#, (9)

where the commanded acceleration vector # is the outputfrom the linear controller designed using the pole place-ment technique. Since a quadrotor is an underactuated sys-tem with four inputs U 2 R4 and six outputs g 2 R6, thedesign of the controller based on feedback linearization isnontrivial. The details of the design and stability analysiscan be found in [19]. The proposed control algorithm isimplemented in MATLAB=Simulink, and the simulationresults validate its efficiency [19].

Adaptive Control for an Unbalanced QuadrotorTo the best of our knowledge, most of the current literaturegenerally assumes a balanced UAV in modeling meaningthat the vector r is invariant and zero. However, there are anumber of applications in robotics where the displacementof the CoG is successfully exploited to improve systemperformance. For instance, dynamic balancing of the CoGis used to achieve better maneuverability in underwaterrobotics [26] or for locomotion of a modular robot overuneven and unknown terrain [27].

One of the drawbacks of feedback linearization is itslack of robustness to model uncertainties, which in ourcase represents the change in the quadrotor CoG. Thismotivates the design of an adaptive controller that is ableto handle changes in the CoG.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Adaptive ControlConsidering the control algorithm (9), we can write

s ¼ C mð Þm þDm þ G gð Þ þ M#, (10)

where the ^ (hat) denotes estimates of the system parame-ters. Now, the tracking error dynamics can be written as

M _m � #½ � ¼ M�M �

#þ C mð Þ � C mð Þ �m

þ G gð Þ � G gð Þ �:

Because quadrotor equations of motion are linear in theparameter vector c ¼ r, we can apply the followingparameterization

U m, gð Þ~c ¼ M�M �

#þ C mð Þ � C mð Þ �m

þ G gð Þ � G gð Þ �:

In the above expression, ~c ¼ c� c 2 R33 1 is the unknownparameter error vector and U m, gð Þ 2 R63 3 is a knownmatrix function of measured signals usually referred to asthe regressor matrix [28]. The parameter update rule(assuming _c ¼ 0) is chosen as

_c ¼ �CUT(m, g)BTPTe,

where e ¼ ½gref � g _gref � _g�T 2 R123 1 is the trackingerror, P ¼ PT > 0 satisfies Lyapunov stability equationfor linear systems, C ¼ CT > 0, and B ¼ 0 M�1½ �T . Byadding the adaptive part to the controller based on feed-back linearization, we succeed in stabilizing an unbalancedquadrotor. The proposed adaptive controller enables aquadrotor to perform agile maneuvers while reconfiguringin real time whenever a change in CoG occurs. Moreover,the control algorithm shows robustness with respect to theexternal disturbance forces and torques exerted by a sus-pended load while dynamic changes in the quadrotor CoGoccur (see Figure 2). Forces and torques exerted by the sus-pended load decrease when the load–displacement anglesare decreased, as shown in Figure 2(c) and (d).

Trajectory Generation for Swing-Free ManeuversTransport of suspended objects using a robot or a crane is acommon application. At the end of a transport motion, thesuspended object naturally continues to swing. Suppres-sion of residual oscillation has been a topic of research formany years. Although both open- and closed-loop strat-egies have been explored, in this article, we focus on anopen-loop technique presented in detail in [29] and [30],which we apply for a quadrotor carrying a suspended load.

Optimal Trajectory GenerationUsing Dynamic ProgrammingWith slight abuse of notation, the subsequent procedurefollows the approach of [29] and outlines the method of

applying dynamic programming to a discrete-time piece-wise linear system. We begin with the general form of thediscrete-time system

qkþ1 ¼ Akqk þ Bkuk, (11)

where qk ¼ ½gL mL gL m�T is the state vector, and Ak

and Bk are the system and input matrices, respectively.The discretization is performed using the Euler algorithm.Given an initial state q0, we would like to find the optimalsequence of inputs that will minimize the scalar objectivefunction

C(q, u) ¼XNk¼1

Ck(qk, uk): (12)

The state and input of the system are consequently given by

q ¼ gL mL g m½ �T , u ¼ _m,

where gL and mL are load–displacement angles and angularvelocities, and g, m, and _m are quadrotor attitude, velocity,and acceleration vectors, respectively. To suppress theresidual oscillations, a penalty weight is introduced in theobjective function

Ck ¼ 12

qTkQkqk þ 2qTkRkuk þ uTk Skuk �þ 1

2pxTF xF :

Matching like terms with those of the general form of theobjective function leads to

fN ¼ 12pxTF xF , yN ¼ �pxF ,

zN ¼ 0, QN ¼ Qk ¼ qpIn,

RN ¼ Rk, SN ¼ Sk ¼ In, (13)

where In is an identity matrix and n is the number of statevariables. The procedure for applying this algorithm is asfollows:1) Determine vN andWN using

fN ¼ cN , mN ¼ yN , WN ¼ QN : (14)

yN and QN can be extracted directly from the objectivefunction.

2) Calculate vi and Wi recursively for i ¼ N � 1 to i ¼ 1using

fi ¼ fiþ1 þ ci �12hT5iH

�13i h5i,

mi ¼ h4i �H2iH�13i h5i,

Wi ¼ H1i �H2iH�13i H

T2i, (15)

and

H1i ¼ Qi þ ATi Wiþ1Ai,

H2i ¼ Ri þ ATi Wiþ1Bi,

H3i ¼ Si þ BTi Wiþ1Bi,

h4i ¼ yi þ ATi viþ1,

h5i ¼ zi þ BTi viþ1, (16)



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






while storing matrices H�13i , HT

2i, and H�13i , h5i in

the process.3) Calculate ui and qi recursively for i ¼ 1 to i ¼ N � 1

usingui ¼ �H�1

3i ½H3iqi þ h5i� (17)

and (11), respectively.

Simulation ResultsThe algorithm described in the previous section requiresthe initial and final states to be known before the

optimization. An initial trajectory estimate is also requiredfor the first optimization pass to compute the required Ak

and Bk matrices. We use cubic polynomial position trajec-tories for the initial simulation of the system because ithas continuous first and second derivatives. Furthermore,we present a set of simulations for using swing-free trajec-tory tracking in cluttered environments. By using theswing-free policy presented above, where the displacementangle of the load is minimized, we obtain the optimaltrajectory depicted in Figure 3. We can see that the quadro-tor tracks the optimal trajectory without collisions in

0 5 10 15 20 25 30 35 40 45 50

0 5 10 15 20 25 30 35 40 45 50

0 5 10 15 20 25 30 35 40 45 50

0 5 10 15 20 25 30 35 40 45 50

−2

0

2

x (m

)

Quadrotor Attitude

−2

0

2

y (m

)

0123

z (m

)

−10

0

10

ψ (

°)φ L

(°)

θ L(°

)

Time (s)

Time (s)

0 5 10 15 20 25 30 35 40 45 50

0 5 10 15 20 25 30 35 40 45 50

0 5 10 15 20 25 30 35 40 45 50

−2−1012

x G (

m)

y G (

m)

y G (

m)

Coordinates of the CoG

−2−1012

−6

−4

−2

0

0 10 20 30 40 50

0 10 20 30 40 50

−20

−10

0

10

20Suspended Load Displacement Angles

−20

−10

0

10

20

0 10 20 30 40 50−0.5

0

0.5

1

FLx

(N

)F

Ly (

N)

FLz

(N

)

TLx

(N

)T

Ly (

N)

TLz

(N

)

Force and Torques Load Excerts to Quadrotor

(a) (b)

(c) (d)

0 10 20 30 40 50−4

−2

0

2

0 10 20 30 40 50−1

−0.5

0

0.5

Time (s)

0 10 20 30 40 50

0 10 20 30 40 50

0 10 20 30 40 50Time (s)

−1−0.5

00.5

1

−1−0.5

00.5

1

−0.5

0

0.5

Cubic TrajectoryOptimal Swing-Free Trajectory



Adaptive CoG Cubic TrajectoryReal CoG Displacement

Adaptive CoG Optimal Swing-Free Trajectory

Figure 2. An adaptive feedback linearization algorithm. (a) Position and heading, (b) estimated and real CoG coordinates,(c) suspended load displacement angles, and (d) forces and torques from the suspended load.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






contrast to tracking the initial cubic trajectories where thedisplacement angles are too large to avoid collisions.

The algorithm presented in this article is not able toreduce the residual oscillations as the trajectory trackingtime reduces. By reducing the tracking time, the linearaccelerations and velocities are higher, which directlyinfluences the magnitude of the load swing (Figure 4).

Experimental VerificationThe experimental verification was performed at the Multi-Agent Robotics Hybrid and Embedded Systems (MARHES)laboratory at the University of New Mexico [31]. Our aerialtest bed consists of three AscTech Hummingbird quadro-tors [32]. Moreover, the laboratory is equipped with amotion capture system (Vicon system [33]) composed ofeight cameras for precision indoor positioning.

System ArchitectureTo track the attitude and position of the quadrotors, thedata are sampled by the motion capture system and areaccessed via Transmission Control Protocol (TCP) andInternet Protocol (IP). As a real-time engine, we use aNational Instruments CompactRIO (NI cRIO) [34],which is a reconfigurable I/O field-programmable gatearray (FPGA)-based core used in a variety of embeddedcontrol and monitoring applications. This FPGA-embed-ded controller offers powerful stand-alone execution forreal-time applications. For wireless communication

between the NI cRIO and quadrotor, we use XBee-embedded radio frequency (RF) modules that providecost-effective wireless connectivity to devices in a ZigBeenetwork. Figure 5(a) shows more details about the inter-connections in the hardware architecture. Through thissystem, we can acquire position data from Vicon, calcu-late control signals using the NI cRIO, and send them tothe quadrotors by the XBee modules at 100 Hz.

14121086420

–5

0

5 –5 –4 –3 –2 –1 0 1 2 3 4 5 6x (m)

y (m)

Load: Optimal Swing FreeLoad: Initial Cubic

Quadrotor: Path

Figure 3. 3-D representation of trajectories considering multiplewaypoints with obstacles.

High-LevelMission Planner

Swing-FreeTrajectoryGenerator

Based on DP

Nonlinear Model ofSuspended Load

Nonlinear Model of the System

Nonlinear Quadrotor Model

ComputedTorque Control

Adaptive Control

Adaptive Controlfor Change

in CoG

PositionController

OrientationController

Baseline Attitude Controller

P

ηL

ηref

ηL

η

η

η,V

τϑ

γ

eη2

eη1

Figure 4. The block scheme of the adaptive control system with swing-free trajectory generation.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






In subsections “Adaptive Control” and “SimulationResults,” we show trajectory tracking using the trackingcontroller based on feedback linearization. Since imple-mentation of this type of controller is nontrivial, forexperimental purposes, we use a lead–lag trackingcontroller without linearizing the system via feedbacklinearization. Since the quadrotor is not performingaggressive maneuvers, we can assume that its dynamicsare close to the linearized model around the equilibriumpoint. The baseline attitude controller is imple-mented in a cascade with the AscTec AutoPilotonboard flight control unit [32]. Subsequently, we havedesigned a position controller and its software archi-tecture implemented in LabVIEW, as shown in

Figure 5(b). The configuration of this controller isas follows:l two lead–lag controllers for the X and Y directional

movement (one lead–lag for the position and one lag forthe velocity)

l one lead–lag controller for the Z directional movement(just for the position error)

l one proportional-integral (PI) for the yaw w angle, i.e.,for the heading.The lead–lag and PI controllers are implemented using

the PID lead–lag virtual instrument (VI) available in theLabVIEW PID and Fuzzy Logic Toolkit. The implementedexperimental setup achieves acceptable trajectory trackingfor our purposes.

LabVIEW Project

VICON Interface VI Quadrotor Interface VI

Host Name

Desired SubjectVIVON

DataStream

txt File

VelocityEstimator

TrajectoryLoader

PositionControllers

andHeadingControl

ControlValue

Transmitter

Running in the NI cRIORunning in the LabVIEW PC Interface

U

xm, ym, zm

xd, yd, zd

xm, ym, zm⋅

xd, yd, zd⋅ ⋅ ⋅

⋅ ⋅

ψm

ψd

VICONCameras

VICONCameras

VICON MXGiganet

VICON PCInterface

LabVIEW PCInterface

XbeeModule/Adapter

NI cRIO

ZigB

ee L

ink

Router

Gigabyte Ethernet

Ethernet 100 Mb/s

Ethernet 100 Mb/s

Eth

erne

t 100

Mb/

s

Rs-232

(a)

(b)

Figure 5. System architecture. (a) Hardware architecture showing the communication links and (b) software architecture scheme;r denotes reference value and m denotes measured value.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Experimental Setup and ResultsIn this section, we report two sets of experiments that showthe performance of our real-time control architecture andverify the swing-free trajectory generation based ondynamic programming. We are currently working on theexperimental validation of the adaptive controller for thechange in CoG.

In the first set of experiments, we show the robustnessof the proposed method with respect to unmodeled actua-tor dynamics, noise, and system delays. In our case, theactuator is a quadrotor whose model and attitude control

design are presented in this article. Moreover, the mass ofthe suspended load used in this experiment is 47 g and thelength of the suspension link is 0.62 m. By implementingthe proposed method on an experimental system, we seethat the performance deteriorates due to imperfect trajec-tory tracking. However, the attenuation of the load–dis-placement angles /L and hL is achieved. Figure 6(a)presents the results for hL since similar results wereobtained for /L. Therefore, the proposed method is robustenough and shows good performance even with the lack ofperfect trajectory tracking.

0 2 4 6−1

−0.5

0

0.5

1

1.5

y (m

)

z (m

)

y (m)x (m)

Initial Cubic

0 2 4 6

0 2 4 6 0 2 4 6

Optimal Swing Free

−20

−10

0

10

20

θ L (

°)

−1

−0.5

0

0.5

1

1.5

y (m

)

−20

−10

0

10

20

θ L (

°)

t (s) t (s)−1.5

−1 −0.50 0.5

1 1.5

−1.5−1

−0.50

0.511.5

0.91

1.11.21.31.41.51.6

Position of the Quadrotor

SimulationExperiment

Initial Cubic TrajectoryOptimal Swing Free Trajectory

(a)(b)

0 5 10 15−0.2

−0.1

0

0.1

0.2

Time Domain Signal φL(t ) (°) Time Domain Signal θL(t ) (°)

Power Spectral Density of φL(W/Hz)

Power Spectral Density of θL(W/Hz)

Time (s)

0 5 10 15

Time (s)

0 1 2 3 4 50

0.2

0.4

0.6

0.8

1

1.2

1.4

−0.2

−0.1

0

0.1

0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

Frequency (Hz)0 1 2 3 4 5

Frequency (Hz)

Initial CubicSwing Free Optimal

(c)

Figure 6. Experimental results for swing-free trajectory tracking. (a) Trajectory tracking considering a single waypoint, (b) quadrotorposition, and (c) power spectral density.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






In the second set of experiments, we try to mimicswing-free trajectory tracking in cluttered environments,motivated in the first section by simulations presented inFigure 3. We built a maze of obstacles shown in Figure 7.The quadrotor is flying above the obstacles while carryingthe suspended load (the same load used for the first experi-ment) through narrow corridors. First, the quadrotor wastracking an initial three-dimensional (3-D) trajectory withcubic profile with respect to time. Then, using the dynamicprogramming-based algorithm, we have computed a 3-Dtrajectory with an optimal swing-free profile with respectto time. The experimental data, including quadrotor posi-tion, are shown in Figure 6.

Since it is hard to analyze raw experimental data just byvisual inspection, we use the tools from signal processing,computing the power spectral density for each signal.Before processing, we extracted the part of the signal thatcorresponds to the swing-free behavior. Power spectraldensity describes how the power of a signal is distributedwith frequency [see Figure 6(c)]. By taking the integral ofthe power spectral density, we compute the total powerfor each signal. We repeated the experiment in four trials

and analyzed them using power spectral density. The dataare presented in Table 1. In all four cases, data show lessenergy for the load displacement while tracking theoptimal trajectory. The table shows the simulation resultstoo. Videos of simulations and experiments can be foundat [31].

ConclusionsThe ability to safely and efficiently manipulate and trans-port loads using UAVs is an extremely useful capability inmany civilian and military applications ranging fromsearch and rescue missions, humanitarian relief opera-tions, and automated construction. In this article, wedescribe a number of key methodologies that enablemicro UAVs to transport loads while satisfying missionconstraints. Due to the small-scale, limited power, andlimited resources of micro UAVs, we are faced with a newset of challenges that need to be addressed to make UAVscommonplace. Static controllers are not sufficient tohandle changes in the CoG of the vehicle. The proposedadaptive control for changes in the CoG alleviates this needby providing a provable adaptation rule to compensate forthe changes in the quadrotor’s CoG. If (practical) swing-free motion is required, the dynamic programmingapproach described in this article is a viable solution totackle this issue. Finally, to better understand the advan-tages and limitations of the proposed approaches, system-atic experimental studies on a hardware test bed aremandatory. To this end, we outlined our multi-UAVrobotic test bed and demonstrated its capabilities to vali-date advanced control algorithms, networking protocols,and cooperative behaviors that would make stable andagile UAV manipulation and transportation using microUAVs a reality.

AcknowledgmentsPartial support was provided by NSF grant ECCS 1027775and by DOE URPR (University Research Program inRobotics) grant DE-FG52-04NA25590.

References[1] L. Merino, F. Caballero, J. Martınez de Dios, I. Maza, and A. Ollero,

“An unmanned aircraft system for automatic forest fire monitoring

and measurement,” J. Intell. Robot. Syst., vol. 65, pp. 533–548, Aug.

2011.

[2] M. Lindermuth, R. Murphy, E. Steimle, W. Armitage, K. Dreger, T.

Elliot, M. Hall, D. Kalyadin, J. Kramer, M. Palankar, K. Pratt, and C. Grif-

fin, “Sea robot-assisted inspection,” IEEE Robot. Automat. Mag., vol. 18,

no. 2, pp. 96–107, June 2011.

[3] T. Honeywell. (2011, Apr.). Hawk aids Fukushima Daiichi disaster

recovery [Online]. Available: http://honeywell.com/News/Pages/Honeywell-

T-Hawk-Aids-Fukushima-Daiichi-Disaster-Recovery.aspx

[4] S. Bouabdallah, P. Murrieri, and R. Siegwart, “Design and

control of an indoor micro quadrotor,” in Proc. IEEE Int. Conf.

Robotics and Automation, Barcelona, Spain, Apr. 2004, vol. 5,

pp. 4393–4398.

•Table 1. Total power Ptot

of the load–displacement signals.

ExperimentalResults

Signal (�) Ptot Init.Traj. (W)

Ptot Opt.Traj. (W)

Trial 1 /L 31.0634 26:7398

hL 36.0171 26.3692

Trial 2 /L 28.3503 26.2140

hL 39.4729 30.3817

Trial 3 /L 34.3385 21.5736

hL 34.0470 28.9020

Trial 4 /L 38.8807 28.5435

hL 35.4629 33.8539

Simulation results /L 37.0453 10.0351

hL 37.1671 10.5550

Figure 7. The Hummingbird quadrotor transporting asuspended load through obstacles. (Photo courtesy of thestudents of MARHES lab, University of New Mexico.)



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_______________________________

http://www.qmags.com/clickthrough.asp?url=http://honeywell.com/News/Pages/Honeywell-T-Hawk-Aids-Fukushima-Daiichi-Disaster-Recovery.aspx&id=17487&adid=P78E1





[5] G. Hoffman, S. Waslander, and C. Tomlin, “Quadrotor helicopter

trajectory tracking control,” in Proc. AIAA Guidance, Navigation and

Control Conf. and Exhibit, Honolulu, HI, Apr. 2008, pp. 1–14.

[6] A. Sch€ollig, F. Augugliaro, and R. D’Andrea, “A platform for dance

performances with multiple quadrocopters,” in Proc. IEEE/RSJ Int. Conf.

Intelligent Robots and Systems—Workshop on Robots and Musical Expres-

sions, 2010, pp. 1–8.

[7] M. Valenti, D. Dale, J. How, and D. Pucci de Farias, “Mission health

management for 24/7 persistent surveillance operations,” in Proc. AIAA

Conf. Guidance, Navigation and Control, Hilton Head, SC, Aug. 2007,

pp. 1–18.


control for precise aggressive maneuvers with quadrotors,” Int. J. Robot.

Res., vol. 31, no. 5, pp. 664–674, Apr. 2012.

[9] S. Lupashin, A. Sch€ollig, M. Sherback, and R. D’Andrea, “A

simple learning strategy for high-speed quadrocopter multi-flips,”

in Proc. IEEE Int. Conf Robotics and Automation, May 2010,

pp. 1642–1648.

[10] N. Michael, J. Fink, and V. Kumar, “Cooperative manipulation and

transportation with aerial robots,” Autonom. Robots, vol. 30, pp. 73–86,

2011.

[11] M. Bernard and K. Kondak, “Generic slung load transportation sys-

tem using small size helicopters,” in Proc. IEEE Int. Conf. Robotics and

Automation, Kobe, Japan, May 2009, pp. 3258–3264.

[12] I. Maza, K. Kondak, M. Bernard, and A. Ollero, “Multi-UAV cooper-

ation and control for load transportation and deployment,” J. Intell.

Robot. Syst., vol. 57, no. 1, pp. 417–449, Jan. 2010.

[13] P. Pounds, D. Bersak, and A. Dollar, “Grasping from the air: Hover-

ing capture and load stability,” in Proc. IEEE Int. Conf. Robotics and Auto-

mation, Shanghai, China, May 2011, pp. 2491–2498.

[14] D. Mellinger, Q. Lindsey, M. Shomin, and V. Kumar, “Design, mod-

elling, estimation and control for aerial grasping and manipulation,” in

Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, San Francisco,

CA, Sept. 2011, pp. 2668–2673.

[15] A. Albert, S. Trautmann, T. Howard, T. Nguyen, M. Frietsch, and C.

Sauter, “Semi-autonomous flying robot for physical interaction with envi-

ronment,” in Proc. IEEE Conf. Robotics Automation and Mechatronics,

Singapore, June 2010, pp. 441–446.

[16] Q. Lindsey, D. Mellinger, and V. Kumar, “Construction of cubic

structures with quadrotor teams,” in Proc. Robot. Sci. Syst., June 2011,

p. 1.

[17] C. Korpela, T. Danko, and P. Oh, “Designing a system for mobile

manipulation from an unmaned aerial vehicle,” in Proc. IEEE Conf. Tech-

nologies for Practical Robot Applications, Apr. 2011, pp. 109–114.

[18] E. Ackerman. (2011, Aug.). Matternet wants to deliver meds with a

network of quadrotors [Online]. Available: http://spectrum.ieee.org/

automaton/robotics/medical-robots/mini-uavs-could-be-the-cheapest-

way-to-deliver-medicine

[19] I. Palunko and R. Fierro, “Adaptive feedback controller design and

quadrotor modeling with dynamic changes of the center of gravity,” in

Proc. 18th IFAC World Congr., Milan, Italy, Aug. 28–Sept. 2, 2011,

pp. 2626–2631.

[20] I. Palunko, R. Fierro, and P. Cruz, “Trajectory generation for swing-

free maneuvers of a quadrotor with suspended payload: A dynamic

programming approach,” in Proc. IEEE Int. Conf. Robotics and Automa-

tion, St. Paul, MN, May 14-18, 2012, pp. 2691–2697.

[21] R. Mahony, V. Kumar, and P. Corke, “Modeling, estimation and

control of quadrotor aerial vehicles,” IEEE Robot. Automat. Mag., vol. 19,

no. 3, pp. 20–32, 2012.

[22] S. Sagatun and T. Fossen, “Lagrangian formulation of underwater

vehicles’ dynamics,” in Proc. Conf. Systems, Man, and Cybernetics, Char-

lottesville, VA, Oct. 1991, vol. 2, pp. 1029–1034.

[23] L. S. Cicolani and G. Kanning, “Equations of motion of slung-load

systems, including multilift systems,” NASA, Office of Management

Scientific and Technical Information Program, Ames Research Center,

Moffett Field, CA, NASA Technical Paper, 3280, 1992.

[24] D. Fusato, G. Guglieri, and R. Celi, “Flight dynamics of an articulated

rotor helicopter with an external slung load,” J. Amer. Helicopter Soc.,

vol. 46, no. 1, pp. 3–14, Jan. 2001.

[25] M. Bisgaard, J. Bendtsen, and A. la Cour-Harbo, “Modelling of a

generic slung load system,” in Proc. AIAAModeling and Simulation Tech-

nologies Conf. and Exhibit, 2006.

[26] I. Vasilescu, C. Detweiler, M. Doniec, D. Gurdan, S. Sosnowski, J.

Stumpf, and D. Rus, “A hovering energy efficient underwater robot capa-

ble of dynamic payloads,” Int. J. Robot. Res., vol. 29, no. 5, pp. 547–570,

Apr. 2010.

[27] M. Moll, P. Will, M. Krivokon, and W.-M. Shen, “Distributed con-

trol of the center of mass of a modular robot,” in Proc. IEEE/RSJ Int. Conf.

Intelligent Robots and Systems, Beijing, China, Oct. 2006, pp. 4710–4715.

[28] J. Slotine and W. Li, Applied Nonlinear Control. Englewood Cliffs,

NJ: Prentice-Hall, 1991.

[29] G. Starr, J. Wood, and R. Lumia, “Rapid transport of suspended

payloads,” in Proc. IEEE Int. Conf. Robotics and Automation, Apr. 2005,

pp. 1394–1399.

[30] D. Zameroski, G. Starr, J. Wood, and R. Lumia, “Rapid swing-free

transport of nonlinear payloads using dynamic programming,” ASME J.

Dyn. Syst. Meas. Control, vol. 130, no. 4, pp. 041001-1–041001-11, July

2008.

[31] (2012, June). Marhes [Online]. Available: http://marhes.ece.unm.

edu/index.php/Ipalunko:Home.

[32] Ascending Technologies GmbH. (2012). [Online]. Available: http://

www.asctec.de/

[33] Vicon Motion Systems Limited. (2012). [Online]. Available: http://

www.vicon.com/

[34] NI CompactRIO [Online]. Available: http://www.ni.com/compactrio/

Ivana Palunko, Department of Electrical and ComputerEngineering, University of New Mexico, Albuquerque,NM, USA. E-mail: [email protected].

Patricio Cruz, Department of Electrical and ComputerEngineering, University of New Mexico, Albuquerque,NM, USA. E-mail: [email protected].

Rafael Fierro, Department of Electrical and ComputerEngineering, University of New Mexico, Albuquerque,NM, USA. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___

_________________

________________

_____________

_______________

_____________________________________

_____________

___

http://www.qmags.com/clickthrough.asp?url=http://spectrum.ieee.org/automaton/robotics/medical-robots/mini-uavs-could-be-the-cheapest-way-to-deliver-medicine&id=17487&adid=P79E2

http://www.qmags.com/clickthrough.asp?url=http://www.ni.com/compactrio/&id=17487&adid=P79E4

http://www.qmags.com/clickthrough.asp?url=http://www.vicon.com/&id=17487&adid=P79E5

http://www.qmags.com/clickthrough.asp?url=http://www.asctec.de/&id=17487&adid=P79E3

http://www.qmags.com/clickthrough.asp?url=http://marhes.ece.unm.edu/index.php/Ipalunko:Home&id=17487&adid=P79E1








Point Cloud LibraryThree-Dimensional Object Recognition and 6DoF Pose Estimation

By Aitor Aldoma, Zoltan-Csaba Marton, Federico Tombari, Walter Wohlkinger, Christian Potthast,

Bernhard Zeisl, Radu Bogdan Rusu, Suat Gedikli, and Markus Vincze

With the advent of new-generation depth sen-sors, the use of three-dimensional (3-D) datais becoming increasingly popular. As thesesensors are commodity hardware and sold at

low cost, a rapidly growing group of people can acquire 3-D data cheaply and in real time.

With the massively increased usage of 3-D data for per-ception tasks, it is desirable that powerful processing toolsand algorithms as well as standards are available for thegrowing community. The Point Cloud Library (PCL) [1]aims at providing exactly these. It is a collection of state-of-the-art algorithms and tools to process 3-D data. Thelibrary is open source and licensed under Berkeley Soft-ware Distribution (BSD) terms and, therefore, free to usefor everyone. The PCL project brings together researchers,universities, companies, and individuals from all aroundthe world, and it is rapidly becoming a reference for any-one interested in 3-D processing, computer vision, and

robotic perception. The PCL core is structured in smallerlibraries offering algorithms and tools for specific areasof 3-D processing, which can be combined to efficientlysolve common problems such as object recognition,registration of point clouds, segmentation, and surfacereconstruction, without the need of reimplementing allparts of a system needed to solve these subtasks. In otherwords, the tools and algorithms provided by PCL allowresearchers and companies to better concentrate on theirspecific areas of expertise.

In this article, we focus on 3-D object recognition andpose estimation. Specifically, our goal is to recognize rigidobjects from a single viewpoint and estimate their positionand orientation in the real world. The objects used to trainthe system are represented as 3-D meshes, and the realobjects are sensed using a depth sensor such as the Kinect.In particular, we review several state-of-the-art 3-D shapedescriptors aimed at object recognition, which are includedin PCL and are publicly available.

As a good descriptor in the end is a small part of anobject recognition system, we present two entire pipelines




©DIG

ITALVISIO

N


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






for our recognition system built using bits and pieces avail-able in PCL: one relies on global descriptors that requirethe notion of objects and hence deploy a specific prepro-cessing step based on segmentation while the other oneuses local descriptors that are computed locally around keypoints and, thus, do not require a presegmentation step.Throughout the article, the advantages and disadvantagesof the different methods used are presented to providethe reader with the necessary insights for designing thefeatures of a recognition system to be used in a specificfront-end application. The tutorial also provides specificguidelines on how to use PCL primitives for the variousmethods presented.

Finally, an experimental evaluation is performed todemonstrate the applicability and the effectiveness of thepresented methods for the task at hand. Moreover, bothtraining and ground truth test data sets are publicly avail-able together with a set of tools to perform evaluations onthese data sets.

Local DescriptorsGenerally, 3-D local descriptors are developed for specificapplications such as registration, object recognition, andlocal surface categorization. For such applications, eachpoint is associated with a descriptor describing the localgeometry of a point.

Signature of Histograms of OrientationsThe signature of histograms of orientation (SHOT)descriptor [2] encodes a signature of histograms repre-senting topological traits, making it invariant to rotationand translation and robust to noise and clutter. Thedescriptor for a given key point is formed by computinglocal histograms incorporating geometric information ofpoint locations within a spherical support structure. Foreach spherical grid sector, a one-dimensional histogramis constructed by accumulating point counts of theangle between the normal of the key point and the normalof each point belonging to the spherical support struc-ture. The final descriptor is formed by orderly juxta-posing all histograms together according to the localreference frame.

Discrete quantization of the sphere introduces aboundary affect when used in combination with histo-grams, resulting in abrupt changes from one histogrambin to another. Therefore, quadrilinear interpolation isapplied to each accumulated element, resulting in anevenly distribution into adjacent histogram bins.Finally, for better robustness toward point-density var-iations, the descriptor is L1-normalized. The dimen-sionality of the used signature is 352. Algorithm 1 givessome guidelines on how to compute SHOT descriptorsusing PCL. In this algorithm, first, an object of the classpcl::SHOTEstimation is created using the templateparameters of our input data (cloud and normals). A kd-tree is provided to the estimator to perform NN

searches. Indices represent the key points from the inputcloud where SHOT descriptors should be computed.The radius search represents the size around each keypoint that will be described. Finally, calling the computemethod returns a point cloud with as many descriptorsas key points.

Fast Point Feature HistogramDescriptors such as point feature histogram (PFH) [3], fastPFH (FPFH) [4], and viewpoint feature histogram (VFH)[5] can be categorized as geometry-based descriptors.These descriptors represent the relative orientation of nor-mals, as well as distances, between point pairs. Point pairsare generated between a point with coordinates pi and thepoints in its local neighborhood with coordinates pj. Thesepoint pairs are represented with the angles a,/, and h,computed in a reproducible fixed coordinate frame. Thecoordinate frame is constructed using the surface normalni of pi, the vector (pi � pj)=(kpi � pjk2) and the crossproduct of these two vectors:

u ¼ ni, (1)

v ¼ u3pi � pjpi � pj

2

, (2)

w ¼ u3 v: (3)

With these, we can calculate the angles a,/, and h asfollows:

a ¼ vT � ni, (4)

/ ¼ uT � pi � pjpi � pj

2

, (5)

h ¼ arctan (wT � ni, uT � ni): (6)

In the case of PFH, the normal orientation angles are com-puted for all point pairs of p and its neighborhood. Toform the descriptor, the estimated values are binned into ahistogram of size 33, representing the divisions of thefeature space. The FPFH descriptor can be computedusing PCL using Algorithm 1 and simply replacing thepcl::SHOTEstimation instantiation with a pcl:FPFHEsti-mation instantiation.

•Algorithm 1. Computing SHOT descriptorsin PCL.

pcl::SHOTEstimation<...>shot;shot.setSearchMethod(tree);//kdtreeshot.setIndices(indices);//keypointsshot.setInputCloud(cloud);//inputshot.setInputNormals(normals);//normalsshot.setRadiusSearch(0.06);//supportshot.compute (*shots);//descriptors



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






3-D Shape ContextThe 3-D shape context (SC) descriptor [6] is an extensionof the 2-D SC descriptor [7] into 3-D. The descriptor for agiven key point is computed utilizing a 3-D spherical grid,centered on the key point, and superimposed over allpoints in the point cloud. Within the support structure, aset of sectors is defined by means of subdivisions along theazimuth, elevation, and radial dimensions, equally spacedfor the first two dimensions and logarithmically spaced forthe radial dimension. The final descriptor is built from thesubdivision of the grid structure and is represented in binsof a 3-D histogram. Each bin of the histogram is associatedwith a value and computed as the weighted sum of thenumber of points falling in the corresponding grid sector.Additionally, the computed weight is inversely propor-tional to the local point density and the bin volume.

The north pole of the 3-D grid is aligned with the normalcomputed on the key point being described, while no princi-pal direction is defined over the normal plane. This remain-ing degree of freedom is dealt with by taking into account asmany orientations of the grid as the number of azimuth sub-divisions, leading to several local reference frames and, inturn, to multiple descriptions for the same key point to dealwith rotations and translations of the point cloud. Algo-rithm 2 shows how to compute the 3-D SC descriptor usingPCL. Again, observe the similarity between this algorithmand the one presented in Algorithm 1, requiring the user toset an additional two parameters.

Unique SCThe unique SC (USC) [8] descriptor is an extension of the3-D SC descriptor [6]. It defines a local reference frame foreach key point so as to provide two principal directionsover the normal plane that uniquely orientate the 3-D gridassociated with each descriptor. This reduces the requiredmemory footprint and possible ambiguities during the suc-cessive matching and classification stage because it avoidsmultiple descriptions for the same key point. In particular,the local reference frame is defined as in [2]. Once the 3-Dspherical grid is uniquely oriented according to the localreference frame, the 3-D histogram yielding the finaldescriptor is built following the approach proposed in [6].To compute USC descriptor using PCL, a variant ofAlgorithm 2 can be used, obtained by replacing pcl::Shape-Context3DEstimation for pcl::UniqueShapeContext.

Radius-Based Surface DescriptorRadius-based surface descriptor (RSD) describes thegeometry of points in a local neighborhood by estimatingtheir radial relationships. The radius can be estimated byassuming each point pair to lie on a sphere and finding itsradius by exploiting that the relation between the distanceof the points d and the angle between the two point nor-mals a is described by

d(a) ¼ ffiffiffiffiffi2r

p ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi1� cos (a))

p� raþ ra3=24þ O(a5): (7)

This relation turns out to be nearly linear fora 2 (0, p=2) with the radius r (in meters) as the scalingfactor. To accurately estimate the minimum and maxi-mum radii r in a neighborhood, linear regression isapplied on the minimum and maximum (a, d) pairs. Theradius takes its extreme value, which is infinity, for a pla-nar surface and lower values for surfaces with a highercurvature. As an example, using the minimum andmaximum radius of a neighborhood allows to distin-guish between spheres and cylinders. For a cylinder, theminimum radius will have a value close to the actualcylinder radius whereas the maximum value will be infi-nite. A detailed description and analysis of the methodcan be found in [9].

In our experiments, we use the local feature for match-ing; to do so, we use the complete histogram as the featurewith 289 values. However, roughly half are nonzero due tothe limited variability of nearby surface normals that areestimated by the current method in PCL. The RSD descrip-tor can be computed using PCL using Algorithm 1 andsimply replacing the pcl::SHOTEstimation instantiationfor a pcl::RSDEstimation instantiation.

Spin ImagesSpin images (SIs) [10] can be used for point clouds withnormal information available for every point. Theapproach operates on a rejection strategy, computing thedifference of the point normal and the source normal nand considers only points with the difference smaller thana specific threshold. Furthermore, two distances are com-puted for each of the points: the distance from n and thedistance from the source point along n. As a final step, ahistogram is formed by counting the occurrences of differ-ent discretized distance pairs. In our implementation, weused a 153-dimensional descriptor. The SI descriptor canbe computed in PCL using Algorithm 1 and simply re-placing the pcl::SHOTEstimation instantiation for pcl::SpinImageEstimation.

Global DescriptorsThe global object descriptors are high-dimensional repre-sentations of object geometry and were engineered for thepurpose of object recognition, geometric categorization,and shape retrieval. They are only usually calculated forsubsets of the point clouds that are likely to be objects.

•Algorithm 2. Computing 3-D SC descriptors in PCL.

pcl::ShapeContext3DEstimation<...>dsc;dsc.setSearchMethod(tree);dsc.setIndices(indices);dsc.setInputCloud(in);dsc.setInputNormals(normals);dsc.setRadiusSearch(radius_);dsc.setMinimalRadius(radius_/10.f);dsc.setPointDensityRadius(radius_/5.0);dsc.compute(*dscs);



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






These object candidates are obtained through a precedingsegmentation step in the processing pipeline.

Point Feature HistogramAs explained earlier, PFH [3] is constructed by calculatingthe angles a,/, and h for all the point pairs in a neighbor-hood and binning them into a histogram. Generally,the number of bins required for a particular divisionof the features space into div divisions for eachdimension is divD, where D is the dimensionality of thefeature space.

In the original article, in addition to the normal angles,the distance between point pairs was used, which turnedout not to be a discriminative feature. The PCL implemen-tation does not use this feature; therefore, the requirednumber of bins only grows as a power of three. As a conse-quence, it becomes computationally feasible to divide thedimensions into five divisions, which leads to a 125-dimensional feature vector.

This feature space captures the correlations between thenormal angles of point pairs in a neighborhood and wasused both as a local and a global feature, and various otherfeatures are based on it. Here we use it as a global featureby considering every point pair in a cluster for building thehistogram. Algorithm 3 shows how to compute a PFHdescriptor using PCL. To obtain a global description of theinput cloud, setRadiusSearch needs to be set according tothe maximum distance between any two points on theinput cloud, and indices contain only a single value in therange of the number of points.

Viewpoint Feature HistogramThe viewpoint feature histogram (VFH) introduced byRusu et al. [5] is related to the FPFH feature. Here, theangles a,/, and h are computed based on a point’s normaland the normal of the point cloud’s centroid pc. Theviewpoint-dependent component of the descriptor isa histogram of the angles between the vector(pc � pv)=(kpc � pvk2) and each point’s normal. It makesthe histogram useful for joint object recognition and poseestimation, the purpose for which it was originally devel-oped. The other component is a simplified point featurehistogram (SPFH) estimated for the centroid of the pointcloud and an additional histogram of the distances of thepoints in the cloud to the cloud’s centroid. In the PCL

implementation, each of those four histograms have45 bins and the viewpoint-dependent component has128 bins, totaling 308 bins. Algorithm 4 shows how tocompute a VFH descriptor using PCL. Because of theglobal nature of VFH, no key points need to be provided,and the output of the estimator will only be a singledescriptor encoding the geometry of the whole inputcloud and its viewpoint.

Clustered Viewpoint Feature HistogramThe clustered viewpoint feature histogram (CVFH)presented in [11] is an extension to VFH, based on the ideathat objects have a certain structure that allows to splitthem in a certain number N of disjoint smooth regions.Each of these smooth regions is then used independentlyto compute a set of N VFH histograms. For instance, thekth VFH histogram uses the centroid pck and normal aver-age nck computed by averaging points and normals belong-ing to region k to build the coordinate frame from whichthe normal distributions of the object are computed. Thesmooth regions for CVFH are easily computed by 1)removing points on the object with high curvatures, indi-cating noise or borders between smooth regions and 2)performing region growing on the remaining points onxyz and normal space. CVFH outputs a multivariate repre-sentation of the object of interest and the histogram k isadditive as long as pck and nck remain unchanged. Thisallows the feature to be more robust to segmentation arti-facts, locally noisy surfaces, and occlusions as long as atleast one of the k regions is still visible (see Figure 1). Theauthors proposed to make the feature scale dependent byavoiding the normalization over the total amount of points

(b)(a)

Figure 1. Centroids and normal averages for three regions of abook. (a) Point cloud obtained from the Kinect. (b) Similarviewpoint of the book used to train the recognition system.

•Algorithm 3. Computing a global PFH descriptorin PCL on an input cloud.

pcl::PFHEstimation<...>pfh;pfh.setSearchMethod(tree);pfh.setInputCloud(input);pfh.setInputNormals(normals);pfh.setRadiusSearch(cloud_radius);pfh.setIndices(indices);pfh.compute(*pfh_signature);

•Algorithm 4. Computing a VFH descriptor in PCLon an input cloud.

pcl::VFHEstimation<...>vfh;vfh.setSearchMethod(tree);vfh.setInputCloud(input);vfh.setInputNormals(normals);vfh.compute (*vfh_signature);



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






on the object, allowing CVFH to differentiate betweenequally shaped objects from different sizes and, underocclusions, the shape of histograms only changes locally.The drawback is that the point clouds used for trainingand recognition need to share a common resolution so thatthe number of points describe the size of the object. From acomputational point of view, the bottleneck to computeCVFH is the region-growing step with a cost n log (n),where n is the amount of points of the object to bedescribed. Algorithm 5 shows how to compute the CVFHdescriptors using PCL. Because of the semiglobal nature ofCVFH, the number of resulting descriptors might be morethan one depending on the geometry of the cloud to beencoded.

Ensemble of Shape FunctionsThe ensemble of shape function (ESF) descriptor intro-duced in [12] is an ensemble of ten 64-bin-sized histo-grams of shape functions describing characteristicproperties of the point cloud. The shape functions consistof angle, point distance, and area shape functions. A voxelgrid serves as an approximation of the real surface and isused to separate the shape functions into more descriptivehistograms representing point distances, angles, and areas,either on the surface, off the surface, or both.

As shown in Figure 2, the ESF descriptor can be effi-ciently calculated directly from the point cloud with nopreprocessing necessary, such as smoothing, hole filling, orsurface normal calculation, and handles data errors such asoutliers, holes, noise, and coarse object boundaries grace-fully. Algorithm 6 shows how to compute an ESF descrip-tor using PCL. The ESF algorithms differ from the otherfeature algorithms as it does not require the use of normalsto describe the cloud.

Global RSDGlobal RSD (GRSD) is the global version of the RSD localfeature, which describes the transitions between differentsurface types (and free space) for an object, constructed byusing the estimated principle radii for labeling the localsurface types. As described in [9], it is based on a voxeliza-tion of the input point cloud, and it differs from the globalFPFH (GFPFH) feature by eliminating the need for a com-plicated local surface classifier through the use of RSD asdescribed earlier. Additionally, RSD is computed only onceper voxel cell, drastically reducing the number of neighbor-hoods that need to be analyzed.

For this work, we counted the transitions between surfacetypes around occupied voxels instead of along lines betweenoccupied voxels. Thismodified version, noted as GRSD,makesthe computation of the transition histogram linear in the num-ber of cells. Together with the computation of RSD for eachcell, the computation is linear in the number of points.

As the feature is independent of resolution, the normal-ization of the histogram can be avoided, making it depend-ent on scale and additive. By using five local surface typesand considering free space, the number of used GRSD his-togram bins is 20.

Recognition PipelinesIn this section, we will present how 3-D shape descriptorscan be used in combination with other components tobuild a full recognition system, outputting the objectspresent in a range image together with their position andorientation (see Figure 3). Without loss of generality, thissection follows our specific scenario in which the represen-tation of world objects is given by 3-D meshes or CADmodels. However, it also applies to other scenarios, as thetraining stage of a system is the only affected part in thepipelines. First, we present common components of boththe global and local descriptors-based pipelines and thenmove to specific parts of each processing pipeline.

TrainingTo train the system, the meshes representing our objects aretransformed to partial point clouds, simulating the input thatwill be given by a depth sensor. For this purpose, a virtualcamera is uniformly placed around the object on a boundingsphere with a radius large enough to enclose the object. Toobtain uniform sampling around the object, the sphere is gen-erated using an icosahedron as starting shape and subdividingeach triangular face with four equilateral triangles. This isdone recursively for each face until the desired level of

•Algorithm 5. Computing CVFH descriptors in PCLfor an input cloud.

pcl::CVFHEstimation<...>cvfh;cvfh.setSearchMethod(tree);cvfh.setInputCloud(input);cvfh.setInputNormals(normals);cvfh.setEPSAngleThreshold(angle);cvfh.setCurvatureThreshold(max_curv);cvfh.setNormalizeBins(false);cvfh.compute(*cvfh_signatures);

•Algorithm 6. Computing an ESF descriptor in PCLon an input cloud.

pcl::ESFEstimation<...>esf;esf.setInputCloud(input);esf.compute (*esf_signature);

Figure 2. Calculation of the shape functions on an examplepoint cloud of a mug. Point distances classified into on thesurface, off surface, and both.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






recursion is reached. The level of recursion indicates howmany triangles will shape the approximated sphere. Theresulting triangles are used to place the camera at their bari-center, and a partial view of the mesh is obtained by samplingthe depth buffer of the graphic card. There are two mainparameters that govern this process: 1) the level of recursionand 2) the resolution of the synthetic depth images. Accord-ing to our experiments, one level of recursion is enough tosample the view space (80 views are generated), and a resolu-tion of 150 3 150 gives an appropriate level of detail over theobject. This functionality is part of the visualization library ofPCL. After the views are generated for each model, thedesired 3-D feature is computed on the partial views and thedescriptors are stored for future use. Moreover, a transforma-tion between model and view coordinates is also saved. Theinverse of such a transformation allows to transform the viewto model coordinates, and, therefore, the full 3-D model canbe obtained by applying the transformation for each view andfusing the transformed views together. Finally, in the case ofglobal 3-D features, a camera roll histogram [11] to retrieve afull 6-DoF pose is also computed for each view and stored, asdetailed in “Correspondence Grouping” section. Figure 4shows the block diagrams for the local and global 3-D pipe-lines, respectively, which can be found in PCL and were usedfor the experimental results presented in this tutorial.

Recognition Pipeline for Local Descriptors

Key Point ExtractionThe first step for every 3-D perception pipeline based onlocal descriptors is represented by the extraction of 3-Dkey points from data. The topic of 3-D key point detectionis new, and recently, there have been several proposalsin literature addressing this problem [14]. The maincharacteristics of a key point detector are repeatabilityand distinctiveness (other than computational efficiency).The former property deals with the capability of a detectorto accurately extract the same key points under a variety ofnuisances, while the latter is the ability to extract keypoints that can be easily described, matched, and/or classi-fied, highly characterizing a surface or a specific object.

Importantly, the distinctiveness of a key point detectordepends on the subsequent description stage: a set of key

points can be more or less salient depending on the traits ofthe local descriptors applied on them. For this reason, toconduct a fair comparison among local descriptors, we havedecided not to adopt any particular technique; instead, weuse a uniform key point sampling over the point cloud, anal-ogously to what was proposed in the 3-D object recognitionpipeline of [10]. Specifically, in our experiments, key pointsare sampled using 3-D voxelization with a width of 15 mm.

Description and MatchingOnce key points are extracted, they are associated to a localdescription. A common trait of all local descriptors is thedefinition of a local support used to determine the subsetof neighboring points around each key point that will beused to compute its description. For fairness, the supportsize is set, for all methods, to the same value (6 cm in ourexperiments—see Figure 5. As for the other parameters

(a)

(b)

Figure 3. (a) Point cloud obtained from Kinect, colors from red toblue representing increasing depth and (b) recognition resultsgiven by the local recognition pipeline. The recognition hypotheseshave been postprocessed using the hypotheses verification stageproposed in [13]. The corresponding 3-D meshes of the recognizedobjects are overlaid as wireframes after being transformed by the6-DoF pose given by the recognition processing pipeline.

LocalPipeline

pcl::UniformSampling

pcl::FPFHEstimationpcl::SHOTEstimation

...

pcl::Searchpcl::Correspondence

Groupingpcl::SampleConsensusModelRegistration

pcl::IterativeClosestPoint

pcl::HypothesisVerification

pcl::CameraRollHistogram

pcl::Searchpcl::CVFHEstimation

...pcl::apps::DominantPlaneSegmentation

GlobalPipeline

Segmentation Description Matching Alignment

Key Point Extraction Description Matching

CorrespondenceGrouping

AbsoluteOrientation

ICPRefinement

HypothesisVerification

Figure 4. Block diagrams for the PCL local and global 3-D pipelines.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






used for each specific technique, we have resorted to thoseproposed by the authors of each method.

Once descriptors are computed for the current scene andeach model in the library, they need to be matched to yieldpoint-to-point correspondences. To handle the case of multi-ple instances of the same model in each scene, each scenedescriptor is matched against the descriptors of all models inour database (and not vice versa). More specifically, descrip-tors are compared using the Euclidean distance, and corre-spondences are set between each scene descriptor and itsnearest neighbor (NN) in the model database. A matchingthreshold is commonly used to discard correspondences pair-ing descriptors that lie far apart in the descriptor space. Incontrast, we do not prune any correspondence in our pipeline(equivalent to using an infinite value for the threshold), leav-ing the task of selecting consistent model hypotheses to thefollowing correspondence grouping stage. This has the bene-ficial effect of eliminating one sensible parameter from thepipeline. Finally, efficient approximated matching schemesare deployed to speed up the matching stage: in particular,the fast approximate NN (FLANN) algorithmwas used [15].

Correspondence GroupingAs a result of the matching stage, point-to-point correspond-ences are determined by associating pairs of model-scenedescriptors that lie close in the descriptor space. A relativelycommon approach within 3-D object recognition methods isrepresented by an additional stage, usually referred to ascorrespondence grouping, where correspondences are dis-carded by enforcing geometrical consistency between them.More specifically, by assuming that the transformationbetween the model and its instance in the current scene isrigid, the set of correspondences related to each model is

grouped into subsets, each one holding the consensus for aspecific rotation and translation of that model in the scene.Subsets whose consensus is too small are discarded, simplyby thresholding based on their cardinality.

A few methods were proposed in literature for this task(see [10], [16], and [17] and citations therein). In thiscomparison, we have used an approach similar to thatproposed in [18], where an iterative algorithm based on sim-ple geometric consistency of pairs of correspondences wasproposed. In our approach, all correspondences are clusteredinto subsets geometrically consistent, i.e., starting from a seedcorrespondence ci ¼ fpmi , psig (pmi and psi being, respectively,the model and scene 3-D key points associated with ci) andlooping over all correspondences that were not yet grouped,the correspondence cj ¼ fpmj , psjg is added to the groupseeded by ci if the following relation holds:

pmi � pmj

2� psi � psj

2

�� < e, (8)

with e being a parameter of this method, intuitively repre-senting the consensus set dimension. After parameter tun-ing, we set e to 8 mm in our experiments.

As previously mentioned, a threshold needs to bedeployed to discard correspondence subsets supported bya small consensus: a minimum of three correspondences isneeded to compute the 6-DoF pose transformation. As thisthreshold trade-offs the number of correct recognitions(true positives) for the number of wrong recognition [falsepositives (FPs)], we have used it to span the reported ROCcurves concerning the local pipeline.

Random Sample Consensus and Absolute OrientationThe previous geometric validation stage, although gener-ally effective in discarding a high number of geometricallyinconsistent correspondences, does not guarantee that allcorrespondences left in each cluster are consistent with aunique 6-DoF pose, i.e., with a unique 3-D rigid rotationand translation of the model over the scene. For this rea-son, it is useful to include an additional step based on a M-estimator, such as random sample consensus (RANSAC)whose aim is to additionally reduce the number of left cor-respondences in each cluster by eliminating those notconsistent with the same 6-DoF pose. The model on whichthe consensus of the estimator is iteratively built is gener-ally obtained by means of an absolute orientation algo-rithm. Given a set of exact correspondences between twopoint clouds, it determines the 3 3 3 rotation matrix (~R)and the 3-D translation vector (~T), which define the rigidtransformation that best explains them. Because no out-liers are assumed in the correspondence set, this is usuallysolved via least square minimization: given a set of N exactcorrespondences c1, . . . , cN , ~R and ~T are obtained as

argminR,T

XNi¼1

psi � R � pmi � T 2

2: (9)

1.0

0.8

0.6

0.4

0.2

0.00.0 0.2 0.4 0.6

1-Specificity

Sen

sitiv

ity

0.8 1.0

SHOT-0.06SHOT-0.04SHOT-0.08

Figure 5. Recognition performance (without pose) usingdifferent support sizes (4, 6, and 8 cm) for the SHOT descriptorevaluated on the test data set without occlusions. From thefigure, a support size of 4 cm does not seem to hold enoughdescriptiveness, while 6 and 8 cm sizes deliver similar results.Therefore, a choice of 6 cm seems reasonable to better dealwith occlusions and clutter without affecting descriptiveness.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Between the several absolute orientation algorithms inliterature [19], we used the one proposed in [20], wherethe rotational component of the transformation is repre-sented as a unit quaternion, given its effectiveness demon-strated in the comparison presented in [19].

Recognition Pipeline for Global Descriptors

SegmentationBecause global features require the notion of object, thefirst step in the recognition pipeline involves a segmenta-tion of the scene to extract the objects in it. Segmentationin itself is a well-researched topic and several techniquesexist [21]–[24]. In this case, we chose a simple buteffective method based on the extraction of a dominantscene plane and a Euclidean clustering step (similar toflood filling) on the remaining points after dominantplane removal. The clustering step is guided by a thresholdt, which indicates how close two points are required to beto belong to the same object. Therefore, for a successfulsegmentation, the method requires different objects to beat least t far away from each other. The method has astrong prior because of its assumption of a dominantplane like a table or a floor. Nevertheless, such an assump-tion is commonly used in robotic scenarios wherethe structure of human-made environment is exploited.PCL provides several components to perform such a taskand the details of this segmentation method can be foundin [25].

Description and MatchingThe output of segmentation represents the potentialobjects in the scene. The shape and geometry of each ofthese objects is described by means of a proper globaldescriptor and represented by a single histogram (exceptfor CVFH where multiple histograms might be obtained aspresented in “Clustered Viewpoint Feature Histogram”section). In all cases, the histogram(s) are independentlycompared against those obtained in the training stage, andthe best N NNs are retrieved. The N NNs represent the Nmost similar views in the training set according to thedescription obtained by means of the desired global featureand the metric used to compare the histograms. N > 1neighbors instead of the single NN are usually retrieved, asthe results can be greatly improved through pose estima-tion and postprocessing. Nevertheless, a descriptive anddistinctive feature helps to efficiently prune lots of unfeasi-ble candidates.

The histogram matching is done using Fast Libraryfor Approximate Nearest Neighbors (FLANN) bymeans of a brute force search, and the distance betweenhistograms is computed using the L1 metric (exceptfor CVFH where the metric proposed in [11] is used) asit is more robust than L2 when dealing with outliers.Brute force might be used in this case without affectingglobal performance as the number of histograms in

the training set is much smaller than in the case oflocal descriptors.

Camera Roll Histogram and 6-DoF PoseThe partial view candidates obtained by matching anobject cluster to the training set can be aligned using theirrespective centroids obtaining a 5-DoF pose. A last degreeof freedom involving a rotation about the roll axis ofthe camera needs to be computed for a full 6-DoF pose.Global features are usually invariant to this rotation due tothe fact that rotations about the roll axis of the camerado not result in changes of the geometry of the visiblepart of the object (assuming no occlusions). Therefore,we adopt the approach presented in [11], where the rollrotation is obtained by means of the camera roll histogramand an optimization step that efficiently aligns twohistograms. The rotation obtained can be then applied tothe 5-DoF pose to obtain a full 6-DoF pose. This step isapplied for each of the N candidates obtained from theprevious step.

PostprocessingBoth the local and the global pipelines can undergo anoptional postprocessing stage to improve the outcome ofthe recognition. One first step within this stage is usuallyrepresented by applying iterative closest point (ICP) to therecognition hypotheses, with the aim of refining theestimated 6-DoF pose. A following step, usually referredin literature as hypotheses verification, aims at reducingthe number of FPs while retaining correct recognitions.This is generally carried out by leveraging on geometricalcues that can be efficiently computed once the modelhypotheses have been aligned to the scene. Typical cuesare the percentage of supporting points (i.e., modelpoints that are close to scene points), as well as the percent-age of outliers (number of visible points belonging tothe models that do not have a counterpart within the scenepoints). Currently, PCL contains an implementation ofthe hypothesis verification algorithm proposed in [13].Figure 3 shows an example where the recognitionhypotheses are postprocessed using this method. Otherverification strategies have been proposed in the literature[17], [26].

Another hypothesis verification method especially con-ceived for global pipelines [11] is based on reordering theN candidates list using a metric based on the number ofinliers between the object cluster in the scene and thealigned training views. The number of inliers can be effi-ciently computed by counting how many points in theobject cluster have a neighbor on the candidate viewswithin a certain distance that we set to 5 mm. From Fig-ure 6, it is clear that postprocessing and reordering thecandidates help to obtain straighter recognition accumu-lated rate graphs. The higher the difference between firstrank and nth rank, more the advantage a feature can obtainfrom such a postprocessing.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






Evaluation of CAD Recognition and6-DoF Pose Estimation

Training Data SetThe database was constructed using physical objects thatare generally available from major retailers. The objectscan be divided into three categories: for the first two cate-gories, all objects were obtained from a single retailer(IKEA and Target, respectively) while the third categorycontained a set of household objects commonly availablein most retail stores.

Test Data SetFor the experimental evaluation, we have collected severalscenes and labeled them with ground truth models and 6-DoF pose. The data set has been split into two differentsets: one where the objects in the scene can be segmented

using the aforementioned segmentation method and with-out occlusions and the second set where the objects mightbe touching each other and present different degrees ofocclusions. The sets contain 31 and 23 scenes, respectively.The full 3-D model meshes, the scene point cloudsobtained from the Kinect, and the ground truth annota-tions can be downloaded from http://users.acin.tuwien.a-c.at/aaldoma/datasets/RAM.zip .

ExperimentsTo better demonstrate the practical applicability of thepresented features and pipelines, an evaluation has been per-formed on the aforementioned data sets. Specifically, we havecarried out several experiments regarding recognition andpose estimation. Due to the segmentation step required byglobal features, they were evaluated only on the first data setwhile local features were evaluated on both data sets. Figure 7presents a subset of the objects that were used in the data set.

Figures 8 and 9 show the ROC curves concerning thecomparison among local features. From these results, itcan be observed that the SHOT and USC features consis-tently outperform other approaches on both data sets.Three-dimensional SC and FPFH performs as well reason-ably good although FPFH performance tends to sink underthe presence of occlusions and nearby objects, as it can beclearly observed in Figure 9.

It is also worth noting that, from a practical point ofview, the dimensionality of the descriptor also plays animportant role. For instance, USC standard size of 1,980required about 6 GB of memory when the whole trainingdescriptors were loaded. For larger data sets, this impliesthe need of out-of-core solutions to perform the feature-matching stage, this having an impact also on the efficiencyof the recognition pipeline. FPFH presents an importantadvantage regarding this factor due to its low dimensional-ity, being just made of 33 bins. The SHOT descriptor, with

1.0

0.8

0.6

0.4

0.2

0.02 4 6 8 10

Rank

Acc

umul

ated

Rec

ogni

tion

Rat

e

12 14

CVFH-POSTCVFHVFHVFH-POST

CVFH-POSTCVFHVFHVFH-POST

Figure 6. Accumulated recognition rate for VFH and CVFH bothwith and without postprocessing.

Figure 7. A subset of the objects used in the recognitionexperiments. The picture shows a screenshot of the point cloudobtained by the Kinect sensor with registered RGB information.

1.0

0.8

0.6

0.4

0.2

0.00.0 0.2 0.4 0.6

1-Specificity

Sen

sitiv

ity

0.8 1.0

SHOTFPFHUSC3DSCSIRSD

Figure 8. ROC curve for recognition (no pose) using localfeatures in the set without occlusions spanning on the size ofthe consensus set.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_____________________

_________________

http://www.qmags.com/clickthrough.asp?url=http://users.acin.tuwien.ac.at/aaldoma/datasets/RAM.zip&id=17487&adid=P88E1





a dimensionality of 352, represents a good tradeoffbetween recognition accuracy and memory footprint, aswell as in terms of matching efficiency.

Another important factor is represented by the amountof false positives (FPs) being yielded by the local recogni-tion pipeline. This is especially true when the recognitionthresholds are set low enough to maximize the number ofcorrect recognitions. From Figures 8 and 9, this seems to bea common trait among all local features and motivates theuse of strong hypotheses verification strategies to reduce thenumber of FPs while retaining correct recognitions.

For what concerns global features, the results arepresented in Figures 6 and 10 and Table 1. As statedearlier, global features were only evaluated on the test dataset without occlusions, where the objects can be segmentedby means of dominant plane extraction and Euclideanclustering. Figure 10 presents a direct comparison amongall global features. Features using normal information todescribe the surface of the objects seem to present a clearadvantage over ESF in this data set. By explicitly takinginto account the size of the objects and the viewpoint infor-mation, CVFH is the best performing feature.

Figure 6 presents how the recognition results can beimproved by means of a suitable postprocessing over areduced number of candidates obtained directly out fromthe matching stage. As observed in Figure 10, GRSDrecognition accuracy improves rapidly when severalcandidates are taken into account; however, the viewpointof the candidates appears to be incorrect more often asseen from the root-mean-square error (RMSE) of the fea-ture, which is about twice bigger as that of the other fea-tures (see Table 1).

The effect of the descriptor dimensionality is lessimportant among global features due to the small numberof descriptors required to describe a set of objects. This alsoresults in an important speedup on the matching stage.

Because of the segmentation step, the number of FPs isdirectly related to the recognition capacity of the feature.In other words, the amount of recognitions (false or truepositives) is bounded by the number of objects found dur-ing the segmentation step. Nevertheless, global featuresmight as well benefit from hypotheses verification stages tofilter those objects that were incorrectly recognized.

Finally, we would like to mention that a direct, fair, andaccurate comparison between global and local features/pipelines is difficult due to their different characteristicsand assumptions. However, it is clear that in scenarioswhere segmentation becomes challenging or occludedobjects need to be recognized, global features should beavoided. Conversely, in controlled environments wherespeed becomes an important factor, global features mightrepresent a good choice.

Classification of Unknown ObjectsA related problem is that of the generalization of thedescriptors to unknown objects of similar shape. While adescriptive feature is preferred if many views of all objectsare known a priori, some robustness to small changes isrequired if similar objects are to be detected (or the sameobjects under different noise levels or resolution). Toprovide the reader with an intuition of how well the

1.0

0.8

0.6

0.4

0.2

0.00.0 0.2 0.4 0.6

1-Specificity

Sen

sitiv

ity

0.8 1.0

SHOTFPFHUSC3DSCSIRSD

Figure 9. ROC curve for recognition (no pose) using localfeatures in the set with occlusions spanning on the size of theconsensus set.

1.0

0.8

0.6

0.4

0.2

0.02 4 6 8 10

Rank

Acc

umul

ated

Rec

ogni

tion

Rat

e

12 14

CVFHESFVFHPFHGRSD

CVFHESFVFHPFHGRSD

Figure 10. Accumulated recognition rate for global features.

•Table 1. Average RMSE for the different globalfeatures among those correctly recognized.

FeatureAverage RMSE[Number of kTP]

CVFH 0.000262 [88]

VFH 0.000197 [81]

GRSD 0.000432 [85]

PFH 0.000220 [79]



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






presented (global) features cope with these changes, wecompared the results on a standard data set to previouslypublished results and obtained similar results using a sim-ple k-NN classifier.

For the empirical evaluation of the features for classifi-cation, we followed the method used by Lai et al. to allow acomparison of results. For training and evaluating the clas-sifiers, we used each of the more than 200,000 scans of 300objects from 51 categories from the database. The task is tolearn a model of the categories in the data set, which gener-alizes to objects and were not part of the training set. Forestimating the performance of the features, we evaluatedeach of them on ten different partitionings of the data setinto training data and test data. To obtain a partitioning,all views of one of the objects from each category waschosen at random and removed from the data set. Theremoved objects formed the test partition and the remain-ing objects the training partition.

Before estimating the features, outliers were removedfrom the point clouds. We considered all point outlierswhose mean distance to the 50 nearest points deviatesmore than three standard deviations from the global dis-tance mean. The point clouds were further subsampledusing a voxel grid filter with a grid size of 5 mm to speedup subsequent computations. An important step in esti-mating all of the point signatures is the estimation of pointnormals. In PCL, the estimation is done via principalcomponent analysis on the covariance matrix obtainedfrom the neighborhood of a point, and sign disambigua-tion is done by orienting all of the normals toward theviewpoint. For all the features under investigation, allpoints with a distance smaller or equal to 2 cm are consid-ered to lie within the neighborhood of a point. We used theFLANN for our k-NN implementation with a K-meansindex with default parameters.

In Table 2, we give the mean accuracy as well as theaccuracy’s standard deviation for all the features andclassifiers used. As CVFH computes a descriptor for eachobject part, it is not a fixed-length feature for a completeobject, so integrating it for this classification task wouldnot be straightforward. Thus, we replaced it with SHOTfor this experiment, by computing a single SHOT featurefor each object and using it as a global feature (we com-puted SHOT for the centroid using the complete cloud asthe neighborhood).

As a comparison, the results reported in [27] are53:1� 1:7 using linear support vector machines (SVM) asclassifier, 64:7� 2:2 using a Gaussian kernel SVM, and66:8� 2:5 using random forests—all using efficient matchkernel [28] features using random Fourier sets computedon SIs. Thus, the features themselves perform well for 3-Dshape-based object categorization but more advanced clas-sifiers are needed for obtaining better results.

ConclusionWe have presented a survey of local and global 3-D shapefeatures publicly available in the PCL. Moreover, weargued that a good feature is a small but important piece ofany object recognition system and, therefore, presentedtwo different complete pipelines (using local and globalfeatures, respectively) built up from different modulesavailable in PCL.

We have experimentally shown how the different pipe-lines can be used to recognize objects and estimate their 6-DoF pose in real scenes obtained with the Kinect sensor.The global features have also been evaluated on a publicdata set for the task of object classification providing goodresults with a simple NN classifier.

We believe that the use of 3-D shape-based features andrecognition pipelines able to yield 6-DoF poses have a strongvalue in many robotic scenarios and applications and hopethat the presented results and available tools will be usefulfor the reader when designing a 3-D perception system.

AcknowledgmentsThe authors thank Kevin Lai for granting them access tothe RGB-D object data set [27] and Florian Seidel for hishelp regarding the evaluation in the “Classification ofUnknown Objects” section. Special thanks to the PCLcommunity (see http://pointclouds.org/about.html) aswell as to the funding institutions of the authors of thisarticle.

References[1] R. B. Rusu and S. Cousins, “3D is here: Point Cloud Library (PCL),”

in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), Shanghai,

China, May 9–13, 2011, pp. 1–4.

[2] F. Tombari, S. Salti, and L. Di Stefano, “Unique signatures of histo-

grams for local surface description,” in Proc. 11th European Conf.

Computer Vision (ECCV ’10), 2010, pp. 356–369.

•Table 2. Classification accuracy on the RGB-D object data set.

k-NN VFH GRSD PFH (G)SHOT

Metric [308d] [20d] [125d] [352d]

L1 56.446 2.66 43.936 2.20 53.006 1.64 55.736 1.52

L2 52.756 2.33 43.106 2.20 51.316 1.86 48.826 1.42

Hellinger 57.201 45.916 2.00 55.956 1.48 54.636 1.75

c2 56.36 2.86 45.956 1.99 55.636 1.49 55.636 1.79

1Building the index took�12 h for this metric, so it was evaluated for only one instance of the data set.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


http://www.qmags.com/clickthrough.asp?url=http://pointclouds.org/about.html&id=17487&adid=P90E1





[3] R. B. Rusu, N. Blodow, Z. C. Marton, and M. Beetz, “Aligning point

cloud views using persistent feature histograms,” in Proc 21st IEEE/RSJ Int.

Conf. Intelligent Robots and Systems (IROS), Nice, France, Sept. 22–26,

2008, pp. 3384–3391.

[4] R. B. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms

(FPFH) for 3-D registration,” in Proc. IEEE Int. Conf. Robotics and Auto-

mation (ICRA), Kobe, Japan, 2009, pp. 3212–3217.

[5] R. B. Rusu, G. Bradski, R. Thibaux, and J. Hsu, “Fast 3-D recognition

and pose using the viewpoint feature histogram,” in Proc. 23rd IEEE/RSJ

Int. Conf. Intelligent Robots and Systems (IROS), Taipei, Taiwan, Oct.

2010, pp. 2155–2162.

[6] A. Frome, D. Huber, R. Kolluri, T. B€ulow, and J. Malik, “Recognizing

objects in range data using regional point descriptors,” in Proc. 8th Euro-

pean Conf. Computer Vision (ECCV 04), 2004, pp. 224–237.

[7] S. Belongie, J. Malik, and J. Puzicha, “Shape matching and object

recognition using shape contexts,” IEEE Trans. Pattern Anal. Machine

Intell., vol. 24, no. 4, pp. 509–522, 2002.

[8] F. Tombari, S. Salti, and L. Di Stefano, “Unique shape context for 3-D

data description,” in Proc. ACM Workshop on 3-D Object Retrieval (3-D

OR ’10), 2010, pp. 57–62.

[9] Z. C. Marton, D. Pangercic, N. Blodow, and M. Beetz, “Combined

2D-3-D Categorization and classification for multimodal perception sys-

tems,” Int. J. Robot. Res., vol. 30, no. 11, pp. 1378–1402, Sept. 2011.

[10] A. E. Johnson, “Spin-images: A representation for 3-d surface

matching,” Robotics Inst., Carnegie Mellon Univ, Pittsburgh, PA, Tech.

Rep. CMU-RI-TR-97-47, Aug. 1997.

[11] A. Aldoma, N. Blodow, D. Gossow, S. Gedikli, R. Rusu, M. Vincze,

and G. Bradski, “CAD-model recognition and 6 DOF pose estimation

using 3D cues,” in Proc. ICCV 2011, 3D Representation and Recognition

(3D RR11), Barcelona, Spain, 2011, pp. 585–592.

[12] W. Wohlkinger and M. Vincze, “Ensemble of shape functions for 3-

D object classification,” in Proc. IEEE Int. Conf. Robotics and Biomimetics

(IEEE-ROBIO), 2011, pp. 2987–2992.

[13] C. Papazov and D. Burschka, “An efficient RANSAC for 3-D object

recognition in noisy and occluded scenes,” in Proc. 10th Asian Conf.

Computer Vision (ACCV), 2010, pp. 135–148.

[14] F. Tombari, S. Salti, and L. Di Stefano, “Performance evaluation of

3-D key point detectors,” Int. J. Comput. Vis., to be published.

[15] M. Muja and D. G. Lowe, “Fast approximate nearest neighbors

with automatic algorithm configuration,” in Proc. Int. Conf. Computer

Vision Theory and Application (VISAPP’09). INSTICC Press, 2009,

pp. 331–340.

[16] F. Tombari and L. Di Stefano, “Hough voting for 3-D object recogni-

tion under occlusion and clutter,” IPSJ Trans. Comput. Vis. Appl. (CVA),

vol. 4, pp. 20–29, Mar. 2012.

[17] A. S. Mian, M. Bennamoun, and R. A. Owens, “On the repeatability

and quality of key points for local feature-based 3-D object retrieval from

cluttered scenes,” Int. J. Comput. Vis., vol. 89, no. 2–3, pp. 348–361, 2009.

[18] H. Chen and B. Bhanu, “3D free-form object recognition in range

images using local surface patches,” Pattern Recog. Lett., vol. 28, no. 10,

pp. 1252–1262, 2007.

[19] D. W. Eggert, A. Lorusso, and R. B. Fisher, “Estimating 3-D rigid

body transformations: A comparison of four major algorithms,”Machine

Vision Appl., vol. 9, no. 5-6, pp. 272–290, 1997.

[20] B. Horn, “Closed-form solution of absolute orientation using unit

quaternions,” J. Opt. Soc. Amer. A, vol. 4, no. 4, pp. 629–642, 1987.

[21] N. Bergstr€om, M. Bj€orkman, and D. Kragic, “Generating object

hypotheses in natural scenes through human-robot interaction,” in

Proc.IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), Sept.

2011, pp. 827–833.

[22] A. K. Mishra and Y. Aloimonos, “Visual segmentation of ‘simple’

objects for robots,” in Proc.Robotics: Science and Systems (RSS), 2011.

[23] C. C. Fowlkes, D. R. Martin, and J. Malik, “Local figure-ground cues

are valid for natural images,” J Vis., vol. 7, no. 8, pp. 1–9, 2007.

[24] D. Comaniciu, P. Meer, and S. Member, “Mean shift: A robust

approach toward feature space analysis,” IEEE Trans. Pattern Anal.

Machine Intell., vol. 24, pp. 603–619, 2002.

[25] R. B. Rusu, N. Blodow, Z. C. Marton, and M. Beetz, “Close-range

scene segmentation and reconstruction of 3-D Point cloud maps for mobile

manipulation in human environments,” in Proc. IEEE/RSJ Int. Conf.Intelli-

gent Robots and Systems (IROS), St. Louis, MO, Oct. 11–15, 2009.

[26] P. Bariya and K. Nishino, “Scale-hierarchical 3-D object recognition

in cluttered scenes,” in Proc. Int. Conf. Computer Vision and Pattern

Recognition (CVPR), 2010, pp. 1657–1664.

[27] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-

view RGB-D object dataset,” in Proc. Int. Conf.Robotics and Automation

(ICRA), 2011, pp. 1817–1824.

[28] L. Bo and C. Sminchisescu, “Efficient match kernels between sets of

features for visual recognition,” in Proc. NIPS, 2009.

Aitor Aldoma, Vision4Robotics Group, Automation andControl Institute, Vienna University of Technology,Vienna, Austria. E-mail: [email protected].

Zoltan-Csaba Marton, Intelligent Autonomous Systems,Technische Universit€at M€unchen, Germany. E-mail:[email protected].

Federico Tombari, Computer Vision Lab, DEIS, Univer-sity of Bologna, Italy. E-mail: [email protected].

Walter Wohlkinger, Vision4Robotics Group, Automationand Control Institute, Vienna University of Technology,Vienna, Austria. E-mail: [email protected].

Christian Potthast, Robotic Embedded Systems Labora-tory (RESL) and Department of Computer Science,University of Southern California (USC), Los Angeles, CA90089, USA. E-mail: [email protected].

Bernhard Zeisl, Computer Vision and Geometry Group,ETH Zurich, Switzerland. E-mail: [email protected].

Radu Bogdan Rusu, Open Perception, Menlo Park, CA,USA. E-mail: [email protected].

Suat Gedikli, Willow Garage, Menlo Park, CA, USA.E-mail: [email protected].

Markus Vincze, Vision4Robotics Group, Automation andControl Institute, Vienna University of Technology,Vienna, Austria. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_______________

__________________

__________________

____________

____________

_______________

_______________

_______________

_______________














•By Marco Bibuli, Massimo Caccia,Lionel Lapierre, and Gabriele Bruzzone

Experiments in

Vehicle Following



Date of publication: 15 February 2012

Virtual target-based path-following techniques are extendedto execute the task of vehicle following in the case ofunmanned surface vehicles (USVs). Indeed, vehicle follow-ing is reduced to the problem of tracking a virtual target mov-ing at a desired range from a master vessel, while separating

the spatial and temporal constraints, giving priority to the formerone. The proposed approach is validated experimentally in a harborarea with the help of the prototype USVs ALANIS and Charlie, devel-oped by Consiglio Nazionale delle Ricerche-Istituto di Studi sui Sis-temi Intelligenti per l’Automazione (CNR-ISSIA).

The 21st century’s scenarios of marine operations, regarding envi-ronmental monitoring, border surveillance, warfare, and defenseapplications, foresee the cooperation of networked heterogeneous U

.S.N

AVYPHOTO

BYMASSCOMMUNICATIO

NSPECIALISTSEAMAN

SCOTTYOUNGBLO

OD/RELE

ASED


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






manned/unmanned air, ground, and marine platforms.Examples are given by the autonomous ocean samplingnetwork, integrating robotic vehicle and ocean models toincrease the capacity of observing and predicting the oceanbehavior and the Barents 2020 vision, optimizing marineresources; thanks to historical and real-time informationcollected by a large network of cooperating vehicles.

In this framework, USVs, given their position at theair–sea interface, can play a key role both in relayingradio-frequency transmissions in air and acoustic trans-missions undersea, as proposed, for instance, in the Euro-pean Commission (EC)-funded ASIMOV project [1], andmonitoring ocean and atmosphere dynamics as well assurface and underwater intrusions. As a consequence oftheir networking capabilities, USVs are naturally seen asa part of flotillas of heterogeneous vehicles executinglarge-scale surveys and supporting rapid environmentalassessment (REA). The result is that an increasing num-ber of prototype vehicles have been developed for science,bathymetric mapping, defense, and general roboticsresearch. For an overview of the developed prototype ves-sels and basic design and research trends and issues, thereader can refer to [2].

In this context, the research presented in the followingdeals with aspects related to cooperative motion control ofunmanned marine vehicles (UMVs), focusing on the theo-retical and experimental study of the problem of a slaveUSV following a master vessel at a predefined range. Thissimple formation configuration, with its natural extensionto a fleet of slaves vehicles following a master vessel, is thebase for a number of different applications. An example isgiven by the execution of morphobathymetric surveys invery shallow water, such as coastal lagoons, combining theuse of vertical incidence echosounders and subbottom chirpdevices [3]. In this case, a flotilla of USVs can constitute aforce multiplier in executing multiple surveys with the samesensor, installed aboard the master and slave vessels, respec-tively, or using different sensors in the same place at thesame time with respect to the spatiotemporal resolution ofthe phenomena under investigation, as in the case of acous-tic devices that cannot work when mounted below the samehull. A team of heterogeneous USVs able to tackle this prob-lem is currently under development at the NationalResearch Council of Italy. Other interesting operational sce-narios include surveys, i.e., periodic bathymetries for evalu-ating the distribution of sediments and classifying theirquality, of harbor areas for driving dredging, coastal land-slides and sand distribution for beach maintenance, andartificial lakes, including dam inspection.

The main contribution of this article relies in the exper-imental validation of guidance techniques, i.e., virtual tar-get-based path following and their extension to handlemultivehicle cooperation as well as in identifying the majorsources of performance limitations. Successful experimen-tal demonstrations, contributing to bridge the gap betweentheory and practice, push the development of operational

marine robots for marine monitoring, surveillance, explo-ration, and exploitation.

In particular, experiments have been carried out in aharbor area using the Charlie USV [4] as a slave vehicleand the dual-mode ALANIS vessel [5], in this case pilotedby a human operator, as a master vehicle. As discussed inthe following, the proposed guidance law privileges thespatial constraint of driving the slave vehicle over the refer-ence path with respect to the temporal requirement ofmaintaining a desired range from the master vessel. Thetarget path, defined by the motion of the master vessel, isfollowed by adopting a conventional nonlinear path-following algorithm of the type discussed in [6].

Problem Definition and State of the ArtIn the literature, motion control scenarios of USVs areusually classified into three main categories (pointstabilization, trajectory tracking, and path following),along with the concept of path maneuvering (see, forinstance, [7]).l Point stabilization: The goal is to stabilize the vehicle

zeroing the position and orientation error with respectto a given target point with a desired orientation (in theabsence of currents). The goal cannot be achieved withsmooth or continuous state-feedback control laws whenthe vehicle has nonholonomic constraints. In the pres-ence of currents, the desired orientation is not specified.

l Trajectory tracking: The vehicle is required to track atime-parameterized reference. For a fully actuated sys-tem, the problem can be solved with advanced nonlinearcontrol laws; in the case of underactuated vehicles, i.e.,the vehicle has less degrees of freedom than state varia-bles to be tracked, the problem is still a very active topicof research.

l Path following: The vehicle is required to converge toand follow a path without any temporal specification.The assumption made in this case is that the vehicle’sforward speed tracks a desired speed profile, while thecontroller acts on the vehicle orientation to drive it tothe path. This typically allows a smoother convergenceto the desired path with respect to the trajectory track-ing controllers, less likely pushing to saturation thecontrol signals.

l Path maneuvering: The knowledge about the vehicle’smaneuverability constraints enables the design of speedand steering laws that allow for feasible path negotiation.In recent years, the above-mentioned scenarios have

been extended to the case of coordinated and/or coopera-tive guidance of multiple vessels, basically introducingthe concept of formation, i.e., geometric disposition of aset of vehicles.

As discussed in [8], a fleet of vessels can be required totrack a set of predefined spatial paths while holding a de-sired formation pattern and speed (cooperative path fol-lowing) to follow (in space) or track (in space and time) amoving target (cooperative target following and tracking,



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






respectively). It is worth noting that these problems canbe solved by converting them into an equivalent virtualtarget-based path-following problem. In particular, theso-called path-tracking scenario, in which the vehicle isrequired to track a target that moves along a predefinedpath, is the basic component of cooperative target-follow-ing/tracking systems. Indeed, with respect to trajectorytracking, path tracking separates the spatial and temporalconstraints, giving priority to the former one, i.e., the

vehicle tries tomove alongthe path and then to zerothe range from the target,as, for instance, in the caseof a virtual target movingat a desired range from amaster vessel (vehicle-fol-lowing scenario).

In this context, a num-ber of preliminary experi-ments on multiple vehiclecooperative guidance wereperformed using combi-nations of USVs, autono-

mous underwater vehicles (AUVs), and manned vessels.Indeed, after first demonstrations carried out with autono-mous kayaks surface craft for oceanographic and underseatesting (SCOUT) in the United States to validate Interna-tional Regulations for Preventing Collisions at Sea (COL-REGS)-based anticollision for UMVs [9], research focusedon vehicle following, cooperative path following and targettracking, as well as mission coordination of multiple vehiclesin the case of poor communication.

In particular, the need of collecting bathymetric data inan REA framework is strongly pushing research in vehiclefollowing to support an operating scenario where a mastervessel is followed on its sides by a flotilla of small USVs.The first full-scale experiment in a civilian setting world-wide, involving as USV a retrofitted leisure boat of length8.5 with a maximum speed of 18 knots and as mannedvehicle a research vessel of length 30 with an upper speedof 13 knots, was performed in Trondheimsfjord, Norway,on September 2008 [10]. In the following year, the ex-periment was replicated with a couple of slave vehiclesfollowing the master vessel [11] (a video describing theexperiment can be found at http://www.youtube.com/watch?v=i_NrA5DwIcc).

Very interesting preliminary demonstrations of cooper-ative control of multiple UMVs, supported by a large theo-retical work, were performed in the framework of the EC-funded project GREX [Instituto Superior Tecnico (IST)-Project No. 035223] about coordination and control ofcooperating heterogeneous unmanned systems in uncer-tain environments. In particular, experiments oriented toevaluate the possibility of coordinating the operations ofmultiple AUVs in the presence of very limited underwateracoustic communications were carried out with the Institut

Français de Recherche pour l’Exploitation de la MER’s(IFREMER’s) AUVs Asterx and AUVortex in the Toulonarea, France, on November 2008 [12]. During these trials,Asterx, the faster AUV, when measuring an excessive coor-dination error, sent the coordinates of a target point thatthe slower AUV, AUVortex, had to reach, while Asterxwas circling around the waiting location.

Preliminary experiments, aiming at validating the exe-cution of vehicle primitives, such as path following andtarget following, were carried out with the DELFIMxautonomous surface vehicle (ASV) following the human-piloted boat Aguas Vivas by the researchers of the IST ofLisbon in Azores in May 2008 [8].

On November 2009, coordinated path-following ex-periments involving the USVs DELFIM and DELFIMx bythe IST of Lisbon, the AUV SEABEE by Atlas Elektronic,and the AUVortex by IFREMER were carried out in Sesim-bra, Portugal [13]. The vehicles that operated on the sur-face communicating through a radio link had to followpaths composed by a segment of line, followed by an arc,and then finalized by a segment of line, while keeping anin-line formation, i.e., aligning themselves along a straightline perpendicular to the paths.

In the meantime, the autonomous kayaks SCOUT wereexploited for evaluating the capacities of autonomouscooperation of AUVs and USVs in executing search tasksat sea, e.g., mine countermeasures [14], as well as foradaptive collection of oceanographic data, e.g., characteri-zation of the sound speed profile with multiple USVs [15].Very interesting experiments on the cooperative maneu-vering of a couple of USVs for capturing a floating objectand shepherding it to a designated position were carriedout by the University of Southern California [16].

Vehicle FollowingThe problem of cooperative path following, where a slavevehicle follows a master maintaining a predefined positionconfiguration, is addressed in this section. In particular,the proposed approach assumes that the slave vehicledoesn’t a priori know the path to be followed: the masterexecutes its motion, i.e., automatically following a path orbeing driven by a human operator and sends basic naviga-tion information to the slave. From this reduced set ofinformation (for instance, master’s position, actual veloc-ity, and orientation), the slave vehicle online reconstructsthe path to be followed. This means that the only con-straints on the master vessel are given by the fact that it hasto be equipped with simple sensors and a communicationsystem to send the navigation data to the slave vehicle.

The goal is to have a slave USV following the path of amaster vessel at a fixed range, measured in terms of curvi-linear abscissa of the desired path or linear distance in aspecific direction in a suitable reference frame (typically,rigidly fixed to the master). Indeed, the problem consists oftracking an online-defined path, giving priority to thespatial constraints with respect to the temporal ones. Thus,

•USVs are seen as a part of

flotillas of heterogeneous

vehicles executing

large-scale surveys and

supporting REA.•



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________

http://www.qmags.com/clickthrough.asp?url=http://www.youtube.com/watch?v=i_NrA5DwIcc&id=17487&adid=P94E1





since the main objective is that the two vehicles follow thesame path, the proposed approach consists of three steps(executed at each control cycle):l reconstructing the master path on the basis of continu-

ously collected navigation datal guiding the vehicle over the reference path according to

a conventional path-following algorithml adapting the vehicle surge speed according to the error

from the desired distance from the master.It is worth noting that, according to research results

presented in [6], the vehicle-following controller isdesigned at the guidance level generating reference yaw rateand surge, that, in the case of the Charlie USV, are trackedby suitable proportional integral (PI) gain-schedulingvelocity controllers. In the operative framework used forvalidating the proposed approach, a slave vehicle is incharge of following exactly the shape of the path executedby a human-driven master vessel, maintaining a desiredposition configuration with respect to it. For instance, theslave has to follow the master’s path keeping a desired dis-tance from its stern or (path-based) curvilinear distancebetween the two vehicles. While moving, the master trans-mits basic navigation information to the slave, i.e., horizon-tal position provided by global positioning system (GPS),when working with surface vessels, or acoustic positioningsystems when underwater vehicles are involved. This basicinformation set can be augmented for instance addingwhen the master follows a predefined path, actual tangentand curvature values. Collecting the data provided by themaster, the slave online generates a reference path that isfollowed using a Lyapunov-based guidance law improvedwith the virtual target approach, as presented in [6] andsummarized in the following subsection. To maintain thedesired range from the master, the slave’s surge velocity isadapted according to a saturated PI function of the desireddistance, linear or curvilinear, between the two vehicles.

Virtual Target-Based Path FollowingA brief description of the adopted path-following guid-ance algorithm for a single vehicle system follows. Allthe details of the proposed technique can be found in[6]. With reference to Figure 1, a Serret-Frenet frame< f > is attached to a virtual target VT moving alongthe path. The error vector connecting the virtual target

VT to the vehicle V, expressed in < f >, is d ¼ ½s1 y1�T .Thus, after straightforward computations, on the hori-zontal plane the error dynamics is given by the follow-ing equation system:

_s1 ¼ �_s 1� ccy1ð Þ þ U cos b,

_y1 ¼ �cc_ss1 þ U sin b,

_b ¼ re � cc _s,

8><>: (1)

where b ¼ we � wf is the angle of approach to the path, r�

and re are, respectively, the rotation rates of the vehicle and

its velocity vector, which has a absolute value U , s repre-sents the position of the virtual target VT over the path(i.e., curvilinear abscissa), and cc ¼ cc(s) is the signedcurvature of the path. Defining the Lyapunov functionV ¼ 1

2 (b� u)2, the following control law for the yaw-rateinput signal is obtained:

r� ¼ 1g(t)

_u� k1(b� u)þ cc_s½ �, (2)

where g(t) embeds the ratio between the angular speed ofthe vehicle’s velocity vector and the vehicle’s yaw rate, k1 isa controller parameter, and u is an odd function definingthe actual angle approach, as a function of the distance y1from the path. A typical choice of u y1ð Þ is

u y1ð Þ ¼ �wa tanh kuy1� �

, (3)

where wa is the maximum approach angle value and ku is atunable function parameter. Moreover, it is worth notingthat the speed _s of the target Serret-Frenet frame < f >constitutes an additional degree of freedom that can becontrolled to guarantee the convergence of the vehicle atthe desired path avoiding possible singularities. Indeed, themotion of the feedback control system restricted to the setE, where _V ¼ 0, i.e., b ¼ u y1ð Þ, can be studied definingthe Lyapunov function VE ¼ (1=2) s21 þ y21

� �. Considering

that in the set E y1 sinu y1ð Þ 0 for the choice of u y1ð Þ in(3), the regulation law for the virtual target speed is com-puted as follows:

_s� ¼ U cos bþ k2s1: (4)

Vehicle Range TrackingAs introduced previously, the vehicle-following approachproposed in this work is based on the single-vehicle

B

U

<b>

<f>

<e>

ψf

ψe

ψ

d

x

P

p

xe

ye

Figure 1. Vehicle’s parameters and frames definition.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






path-following guidance technique (2), combined with thecontinuous adaptation of the surge speed of the slave vehi-cle, with the aim of forcing the intervehicle distance to con-verge to and be maintained at a desired value.

The intervehicle distance D can be defined in differentways according to the mission requirements: the mostcommon establishes the linear range between the vehicles,i.e., D ¼ kxmaster � xslavek, or the curvilinear distancebetween the master and slave, i.e., the differencebetween the respective curvilinear abscissas D ¼ Ds ¼smaster � sslave. Thus, the range tracking task consists ofdesigning a control law to reach a desired distance D�.Defining the distance error es ¼ D� D�, the simplestimplemented solution to make es ! 0 is a PI control lawthat generates a surge speed reference signal u�. This basicsolution can be easily improved by introducing a continu-ous saturation function to constrain the computed surge-speed reference within a minimum and maximum value,generating a feasible reference signal u�sat for the lowerorder surge speed controller:

u� ¼ uff þ Kpes þ KiResdt,

u�sat ¼ C þ umax�umin2 tanh ku� � Cð Þ

�, (5)

where Kp and Ki are, respectively, the proportional andintegral gains of the controller, k is a gain factor, andC ¼ umin þ ((umax � umin)=2). The feed forward of thevelocity of the master vehicle uff can be transmitteddirectly from the vessel itself or computed by the slaveusing, for instance, classic numerical derivation or somesort of filter. A minimum surge speed limit umin, usuallygreater than zero, is needed to guarantee maneuverability,whereas the maximum speed limit umax takes into accountthe physical constraints of the thrust actuation.

Sensing IssuesAs introduced previously, the motion of the master andslave vehicles is estimated online on the basis of the

measurements of aboard GPS and compass. Dependingon the quality of the sensor measurements, issues in thesmoothness of the estimated path or in the groundtruthing of the estimated positions of the vehicles canturn up.l Estimation of the target path: The steering control

action, which is a function of the master path tangentand curvature as in (2), can be affected by the noise intheir estimates. Indeed, a direct computation of the pathtangent and curvature amplifies the noise of the GPSposition measurements. Anyway, when the slave vehiclecan be assumed to keep a certain distance from themaster, a local smoothing for estimating the referencepath is possible, thus reducing the impact of disturbancein position measurements. On the other hand, wherethe vehicles are required to maintain a parallel forma-tion, only causal filtering techniques can be used for esti-mating the master’s path.

l Consistency of position measurements: Although ad-vanced guidance techniques usually guarantee that avehicle follows a path with a desired precision, this istrue for the estimated position of the vehicle that, as aconsequence of disturbance on GPS measurements,could differ from the actual one also of some meters.Indeed, when a slave vehicle is required to follow thepath of a master vehicle to collect data in the same pla-ces, the precision in executing this task is not only afunction of the performance of the guidance and con-trol modules but also of the consistency of the positionmeasurements collected aboard the two vehicles.Thus, special attention to ground-truth verificationof the followed path has to be paid when performingfield trials.

Experimental SetupExperiments have been carried out with the Charlie USVand ALANIS dual-mode vessel in a rowing regatta fieldinside the Genova Pr�a harbor, Italy, on July 2009. Accord-ing to the requirements specified by the HydrographicInstitute of the Italian Navy, the experiments focused onthe high-precision vehicle following to provide, althoughin a protected environment, a preliminary feasibilitydemonstration of the proposed technology and to validatethe basic architecture requirements in terms of control,communication, and sensing systems. In the following,after a short introduction of the basic characteristics of thevehicles involved in the demonstration, a detailed presen-tation of their adaptation, required to support the experi-ments, will be given together with a description of theadopted navigation sensors.

Charlie USVThe Charlie USV [4] is a small autonomous catamaranprototype that is 2.40 m long, 1.70 m wide, and weighsabout 300 kg in air (see Figure 2). The vessel, originallydesigned and developed by CNR-ISSIA, Genova, forFigure 2. A view of the Charlie USV in the Genova Pr�a harbor.

CNR-ISSIA



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






sampling sea surface microlayer and collecting data on theair–sea interface in Antarctica, is propelled by two dcthrusters whose revolution rate is controlled by a couple ofservo amplifiers, closing a hardware speed control loopwith time constant negligible with respect to the system.With respect to the original version, where steering wasguaranteed by the differential revolution rate of the propel-lers, the vehicle has been upgraded with a rudder-basedsteering system constituted by two rigidly connected rud-ders, positioned behind the propellers, and actuated by abrushless motor. The standard vessel navigation package isconstituted by a GPS Ashtech GG24C integrated with aKVH Azimuth Gyrotrac providing the true north. Theelectrical power supply is provided by four 12 V at 40 Ahlead batteries integrated with four 32 W triple junction,flexible solar panels.

Communications with the remote control and supervi-sion station are guaranteed by a radio wireless LAN at2.4 GHz with a maximum data transfer rate of 3 Mb/s, sup-porting robot telemetry, operator commands, and videoimage transmission. Owing to poor performance, mainlyin terms of reliability, offered by commercial, relatively lowcost, wireless line-of-sight (LOS) links, the communicationsystem has been upgraded with a radio modem working at169 MHz with a transfer rate of 2,400 b/s, guaranteeing asafe transfer of commands and basic telemetry. Indeed, theradio modem link acts as a backup channel, due to the fre-quent and unpredictable main wireless link disconnec-tions, allowing to send a basic command set to drive orrecover the vehicle.

The human operator station is formed by a laptopcomputer, running a human computer interface, imple-mented originally in C++ and then in Java, and the powersupply system, which integrates a couple of solar panels(32 W at 12 V) and one lead battery (100 Ah at 12 V), thus,guaranteeing its full autonomy and portability.

ALANIS Dual-Mode USVThe ALANIS USV [5] is a 4.50-long, 2.20-m wide rubberdinghy-shaped aluminum vessel with a 40 HP Honda out-board motor (see Figure 3). It weighs 600 kg for a loadcapacity of 800 kg and has an autonomy of about 12 hguaranteed by a fuel capacity of 65 L. A motorized winchcan be mounted on board for automatic deployment andrecovery of scientific instrumentation through a stern holeof 0.20 m diameter. The basic navigation package isformed by a Garmin GPS 152 with 12 parallel channels, anavicontrol smart compass SC1G, and a dual-axis appliedgeomechanics IRIS MD900-TW wide-angle clinometerproviding accurate pitch and roll measurements. A man-ually (dis)connectible electromechanical system for ser-voactuating the vessel steering and throttle allows the dualuse of the vehicle as a manned and an unmanned platform.Indeed, the possibility of having a crew onboard and fastswitching control to a human pilot has been motivated bythe lack of rules for operating unmanned vehicles at sea.

For these reasons, when working in the automatic mode,the human–computer interface, which has the same archi-tecture as the operator station of the Charlie USV, is keptaboard the vessel itself. The basic navigation, guidance,and control system implemented on a single-boardcomputer-based architecture running GNU/Linux OSconsists of proportional derivative auto heading and LOSway-point guidance.

Charlie and ALANIS AdaptationTo implement a master–slave vehicle-following scheme ofthe class discussed in the “Vehicle Following” section, themaster vessel has to communicate its basic navigation infor-mation to the slave vehicle. This implies the installation ofa radio link supporting the transmission of ALANIS naviga-tion data to the Charlie USV. This additional link, aradio modem channel working at 436 MHz with a transferrate of 2,400 b/s, is seen by the slave control system as anadditional sensor providing the measurements required bythe vehicle-following guidance module, i.e., GPS position,course, and speed. The resulting communication scheme isdepicted in Figure 4.

It is worth noting that due to safety reasons, i.e., to haveboth the vehicles under strict visual control by the humansupervisor when executing automatic coordinated maneu-vers in an area with recreational traffic, the basic operatorstation of the Charlie USV, consisting of a laptop and awireless communication link, has been mounted onboardthe ALANIS vessel to perform the experiments. More-over, to improve the localization performance and navi-gation accuracy, according to the issues discussed in the“Sensing Issues” section, the two vehicles have beenequipped with a couple of Omnistar HP-8300 high-posi-tioning GPS receivers with a 95% accuracy, supplied bythe Hydrographic Institute of the Italian Navy.

Test SiteExperimental tests have been carried out in the GenovaPr�a harbor, a calm water channel devoted to rowing races

Figure 3. ALANIS USV.

CNR-ISSIA



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






(44�2503200 N, 8�4604800 E), at the end of July 2009, in a daywith no significant wind disturbance. As shown in the fol-lowing, the presence of white buoys delimiting the lines ofthe regatta field has been very useful for visual ground-truth evaluation of the system performance.

Experimental ResultsField trials, aiming at validating the proposed approach,have been carried out with the slave Charlie USV followingthe master ALANIS vessel piloted by a human operator. Asdiscussed previously, the master vessel sends to the slave

vehicle its fundamental navigationdata (position, course, and speed).The result is that the slave followsthe actual path of the master at lesserrors than in the intercalibration ofthe GPS receivers mounted on thetwo vehicles. To experimentallyevaluate the amount of this intercali-bration error using GPS devices ofdifferent classes, dedicated prelimi-nary tests have been performed.

GPS PerformanceTo evaluate the GPS performance, interms of measurement noise andtime-variable offset between two dif-ferent devices, the measured rangebetween a couple of GPS antennaspositioned at a constant distance has

been evaluated. Indeed, since the main goal of the guidancetask is to force the two vehicles to navigate along the samepath, a constant bias in measurements carried out by dif-ferent devices is required. Preliminary tests, performed inthe framework of the ALANIS project, demonstrated that,using different conventional low-cost devices the differ-ence between simultaneous measurements of positioncould be of the order of some meters. During the experi-ments, three devices, i.e. a Garmin GPS 152, a GPS Ash-tech GG24C, and a Trimble GPS Pathfinder Pro XRS,made available by the Hydrographic Institute of the ItalianNavy, were mounted on the ALANIS USV maneuveringinside the harbor.

600 700 800 900 1,000 1,100 1,200

600 700 800 900 1,000 1,100 1,200

600 700 800 900 1,000 1,100 1,200

−5

0

5Position Difference: Measurement Noise

d (m

)

−5

0


d (m

)

−5

0


(a)

(b)

Time (s)(c)

d (m

)

Figure 5. The measured distance error between different GPSreceivers. (a) Garmin versus Ashtech, (b) Garmin versus Trimble,and (c) Trimble versus Ashtech.

200 300 400 500 600 700 800−0.1

−0.05

0

0.05

0.1

Omnistar HP−8300Position Difference: Measurement Noise

Time (s)

d (m

)

−0.1 −0.05 0 0.05 0.10

20406080

100

d (m)

(a)

(b)

Num

ber

of S

ampl

es

Figure 6. The measured distance error between two identicalOmnistar HP-8300 high-positioning GPS receivers. (a)Measurement record in time and (b) range histogram.

Radiomodemat 436 MHz

Radiomodemat 169 MHz

Wireless LAN at 2.4 GHz

CharlieControlSystem

ALANISControlSystem

ALANISHCI

Local LAN

CharlieHCI

Figure 4. Charlie–ALANIS network configuration.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






As shown in Figure 5, their measured distances werenot constant, varying up to 5 m in the case of Garmin andAshtech devices. Further tests, carried out with a couple ofidentical Omnistar HP-8300 high-positioning GPS, sup-plied by the Hydrographic Institute of the Italian Navy,revealed a dramatic performance improvement, obtaining,as depicted in Figure 6(b), the measured range errorbetween the devices that was always lower than 0.1 m.

Vehicle FollowingAs previously discussed, experimental tests were manuallyperformed driving the ALANIS vessel in the Genova Pr�aharbor at an advance speed of about 1 m/s. It is worth not-ing that the human pilot had to be very careful in executinga path as free as possible of high curvature stretches. Indeed,since the slave Charlie USV has a slower steering dynamicsthan the master ALANIS vessel, narrow or tricky maneuverscould lead to a divergence from the reference target causingoscillating motions of the slave to recover the desired path.Oscillations of the vehicle motion around the referencepath, originated by a significant overshoot when convergingto a path at a high speed with respect to the steering dynam-ics [6], could be very dangerous when working in restrictedareas in the presence of fixed obstacles (e.g., rocks, parkedboats, and quays) and recreational traffic.

To evaluate the performance of the proposed algorithm,different path shapes have been considered focusing atten-tion on bending maneuvers and the possibility of executingrepetitive tests in similar operating conditions. For instance,Figure 7 shows the path followed by the master vessel ALA-NIS and the slave USV Charlie while executing a U-turnmaneuver presenting a reduction in the curvature radiustoward the end of the bend. This induced a slight sliding ofthe Charlie USV toward outside, clearly visible in the log ofthe lateral range y1 from the target path [see Figure 8(a) inthe time interval between 2,650 and 2,720 s].

The trend of the range between the master and slavevehicles, plotted in Figure 8(b), reveals the difficulties inaccomplishing the secondary task of the path-trackingproblem, i.e., satisfying the time constraints, while guaran-teeing high precision by following the desired curvilinearpath. Indeed, to remain on the desired track, the slave ves-sel reduced its speed while bending to accelerate whencurvature decreases. The precision of the proposed systemin tracking the master path was evaluated performing akind of slalom between a sequence of buoys delimiting theregatta field lanes. Repetitive tests were performed, wherethe presence of the buoys allowed a visual ground-truthverification of the performance of the proposed vehicle-following system, including the validation of the consis-tency of the position measurements supplied by the GPSdevices aboard the two vehicles.

An example is reported in Figure 9 where the passageof the master and slave vehicles between a couple ofwhite buoys is shown. (A video documentation of the tri-als with simultaneous views of the vehicles and their

estimated path is available on the Web at http://www.umv.ge.issia.cnr.it/video/vessel_following.html) As faras the repeatability of system performance in similarconditions is concerned, the master vessel was guidedby the human pilot through the buoys approximatelyalong the same path in different passages. As shown inFigure 10, where a couple of passages are shown, theALANIS pilot (Mr. Edoardo Spirandelli by CNR-ISSIA)

−400

−390

−380

−370

−360

−350

−340

−330

−320

−310

−50

−40

−30

−20

−10

0

10

Vehicle Following: ALANIS Versus Charlie—Bend Path

y (m)

x (m

)

Figure 7. The vehicle-following experimental results: path ofthe master vessel ALANIS and the slave USV Charlie whileexecuting a large radius U-turn.

2,560 2,580 2,600 2,620 2,640 2,660 2,680 2,700 2,720

2,560 2,580 2,600 2,620 2,640 2,660 2,680 2,700 2,720

2,560 2,580 2,600 2,620 2,640 2,660 2,680 2,700 2,720

2,560 2,580 2,600 2,620 2,640 2,660 2,680 2,700 2,720

−2024

Vehicle Following: Path-Tracking Errorsand Charlie USV Speed

y1 (

m)

304050

D (

m)

−10

0

10

r (°

/s)

1

1.1

1.2

(d)Time (s)

(c)

(b)

(a)

u (m

/s)

Figure 8. The vehicle-following experimental results—largeradius U-turn: path-tracking errors and the slave USV speed. (a)Lateral range error, (b) master–slave distance, (c) Charlie USVyaw-rate, and (d) Charlie USV surge.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________________________

http://www.qmags.com/clickthrough.asp?url=http://www.umv.ge.issia.cnr.it/video/vessel_following.html&id=17487&adid=P99E1





was able to execute very precise maneuvers; thanks tothe visual help provided by buoys. As shown in Figure 11,the lateral shift in following the path of the master vessel,during these maneuvers, was higher than 1 m (often higherthan 0.5 m).

In particular, the mean precision �Ass of the guidancesystem in the steady state has been defined as the area Ass

between the actual and the desired path normalized withrespect to the length Dsss of the reference path (see [6]for more details). The computed values for the paths1 and 2, represented in Figure 10, in the intervaly 2 �240m, �140m½ � are reported in Table 1. It is worthnoting than the computed values of �Ass are similar to theones computed in the path-following experimentsreported in [6] where a mean value of 0.74 was computed.

At the end of the maneuver shown in Figures 7 and 8,the increasing range between the vessels is visible when themaster accelerates to go along a low curvature line whilethe slave is still turning. The higher lateral shift at thebeginning of path 2 [time between 3,150 and 3,180 s inFigure 11(c)] shows the behavior of the Charlie USV

during a transient phase when coming from a narrow-range U-turn that is visible in Figure 12. The differentcapabilities in low surge turning of the Charlie and ALA-NIS vessels are clearly visible.

The research and experiments described in this articlepresent significant analogies with the work carried out inthe GREX project with the DELFIMx ASV following thehuman-piloted boat Aguas Vivas [8]. In the case discussedhere, no vehicle primitives, e.g., straight lines and arcs,were defined, representing any generic path as a simplesequence of points, close to each other, with the associatedlocal tangent and curvature. Anyway, the differences inthe performance with respect to the result presented in [8]are evaluable in a difficult way due to different experi-mental conditions: the higher precision in path followingof the experiments presented here can be likely due tothe smoother sea state inside the harbor than along theAzores coastline, besides the different dynamics of the ves-sels involved.

Lessons LearnedThe above-presented research with the theoretical andexperimental results, independently achieved by the Nor-wegian University of Science and Technology and thecompany Maritime Robotics and the Instituto SuperiorTecnico of Lisbon, Portugal, demonstrates that the basicissues concerning the task of USV following have beensolved. In particular, this research as well as the resultsobtained in the GREX project and the examples reportedin [17] demonstrate how the virtual target-based guidance,

(a)

(b)

Figure 9. The vehicle-following ground truthing: (a) passage ofthe master vessel ALANIS and (b) the slave USV Charlie betweenthe same couple of white buoys.

CNR-ISSIA

−300 −250 −200 −150 −100

−100

−80

−60

Vehicle Following: ALANIS VersusCharlie—Slalom Path 1

Vehicle Following: ALANIS VersusCharlie—Slalom Path 2

y (m)

y (m)

(a)

(b)

x (m

)x

(m)

−280

−260

−240

−220

−200

−180

−160

−140

−120

−100

−100

−80

−60

Figure 10. The vehicle-following experimental results—Anexample of the path followed in two similar (repetitive)experiments. Red and blue lines denote the ALANIS and Charliepaths, respectively.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






originally formulated for wheeled ground robots in [18], isvery effective and practical for handling multivehicle coop-eration in marine environment.

As discussed in the “Experimental Results” section,although the performance is mainly limited by the qualityof the sensor data, the last-generation high-positioningGPS devices guarantee a satisfactorily accuracy in themeasured position for many applications without requir-ing the installation of a base station for a differential sys-tem. Further research efforts are required for improvingthe speed control of the slave vehicle, i.e., the weak aspectpointed out by the experiments presented in this article,introducing, if necessary, some heuristics to increase thesystem performance in executing the secondary task of thepath-tracking problem, i.e., satisfying the time constraintsand minimizing the effects of oscillations around thedesired path. Moreover, accurate studies for adapting theproposed guidance algorithms to the presence of signifi-cant wind disturbance, including the definition of enoughaccurate models of the vehicle behavior in those condi-tions, should be carried out.

In-field experimental activity revealed the fundamentalrole played by the availability of reliable robotic platformsand communication infrastructure as well as the largeamount of human resources devoted to their development,adaptation, and integration. In addition, critical issues wereencountered when trying to execute good experiments interms of defining practical procedures, metrics, and experi-mental conditions and performing repeatable trials:l Ground truthing: The availability of fixed buoys, as well

as the synchronization of the vehicle telemetries with

the recorded videos through the use of the USV siren,logged in the USV telemetry and clearly audible in thevideo audio track, allowed an immediate, at first glance,evaluation of the system accuracy when executing trialsin restricted waters, e.g., harbor area. In any case, theperformance of suitable GPS devices is such that aninstrumental validation of system accuracy can be suffi-cient, although less impressive to an external evaluator.

l Performance metrics: Metrics, defined in [6] for evaluat-ing the path-following performance, i.e., satisfying thespatial constraints, are reasonable and easy to beapplied. Further metrics for evaluating the performancein satisfying the time constraints have to be defined andtheir computation has to be implemented.

l Test repeatability: As shown in Figure 10, with the help ofvisual landmarks such as fixed buoys, the human pilotcould approximately drive the master vessel along thesame path during different experiments. Anyway, sincethe effectiveness of the communication infrastructureand GPS measurement consistency have been demon-strated, further experiments, mainly devoted to improveperformance in terms of satisfaction of the time con-straints, could be executed providing as input to the slaveUSV previously recorded trajectories of the master vessel.For this aim, the trajectory of the ALANIS vessel duringthe experiments reported in this article is made available

•Table 1. A path-following performanceindex—Slalom paths.

Ass Dsss �Ass

Path 1 68.48 113.52 0.60

Path 2 83.10 113.84 0.72

−110 −100 −90 −80 −70 −60 −50 −40

−140

−130

−120

−110

−100

−90

Vehicle Following: ALANIS VersusCharlie—Narrow U-Turn Path

y (m)

x (m

)

Figure 12. The vehicle-following experimental results—Exampleof narrow U-turn of the master vessel. The red and blue linesdenote the ALANIS and Charlie paths, respectively.

2,300 2,350 2,400 2,450 2,500

2,300 2,350 2,400 2,450 2,500

−505

Vehicle Following: Path-Tracking Errors (Path 1)

Vehicle Following: Path-Tracking Errors (Path 2)

y1

(m)

20304050

D (

m)

3,150 3,200 3,250 3,300 3,350

3,150 3,200 3,250 3,300 3,350

−505

y1

(m)

20304050

Time (s)

(b)

(c)

(a)

(d)

D (

m)

Figure 11. The vehicle-following experimental results. (a)–(d)Path tracking errors (lateral shift) and the vessel range in twosimilar (repetitive) experiments.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






at http://www.umv.ge.issia.cnr.it/video/vessel_following.html, thus providing a small contribution to the diffusionof data sets about marine robotics applications.

ConclusionsPreliminary experimental results demonstrating the effec-tiveness of a vehicle-following guidance system for USVsbased on the concept of virtual target have been presentedand discussed, after a brief presentation of the proposedapproach. The above-presented research as well as a fewother similar demonstrations cited in the text contribute tobridge the gap between theory and practice in the field ofUMVs encouraging the application of this emergingtechnology not only in military scenarios but also in civil-ian applications.

AcknowledgmentsThis research has been partially funded by the CNR-Centre National de la Recherche Scientifique (CNRS)bilateral agreement Coordinated mission control for auton-omous marine vehicles and Parco Scientifico e Tecnologicodella Liguria s.c.p.a.

The authors thank the Hydrographic Institute of theItalian Navy for supplying high-positioning GPS devicesand participating in the definition of the multivehiclemaneuvers. Moreover, the authors also thank GiorgioBruzzone and Edoardo Spirandelli for their activity in thedevelopment and operation of the ALANIS and CharlieUSVs, Riccardo Mantovani for the video documentationof the experiments, and the personnel of A.S.D.P.S. Pr�aSapello for their kind support to the execution of sea trials.

References[1] A. Pascoal, C. Silvestre, L. Sebastiao, M. Rufino, V. Barroso, J.

Gomes, G. Ayela, P. Coince, M. Cardew, A. Ryan, H. Braithwaite, N.

Cardew, J. Trepte, N. Seube, J. Champeau, P. Dhaussy, V. Sauce, R.

Moiti, R. Santos, F. Cardigos, M. Brussieux, and P. Dando, “Robotic

ocean vehicles for marine science applications: The European ASI-

MOV project,” in Proc. Oceans, Providence, RI, Sept. 2000, vol. 1,

pp. 409–415.

[2] J. Manley, “Unmanned surface vehicles, 15 years of development,” in

Proc. MTS/IEEE Oceans’08, Quebec City, Canada, Sept. 2008, pp. 1–4.

[3] L. Gasperini, “Extremely shallow-water morphobathymetric surveys:

The Valle Fattibello (Comacchio, Italy) test case,” Marine Geophys. Res.,

vol. 26, no. 40270, pp. 97–107, 2005.

[4] M. Caccia, M. Bibuli, R. Bono, G. Bruzzone, G. Bruzzone, and E.

Spirandelli, “Unmanned surface vehicle for coastal and protected water

applications: The Charlie project,” Marine Technol. Soc. J., vol. 41, no. 2,

pp. 62–71, 2007.

[5] M. Caccia, M. Bibuli, and G. Bruzzone, “Aluminim autonomous navi-

gator for intelligent sampling: The ALANIS project,” Sea Technol.,

vol. 50, no. 2, pp. 63–66, 2009.

[6] M. Bibuli, G. Bruzzone, M. Caccia, and L. Lapierre, “Path-following

algorithms and experiments for an unmanned surface vehicle,” J. Field

Robot., vol. 26, no. 8, pp. 669–688, 2009.

[7] M. Breivik and T. Fossen, “Guidance laws for planar motion control,”

in Proc. 47th IEEE Conf. Decision and Control, Cancun, Mexico, Dec.

2008, pp. 570–577.

[8] A. Aguiar, J. Almeida, M. Bayat, B. Cardeira, R. Cunha, A. H€ausler, P.

Maurya, P. Oliveira, A. Pascoal, A. Pereira, M. Rufino, L. Sebastiao, C.

Silvestre, and F. Vanni, “Cooperative control of multiple marine vehicles,”

in Proc. 8th IFAC Int. Conf. Manoeuvring and Control of Marine Craft,

Guaruj�a, Brazil, Sept. 16–18, 2009 [CD-ROM].

[9] M. Benjamin, J. Curcio, and P. Newman, “Navigation of unmanned

marine vehicles in accordance with the rules of the road,” in IEEE Proc.

Int. Conf. Robotics and Automation, 2006, pp. 3581–3587.

[10] M. Breivik. (2009). Formation control with unmanned surface

vehicles. Centre for Ships and Ocean Structures, Norwegian Univ. Sci.

Technol., Tech. Rep. [Online]. Available: http://www.ivt.ntnu.no/imt/

courses/tmr4240/literature/formation_control_usvs.pdf.

[11] M. Breivik, “Topics in guided motion control of marine vehicles,”

Ph.D. dissertation, Dept. Eng. Cybern., NorwegianUniv. Sci. Technol., 2010.

[12] T. Alves, L. Brignone, M. Schneider, and J. Kalwa, “Communication

infrastructure for fleets of autonomous marine vehicles: Concepts and first

results,” in Proc. 8th IFAC Conf. Manoeuvring and Control of Marine

Craft, 2009 [CD-ROM].

[13] J. Kalwa, “Final results of the European Project GREX: Coordination

and control of cooperating marine robots,” in Proc. 7th IFAC Symp. Intel-

ligent Autonomous Vehicles, 2010 [CD-ROM].

[14] A. Shafer, M. Benjamin, J. Leonard, and J. Curcio, “Autonomous

cooperation of heterogeneous platforms for sea-based search tasks,” in

Proc. Oceans, 2008, pp. 1–10.

[15] J. Curcio, T. Schneider, M. Benjamin, and A. Patrikalakis,

“Autonomous surface craft provide flexibility to remote adaptive oceano-

graphic sampling and modeling,” in Proc. Oceans, 2008, pp. 1–7.

[16] F. Arrichiello, H. Heidarsson, S. Chiaverini, and G. Sukhatme,

“Cooperative caging using autonomous aquatic surface vehicles,” in Proc.

IEEE Int. Conf. Robotics and Automation, 2010, pp. 4763–4769.

[17] M. Bibuli, L. Lapierre, and M. Caccia, “Virtual target based coordi-

nated path-following for multi-vehicle systems,” in Proc. 8th IFAC Conf.

Control Applications in Marine Systems, 2010 [CD-ROM].

[18] L. Lapierre, D. Soetanto, and A. Pascoal, “Adaptive, non-singular

path-following of dynamic wheeled robots,” Proc. 42nd IEEE Conf. Deci-

sion and Control, Maui, HI, Dec. 2003, vol. 2, pp. 1765–1770.

Marco Bibuli, Istituto di Studi sui Sistemi Intelligentiper l’Automazione, CNR, Genova, Italy. E-mail: [email protected].

Massimo Caccia, Istituto di Studi sui Sistemi Intelligentiper l’Automazione, CNR, Genova, Italy. E-mail: [email protected].

Lionel Lapierre, Laboratoire d’Informatique, Robotiqueet Micro-�electronique de Montpellier of CNRS, Montpellier, France. E-mail: [email protected].

Gabriele Bruzzone, Istituto di Studi sui Sistemi Intelligentiper l’Automazione, CNR, Genova, Italy. E-mail: [email protected].



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___

______________________________

________

_____

____________

_______

______

______________

_____

http://www.qmags.com/clickthrough.asp?url=http://www.ivt.ntnu.no/imt/courses/tmr4240/literature/formation_control_usvs.pdf&id=17487&adid=P102E2

http://www.qmags.com/clickthrough.asp?url=http://www.umv.ge.issia.cnr.it/video/vessel_following.html&id=17487&adid=P102E1











M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_____________

http://www.qmags.com/clickthrough.asp?url=www.case2013.org&id=17487&adid=P103A1

http://www.qmags.com/clickthrough.asp?url=www.case2013.org&id=17487&adid=P103A2

http://www.qmags.com/clickthrough.asp?url=www.ieee.org/t-ase&id=17487&adid=P103A3





STUDENT’S CORNER •

Innovation for Tomorrow’sNeeds:StudentActivities andCreativity at ICRA2012

By Laura Margheri and Ludo Visser

The 2012 IEEE InternationalConference on Robotics andAutomation (ICRA) provideda range of unique opportu-

nities. From meeting, socialization,and discussions among our student

groups to thechance to havelunch with lead-ers from research,education, andindustry, thisgathering of ourpeers and pro-fessionals was agreat experiencefor the 650 stu-dents in attend-ance this year.

The IEEE Robotics and Automa-tion Society’s (RAS’s) Student Activ-ities Committee (SAC) scheduleda series of social events amid the

tutorials, workshops, interactive ses-sions, and plenary sessions of ICRA.Jorge Cham, author of PhD Comics,which chronicles life in graduateschool in a humorous manner, wherePhD stands for “piled higher anddeeper,” gave an entertaining andhumorous presentation at theStudent Reception. Following thelaughter, more than 300 studentshad a chance to get Cham’s auto-graph. The reception opened withremarks from Nikos Papanikolo-poulos, ICRA general chair, andDavid Orin, RAS president, and wasplanned through the efforts of theLocal Arrangements Chair VolkanIsler, the Program Chair Paul Oh,and the Student’s Local CommitteeCochairs Daniel Lofaro and PratapTokekar.

An impromptu gathering of 50students for lunch at the Eagle Street

Grille immersed us in the history ofSt. Paul. While conversations rangedfrom our studies, research, andICRA, these topics were far fromreference of the mob-themed menuthat referred to the 1930s and thegangsters who gave the city a notori-ous reputation. Later that evening,we joined other ICRA attendees atthe Minnesota Science Museum,where we had a chance to explorehands-on displays and the featuredReal Pirates: The Untold Story of theWhydah from Slave Ship to PirateShip exhibit.

One of the most compellingevents was the Lunch with Leaders,which was aimed at enhancing therelationship between the studentsand leaders in robotics and automa-tion. It gave more than 120 studentsa chance to ask questions, get advice,and reach out to 12 mentors, including




Student reception photos.

•The goal of Student

Reviewer Program is

to introduce students

andyoung researchers

to the reviewing

process in a controlled

and supervisedway.

•


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






RAS past presidents and IEEE Presi-dent-Elect Peter Staecker. Topicsincluded the lifelong career benefitsprovided through involvement inRAS, and this luncheon was a per-fect example. While the Windowson the River room, where the eventwas held, featured grand views of the

river and city, the chance to meetand converse with such prominentpeople in the field was far moreimpressive and is something that wecan take with us through ourcareers. Thanks to all of the leadersfor participating and creating suchan amazing experience.

Seeing the sights was also a partof ICRA, so RAS sponsored theICRA 2012 Student Photo Contest,where winners received RAS mem-bership for 2013. After a week in St.Paul, more than 50 submissions werereceived and judged by RAS PastPresident Bruno Siciliano and RASAdministrator Kathy Colabaugh. Ihope you enjoy the photos and con-gratulations to the winners: TheEscape by Martin Felis at the WaveGenerator at St. Paul Science Museum,Perspective Informed Grasping forFruit of Knowledge Applications by

Austin Buchan, Lael Odh-ner, and David Rollinson atThe Walker Art Center inMinneapolis, and CharlieGoes to ICRA by Pedro Mor-eira, and Mariana Bernardesat Landmark Plaza, St. Paul.

SAC and representa-tives, like ourselves, alsoattended RAS meetingsthroughout the week. Underthe Membership ActivitiesBoard, we were a part of the

meeting sharing what we had accom-plished, reporting on ICRA eventsand our plans for the year ahead.With the objective of growing thecommunity of students and enhanc-ing our activities, our projectsinclude communications in IEEERobotics & Automation Magazineand the newsletter, relations withother IEEE Societies, events at morerobotics and automation conferen-ces, enhancements to the Web siteand social media, collaborationwith the Education Committee andIndustrial Activities Board, and de-veloping ideas for career enhance-ment. We also sat in a number ofother meetings, including the Admin-istrative Committee. By attendingsuch meetings, we were able to con-tact, update, and coordinate withother board members for endeavorsof the SAC.

I encourage you to take thechance to participate in RAS eventsand meetings to be prepared andready to become the new genera-tion of leaders in robotics andautomation.

Plans are underway for the IEEE/RSJ International Conference onIntelligent Robots and Systems(IROS) 2012 in October in Vila-moura, Portugal, so be sure to checkfor updates and more information byvisiting our Web site at http://wiki.ieee-ras.org/mab/sac and socialnetworks. We are looking forward towelcoming those of you joining us inVilamoura.

•

ICRA 2012 photo contest winner: TheEscape by Martin Felis at the WaveGenerator at St. Paul Science Museum.

ICRA 2012 photo contest winner:Perspective Informed Grasping for Fruitof Knowledge Applications by AustinBuchan, Lael Odhner, and DavidRollinson at The Walker Art Center inMinneapolis.

Students and RAS past presidents during the lunch with leaders.

ICRA 2012 photo contest winner: CharlieGoes to ICRA by Pedro Moreira andMariana Bernardes at Landmark Plaza,St. Paul. (continued on page 122)



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________________

____

http://www.qmags.com/clickthrough.asp?url=http://wiki.ieee-ras.org/mab/sac&id=17487&adid=P105E1





SOCIETY NEWS •

RASCelebrations andAwardWinners

IEEE International Conferenceon Robotics and AutomationThe 2012 IEEE International Conferenceon Robotics and Automation (ICRA)was held in Saint Paul, Minnesota, 14–18Maywithmore than 1,700 registrants.

Because of the record number of2,032 submissions from more than 40countries, the Conference EditorialBoard and the Senior Program Com-mittee were forced to make difficultdecisions in selecting papers to main-tain the quality and balance of thetechnical program. The use of iThenti-cate was introduced with the goal ofreducing overlap with previous papersubmissions. An aggressive approachwas followed for each paper to receiveat least two meaningful and construc-tive reviews. A total of 818 papers wereselected, and some of these papers werepresented in interactive sessions.

A number of new initiatives wereintroduced at ICRA. First, all presenta-tions with the authors’ approval werevideotaped and archived. These pre-sentations are available at www.ICRA2012.org. In addition, conferenceproceedings were included on a USBdrive and distributed to registrants.

Technical sessions were accompa-nied by special tutorials, where theorganizers invited experts to sharetheir knowledge. These tutorials werefree to participants and covered vari-ous robotics and automation topics ofcurrent research interest. The IEEERobotics and Automation Society(RAS) Technical Committees (TCs)supported and proposed some of theorganized workshops that corre-sponded to their areas of interest.

Supplementing the technical pre-sentations, the program was high-lighted by plenary talks delivered by

distinguished scholars: Bradley Nelson“Robotics in the Small,” Harry Asada“Bio-Bots,” and Jun-Ho Oh “HumanoidRobot HUBO II.”

ICRA 2012 had one of the largestexhibitions in our recent history

with representatives from industry,research institutions, and national lab-oratories (see “ICRA Exhibitors”). ICRA2013: Anthropomatics—Technologies forHumans will be held 6–10 May inKarlsruhe,Germany.Visitwww.ICRA2013.org for complete details.

Past Presidents CelebrationIn anticipation of the 30th anniver-sary of the founding of the IEEERobotics and Automation (RA)Council in 2013, the RAS organized aPast Presidents Celebration at ICRA,bringing all the living Society presi-dents and one of the past RA councilpresidents to St. Paul. They were ableto attend the conference sessions andsocial events and enjoy a special dinner




Bruno Siciliano, ICRA honorary chair and RAS past president, at the lunch with leaders.

Kazuhiro Kosuge, ICRA honorary chair and RAS junior past president, at the lunch with leaders.

ICRA leaders. From left: Raj Madhavan(exhibit chair), George Lee (honorarychair), Kazuhiro Kosuge (honorary chair),Paul Oh (program chair), and NikosPapanikolopoulos (general chair).


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


__________

__

__________

____

http://www.qmags.com/clickthrough.asp?url=www.ICRA2012.org&id=17487&adid=P106E2

http://www.qmags.com/clickthrough.asp?url=www.ICRA2013.org&id=17487&adid=P106E1





with current and past officers, theRAS publications editors, and otherSociety leaders. Many past presidentsalso participated in the student lunchwith leaders and chatted with studentconference attendees about the historyand future of robotics and automation.

During the dinner, past presidentsand other guests enjoyed a slide showprepared by Bruno Siciliano, includingthe photos of members from as farback as 1979. Each past president spokebriefly, recalling events of his term andremarking on the changes that havetaken place over the past 30 years.

George Bekey recalled how the lateGeorge Saridis, the first president ofRA Council, urged him to form a newIEEE Journal on Robotics and Auto-mation and worked to organize theRA Council to support the journaland the new ICRA. Antal “Tony”Bejczy described the arduous processinvolved in establishing the council in1984 and then forming a new Society.

In their remarks, the speakers recalledaccomplishments and crises for the lastthree decades. Among the many note-worthy accomplishments were the firstICRA held outside the United States(Nice, 1992), the first RAS presidentfrom outside the United States (ToshioFukuda, 1998–1999), the introduction ofthe first RAS Web site (1998), the intro-duction of IEEE Xplore in 2000, the firstfemale Administrative Committee(AdCom) member (Jill Crisman), theestablishment of IEEE Robotics & Auto-mation Magazine (RAM) (1994), IEEETransactions on Automation Science and

Engineering (2004), and the IEEE Inter-national Conference on Automation Sci-ence and Engineering (2005).

Challenges included the SARSepidemic, which required that ICRA2003 in Taipei to be postponed,the H2N1 virus, which caused theJapanese government to provide allattendees of ICRA 2009 with sur-gical masks, and the earthquake andtsunami in 2011, which personallyaffected many RAS members in Japan,including President Kazuhiro Kosuge.

The past presidents are part of arobotics history project, which is wellunderway. Indiana University histo-rian, Selma Sabanovic, conductedhour-long interviews with the pastpresidents. The interviews of morethan 30 robotics pioneers are availableat http://roboticshistory.indiana.edu/narratives with more to follow in thecoming months. Past presidents inattendance included the following:l RA Council, Antal Bejczy, 1987l RAS, Arthur Sanderson, 1989–1990l Norman Caplan, 1991l T.J. Tarn, 1992–1993l Richard Klafter, 1994–1995

l George Bekey, 1996–1997l Toshio Fukuda, 1998–1999l T.C. Steve Hsia, 2000–2001l Paolo Dario, 2002–2003l Richard Volz, 2006–2007l Bruno Siciliano, 2008–2009l Kazuhiro Kosuge, 2010–2011.

AwardsRAS recognized the outstandingachievements of the following awardwinners at the ICRA Awards Cere-mony on 17May 2012:l IEEE Robotics and Automation

Award was given to Bernard Roth, aprofessor at Stanford University andacademic director of the HassoPlattner Institute of Design (alsoknown as the d.school) for funda-mental contributions to robot kine-matics, manipulation, and design.This award, established in 2002, isthe highest honor given by RAS.

l “Human-Like Adaptation of Forceand Impedance in Stable andUnstable Interactions” by C. Yang,G. Ganesh, S. Haddadin, S. Parusel,A. Albu-Sch€aeffer, and E. Burdet(IEEE Transactions on Robotics,

The group of past presidents. From left: David Orin, Dick Volz, Antal Bejczy, GeorgeBekey, Richard Klafter, Paolo Dario, Arthur Sanderson, T.C. Steve Hsia, Kazuhiro Kosuge,Bruno Siciliano, T.J. Tarn, Norman Kaplan, and Toshio Fukuda.

Past President Antal Bejczy and AdCommember Oussama Khatib.

RAS Past Presidents Richard Klafter andNorman Caplan.

Robotics Auto Tech Field Awardpresented to Bernard Roth by IEEEDivision X Director Vincenzo Piuri.

•



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_______

http://www.qmags.com/clickthrough.asp?url=http://roboticshistory.indiana.edu/narratives&id=17487&adid=P107E1





vol. 27, no. 5, pp. 918–930, Oct. 2011)received the IEEE Transactions onRobotics King-Sun Fu Memorial BestPaper Award.

RAS Award RecipientsThe recipients of the RAS Awards areas follows:l Jean-Claude Latombe, Stanford

University, received the Pioneer inRobotics and Automation Awardfor pioneering contributions torobot motion planning and theapplication of robotics methodsto computational biology andmedicine and fundamental contri-butions to the foundations of con-trol of robots and teleoperators,and for contributions to roboticseducation.

l George Bekey, University of SouthernCalifornia, USA, was the recipient ofRAS George Saridis LeadershipAward in Robotics and Automationfor his leadership in the robotics andautomation community, realizedthrough innovative research, educa-tion, and service contributions.

l John Hollerbach, University of Utah,USA, received the RASDistinguishedService Award for his dedication,professionalism, and commitment totransparency in different tasks for theSociety, especially for conference andtechnical activities.

l Vijay Kumar, University of Penn-sylvania, USA, was the recipient ofRAS Distinguished Service Awardfor his leadership in improvingRAS conferences and his continu-ing service in promoting majorrobotics research initiatives.

l Pieter Abbeel, University of Califor-nia Berkeley, USA, received the EarlyAcademic Career Award for contri-butions to apprenticeship learningand deformable object manipulationand their application to autonomoushelicopter flight and surgical andpersonal robotics.

l Katherine Kuchenbecker, StanfordUniversity, USA, received the EarlyAcademic Career Award for contri-butions to haptic interfaces andtouch perception for robotic andtelerobotic systems.

l Rainer Bischoff, Kuka Laboratories,was the recipient of Early Govern-ment/Industry Career Award forleadership and outstanding contri-butions to the cooperation ofacademia and industry and formanaging and promoting signifi-cant technology transfer in the areaof industrial and service robotics.

l The most active Technical Commit-tee of RAS was awarded to theTechnical Committee on Safety, Secu-rity, and Rescue Robotics CochairsHideyuki Tsukagoshi, Andreas Birk,and Julie Adams for outstandingquantity, diversity, and quality ofactivities, such as helping to the disas-ter response and recovery in greatEastern Japan earthquake, organizinga rescue robotics camp and exercise,arranging many workshops and jour-nal special issues, and participatingactively in conferences, reviewing,and organizing special sessions.

l The winner of the RAS Chapter ofthe Year is IEEE Joint Chapter of

the RAS and IEEE Control SystemsSociety for outstanding contribu-tions to the local growth of roboticsand automation in Hong Kong,student programs and activities,and international relationship de-velopment with other regions,especially in mainland China. Wei-Hsin Liao is the chair of the IEEEHong Kong Section Joint Chapter.

ICRA 2012 Award WinnersThe award winners for the ICRA 2012are as follows:l Best Conference Paper: M. Hagi-

wara, T. Kawahara, T. Iijima, Y.Yamanishi, and F. Arai for thepaper titled “High Speed Microro-bot Actuation in a MicrofluidicChip by Levitated Structure withRiblet Surface”

l Best Manipulation Paper (endowedby Ben Wegbreit): Z. McCarthy andT. Bretl for the paper titled“Mechanics and Manipulation ofPlanar Elastic Kinematic Chains”

l Best Vision Paper (endowed by BenWegbreit): Michael J. Milford andGordon F. Wyeth for the paper titled“SeqSLAM: Visual Route-BasedNavigation for Sunny Summer Daysand StormWinter Nights”

l Best Automation Paper (sponsoredby United Technologies ResearchCenter): A. Stolt, M. Linderoth, A.Robertsson, and R. Johansson forthe paper titled “Force ControlledRobotic Assembly Without a ForceSensor”

l Best Medical Robotics Paper (spon-sored by Intuitive Surgical): A.Gosline, N.V. Vasilyev, A. Veera-mani, M. Wu, G. Schmitz, R. Chen,V. Arabagi, P.J. del Nido, and P.E.Dupont for the paper titled “MetalMEMS Tools for Beating-HeartTissue Removal”

l Best Service Robotics Paper (spon-sored by KUKA): H. Tsukagoshi, Y.Mori, and A. Kitagawa for thepaper titled “Fast Accessible RescueDevice by Using a Flexible SlidingActuator”

l Best Video Award: A.S. Boxer-baum, A.D. Horchler, K.M. Shaw,H.J. Chiel, and R.D. Quinn for the

Pieter Abbell and KatherineKuchenbecker each receive the EarlyAcademic Career Award.

Rainer Bischoff is the recipient of EarlyGovernment/Industry Career Award withRAS President David Orin.

•



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






paper titled “The Worm Turns . . .and Runs”

l Best Cognitive Robotics Paper(sponsored by CoTeSys): M. Ten-orth, A. Perzylo, R. Lafrenz, and M.Beetz for the paper titled “TheRoboEarth Language: Representingand Exchanging Knowledge AboutActions, Objects, and Environments”

l Best Student Paper: Ying Mao andSunil K. Agrawal for the paper titled“Transition from Mechanical Armto Human Arm with CAREX: ACable-Driven Arm Exoskeleton(CAREX) for Neural Rehabilitation”

l Outstanding Associate EditorAward was given to Nabil Simaan,Vanderbilt, USA, for his dedicatedservice to ICRA 2012 and RAS

l Best Reviewer Award was receivedby Aude Billard, Ecole Polytech-nique Federale de Lausanne,Switzerland, and Tim Bailey,University of Sydney, Australia,for their dedicated service toICRA 2012 and RAS.

Twin CitiesTeen Robotics EventRAS sponsored its second TeenRobotics Program at the ICRA 2012.The program, designed to give highschool students insight into the excitingchallenges and opportunities in the fieldof robotics, took place on 17May 2012.

RAS contacted local public schools,the first Robotics Organization, theExplorer Scouts, and the IEEE TwinCities Section to recruit motivated highschool juniors and seniors to attend.Even though mid-May is a busy timefor teenagers with proms, exams, andgraduation, RAS managed to recruit adozen teenagers and one teacher toparticipate in the six-hour program.

The teenagers learned about theexciting field of robotics during theirvisit. After Wesley Snyder of NorthCarolina State University gave anoverview on robotics, the studentswatched a technology, entertainment,design (TED) talk video featuringVijay Kumar of the University ofPennsylvania and his flying robots.Much to the students’ surprise, Kumarstrolled into the room and finished the

talk in person. Then, ICRA GeneralChair Nikos Papanikolopoulos of theUniversity of Minnesota spoke to thestudents about the robotics programsunderway in their home state.

The students also had the chanceto see some real robots in action. Theymet Steve Cousins, CEO, of WillowGarage, a company that developshardware and open-source softwarefor personal robotic applications. Wil-low Garage, the ICRA Robot Chal-lenge sponsor, provided several of itssophisticated, commercially availablePR2 robots for the challenge that tookplace the previous day.

Cousins demonstrated “Yester-day’s Sushi,” which featured robotsclearing and resetting a full-sizedtable. This reenactment of the MobileManipulation Challenge, also knownas “The Sushi Boat Challenge,” wasone of the ICRA competitions. Thestudents spoke with a group of gradu-ate students from the University of

Michigan who participated in theChallenge and saw the team’s robot.

Meeting with representatives frommany of the exhibitions, includingThe Defense Advanced ResearchProjects Agency, National Aero-nautics and Space Administration,iRobot, ABB, KUKA, and BarrettTechnologies, the students not onlywitnessed robots in action but also gotto operate them.

The event was originally scheduledto end at noon, but several of the stu-dents accepted an invitation to attendthe ICRA plenary talk on the Devel-opment Outline of the HumanoidRobot: HUBO II by Prof. Jun Ho Ohof the Korea Advanced Institute ofScience and Technology, Korea.Impressed with the cool robots as wellas the enthusiasm of the academicand industry professionals they met,the students were provided a greatopportunity to see robotics and auto-mation in action.

Winners of the Best Automation Paper.

Winners of the Best Medical Robotics Paper.King-Sun Fu Memorial Award for the IEEETransactions on Robotics Best Paper for 2011.

•



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






ICRA 2012 Robot ChallengeThe goal of the ICRA 2012 Robot Chal-lenge, sponsored by Willow Garage, wasto make the challenge accessible to allmembers of the ICRA community, tointegrate it tightly with the technicalaspects of the conference, and to encour-age as many participants as possible tobring their teams and participate.

Through the Sushi Challenge,state-of-the-art integrated perception

and human-scale manipulation wasdemonstrated. Participants fromBrown University, USA; KU Leuven,Belgium; TU Munich, Germany;University of California, Berkeley,USA; and Willow Garage were chal-lenged with clearing a table, setting atable, and serving from a rotatingtable with their own robots, WillowGarage’s PR2 robots, or KUKA’s You-Bot robots.

The Modular Robotic Challengesimulates an unexpected problem,where a robotic solutionmust be quicklydeveloped and deployed using onlyexisting resources. The intent of thisevent is to develop versatile robotic sys-tems and software that can be adaptedquickly to address unexpected events.

A hockey-playing robot namedJennifer and a team from the Auto-nomous Agents Laboratory at theUniversity of Manitoba, Canada, wonthe DARwin-OP Humanoid Applica-tion Challenge.

In the fourth year of this simula-tion-based challenge, which is de-signed to stimulate research inrobotics dealing with problemsrelated to mixed-palletizing and intra-factory package delivery and logistics,the Virtual Manufacturing Auto-mation Challenge winners includedMixed Palletizing Jacobs University,Germany, and Intrafactory MobilityUniversity of Zagreb, Croatia, with anHonorable Mention Mixed PalletizingDrexel University, USA.

AdCom HighlightsThe RAS AdCom met on 19 May2012. In addition to reports from theboards and committees, the volun-teers selected Singapore as the sitefor ICRA 2017. The general chairis I-Ming Chen from Singapore’sNational Technical University.

In an effort to expand RAS human-itarian efforts, funding was approvedto establish a Special Interest Groupon Humanitarian Technologies.Additionally, funds for three prizes tobe awarded to the 2012 AfricanRobotics Network (AFRON) Design

Mobile Microrobotics Challenge, Micro Assembly Task. Second Place: The U.S. Naval Academy.

Teen robotics.

•

•Call for Nominations: RAM Associate EditorsRAM is soliciting nominations for two new associate editors, tobegin in January 2013. The associate editors play an importantrole inmaintaining the caliber of the magazine by

l ensuring the quality of published articles by implementingreviews of technical features according to IEEE guidelines

l soliciting interesting and topical material articles for publi-cation in the magazine

l guiding the overall direction of the publication and pro-viding feedback from the readership through e-mail con-versations, teleconferences, and twice-yearly in-personmeetings held in conjunction with ICRA and the IEEE

International Conference on Intelligent Robots andSystems.

Associate editor terms normally consist of a one-year proba-tion period followed by two years of additional service ifperformance is satisfactory. Applicants should have a strongtechnical background and excellent English language skills.Nominations should include a resume (not to exceed three

pages), previous experience with publications as a reviewer orin other capacities, and areas of technical expertise. Pleasesubmit nominations as a single pdf file to Rachel O. Warnick [email protected] by 30 September 2012.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


____________







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


__________________

http://www.qmags.com/clickthrough.asp?url=www.ieeeghn.org&id=17487&adid=P111A1





Challenge winners were approved.AFRON is a community of institu-tions, organizations, and individualsengaged in robotics in Africa, seekingto promote communication and col-laborations that will enhance robotics-related education, research, and industryon the continent.

In accordance with the new guide-lines for establishing a TC, a review ofnewly established TCs will take placeafter 18months to determine continua-tion or termination of the committee.With regard to RAS fully owned publi-cations, RAM, IEEE Transactions onRobotics, and IEEE Transactions onAutomation Science and Engineering,initiatives were approved to implementthe iThenticate plagiarism checkingsoftware system for all three pub-lications and add a voting feature to

RAM digital articles to track articlepopularity.

For complete details of the AdCommeeting, please see the minutesavailable at http://www.ieee-ras.org/administration/minutes.html.

RAS WelcomesNew Chapters and TCsRAS is pleased to announce that severalnew Chapters have been approved,including the IEEE Huntsville SectionControl Systems Society and RAS JointChapter, Gary Kendrick, chair (effec-tive 9 February 2012) and the IEEEColombia Section (Tunja) ElectronDevices Society and RAS Joint Chapter(approved 27 April 2012).

The RAS Member Activities Boardwill be reviewing proposals for localChapter grants at their October 2012

meeting. Proposals must be submittedto [email protected] by 1 October to be con-sidered. Please use the template foundat www.ieee-ras.org/member/chaptersfor your proposal. A maximum ofUS$2,000 is available per proposal.

RAS TCs sponsor workshops, tutori-als, and special sessions at conferences;organize special issues of the magazine;the transactions, and other journals;and provide a focal point for research-ers interested in finding collaborators innew and emerging fields of research.

At the May Advisory Committeemeeting, the following committees wereapproved: Automation in Logistics,Chair Maria Pia Fanti; Smart Buildings,Chair Qianchuan Zhao; and SustainableProduction Automation, Chair JingshanLi. Please visit www.ieee-ras.org for com-plete details and the latest RAS news.

•ICRA ExhibitorsABBAdept MobileRobotsAldebaran RoboticsATI Industrial AutomationBarrett Technology Inc.Butterfly Haptics, LLCCambridgeUniversity PressCyberboticsDARPA ARMOutreach Team

Hokuyo Automatic Co., Ltd.InTech–open science|openmindsIntuitive Surgical, Inc.Kinova TechnologyMEKA RoboticsNASA/Johnson SpaceCenterNational Instruments

OPRosPaR Systems, Inc.ReconRobotics, Inc.RoadNarrows LLCROBOTIQRobotis Co., Ltd.SAGE Publishers Ltd.SCHUNKSimLab. Co. Ltd.

Springer PublishersSunrise Auto SafetyTechnologySynapticon, Inc.Syntouch LLCWillow Garage, Inc.youBot

•RAS AdCom Elections–Voting Is UnderwayElection schedule is from 17 August to 28 September. On 4 September, voting began for RAS members to elect six newmembersto the Society’s AdCom, to serve three-year terms beginning 1 January 2013. Voting will conclude 15 October. Members may voteelectronically, by fax or by postal ballot (on request). Postal ballots must be postmarked 15 October. In August, voting members(graduate students and higher grade members) should have received their AdCom Election information packages delivered by e-mail (or postal mail if requested or valid e-mails are not available) with the slate of candidates, their biographies, and their posi-tion statements. The candidate information is also posted on the RAS Web site at http://www.ieee-ras.org.Members may vote for candidates from any region. The candidates for the six positions are:

William (Bill) HamelUniversity ofTennessee, USA

Peter B. LuhUniversity of Connecticut,USA

Ning XiMichigan StateUniversity, USA

Jing XiaoUniversity of North Carolina,USA

Martin BussTechnische UniversitatMunchen, Germany

Darwin G. CaldwellItalian Institute ofTechnology, Italy

Maria Pia FantiPolytechnic of Bari, Italy

Danica KragicThe Royal Institute ofTechnology (KTH), Sweden

Jianwei ZhangUniversity of Hamburg,Germany

Li-Chen FuNational Taiwan University(NTU), Taiwan

Tae-Eog LeeKorea Advanced Instituteof Science and Technology(KAIST), Korea

Max Qing HuMengChinese University ofHong Kong, Hong Kong

Kazuhito YokoiNational Institute ofAdvanced Industrial Scienceand Technology (AIST), Japan

Don’t forget to vote.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


____________________

_________

http://www.qmags.com/clickthrough.asp?url=www.ieee-ras.org&id=17487&adid=P112E4

http://www.qmags.com/clickthrough.asp?url=www.ieee-ras.org/administration/minutes.html&id=17487&adid=P112E3

http://www.qmags.com/clickthrough.asp?url=www.ieee-ras.org/member/chapters&id=17487&adid=P112E2

http://www.qmags.com/clickthrough.asp?url=http://www.ieee-ras.org&id=17487&adid=P112E1







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________________________

http://www.qmags.com/clickthrough.asp?url=www.ieee.org/proceedings&id=17487&adid=P113A1





HISTORY •

George CharlesDevol, Jr.By Leslie Anne Ballard, Selma Sabanovi�c, Jasleen Kaur, and Stasa Milojevi�c

George Charles Devol, Jr.,opened his first manufactur-ing company when he wasjust 19 years old, in the

depths of the Great Depression. It wasthe first step in a lifelong commitmentto transforming factory efficiency andsafety. With his revolutionary conceptof universal automation—Unima-tion—he cofounded the first and larg-est robotics company in the world. Heheld 36 patents, including the firstindustrial robot. Sadly, he passed awayin August 2011 at the age of 99. Earlierthat year, he agreed to this, his finalinterview. He was robust, living inde-pendently in his own home, and driv-ing his own car; his recall of his careerand life was complete and precise. Hegenerously gave access to his privatephoto and document archive, and thephotographs included here are his. Allvisualizations of patent data are basedon information gathered from U.S.Patent and Trade Office records andGoogle Patents.

George Devol never had much inter-est in robots until he patented onethat would change the world. Henever saw Karel Capek’s play Ros-sum’s Universal Robots; he never con-sidered Isaac Asimov’s Three Laws ofRobotics, although these men were hiscontemporaries. In fact, he dislikedscience fiction, and said of himself,“I’m no writer, and I only read what Ineed to.” Devol humbly describedhimself as “just an inventor and abusinessman—in that order.”

It is ironic then, that in 1961 hepatented the first industrial robot: hisUnimate robot transformed manufac-turing, particularly the Americanautomobile assembly line. Devol wasa passionate problem solver, andhis “Programmed Article Transfer”robotic arm was his solution to theproblem of the tedious and oftendangerous tasks workers performedin factories and the inefficient manualsystems he encountered every day.

Riordan AcademyBorn in Louisville, Kentucky, in 1912,Devol was caught up in the technolog-ical revolution of his time, fascinatedwith all things mechanical. When hewas 15, he rebuilt the transmission inthe family car with no instruction. Hisfather, a railway traffic control consul-tant, recognized Devol’s talents andsent him to the prestigious RiordanPreparatory Academy in New York.There, he met “the kind of people youcouldn’t meet otherwise—sons of dip-lomats and heads of state, politicians,the very wealthy.” Comfortableamong the elite, he still developed apreference for the hands-on work ofthe inventor he would become. Rior-dan’s freestyle curriculum suitedDevol. “I wasn’t much of a student,and I didn’t like sports, but they letme run the electrical power station,and I read everything I could find onengineering and mechanics.”

Cinephone and TalkiesIn 1931, most Americans had justabout forgotten what a dollar billlooked like, as the Great Depressionstaggered the country. But for an opti-mistic 19-year-old Devol, it was time

to start a company. He persuaded hisparents that he should circumventcollege. “It was a shock for them. I’dbeen accepted and was all set to go toMIT, but my father finally gave in. Hesaid, ‘OK, go ahead and see if you cando it.’ And I did do it!”

Growing up, Devol had witnessedan unparalleled range of innovations:television, commercial aviation, peni-cillin, gyroscopes, arc welders, loud-speakers, analog computers, jetengines, and car radios. In addition,Ford’s assembly line had madeautomobiles affordable to the masses,and the world was agog over Ein-stein’s new Theory of Relativity. Of allthese advances, the one that capturedDevol’s imagination was “talkingpictures.”

Before the 1930s, motion pictureswere silent with brief subtitles for dia-log. But soon, talkies were a globalphenomenon. Reliable synchroniza-tion was difficult to achieve with earlysound-on-disc systems, so Devol, theproblem solver, devised his ownprocess for variable-area recording.

“Everybody was excited about‘talkies.’ I just approached investors,saying I had a new way of puttingsound on film. I got [US]$25,000 here,[US]$50,000 there, and pretty soon Ihad enough for my own company.And this was during the Depression! Ican’t figure out how I did it myself.”This power of persuasion would servehim well and often in his career.

Within a year of inception, Devolopened his first manufacturing plant,Cinephone, in Torrington, Connec-ticut. But he soon realized that hecould not compete with other sound-to-film ventures. “Being so young,





M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






I didn’t even think about who I wasup against; Western Electric, RCA,AT&T. It was impossible.”

Photocells and Vacuum Tubes“I started thinking about what else wecould do with all these photocells andvacuum tubes. I came up with the ideaof making photoelectric controls forelectric doors. And with that patent, Iformed United Cinephone Corpora-tion, as far as I know, the first indus-trial control company in the U.S.”Now ubiquitous, this was the world’sfirst automatic door opener. Helicensed it to Yale & Towne for their“Phantom Doorman” (Figure 1).

At the age of 26, with a new, largerplant in Long Island, Devol startedpatenting his ideas, beginning withOrthoplane lighting [1938, “Designof a Lighting Fixture” U.S. PatentD112,634], then registration controls,wireless control devices, and phono-graph amplifiers.

Early Barcode SystemEven an entrepreneur like GeorgeDevol will occasionally miss anopportunity. In 1935, Firestone Tireand Rubber Co. approached Devol todevise a package sorting system. Hecreated an optical barcode systemusing photocells, a forerunner to thesame system used by shipping compa-nies and retailers today. Consideringit simply as one component of thelarger system, Devol never patentedhis barcode system. “Now it’s bigbusiness, but then, we didn’t knowwhat we had. As far as I know, wewere using it long before anyone else.”

EvelynAt about this time, Devol had a blinddate with a beautiful Hunter Collegecoed named Evelyn Jahelka. It waslove at first sight; they married onNew Year’s Eve in 1938. For the next65 years, Evelyn was by Devol’s sidethrough good and times. To theirfour children, she was the backboneof the family, “particularly throughthe hard times when he was trying toraise money to launch the first indus-trial robot.” She supported Devol’s

creativity and even named his auto-mation system and his companywhen she coined the word“Unimation” from “universal” and“automation” (Figure 2).

Radar and the Robot BrainBy 1939, American manufacturingturned its focus to World War II.Realizing that United Cinephone wasnot big enough to handle militarycontracts, Devol sold his interest inhis company to explore new horizons.“I gave them a very good deal, but Ididn’t think it all the way through:‘Now what!?’”

In a real way, all of George Devol’sexperience before and during WorldWar II culminated in the creation of

the first industrial robot. Stints atRCA, Sperry Gyroscope, and Reming-ton Rand contributed to aspects of hisbrainchild.

Wanting to do his part for the wareffort, he approached the head of Sev-ersky Aircraft in Long Island. “I didn’tknow anything about airplanes, butI knew all about production.” Tohis surprise, by 5:00 that afternoonhe was the company’s head of pro-duction. However, he felt unchal-lenged, and within the year moved toSperry Gyroscope as manager of theSpecial Projects Department, develop-ing military radar devices and micro-wave test equipment.

Then came another missed oppor-tunity: “At Sperry, we made radarscanners. One day, I was up in theplane, testing a new scanner, andnoticed the unit was getting too hot. Italmost burning my hand. So, I turnedit off and right away it cooled. Istarted thinking along the lines of amicrowave oven, long before anyonepatented it, but I was too busy to focuson that. I never got a chance to applyfor a patent on that one. And look at ittoday!”

Radar defense technology in-trigued Devol, and he had some ideasto develop. In 1943, he approachedthe head of Auto Ordinance Com-pany, convincing him of the AlliedForces’ urgent need for radar counter-measure devices. “By the time I lefthis office, I had US$5 million to start anew division, and he would havegiven me just about any amount!” Heestablished General Electronic Indus-tries (GEI) in Greenwich, Connecti-cut. With Devol as general manager,the company grew to 3,000 em-ployees, one of the world’s largestmanufacturers of radar equipment.However, when the biggest militarycontract in history was offered, Devolbalked, knowing they could not fill it.Against his advice, GEI took the con-tract and Devol resigned.

Following GEI, there was a brief,uninspired stint at RCA, managingthe electronic products division, andit was then that Devol’s restless mindstarted to formulate the principles

•

Figure 1. One of Devol’s ideas forCinephone became the world’s firstautomatic door opener. He licensed it toYale & Towne, for their “PhantomDoorman.”

Figure 2. Married for 65 years, Evelynnamed Devol’s concept and company bycombining “universal” and “automation”and coining “Unimation.”



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






that would lead to his patent for thefirst industrial robot.

With the war winding down, Devolwas again looking for the Next BigIdea to develop. He had done someresearch into magnetic recording andnow turned his attention back to it.“There were all kinds of ac recordingtechniques, but no one was doing any-thing with dc recording. So I decidedto apply for patents on dc magneticrecording. This is recording at restrather than in motion. Normal mag-netic recording has a magnetic pulse,which is read as the tape passes themachine head, but it won’t workstanding still—something has tomove. My idea was the opposite—nocontinuous tape,” he said.

What happened next led him todevelop the brains of the Unimaterobot: In 1946, he patented his mag-netic recording system for controllingmachines and a digital playbackdevice. Needing to develop thisfurther, he approached retired Lt.Gen. Leslie Groves, who had recentlybecome a VP at Remington Rand.Groves had been the director of theManhattan Project, and Devol idol-ized him as a war hero. Groves offereda development team at Rand. Whilethere, Devol patented what he consid-ered the capstone of his career: amethod of high-speed printing usingmagnetic tape and one line of print

members. At 1,000 sheets per minute,it was the fastest printing press everbuilt at that time. The company’slegal department issued warningsto guard the secrecy of the project.Devol liked to say, “General Groveshad two top-secret projects: theatomic bomb and our printer.” In1957, Devol was awarded threepatents relating to printing at highspeed: U.S. Patent 2,811,102, U.S.Patent 2,811,101, and U.S. Patent2,918,864 (Figure 3).

The Teachable MachineWith his dc magnetic recording pat-ent licensed to Remington Rand,Devol’s team got started. What theycreated was groundbreaking: a teach-able machine. “We would hook up therecording system to a machine, a lathefor example. We turned out whateverparts we wanted, and, in the processof making them, we magneticallyrecorded all the lathe’s actions. Afterthat, the lathe could automaticallyproduce identical results. We appliedfor a patent for this “teachable ma-chine.” Then we thought, why notmake a manipulator? If we put a handon it, we can move parts around.”

UnimationBy this time, Devol was consideringhow to update manufacturing meth-ods. Seeing tons of manufacturing

tooling in scrap yards, made obsoleteby-product design changes, he con-cluded that this was “wasteful andno way to run a business.” Devol’ssolution was universal automation—automation that would not becomeobsolete but could adapt to productchanges and new products. He devel-oped the idea and obtained U.S.Patent 2,988,237 for “ProgrammableArticle Transfer” in 1956. This is hismost cited patent (Figure 4) and thefirst patent granted for a digitally pro-grammable robot arm, or as Devolreferred to it, “a more or less generalpurpose machine that has universalapplication to a vast diversity of appli-cations where cyclic digital control isdesired” (U.S. Patent 2,988,237). Heproposed this system as an alternativeto the existing manual and cam-controlled machines used for articletransfer.

Devol was disturbed by worker’sconditions: “factories were treacher-ous places.” Workers were oftenforced to operate in cramped, dirtyconditions, surrounded by toxicchemicals and dangerous machinery.He envisioned robots handling thesedangerous and monotonous tasks;devices that could perform repetitivetasks with greater precision andendurance than the human worker,do so at less cost, and be retrainablefor new tasks as production required.

Motor Generator

Capacity-ControlledTextile Press

Sensing Device for MagneticRecord

Magnetostrictive TypePrinting Device

Binary-Code-ControlledApparatus

Programmed Apparatus

Ferroresonant Device

Coaxial Line Coupling

Magnetic Process Control

Magnetic Storage andSensing Device

Programmed Article Transfer

Events in the Life of George C. Devol

Program-Controlled Equipment

Dual-Armed MultiaxesProgram-Controlled

Manipulators EncodingApparatus

1938 1943 1948 1953 1958 1963 1968 1973 1978 1983 1988

Design for a Lighting Fixture

Figure 3. Select patents granted to Devol from 1938 to 1984, along with their relative frequency of citation by other patentsexpressed by the height of the lines. Only patents that have at least ten citations are depicted.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






With his “Programmed ArticleTransfer” patent in hand, Devol waseager to develop his idea. Providen-tially, an executive at Manning,Maxwell & Moore (MM&M) inBridgeport, Connecticut, introducedDevol to a young Columbia Univer-sity graduate, Joseph Engelberger, achief engineer of MM&M’s AircraftProducts division. The two were animmediate symbiotic match: Devolhad patents, and Joe had a postwarmanufacturing division with nocontracts.

As with many innovations, secur-ing funding for the nascent roboticfield would be an uneven, dauntingprocess with years of ups and downs.“We made our first contract on thebasis of the magnetic recording pat-ents since robots were an expensiveproposition. MM&M licensed them,so we could do further developmentwork. In the meantime, DresserIndustries bought MM&M, and theykilled off Joe’s aircraft division. Weneeded to find a buyer to continuethat division,” Devol said.

Devol started a search for financialbacking for his venture. “I talked toNorm Schafler, the president of Con-solidated Diesel Electric Corporation(Condec). They bought the division,and in 1958, we set up a shop in

Danbury, Connecticut. In 1961, wesold the first Unimate to GeneralMotors (GM),” he said (Figure 5).

Although both Devol and Engel-berger initially worked in tandem tosell and promote the Unimate, thedivision of labor between the twobecame synergistic. Joe, the presidentof Unimation, a charismatic extrovertin his signature bow tie, promotedthe Unimate (and thus, robotics).Devol, the inventor, secured thecompany’s future, developing new

patents through his own patentcompany.

In Pity the Pioneer: The Rise andFall of Unimation, the as-yet-unpub-lished account of Unimate design byengineer George E. Munson, “GeorgeDevol had learned long ago thatownership of patents is a valuableasset from which we benefited hand-somely. They protected our intellectualproperties and helped us develop astrong licensing position for a numberof years.”

•

Sapal, Societe Anonyme DesPlieuses Automatiques

Companies That Cited the “Programmed Article Transfer”

Mitsubishi Jukogyo KabushikiKaisha

Advanced ManufacturingSystems, Inc

Rimrock Corp.

Kao Corp.

Virgina Intl. Terminals

Texas Instruments Inc.

Shibuya Kogyo Co. Ltd.

Hitachi, Ltd.

Kobe Steel, Ltd.

Molins PLC G.D. Societa per Azioni

American Telephone & Telegraph Co.

Memeison Medical, Education & Res Foundation

STIWA-Fertigungstechnik Sticht GesmbH

Murata Kikai Kabushiki Kaisha

1974 1978 1982 1986 1990 1994 1998

Figure 4. Various companies have referenced Devol’s 1961 patent on “Programmed Article Transfer” over the years, including manycompanies from Japan.

Figure 5. Devol with his engineers and the first Unimate in the Danbury, Connecticutworkshop, circa1960.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






In fact, after being granted theUnimation patent in 1961, Devol con-sistently continued to produce in-ventions relating to robotics andautomation (see Figures 3 and 6). In1966, he was awarded three patentsrelating to robotics: a precisely con-trollable and reproduceable microma-nipulator (U.S. Patent 3,233,749), aprogrammed article-handling device(U.S. Patent 3,279,624), and a co-ordinated conveyor and programmedapparatus (U.S. Patent 3,283,918).The 1966 patents presented an im-provement on the digital systemdescribed in Devol’s 1961 patent byproviding an arm with a gripper thatcan produce in-and-out longitudi-nal motion, sweep through a hori-zontal angle, and swing through avertical arc using hydraulic actua-tors; it was also controlled using ateachable analog mechanical pro-gram controller that allowed forquick, easy, and accurate produc-tion of new programs (USPTO Pat-ent 3,279,624).

Devol’s 1967 patents on pro-grammed apparatus continued toextend his work in robotics by provid-ing an apparatus that can be taughtthe desired programs by operating theapparatus under manual control (U.S.Patent 3,306,471). A second patentproduced that year described a pro-gram-controlled apparatus that couldperform any number of possibleactions in response to environmentalconditions outside of itself (such as

the weight, color, and thickness ofarticles it is handling) that it has thecapability to detect rather than in a setpreprogrammed order (U.S. Patent3,306,442).

In 1958, with a six-man crew,Devol began developing his robotin the Danbury shop. According toMunson:

We were to soon learnthat the trailing technolo-gies were everything thatwent into such a device—digital control, memory,andmuscle. Thus, our taskwas not only to developthe concept, but to designand build all that madeit tick! The engineeringdesign tasks we werefaced with included thedevelopment of1) a digitally controlled

system based on thebinary numbering sys-tem (remember, thiswas 1956!)

2) a nonvolatile solid-statememory system (whichdidn’t exist)

3) shaft position digitalencoders, preferably, opti-cal high-speed perform-ance (which also, as itturned out, didn’t exist)

4) a custom-designed high-performance digital servocontrol system capable ofdynamic control with a

wide range of payloads(e.g., a few pounds to acoupleofhundredpoundsat varying torques)

5) high-performance hy-draulic servo valves(thank goodness forthe military)

6) self-contained electricaland hydraulic powersupplies

7) an attractive envelopeandmyriad other details.

Under Devol, we devel-oped a ferroresonant sen-sor, the basis for a self-styled memory system,patented as Dynastat. Wealso needed an opticalshaft position encoder toprovide the necessaryposition feedback to closethe loop between the robotarm’s actual position andits command positions. By1965, we had perfected anoptical Gray code encoderwe called Spirodisk.

[For a detailed engineering ac-count of the development of theUnimate robot, see George E. Mun-son’s “The Rise and Fall of Unima-tion,” Robot Mag., Sept./Oct. 2010,published in the articles section atwww.botmag.com.]

“Just Call It a Robot”At first, Engelberger was disinclined touse the word robot. Fearing it would

•

Figure 6. Devol’s patents were classified under 47 different U.S. Patent and Trademark Office (USPTO) classes. This “wordle” depictsthe frequency of usage of these classifications in Devol’s patents, in which the size of a word corresponds to its relative frequency ofuse. The classifications often used by Devol are “material or article handling (n = 18), “robots” (n = 17), “electricity: motive powersystems” (n = 11), and “dynamic magnetic information storage and retrieval” (n = 6).



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


http://www.qmags.com/clickthrough.asp?url=www.botmag.com&id=17487&adid=P118E1





sound like hokey sci-fi to prospectiveclients, he preferred manipulator.Devol believed otherwise, telling him,“If you want to sell something, youhave to give it a name people recog-nize. Nobody knows what a manipu-lator is. Just call it a robot,” saidDevol. And so they did.

As Unimation grew, with opera-tions in Europe and Japan, Joe Engel-berger traveled the world furtheringthe robotic cause. He and the Unimateappeared on The Tonight Show withJohnnie Carson, The Merv GriffinShow, and Walter Cronkite’s TheTwentieth Century. They were atevery trade show and expo. His omni-presence gained him the title “thefather of robotics.” Devol takes a prac-tical view of this: “He deserves a lot ofcredit. He saw my vision when othersdidn’t. People can call him the ‘father,’but I’m the inventor. The patent officesays so” (Figure 7).

Unimates as CoworkersThe automotive industry was drivingthe economy in the late 1950s, soEngelberger and Devol concentratedtheir energies there. They first tried tointerest the Ford Motor Company inthe robot. They submitted the specifi-cation for the Unimate to Ford, whereit got the attention of Del Harder, whois credited with coining the word auto-mation in 1947. Harder proclaimedthat he could use thousands of them.But FoMoCo itself did not see thepotential until much later. Engelbergerwas also cultivating executives at GM,and in 1961, Unimate number 001was delivered to GM’s diecasting plantin Trenton, New Jersey (Figure 8).With the perception that robots woulddisplace factory workers, the Unimatewas not going to be a popular new-comer, and the consensus among thediecast machine operators was that therobot was just a curiosity, destined tofail. However, aside from spot welding,no other industry has encouraged theproliferation of the industrial robot likediecasting. Eventually, some 450 Unim-ate robots were employed in diecasting.

In 1966, with nearly 100 Uni-mates, GM opened its state-of-the-art

Lordstown, Ohio, plant, producing110 cars per hour—twice the rate ofany plant then in existence. Otherauto manufacturers soon followedsuit and installed Unimates: Ford,Fiat, Chrysler, Nissan, and Volvo, aswell as other industries worldwide.

At first, autoworker unions foughtagainst the use of robots, calling themman replacers. Unimation did its partto alleviate these concerns, trainingthese same union workers to trouble-shoot and repair robots. As robotseliminated dangerous jobs, there wasno denying that they performeda valuable service and were soonwelcomed.

Devol kept up the pace, develop-ing new applications for Unima-tion’s automated system and itsrobots. The year 1970 was one of hismost productive periods: he wasgranted four more patents relating torobotics, including a patent for anorientation station to check theright-side up position of an articlebeing transferred and manipulatingit into the correct positioning, anautomatic apparatus for servicing

die-casting machines, and two work-heads for an automatically con-trolled arm.

The Rise and Fall of UnimationFor several years, as the first and larg-est robotic company in the world,Unimation was the industry leader.But with competition from othercompanies such as Cincinnati Mila-cron, Asea, and GMF Robotics, and afuriously booming robotics industryin Japan, sales dropped. Also, whilethere was high demand for electric-driven robots, the Unimate washydraulic, a fact that would mark thefinal blow for the company’s for-tunes. Says Devol, “we had to dosomething. We were getting swamp-ed!” He wanted to find a way to refi-nance the company, a buybackarrangement.

As he had done years before withMM&M’s aircraft division, Engel-berger went looking for someone tobuy and save his company. In 1982,Westinghouse paid US$107 million,buying Unimation, but keepingEngelberger as president. Devol wasagainst the deal, but acquiesced.Unimation never again showed aprofit, and in 1988, Westinghousesold what little was left of it to Staubliof France.

Devol ResearchAfter the demise of Unimation, Devolcontinued with Devol Research, hisown robotics/automation systemsconsulting company. He focused onhis love of boats, running a successfulboat brokerage in Fort Lauderdale,spending summers with his family inConnecticut and winters in Florida.Sometime in his 90s, he got hookedon day trading, manning three ormore computers at a time to keep upwith the ever-changing stock market.He would enthusiastically share hiscan’t-miss methods and give invest-ment tips. His warm parting hand-shake came with one admonishment:“Now don’t write anything floweryabout me. I don’t like compliments.Just the facts. The facts are goodenough.”

•

Figure 7. In a 1962 promotionalappearance on CBS’s The TwentiethCentury with Walter Cronkite, Unimationcofounders Joe Engelberger and GeorgeDevol enjoy a drink served up by theirfavorite bartender—the Unimate.

Figure 8. An early Unimate at work.



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






WOMEN IN ENGINEERING •

GrowingAround theGlobeBy Xiaorui Zhu

It is with great pleasure that I writethe first “Women in Engineering(WIE)” column in IEEE Robotics &Automation Magazine. This is also

the first year of my term as a WIE liai-son for the IEEE Robotics and Auto-mation Society (RAS). Special thanksfor all the efforts of our previous liai-sons Mihoko Otake and Aude Billardin making this all happen.

With the membership of womenwithin the IEEE growing around theglobe, the IEEE WIE was officiallyformed in 1994 and is now a commit-tee of the IEEE Member and Geo-graphic Activities Board. Devoted topromoting the advancement ofwomen in all IEEE technical fields aswell as encouraging youth to pursuecareers in science and engineering,the mission is to facilitate the recruit-ment and retention of women intechnical disciplines globally such thatIEEE women and men can collectivelytake advantage of their diverse talentsfor technical innovation to benefit thehumanity. Women’s membership inthe IEEE from locations outside ofthe United States is rapidly growingand will likely double this decade,showing an increase of more than4,000 members since 2008.

Under the direction of the Mem-bership Activities Board, there is anRAS WIE group. The current RASfemale membership is approximately7%, and the growing, comparatively,overall IEEE female membership isaround 3%. Within RAS, women holdvaluable roles, serving as Administrative

Committee members, Tech-nical Committee chairs,and more.

With a broad goal ofstrengtheningwomen’s com-munication within RAS,WIE has been working to makeconnections both virtually and face toface. To share and disseminate infor-mation, Mihoko Otake set up an e-mail list that reaches 714 RAS WIEmembers. A second list was createdfor communication among the RASWIE organizers. If you are notincluded, please join our e-mail alias bycontacting us at [email protected]. Most recently, the RAS WIEgroup on Linkedin In was created.

Another importantinitiative of the group isthe WIE Luncheon heldduring the IEEE Interna-tional Conference on Ro-

botics and Automation(ICRA) and the IEEE Interna-

tional Conference on IntelligentRobots and Systems (IROS). Begin-ning at ICRA 2007, prestigious femaleprofessionals have participated, sup-ported, and presented these successfulevents. Nancy Amato, Leslie PackKaelbling, Lynne Parker, and MeloneeWise are a few of the notable womenwho have shared their experiencesand advice. Most recently, at ICRA2011 and 2012, Willow Garage has




(a) (b)

(c) (d)

(a) (b)

(c) (d)

WIE Luncheon at (a) and (b) ICRA 2010 and 2011. (c) and (d) WIE Luncheon at ICRA2012. Guest speakers at the luncheon: Lynne Parker, professor at the University ofTennessee and Melonee Wise, manager of Robot Development at Willow Garage,addressed the nearly 50 women in attendance.


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_____________






IEEE Transactions on

Automation Science andEngineering

CALL FORPAPERSSpecial Issue

IntegratedOptimizationof Industrial Automation

The central theme of the Special Issue will be emerging opportunities and future directions in automation scienceand engineering for the whole production lines widely seen in various process industries. Examples are the auto-mation in steel making, mineral processing, metal processing, papermaking and petro-chemical plants. The goalsof the special issue are to

(1) present the state-of-the-art research in science, engineering and methodologies for automation for the wholeproduction lines, and

(2) provide a forum for experts to disseminate their recent advances and views on future perspectives in this field.

The special issue aims to publish original, significant and visionary automation papers describing scientific methods andtechnologies that improve planning and scheduling, product quality, production efficiency and energy consumptions. Inte-grated automation will also be included together with condition monitoring and fault tolerant control for complex indus-trial processes. Submissions of scientific results from experts in both academia and industry worldwide are stronglyencouraged. Topics to be covered include, but are not limited to,

Planning and scheduling for the whole production lines

Plant-wide operational automation and optimization

Data drivenmodeling and operational automation

High volume data reduction

Multi-objective operational optimization

Hybrid computer networked control systems for DCS architecture

Fault diagnosis and prediction for the whole production lines,

Prognosis and heathmanagement for the whole production line

Integrated and collaborative fault tolerant control

Product quality monitoring for the whole lines

Dynamic performance assessment for whole production lines

LeadGuest Editor: Tianyou ChaiPaper SubmissionDeadline: 15 January 2013

Publication: Fall/Winter 2014

Digital Object Identifier 10.1109MRA.2012.2206674



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM






become an important sponsor andindustrial partner. The involvementof the industry is expected to help ourfemale members widen the visionin terms of research and careerdevelopment.

I believe this is just a start for WIE-RAS. Strongly supported by RASmembers like David Orin, StefanoStramigioli, Nancy Amato, LynneParker, and Peter Luh, approval was

given to hold more WIE activities.Through the participation of ourWIE organizing committee, whichcurrently consists of 14 seniorfemale professionals from all overthe world and student volunteers,we are confident in serving thefemale members of our Society innew ways.

I strongly encourage more femalemembers from academia, industry,

and students to join in and speak outfor women engineers in this column.Remember, this is YOUR home (see“Birds of a Feather”).

Please join us at these upcomingWIE activities including the WIELuncheon at IROS 2012, ICRA 2013,and IEEE Conference on AutomationScience and Engineering (CASE)2013.

•

Birds of a FeatherThe Birds of a Feather Women’s Luncheon was held on 15 May2012. As a social event in conjunction with ICRA, the luncheonoffers young female researchers the opportunity to gather andconverse with their peers as well as senior robotics and auto-mation professionals. In an effort to grow the RAS WIE commu-nity and foster interaction, attendees were invited to join the

WIE e-mail list. The hope is that future communications to theWIE community provide a platform to continue discussionsinvolving career development, research experiences, andmore.Xiaorui Zhu, WIE liaison, organized the luncheon, sponsored

by the Member Activities Board and WIE.For more information visit www.ieee-ras.org/mab/wie.

•STUDENT’S CORNER (continued from page 105)

Student Reviewer ProgramThe goal of Student ReviewerProgram is to introduce studentsand young researchers to thereviewing process in a controlledand supervised way. With thisaim, we are organizing partici-pation rounds, in which studentsare paired with experiencedmentors in reviewing a confer-ence paper.

Introduced in conjunction with ICRA2012, we saw a successful round of theStudent Reviewer Program (SRP),where 13 students reviewed confer-ence paper submissions with the helpof four experienced volunteer men-tors. Following the success, a new SRPparticipation round was organized inconjunction with the IEEE Interna-

tional Conference on BiomedicalRobotics and Biomechatronics (Bio-Rob) 2012, held in June in Rome.

In this round, nearly 50 studentsapplied. Due to the overwhelmingresponse, there were not enoughmentors to support all the students.Ultimately, 30 students were selectedon the basis of motivation and re-search interests. These students werethen paired up with ten mentors tohelp with their review of BioRobsubmissions.

After having submitted all studentreviews, they were compared with theofficial conference reviews of associateeditors, and the students were gradedby their mentors based on a personalevaluation form and the quality oftheir reviews. Almost all studentsdelivered high-quality reviews, withan average grade of 7.8 (on a scale of1–10). The reviews revealed that the

students are typically thorough andhave a tendency to be a little stricterin grading the papers than their asso-ciate editors.

We also asked participants to com-plete a short survey about their ex-periences. Overall, students weresatisfied with the opportunity SPRprovided to improve their skills, andthe participation round was valued at4.2 (on a scale of 1–5).

After the two SRP participationrounds, we received many requestsasking to see the official reviews, andwe will do our best to provide theopportunity. In the near future, wehope to integrate the SRP with Paper-Plaza, allowing more flexibility. Thiswould give the students a chance tolearn from the other reviews aswell. Stay tuned for more SRP athttp://sites.ieee.org/ras-srp/.Digital Object Identifier 10.1109MRA.2012.2213903




M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


http://www.qmags.com/clickthrough.asp?url=http://sites.ieee.org/ras-srp/&id=17487&adid=P122E2

http://www.qmags.com/clickthrough.asp?url=www.ieee-ras.org/mab/wie&id=17487&adid=P122E1






M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


______________

http://www.qmags.com/clickthrough.asp?url=www.ieee.org/learning&id=17487&adid=P123A1





Turning Point (continued from page 128)

EG: One of the tasks requires arobot to drive a vehicle. Why is thatpart of the challenge, and is that away of imposing human size andfeatures to the robot?GP: That was not the intention. It isreally a question of available tools. As anexample, in the Fukushima disaster, theyhad fire trucks on hand, but after the firstreactor explosion, they could not havepeople drive those fire trucks because thecontamination was high. We want theserobots to really be compatible with alltools, from earth-moving machinery allthe way down to a screwdriver. Weknow that in a disaster, presently, mostof those tools are not going to be set upto be driven by a robot or teleoperated—they are mostly going to be the tools thatare at hand, and that includes vehicles. Ifyou look at [a typical human] environ-ment, the number of vehicles is verylarge, so they are what we consider a toolthat is a resource in a disaster that tendsto get used. Notice that this is not arepeat of the other DARPA challengesthat developed cars that drove by them-selves. The robot will be under supervi-sory control from a person, and so it isnot that we expect the robot to drive ona path all on its own. This is to show itsdexterity in using various tools andvehicles are within that set of tools. Nowthere is a second benefit that we get fromit, which has to dowith the power supplyfor the robot itself. We are not going todisallow tethers to power the robot, sothe vehicle can be amovable base that therobot can operate a certain distance from.

EG: And is a robot allowed todivide its body into more than oneunit to complete tasks? Can a robotleave parts of itself behind?GP: Robots competing in the chal-lenge cannot leave parts behind. Wewant to give maximum freedom to theperformers to choose the topology ofthe robot that they think is best. So ifduring an event, it is best to have a smallpart move off and then perform a taskand then come back, that is perfectlyfine. But we want to exclude teamsdeveloping a robot that is specializedfor each task and then just using one ata time for each. So, it is a one-robotchallenge, but the robot can split apartand come back together if desired.

EG: In the program, some teamswill get to test their simulation soft-ware on an actual robot, which theU.S. firm Boston Dynamicswill buildfor DARPA. But how can you be surethat this DARPA robot will be capa-ble of performing all the tasks thatare part of the challenge? It looks likeyou are trying to solve the challengethat you are proposing . . .GP: We are not solving the challengeby developing a robot for the softwareteams. The robot itself is only part of thechallenge. As an example, a robot thathas two legs that walks on its own anddoes the balancing task, that has beendone; it is been done by the Japaneseand by several U.S. groups also. But thatis not what the challenge is. The chal-lenge is going to be, given these

particular tasks, where does the robotplaces its feet, where does it places itshands, how does it turns the valve, andhow does it opens the door. The loco-motion part of it, which is to balanceand decide where the feet are going to beplaced to walk forward, that is actuallynot the hard part. In particular, how doyou make best use of the human [opera-tor] in this team, to give the supervisorycommands to the machine, to say, grabthat handhold, turn that valve, pick upthis bolt those are themore difficult parts.

EG: Still, from the scenario, it lookslike the technology required is toofar off, and though you might beable to have a robot do one or twotasks, performing them all with asingle robot seems really far-fetched.GP: We think that it is actually“DARPA hard,” but not an impossiblething to do. And the reason that we arespending the funds on this is actually topush the field forward and make thiscapability a reality. We are also trying towiden the supplier base for the capabilitythat would help here. So, we picked apretty hard goal that is absolutely true. Itis a goal that has a lot of risk, but a lot ofreward as well, and that is really thetheme of what DARPA tries to do. If welook at the driverless-car world beforethe DARPA Challenges occurred, therewere a lot of research efforts that showedthe cars moving a small distance downthe road, on a curve, and maybe recog-nizing some fraction of the time wherethey were with respect to the road. And Ithink that the previous challenges reallypushed the field forward to the statewhere now other firms have picked thisup and are making those cars. Some day,not too far from now, we will just get inour car and sit and talk to the personwho is next to us and not worry abouthow to drive. And that would be anamazingly great thing. I expect the samesort of thing will happen with the newchallenge we are launching.

A version of this interview appearedoriginally in IEEE Spectrum’s roboticsblog Automaton.

•

Figure 2. Illustration of a disaster response scenario part of the DARPA RoboticsChallenge. The robot on the right uses a power tool to break through a wall, and theone on the left turns a valve to close a leaking pipe. (Image courtesy of DARPA.)



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


________________________________________

http://www.qmags.com/clickthrough.asp?url=http://www.icra2013.org&id=17487&adid=P125A1





CALENDAR •

20129–11 JulyReMAR 2012: 2nd ASME/IEEEInternational Conference onReconfigurable Mechanisms andRobots. Tianjin, China. http://www.ReMAR2012.com

11–14 JulyAIM 2012: IEEE/ASME InternationalConference on Advanced IntelligentMechatronics. Kaohsiung, Taiwan.http://www.aim2012.org

14–15 JulyICBM: BMI International Confer-ence on Brain-Mind. East Lansing,Michigan, USA. http://brain-mind-institute.org/program-summer-2012.html

25–27 JulyCCC 2012: Chinese Control Confer-ence. Hefei, China. http://ccc.amss.ac.cn/2012/en/

28–31 JulyICINCO: 9th International Confer-ence on Informatics in Control,Automation, and Robotics. Rome,Italy. http://www.incinco.org

5–8 AugustICMA 2012: IEEE InternationalConference on Mechatronics andAutomation. Chendu, China. http://www.ieee-icma.org/

15–17 AugustICAL 2012: IEEE InternationalConference on Automation andLogistics. Zengzhou, China. http://ieee-ical.net/home

20–23 AugustSICE2012: 51st Annual Conferenceof the Society of Instrument andControl Engineers of Japan. Akita,Japan. http:www.sice.or.jp/sice2012

20–24 AugustIEEE-CASE 2012: IEEE Conferenceon Automation Sciences and Engi-neering. Seoul, Korea. http://www.case2012.org

27–30 AugustMMAR 2012: 17th InternationalConference on Methods and ModelsinAutomation andRobotics. Miedzyz-droje, Poland. http://www.mmar.edu.pl/

13–15 SeptemberMFI 2012: IEEE International Confer-ence on Multisensor Fusion and Inte-gration for Intelligent Systems.Hamburg, Germany. http://www.mfi-2012.org

7–12 OctoberIROS 2012: IEEE/RSJ InternationalConference on Intelligent Robots andSystems. Vilamoura-Algarve, Portugal.http://www.iros2012.org

17–21 OctoberICCAS: 12th International Confer-ence on Control, Automation andSystems. Jeju Island, Korea. http://2012.iccas.org

4–7 NovemberMHS: International Symposium onMicro- NanoMechatronics andHuman Science. Nagoya, Japan.http://www.mein.nagoya-u.ac.jp/mhs/MHS2012-Top.html

5–8 NovemberSSRR: International Symposiumon Safety, Security, and RescueRobotics. College Station, Texas, USA.

26–28 NovemberURAI: 9th International Conferenceon Ubiquitous Robots and AmbientIntelligence. Daejeon, Korea. http://www.kros.org/urai2012

29 November–1 DecemberHumanoids 2012: IEEE-RAS Inter-national Conference on HumanoidRobots. Osaka, Japan. http://www.humanoidrobots.org/humanoids2012/

5–7 DecemberICARCV 2012: 12th InternationalConference on Control AutomationRobotics and Vision. Guangzhou,China. http://www.icarcv.org/2012

7–9 DecemberICIES: 1st International Conferenceon Innovative Engineering Systems.Alexandria, Egypt. http://www.ejust.edu.eg/ies2012

11–14 DecemberRobio2012: IEEE InternationalConfer-ence on Robotics and Biomimetics.Guangzhou, China. http://www.ualberta.ca/~robio12

16–18 DecemberSII2012: IEEE/SICEInternationalSym-posiumonSystemIntegration.Fukuoka,Japan.www.si-sice.org/SII2012/

20133–6MarchHRI: ACM/IEEE International Con-ference on Human-Robot Interac-tion. Tokyo, Japan.

4–7 AugustICMA2013: IEEE InternationalConfer-ence onMechatronics and Automation.Takamatsu, Japan.

17–20 AugustCASE2013: IEEE International Con-ference on Automation Science andEngineering. Madison, Wisconsin,USA.





M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_____

_________________

__________

_______

_________

______________

____

____

________

________

____

____________

http://www.qmags.com/clickthrough.asp?url=http://www.kros.org/urai2012&id=17487&adid=P126E15

http://www.qmags.com/clickthrough.asp?url=www.sice.or.jp/sice2012&id=17487&adid=P126E8

http://www.qmags.com/clickthrough.asp?url=http://ieee-ical.net/home&id=17487&adid=P126E7

http://www.qmags.com/clickthrough.asp?url=http://www.mein.nagoya-u.ac.jp/mhs/MHS2012-Top.html&id=17487&adid=P126E14

http://www.qmags.com/clickthrough.asp?url=http://www.ieee-icma.org/&id=17487&adid=P126E6

http://www.qmags.com/clickthrough.asp?url=http://2012.iccas.org&id=17487&adid=P126E13

http://www.qmags.com/clickthrough.asp?url=www.si-sice.org/SII2012/&id=17487&adid=P126E20

http://www.qmags.com/clickthrough.asp?url=http://www.icinco.org&id=17487&adid=P126E5

http://www.qmags.com/clickthrough.asp?url=http://www.iros2012.org&id=17487&adid=P126E12

http://www.qmags.com/clickthrough.asp?url=http://www.ualberta.ca/~robio12&id=17487&adid=P126E19

http://www.qmags.com/clickthrough.asp?url=http://ccc.amss.ac.cn/2012/en/&id=17487&adid=P126E4

http://www.qmags.com/clickthrough.asp?url=http://brain-mindinstitute.org/program-summer-2012.html&id=17487&adid=P126E3

http://www.qmags.com/clickthrough.asp?url=http://www.ejust.edu.eg/ies2012&id=17487&adid=P126E18

http://www.qmags.com/clickthrough.asp?url=http://www.mfi-2012.org&id=17487&adid=P126E11

http://www.qmags.com/clickthrough.asp?url=http://www.aim2012.org&id=17487&adid=P126E2

http://www.qmags.com/clickthrough.asp?url=http://www.icarcv.org/2012&id=17487&adid=P126E17

http://www.qmags.com/clickthrough.asp?url=http://www.mmar.edu.pl/&id=17487&adid=P126E10

http://www.qmags.com/clickthrough.asp?url=http://www.ReMAR2012.com&id=17487&adid=P126E1

http://www.qmags.com/clickthrough.asp?url=http://www.humanoidrobots.org/humanoids2012/&id=17487&adid=P126E16

http://www.qmags.com/clickthrough.asp?url=http://www.case2012.org&id=17487&adid=P126E9





AD INDEX•

The Advertisers Index contained in this issue is compiled as a service to our readers and advertisers: the publisher is notliable for errors or omissions although every efforts is made to ensure its accuracy. Be sure to let our advertisers knowyou found them through IEEE Robotics& Automation Magazine.

Advertiser Page URL Phone

Adept Mobile Robots CVR2 www.mobilerobots.com þ1 603 881 7960

ATI Industrial Automation 13 www.ati-ia.com/ras

Barrett Technology Inc. CVR4 www.barrett.com þ1 617 252 9000

Butterfly Haptics, LLC 15 http://butterflyhaptics.com

Cyberbotics Ltd. 5 www.cyberbotics.com

Force Dimension CVR3 www.forcedimension.com þ41 22 362 6570

ICRA 2013 125 www.icra2013.org

IEEEMarketing Department 11 www.ieee.org/tryieeexplore

KUKA youBot 3 www.youbot-store.com þ49 1805 968 268

MotionLab, LLC 17 www.motionlab.com

Reflexxes GmbH 10 www.reflexxes.com

Robotis Co., Ltd. 7 www.robotis.com þ1 949 333 3635


AD SALES OFFICES•

James A.VickSr. Director, Advertising BusinessPhone: +1 212 419 7767Fax: +1 212 419 [email protected]

Marion DelaneyAdvertising Sales DirectorPhone: +1 415 863 4717Fax: +1 415 863 [email protected]

Susan SchneidermanBusiness Development ManagerPhone: +1 732 562 3946Fax: +1 732 981 [email protected]

ProductAdvertising

MidAtlanticLisa RinaldoPhone: +1 732 772 0160Fax: +1 732 772 [email protected], NJ, PA, DE, MD, DC,KY, WV

New England/South CentralEastern CanadaJody EstabrookPhone: +1 774 283 4528Fax: +1 774 283 [email protected], ME, VT, NH, MA, RI,AR, LA, OK, TXCanada: Quebec, Nova Scotia,Newfoundland,Prince Edward Island,New Brunswick

SoutheastThomas FlynnPhone: +1 770 645 2944Fax: +1 770 993 [email protected], NC, SC, GA, FL, AL, MS, TN

Midwest/CentralCanadaDave JonesPhone: +1 708 442 5633Fax: +1 708 442 [email protected], IL, IA, KS, MN, MO, NE, ND,SD, WICanada: Manitoba, Saskatchewan,Alberta

Midwest/Ontario, CanadaWill HamiltonPhone: +1 269 381 2156Fax: +1 269 381 [email protected], MI. Canada: Ontario

West Coast/Mountain StatesWestern CanadaMarshall RubinPhone: +1 818 888 2407Fax: +1 818 888 [email protected], CO, HI, NM, NV, UT,AK, ID, MT, WY, OR, WA, CA.Canada: British Columbia

Europe/Africa/Middle EastHeleen VodegelPhone:þ44 1875 825 700Fax:þ44 1875 825 [email protected], Africa, Middle East

Asia/Far East/Pacific RimSusan SchneidermanPhone:þ1 732 562 3946Fax:þ1 732 981 [email protected], Far East, Pacific Rim,Australia, New Zealand

RecruitmentAdvertising

MidatlanticLisa RinaldoPhone: +1 732 772 0160Fax: +1 732 772 [email protected], NJ, CT, PA, DE, MD, DC,KY, WV

New England/Eastern CanadaLiza ReichPhone: +1 212 419 7578Fax: +1 212 419 [email protected], VT, NH, MA, RICanada: Quebec, Nova Scotia,NewfoundlandPrince EdwardIsland, New Brunswick

SoutheastCathy FlynnPhone: +1 770 645 2944Fax: +1 770 993 [email protected], NC, SC, GA, FL, AL,MS, TN

Midwest/South CentralCentral CanadaDarcy GiovingoPhone: +1 847 498 4520Fax: +1 847 498 [email protected], IL, IN, IA, KS, LA,MI, MN, MO, NE, ND,SD, OH, OK, TX, WI.Canada: Ontario, Manitoba,Saskatchewan, Alberta

West Coast/Southwest/Mountain States/AsiaTimMattesonPhone: +1 310 836 4064Fax: +1 310 836 [email protected], CO, HI, NV, NM,UT, CA, AK, ID, MT,WY, OR, WA. Canada: BritishColumbia

Europe/Africa/Middle EastHeleen VodegelPhone:þ44 1875 825 700Fax:þ44 1875 825 [email protected], Africa, Middle East



M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___________

____________

______________________

___________

________

___________

___________

___________

___________

____________

___________

____________

___________

____________

___________

___________

http://www.qmags.com/clickthrough.asp?url=www.robotis.com&id=17487&adid=P127E12

http://www.qmags.com/clickthrough.asp?url=www.reflexxes.com&id=17487&adid=P127E11

http://www.qmags.com/clickthrough.asp?url=www.motionlab.com&id=17487&adid=P127E10

http://www.qmags.com/clickthrough.asp?url=www.youbot-store.com&id=17487&adid=P127E9

http://www.qmags.com/clickthrough.asp?url=www.ieee.org/tryieeexplore&id=17487&adid=P127E8

http://www.qmags.com/clickthrough.asp?url=www.icra2013.org&id=17487&adid=P127E7

http://www.qmags.com/clickthrough.asp?url=www.forcedimension.com&id=17487&adid=P127E6

http://www.qmags.com/clickthrough.asp?url=www.cyberbotics.com&id=17487&adid=P127E5

http://www.qmags.com/clickthrough.asp?url=http://butterflyhaptics.com&id=17487&adid=P127E4

http://www.qmags.com/clickthrough.asp?url=www.barrett.com&id=17487&adid=P127E3

http://www.qmags.com/clickthrough.asp?url=www.ati-ia.com/ras&id=17487&adid=P127E2

http://www.qmags.com/clickthrough.asp?url=www.mobilerobots.com&id=17487&adid=P127E1






















TURNING POINT •

Robotics ChallengeBy Erico Guizzo

In this issue, Erico Guizzo (EG)interviews Gill Pratt (GP) (Figure 1),a program manager at the U.S.Defense Advanced Research Proj-

ects Agency (DARPA). Early this year,Pratt unveiled the DARPA RoboticsChallenge, an ambitious programaiming to revolutionize disaster-response robots. The program isoffering tens of millions of dollars infunding to teams from anywhere inthe world to build robots capable ofperforming complex mobility andmanipulation tasks such as walkingover rubble and operating powertools. It all will culminate with anaudacious competition with robotsdriving trucks, breaking throughwalls, and attempting to performrepairs in a simulated industrial-disaster setting. The winner takes all:a US$2 million cash prize. Guizzospoke with Pratt—a roboticist andeducator who coinvented the series-elastic actuator—about the goals ofthe program and how it could changerobotics in a big way.

EG: DARPA funds lots of roboticsprograms. What is the goal andfocus of this new effort?GP: The program is really aimed atdeveloping human–robot teams to beable to help in disaster response. Here,the human is at a distance from therobot and will supervise the robot todo a number of tasks that are quitechallenging. And we think it will bevery exciting. It is important to notethat this is not just for a nuclear powerplant situation. The next disaster may

not be a nuclear plant. Forthat reason, we want toleverage the human toolsthat are likely to be outthere. It is all about adapt-ability—what is the mostadaptable system that canbe used during that firstday or two of the disasterwhen you have a chance toreduce the scope of thedisaster by taking action.That is what the challengeis about.

EG: Is the program designed toadvance humanoid robot tech-nology? Do robots entering thechallenge have to be humanlikemachines?GP: The DARPA Robotics Challengeis decidedly not exclusive to human-oid systems. The three big ideas hereare: first, we need robots that arecompatible with shared environ-ments, even though the environmentsare degraded; second, we need robotsthat are compatible with human tools;the third is compatibility with humanoperators in two ways, one is that therobot is easy to operate without par-ticular training and second is that thehuman operator can easily imaginewhat the robot might do. For that tobe true, the robot needs to have a formthat is not too different from thehuman form. But, I think that somevariation actually might work. Forinstance, if it had more arms than wehave or if it had more legs than wehave, or if it had a mobility platformthat was different than legs and couldget around in the same environmentand use the same tools that we use,

that would be fine to dothose types of tasks. Weare not pushing a particu-lar robot architecture ortype; rather we are sayingwhat the interface needsto be like, both for theoperator and for the toolsand environment.

EG: The disaster re-sponse scenario you cameup involves robots driv-ing trucks,walking through

rubble, crashing through walls, andoperating power tools (Figure 2).Is it realistic to expect teams willsucceed?GP: Some people have said, incor-rectly, that we expected that teamswould not be able to complete the firstchallenge, but that is actually not true.[Editor’s Note: The program will con-sist of two phases, each ending with acompetitive challenge. Phase 1 willlast 15 months from October 2012 toDecember 2013. Phase 2 will last 12months from January 2014 to Decem-ber 2014.] The challenge will beadjusted as we gain experience withthe teams over this first phase, beforethe first live challenge in December2013. What we are going to make sureis that the live challenge is difficult butnot impossible. And then we expectthat in the second live challenge wewill be doing the same thing, and thatin fact we will show off skills andperformance that are better than whatwe had before.



Date of publication: 10 September 2012 (continued on page 124)

Figure 1. Dr. Gill Prattjoined DARPA as aprogram manager inJanuary 2010. Heoversees severalDARPA programs in thefield of robotics. (Imagecourtesy of DARPA.)


M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


_________________

__________________

http://www.qmags.com/clickthrough.asp?url=www.forcedimension.com&id=17487&adid=PCOVER 3A1







M AGAZ INE

oboticsutomation


M AGAZ INE

oboticsutomation

qqM

Mq

qM

MqM


qqM

Mq

qM

MqM


___________________________________

______________

http://www.qmags.com/clickthrough.asp?url=www.barrett.com&id=17487&adid=PCOVER 4A1

http://www.qmags.com/clickthrough.asp?url=www.barrettcommunications.com&id=17487&adid=PCOVER 4A2





ram_20120901_sep_2012

Documents