Benchmarking of Cyber-Physical Systems in Industrial Robotics

The RoboCup Logistics League as a CPS Benchmark Blueprint

Tim Niemueller and Gerhard LakemeyerKnowledge-Based Systems Group

RWTH Aachen University{niemueller,lakemeyer}@kbsg.rwth-aachen.de

Sebastian Reuter and Sabina JeschkeInstitute Cluster IMA/ZLW & IfU

RWTH Aachen University{sebastian.reuter,sabina.jeschke}@ima-zlw-ifu.rwth-aachen.de

Alexander FerreinMobile Autonomous Systems & Cognitive Robotics Institute

Aachen University of Applied Sciencesferrein@fh-aachen.de

Abstract

In the future, we expect manufacturing companies to follow a new paradigm thatmandates more automation and autonomy in production processes. Such smart factorieswill offer a variety of production technologies as services that can be combined ad-hocto produce a large number of different product types and variants cost-effectively even insmall lot sizes. This is enabled by cyber-physical systems that feature flexible automatedplanning methods for production scheduling, execution control, and in-factory logistics.

During development, testbeds are required to determine the applicability of integratedsystems in such scenarios. Furthermore, benchmarks are needed to quantify and comparesystem performance in these industry-inspired scenarios at a comprehensible and man-ageable size which is, at the same time, complex enough to yield meaningful results.

In this chapter, based on our experience in the RoboCup Logistics League (RCLL)as a specific example, we derive a generic blueprint how a holistic benchmark can bedeveloped, which combines a specific scenario with a set of key performance indicators asmetrics to evaluate the overall integrated system and its components.

Keywords—Smart Factory, Industry 4.0, Cyber-physical Systems, Multi-Robot Systems, Autonomous

mobile robots, Production intra-logistics, Benchmarking and Performance Evaluation, Key Perfor-

mance Indicators, RoboCup, Factory of the Future, Context-aware production

1 Introduction

Manufacturing industries are on the brink of widely accepting a new paradigm for organizingproduction by introducing perceiving, active, context-aware, and autonomous systems. Thisis often referred to as Industry 4.0 [8], a move from static process chains towards moreautomation and autonomy. The corner stones for this paradigm shift are smart factories,

which are context-aware facilities in which manufacturing steps are considered as servicesthat can be combined efficiently in (almost) arbitrary ways allowing for the production ofvarious product types and variants cost-effectively even in small lot sizes, rather than the moretraditional chains which produce only a small number of product types at high volumes. Thesesmart factories are made possible by means of cyber-physical systems (CPS) which combinecomputational with sensing and actuation capabilities to directly observe and influence theproduction process in a more decentralized fashion. They can communicate vertically withinthe overall business processes (e.g., to retrieve information like the priority of an order) andhorizontally with other machines on the factory floor. Such smart factories require a capablelogistics system, which efficiently maintains the material flow and highly parallel execution ofindividual productions. Autonomous mobile robots are a natural choice for this task, as theycan apply multi-robot coordination to the emerging problem of supply chain optimization1

(SCO) and even take over some parts of the machine handling. To develop such systems,a testbed is required to ensure the fitness of a system for a specific task. Furthermore,benchmarks are needed, which allow to quantify the performance by a given set of aspects tocompare systems with each other.

In this chapter, we will outline a method for designing such benchmarks by identifyingKey Performance Indicators (KPI) from the underlying industrial context and using suchmetrics for the evaluation of integrated systems within an environment of comprehensible andmanageable size. We focus in particular on multi-robot systems as one of the most challengingCPS categories. As a specific example, we introduce the RoboCup Logistics League (RCLL)which models the task of in-factory logistics and is inspired by actual industrial contexts. Wedescribe how the analysis of data recorded during actual test runs (competitive games) leadto the better understanding and identification of relevant KPIs that will hopefully allow toquantify the overall system performance by fine-grained criteria in the future. This specificinstance of the workflow to create a holistic benchmark, that is a benchmark that combinesthe assessment of an integrated (multi-robot) system with a domain-specific set of KPIsto evaluate the performance, can serve as a blueprint to create benchmarks also for otherindustry-inspired scenarios.

The chapter is structured as follows. In Sect. 2 we define the terminology and the examplelogistics scenario. Following, in Sect. 3 we introduce the RoboCup Logistics League as anexample for the logistic robot scenario. We describe general criteria for a benchmark in Sect. 4and other scenarios which exist today. In Sect. 5 we present the RCLL as a benchmark and ourquantitative data analysis. KPIs and their exemplary application to the RCLL are describedin Sect. 6 before we discuss the transferability of our approach to other scenarios in Sect. 7and conclude in Sect. 8.

2 Cyber-Physical Systems (CPS) in a Smart Factory

Cyber-Physical Systems (CPS) integrate computation and physical processes [10]. The termhas undergone several development stages starting with simple identification devices suchas RFID2 chips, followed by devices with sensors and actuators that required centralizedprocessing and instruction. The latest generation – considered as the canonical CPS [5] – are

1Supply chains describe logistic networks which comprise interlinked logistic actors [20], in this chapter, wefocus on intra-logistics for the material flow of a smart factory by a group of autonomous mobile robots.

2Radio Frequency Identification, passive information retrieval via wireless communication.

2

embedded systems that extend this by integrating computation and can be networked. Inthe context of this paper, we limit the meaning to the most complex version of such CPS,that is autonomous mobile robots. Such systems have a variety of sensors and actuators toperceive, manipulate, move within the environment, and coordinate to form a multi-robotsystem. A Smart Factory (SF) is a context-aware production facility that considers, e.g.,object positions or machine status to assist in the execution of manufacturing tasks [14]. It candraw information from the physical environment or a virtual model, e.g., a process simulation,order, or product specification. It is designed to cope with the challenges that arise from thedesire to produce highly customized goods which result in the proliferation of variants [14] andtherefore smaller lot sizes. A CPS, in the sense of an autonomous mobile robot, requires a task-level executive which is the highest level decision-making software component that determinesthe tasks and steps necessary to achieve its intended goal, and which executes and monitorsthe required behaviors. Typical approaches can be roughly divided in three categories [19]:state machine based controllers like SMACH [2] or XABSL [13], reasoning systems fromProcedural Reasoning Systems [6] or rule-based agents [18] to more formal approaches likeGolog [11], and finally automated planning systems with varying complexity and modelingrequirements, for example based on PDDL [15] and its various extensions. In this sense, thetask-level executive is what makes a CPS an intelligent system.

For the remainder of this chapter, we envision a SF that offers various manufacturing ser-vices, i.e., a number of machines that can refine, assemble, or modify a workpiece towards afinal product. Given a product specification, the SF determines the required processing stepsand involved machines. Such a SF rather offers a number of special production technologiesthan just production types [5] and requires more complex logistics for efficient parallel pro-duction. The challenge then becomes developing a multi-robot system capable of efficientlydetermining the assignment of robots to logistics tasks, and machine handling. The robotsmust coordinate in order to avoid resource conflicts (e.g., two workpieces requiring processingat the same machine) and cease the opportunity for cooperation (e.g., to meet a tight schedulefor a high value product). This entails to study various methods for the task-level executiveand coordination, and to balance the trade-off between centralization and distribution.

3 The RoboCup Logistics League (RCLL)

Figure 1: Teams Carologistics (robots with ad-ditional laptop) and Solidus (pink parts) dur-ing the RCLL finals at RoboCup 2015.

RoboCup [9] is an international initiativeto foster research in the field of roboticsand artificial intelligence. Besides roboticsoccer, RoboCup also features application-oriented leagues which serve as commontestbeds to compare research results. Amongthese, the industry-oriented RoboCup Lo-gistics League3 (RCLL) tackles the problemof production logistics in a smart factory.Groups of three robots have to plan, execute,and optimize the material flow and deliverproducts according to dynamic orders in asimplified factory. The challenge consists of

3RoboCup Logistics website: http://www.robocup-logistics.org

3

BS RS 1 RS 2 RS 2 CS 2

Base Element(specific color)Base Element(any color)

Up to 3 rings(four colors)

Top-most cap

CS Machine Type{BS,DS,CS,RS}

Figure 2: Steps for the production of a high complexity product in RCLL 2015 (legend onthe right).

creating and adjusting a production plan and coordinate the group [16].The RCLL competition takes place on a field of 12 m× 6 m partially surrounded by walls

(Fig. 1). Two teams of three robots each are playing at the same time. The game is controlledby the referee box (refbox) [20], a central software component for sending orders to the robots,controlling the machines and collecting teams’ points. Additionally, log messages and gamereports are sent to the refbox, which allows for detailed game analyses and benchmarks. It isalso used in a simulation of the RCLL [25] After the game is started, no manual interferenceis allowed, robots receive instructions only from the refbox and must act fully autonomous.The robots must plan and coordinate their actions to efficiently fulfill their mission (cf. [19]for a characterization of the RCLL as a planning domain). The robots communicate amongeach other and with the refbox through wifi. Communication delays and interruptions arecommon and must be handled gracefully.

Figure 3: Robot approach-ing a ring station in produc-tion

Each team has an exclusive set of six machines of four differ-ent types of Modular Production System (MPS) stations. Therefbox assigns a zone of 2 m× 1.5 m to each station (positionand orientation are randomized within the zone). Each stationaccepts input on one side and provides processed workpieces onthe opposite side. Machines have markers that uniquely iden-tify the station and side. A signal light indicates the currentstatus, such as ”ready”, ”producing”, or ”out-of-order”.

The game is split in two major phases. In the explorationphase robots need to roam the environment and discover ma-chines assigned to their teams. For such machines, it must thenreport the marker ID, position zone, and light signal code. Forcorrect reports, robots are awarded points, for incorrect reportsnegative points are scored.

In the production phase, the goal is to fulfill orders accord-ing to a randomized schedule by refining raw material throughseveral processing steps and eventually delivering it. An orderstates the product to be produced, defined as a workpiece con-sisting of a cup-like cylindrical colored base, zero, or up to threecolored rings, and a cap. An exemplary production chain for a product of the highest com-plexity is shown in Fig. 2. All stations require communication through the refbox to prepareand parametrize them for the following production step. Using different types of machines,the robots have to assemble different final products.

The robot used in the competition is the Robotino 3 by Festo. It features a round basewith a holonomic drive, infrared short-distance sensors, and a webcam. Teams are allowed to

4

add additional sensors and computing devices as well as a gripper for workpiece handling.

4 Robot Benchmarking

In the chosen example, the goal is to evaluate a robotics scenario in an industrial context.Therefore, we now describe aspects and criteria of a robotics benchmarks in general, verifythat our specific scenario indeed constitutes a suitable benchmark, and mention some otherexisting robot benchmarks.

4.1 Robot Benchmark Features

The key question is whether a scenario fulfills some general criteria to consider it a properrobot benchmark like (cf. [4]): (1) the robot needs to perform a real mission; (2) the bench-mark must be accepted in the field; (3) the task has a precise definition; (4) repeatability,independence, unambiguity of the test, and (5) collection of ground-truth data.

Robotic systems are integrated from a large number of hardware and software components.Other questions that arise are whether specific or smaller groups of components (modules)can and should be tested, whether the analysis of the full system is of interest, or to aim for acombination of both [1]. In the context of a multi-robot system another important dimensionis the overall fleet-level performance, i.e., how does the robot group perform as a whole.

A final relevant question is whether to run fully reproducible tests, or whether to testthe flexibility of the approach. In the former case, robots would be presented with the sameenvironment, task, and parametrization every time. This especially allows to benchmark theoptimization capabilities of a system or module, i.e., how well can it adapt to the environ-ment over time and what is the best or worst case performance (depending on the chosenscenario). If testing for flexibility, those three elements could be varied (within a certaindomain-specific range), for example introducing uncertainty in the task structure (varying,e.g., time parameters when something must be started or completed), or other agents. Thisin particular allows to test the robustness and average performance of systems and modules.

A crucial part for a benchmark are metrics, which allow to quantify the performance andcompare different methods and systems. These metrics are often neglected when benchmarksare described, as it can be hard to find the proper method to calculate a meaningful anddecisive value. The metrics are also not intrinsic to an environment or system, but they ratherdescribe the criteria (optimization, robustness, etc.) relevant for the benchmark operator.

4.2 RCLL as a Benchmark

First, we verify the general requirements outlined by Dillmann [4]: (1) the robots’ task isto maintain and optimize the material flow in the factory and they thus accomplish a realmission; (2) the scenario has been accepted as a full RoboCup league with participants (in theRCLL alone) from all over the globe, thus we can state that the benchmark has been acceptedin the field; (3) the rulebook provides a detailed specification [3], furthermore the refereebox provides accountable agency to the environment (and simulation), the task thereforehas a precise definition; (4) the scenario can be run many times, either with the same orwith a randomized parametrization, it is independent from the observer and the results areunambiguous, and (5) the referee box logs messages sent over the network, game reports, andupdates to its fact base comprising information about the game available and relevant for

5

the refbox, additionally a prototype was developed using field-mounted sensors for producttracking [20], thus ground-truth data is available.

An important aspect that can be tested is the system-level performance of a single robotsystem, i.e., what difference does it make to add or remove a robot. Necessarily, this alsoincludes module-level aspects, e.g., how good are the path planning or collision avoidancecapabilities of the robot while being deployed in a real task. But most importantly, thefleet-level performance is crucial, which is determined in particular by the effectiveness ofthe coordination strategy and cooperation capabilities. Unlike approaches like RoCKIn (seeSect. 4.3), however, the performance is evaluated within an industry-inspired scenario, ratherthan crafting the scenario such that the benchmark requirements are met and potentiallydiverting much farther from an actual industrial scenario.

The referee box allows to run the scenario either with the same environment and parametriza-tion, or randomizing machine positions and order time windows each time. This way, opti-mization for specific scenarios can be evaluated for best and worst case runs, or the averageperformance and robustness without bias can be evaluated.

The RCLL’s metric is implicitly given by its grading scheme. Complex production stepsand delivery are awarded with points and thus drive the performance. We will have a closelook at the evaluation criteria and ideas for improvement in Sect. 5.

4.3 Other Testbed and Benchmark Scenarios

The RoCKIn4 project is another recent and prominent benchmarking approach with an in-dustrial aspect. There, the goal is to combine system-level and module-level benchmarkingresults into a single system [1]. While there are key differences between the RCLL and theRoCKIn@Work approach (e.g., single vs. multi-robot, module-level aspects vs. focus onoverall integrated system evaluation, scenario design vs. solution design based on industry-inspired scenario), we deem compatible elements in both approaches [17]. An obstacle in thisregard might the difference that RoCKIn@Work focuses on multiple short-term tasks, whilethe RCLL strives for long-term autonomy by using a single scenario for enduring test runs.Other interesting Benchmark Scenarios making use of robotic competitions is, for instance,the European Land Robot Trials (ELROB) [24], or RoboCup@Home [7].

5 A Benchmark for Logistic Robots in a Smart Factory

Figure 4: Carologistics (Robotino 2 with lap-tops) and BavarianBendingUnits (Robotino 3)during the RCLL finals at RoboCup 2014.

In this section, we combine our generalconsiderations of CPS in smart factories(Sect. 2) and its special instance RCLL withthe criteria from Sect. 4 to form a relevantbenchmark.

5.1 RCLL 2014 Evaluation

Recently, we have presented a data analy-sis based on 75 GB of refbox data organizedusing MongoDB [21] of the RoboCup com-petition 2014 [22]. As an example, we focus

4RoCKIn website: http://rockinrobotchallenge.eu/

6

on the RoboCup 2014 final of the RCLL between the Carologistics (cyan) and the BBUnits(magenta) which ended with a score of 165 to 1245. While the overall theme of the game wasthe same in 2014, the game was played on a more constrained field with simpler machinescompared to 2015 as shown in Fig. 4. Products were represented by pucks moved on theground and placed at machines. Each team had 16 machines of 7 different types and therewere only three different kinds of products. We briefly introduce the figures that make upour analysis before we draw conclusions based on our observations.

5.1.1 Analysis – Description

Fig. 5 shows the machines (M1–M24) grouped per team above the horizontal game-time axis.Each row expresses the machine’s state over the course of the game. Gray means it is currentlyidle. Green means that it is actively processing (busy) or blocked while waiting for the next

5Video of the final is available at https://youtu.be/_iesqH6bNsY

Deliveries

Deliveries

Carologistics

BBUnits

Legend

Idle

Busy

Blocked

Waiting

Down

Imprecise

Busy Cyan

Waiting Cyan

Busy Magenta

Waiting Magenta

Game Time(min)1 2 3 4 5 6 7 8 9 10 11 12 13 14

M1 T3

M2 T2

M3 T1

M16 T5

M5 T1

M6 T2

M19 T2

M8 T3

M21 T1

M10 T4

M11 T1

M12 T4

M13 T3

M14 T2

M15 T1

M4 T5

M17 T1

M18 T2

M7 T2

M20 T3

M9 T1

M22 T4

M23 T1

M24 T4

P3

P3 P3

P3

P2

Busy Machines(# machines)1

234567

Figure 5: Machine states over the course of the final game at RoboCup 2014. The lowergraph shows the occupied machines per 20 sec time block.

7

Carologistics

BBUnits

Legend

Completed

Partial

Unfulfilled

Delivery

Game Time(min)1 2 3 4 5 6 7 8 9 10 11 12 13 14

1x P3

1x P3

2x P3

1x P1

1x P2

1x P3

1x P3

2x P3

1x P1

1x P2

Figure 6: Adherence to delivery schedule (finals RoboCup 2014).

input. After a work order has been completed, the machine waits for the product to bepicked up (orange). The machine can be down for maintenance for a limited time (dark red).Sometimes it is used imprecisely (yellow), that is, the product is not placed properly underthe RFID device. The row ’Deliveries’ shows products that are delivered at a specific time.Below the time axis, Fig. 5 shows the busy machines over time. Each entry consists of a cyanand magenta column and represents a 20 second period. The height of each column showsthe number of machines that are producing (bright team color) or waiting for the product tobe retrieved (dark team color).

Fig. 6 shows the adherence to the delivery schedule. The horizontal axis denotes thegame-time and each row a particular order. Recall that an order is defined by a product typeand amount (P1 to P3) and the start and end time for the delivery. In the diagram, redboxes indicate orders which have been missed. Light green boxes show fulfilled orders, whiledark green orders indicate that only some of the total amount of requested products has beendelivered. Red dots indicate the actual time of delivery. Robots send periodic beacon signals

Figure 7: 2D histogram (heatmap) for the positions of all three robots of one team combinedon the playing field over the course of a full game in simulation. Red indicates areas offrequent travel fading according to the scale to blue for less time spent at a position.

8

which may include the position of the robot. This allows for analyzing the typical paths andhotspots on the field. As one example, Fig. 7 visualizes the data of a game in simulation bythe Carologistics team as a 2D histogram (heatmap). Regions of the map without overlayhave not been visited by any robot. Red areas indicate frequent operation of any of the robotsin that area fading to blue according to the given scale for less frequent travel.

5.1.2 Analysis – Conclusions

Fig. 5 shows that the production strategies of both teams differ significantly. The cyan teamis goal oriented towards moving workpieces fast through the production steps, indicated byshorter busy and blocked times (green) as well as fewer and very short periods where amachine is waiting for the output to be picked up (orange). The magenta team, on the otherhand, has on average more machines occupied (bar chart at the bottom) resulting in longertimes that workpieces remain in a machine (dark green areas when waiting for the next inputworkpiece and orange areas when waiting for output to be picked up).

Already in anticipation of the next section, we recognize that the cyan team achieved asignificantly lower throughput time, that is the time that a workpiece requires from the firstto the last processing step. The magenta team, however, had a higher number of machines inuse at a given time. Only the lower throughput time enabled the cyan team to deliver the P2product (Fig. 6). Additionally, it allowed for more recycling of material (recyclable materialis only available after a more complex machine has completed a work order).

The major observation here is that the grading implicitly favors low throughput production– which was not intended. Recognizing this lead to reasoning about alternative and more fine-grained evaluation metrics that we introduce in the next section.

A secondary observation based on Fig. 7 is the existence of areas which are visited fre-quently. One such place, naturally, is the blue insertion area, where robots need to go topickup raw material. The heatmap also reveals corridors used frequently for traveling. Inparticular, the route from the blue insertion area through M3 and M6 and then M2 and M4is used often. Such a hotspot can slow down the team more than using sub-optimal, but lessfrequently traveled routes.

6 Key Performance Indicators (KPI) for the RCLL

Performance evaluation requires metrics to quantify how well a task was performed and tocompare different methods and systems. In industrial contexts Key Performance Indicators(KPI) are used to analyze the performance of systems in terms of deviation of operationalplanning and execution. These metrics are highly specific to the problem domain and evendepend on the particular department performing the evaluation. In the following, we willdescribe KPIs for logistics in production and their adaptation and application in the RCLL,as an example of how existing well-known KPIs can be transferred to a related roboticsscenario.

6.1 Key Performance Indicators in Logistics

High efficiency of logistics systems can be achieved by maximizing performance, or by mini-mizing costs [23]. Using KPIs, an operationalization of the logistic objective is possible. Fig. 8shows exemplary indicators assigned to such objectives. As an example, we describe two of

9

Logistic efficiency

High logistic performance Low logistic costs

(short)throughput

time

(high)deliveryreliability

(low)deliverylateness

(low) workin progress

(WIP)

(high)utilization

(low)cost of

late delivery

Figure 8: Exemplary Key Performance Indicator (KPI) within Production Logistics

the most important logistic KPIs throughput time (the time span for an order to be created)and utilization (of production machines) in more detail and point out some intricacy knownas the scheduling dilemma of logistics. For a more detailed discussion of KPIs for logisticprocesses relevant for the RCLL we refer to [22].

The throughput time TTP of a process is defined as duration from the start to the endof processing an order [12]: TTP = Tproc. end − Tproc. start. As an example, Fig. 9 shows thethroughput of a product of type P2 which consists of a raw product S0 and two intermediateproducts S1 and S2. The intermediate products have to be manufactured in advance in orderto assemble the product P2. The manufacturing of an intermediate product S2 consists of twooperations on the machines T1 and T2 as well as the time span needed for transportation andwaiting times. While the final assembly of product P2 can only be done after all necessaryintermediate products are ready, the manufacturing of the intermediate products S1 and S2can be parallelized. Thus, the resulting throughput time of product P2 is formed by thethroughput time of the intermediate product S2 and the final assembly.

The utilization U is the usage ratio of production resources such as production machinesor transportation entities and the operation period. Thus, the utilization in terms of aproduction machine describes the ratio of a machine’s operating time and a duration of thedesired reference period [12]: U =

∑Toperations/Tduration of reference period ∗ 100%.

Effective logistic processes strive for the maximization of logistic performance while min-imizing costs. In terms of the introduced KPIs, this demands for short throughput timesand high utilization of logistic resources. To maximize throughput time of an order, the orderhas to be processed as soon as it arrives by triggering the manufacturing of all necessary

throughput time(Product P2)

throughput time(Intermediate Product S2)

S0

S1 T1

S2 T2T1

Assembly of P2

T4

setup processing waiting transport waiting setup processing waiting transport waiting

Time

Figure 9: Throughput Time Components

10

intermediate products, the transport among the production machines as well as the assemblyof the final product. For an undelayed flow of the order through the production facility, allproduction and logistic resources should be idle and waiting for processing the active order.In contrast, a high utilization of resources demands for a continuous in- and output of ordersto minimize waiting and idle-times. This can only be achieved by continuously feeding pro-duction machines in order to prevent shortages in the material flow. But this will slow downthe throughput time as orders have to wait to be processed. Hence, high resource utilizationand short throughput times cannot be achieved together. This conflict among the objectiveslogistic performance and logistic costs is called the scheduling dilemma of logistics [23]. Asone strives to maximize logistic performance by decreasing throughput time, logistic costsincrease as this can only be achieved by decreasing the utilization of production resources. Itis up to the benchmark operator to balance this trade-off.

6.2 Application of KPIs in the RoboCup Logistics League

The aim of RCLL is to simulate a realistic, yet simplified, production environment. With alimited set of resources (mobile transportation robots and stationary production machines)the teams have to produce and deliver products according to a dynamic production schedule.By means of the referee box all necessary data for calculating throughput time as well asresource utilization (and further KPIs) are available. In the following we describe how thethroughput time and resource utilization KPIs can be adapted and applied in the RCLL.

The utilization U of a resource is calculated by dividing its actual processing time by theoverall evaluation time (e.g. game time). Resources are MPS stations as well as three mobilerobots for material transportation. For MPS stations, the utilization can be calculated bydividing the actual busy time (bright green areas in Fig. 5) by the overall game time. Forrobots, the sum of times while transporting a product divided by the game time denotes theutilization.

The throughput time TTP is the time from the start of the first to the completion of thefinal processing step (also known as makespan). For example, in Fig. 5 in the second line forM2, the Busy-Blocked-Busy cycle is the TTP for a production of 84 sec.

From our data analysis of tournament games it became clear that the current gradingscheme’s dominant metric is minimizing the throughput time (cf. Sect. 5.1). With the betterunderstanding of relevant KPIs and mapping them to the RCLL (cf. [22]) the productionschedule could now be adjusted and a more fine-grained grading scheme developed, e.g., toaward certain best-in-class capabilities. For example, to simulate large volume production(which still can benefit from dynamic autonomous logistics, for example for variant produc-tion, where robots are used only for some processing steps), a standing order for a certainproduct would benefit from the maximization of machine utilization.

7 Transferring Holistic Benchmarks to other Domains

A holistic robot benchmark is based on an industry-inspired scenario at a comprehensiblescale where robotic task performance is evaluated with existing or slightly adapted KPIs.When introducing robotics in an industrial scenario, the goals are usually efficiency and/orflexibility. In the context of smart factories, additionally more process autonomy is desired.To verify whether these goals have been reached, valid metrics are necessary to quantifythe impact. Several factors must be considered, e.g., whether the overall factory performs

11

better with the help of robots, or whether more robots increase performance or lead tounexpected bottlenecks. At the example of the RCLL, we have presented an idea how this canbe accomplished for autonomous in-factory logistics. This can be extended into a blueprintfor generally applying robotic benchmarks in industrial scenarios:

1. Take the relevant scenario and scale it down to a comprehensible size introducing robots(or CPS in the more general sense) in certain areas.

2. Integrate the system on a small scale and verify its feasibility. A simulation environmentadequately representing the environment can be a viable option.

3. Apply existing or slightly adapted KPIs relevant to the scenario that have been usedbefore.

4. If the results are satisfactory, scale up the approach monitoring the development of theKPIs.

The latter step seems especially important to small and medium enterprises, for which theintroduction of complex technologies can pose a high risk. Starting out at a smaller scale, orin simulation, and continuous evaluation, allows for an informed decision of the viability ofthe project early in the process.

8 Conclusion

We expect autonomous mobile robots to play an important role in smart factories in thecontext of Industry 4.0. Mobile robots are complex cyber-physical systems that combinesensors, actuators as well as computing devices that enable them to reason about and interactwith the environment. The ability of these systems to reason about their environment and toselect a corresponding behavior poses new challenges for evaluation and certification. On theone hand, the autonomy of these systems expands their operational envelop and make themsuitable for a broad area of applications. On the other hand, companies selling mobile robotshave to give reliable predictions of the systems behavior. Therefore, extensive tests have to beconducted. As the systems behavior evolves in combination with other autonomous systems,a test of all possible situations becomes infeasible.

In this paper, We have introduced the concept of a holistic benchmark, which combinesan environment and task inspired by an industrial context with selected key performanceindicators (KPI), which are metrics relevant for the specific scenario.

As a specific example for such a logistics benchmark scenario we described the RoboCupLogistics League (RCLL). It has a proper balance of medium complexity, comprehensible size,and yet the necessary challenges to yield meaningful results. Based on the analysis of datarecorded through several RCLL competitions and simulations we explained the insight, thatthe current grading scheme prefers low throughput production over a high machine utilization.These are two of several KPIs that we have identified as being relevant and interesting, andcould be used for further develop the league as a benchmark. It shows that it is importantto choose both, the scenario and metrics with respect to the intended industrial context.

The general idea of a holistic benchmark and its example implementation for the RoboCupLogistics League can serve as a blueprint for further testbeds and benchmarks.

12

Acknowledgments

T. Niemueller was supported by the German National Science Foundation (DFG) research unit FOR

1513 on Hybrid Reasoning for Intelligent Systems (http://www.hybrid-reasoning.org).

References

[1] F. Amigoni, A. Bonarini, G. Fontana, M. Matteucci, and V. Schiaffonati. Benchmarkingthrough competitions. In European Robotics Forum – Workshop on Robot Competitions:Benchmarking, Technology Transfer, and Education, 2013.

[2] J. Bohren and S. Cousins. The SMACH High-Level Executive. Robotics AutomationMagazine, IEEE, 17(4), 2010.

[3] R. T. Committee. The robocup logistics league rulebook for 2015. Technical report,RoboCup Logistics League, 2015.

[4] R. Dillmann. Benchmarks for robotics research. Technical report, EURON, 2004.http://www.cas.kth.se/euron/euron-deliverables/ka1-10-benchmarking.pdf.

[5] M. Hermann, T. Pentek, and B. Otto. Design Principles for Industrie 4.0 Scenarios: ALiterature Review. Working paper 01/2015, Technische Universitat Dortmund, 2015.

[6] F. Ingrand, R. Chatila, R. Alami, and F. Robert. PRS: A High Level Supervision andControl Language for Autonomous Mobile Robots. In IEEE International Conferenceon Robotics and Automation (ICRA), volume 1, 1996.

[7] L. Iocchi, D. Holz, J. Ruiz-del Solar, K. Sugiura, and T. van der Zant. Robocup@home:Analysis and results of evolving competitions for domestic and service robots. ArtificialIntelligence, 229:258–281, 2015.

[8] H. Kagermann, W. Wahlster, and J. Helbig. Recommendations for implementing thestrategic initiative INDUSTRIE 4.0. Final Report, Platform Industrie 4.0, 2013.

[9] H. Kitano, M. Asada, Y. Kuniyoshi, I. Noda, and E. Osawa. Robocup: The Robot WorldCup Initiative. In 1st Int. Conference on Autonomous Agents, 1997.

[10] E. Lee. Cyber Physical Systems: Design Challenges. In 11th IEEE International Sym-posium on Object Oriented Real-Time Distributed Computing (ISORC-08), 2008.

[11] H. J. Levesque, R. Reiter, Y. Lesperance, F. Lin, and R. B. Scherl. Golog: A logicprogramming language for dynamic domains. Journal of Logic Programming, 31(1-3),1997.

[12] H. Lodding. Handbook of manufacturing control: Fundamentals, description, configura-tion. Springer Science & Business Media, 2012.

[13] M. Loetzsch, M. Risler, and M. Jungel. XABSL - A Pragmatic Approach to BehaviorEngineering. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 2006.

13

[14] D. Lucke, C. Constantinescu, and E. Westkamper. Smart factory – a step towards thenext generation of manufacturing. In Manufacturing Systems and Technologies for theNew Frontier, The 41st CIRP Conference on Manufacturing Systems, 2008.

[15] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, andD. Wilkins. PDDL – The Planning Domain Definition Language. Technical report,AIPS-98 Planning Competition Committee, 1998.

[16] T. Niemueller, D. Ewert, S. Reuter, A. Ferrein, S. Jeschke, and G. Lakemeyer. RoboCupLogistics League Sponsored by Festo: A Competitive Factory Automation Testbed. InRoboCup Symposium 2013, 2013.

[17] T. Niemueller, A. Ferrein, S. Reuter, S. Jeschke, and G. Lakemeyer. The robocup logisticsleague as a holistic multi-robot smart factory benchmark. In Open forum on evaluation ofresults, replication of experiments and benchmarking in robotics research at IROS 2015,2015.

[18] T. Niemueller, G. Lakemeyer, and A. Ferrein. Incremental Task-level Reasoning in aCompetitive Factory Automation Scenario. In AAAI Spring Symposium 2013 - DesigningIntelligent Robots: Reintegrating AI, 2013.

[19] T. Niemueller, G. Lakemeyer, and A. Ferrein. The RoboCup Logistics League as aBenchmark for Planning in Robotics. In WS on Planning and Robotics (PlanRob) atInt. Conf. on Aut. Planning and Scheduling (ICAPS), 2015.

[20] T. Niemueller, G. Lakemeyer, A. Ferrein, S. Reuter, D. Ewert, S. Jeschke, D. Pensky,and U. Karras. Proposal for Advancements to the LLSF in 2014 and beyond. In ICAR– 1st Workshop on Developments in RoboCup Leagues, 2013.

[21] T. Niemueller, G. Lakemeyer, and S. S. Srinivasa. A Generic Robot Database and itsApplication in Fault Analysis and Performance Evaluation. In IEEE/RSJ Int. Conf. onIntelligent Robots and Systems (IROS), 2012.

[22] T. Niemueller, S. Reuter, A. Ferrein, S. Jeschke, and G. Lakemeyer. Evaluation of theRoboCup Logistics League and Derived Criteria for Future Competitions. In RoboCupSymposium, 2015.

[23] P. Nyhuis and H.-P. Wiendahl. Logistic production operating curves–basic model of thetheory of logistic operating curves. CIRP Annals-Manufact. Tech., 55(1), 2006.

[24] F. E. Schneider, D. Wildermuth, B. Bruggemann, and T. Rohling. European land robottrial (elrob) towards a realistic benchmark for outdoor robotics. AT&P Journal Plus, 2,2010.

[25] F. Zwilling, T. Niemueller, and G. Lakemeyer. Simulation for the RoboCup LogisticsLeague with Real-World Environment Agency and Multi-level Abstraction. In RoboCupSymposium, 2014.

14