the robot capable of understanding human-like instructions · the robot capable of understanding...

The robot capable of understanding human-like instructions

The European Robot Initiative for Strengthening the Competitiveness of SMEs in Manufacturing

2SMErobot Final Project Workshop

May 7th, 2009

Contents of this Presentation

• Introduction

• Human robot interaction devices and methods investigated

• General topics: technologies, that are applicable in a variety of use cases

• Success stories: solutions for specific tasks that allow human-like instructions

• Summary


May 7th, 2009

Common Robot Programming

„Online“ programming using the

teach pendant

Requires training

Requires experience

Takes a lot of time on the

robot (approx. 1 min / point)

„Offline “ programming in virtual

environments

Requires training

Requires experience

Requires equipment

Takes a lot of timeNeeds:Fast and intuitive

Programming!


May 7th, 2009

Outlines and Goals

Success StoriesIntegration in Demonstrations

High Level Programming MethodsHow can the information be processed (high-level)?

Device Interface, Multimodal InterfaceHow can the information be transferred?

Interaction DevicesHow can this data be retrieved and conveyed (low-level)?

Use Case AnalysisWhat data do we need?


May 7th, 2009

Use Case Analysis

• Use Cases • Analysis of user scenarios and interaction methods

• Conducted SME visits:

– System integrator: Automazioni Industriali, Kine,Friul Robot

– Joiners workshop: Som, Mohring

– Locksmithery: Petry, Schmidt

– Other: K.Met, Europress, Treffler, SDGmbH, ZF-Lenksysteme, …

• Suitable use cases

– Milling of wooden kitchen elements

– Sanding of wooden stair elements

– Marking sheet metal for punching

– Handling / Cooperation in ramp-up phase

– Welding and bending in small batch sizes

Locksmith Schmidt

Joiner Mohring

Metal works Treffler


May 7th, 2009

Overview on Methods and Devices

• Programming devices and methods in different areas have been investigated

– Multimodality:

• Tactile

• Verbal

• Visual

– Different Intelligence levels:

• Pure point to point teaching

• Integration of process know how

• High Level Programming

• Automatic Programming

• Investigation in test beds, integrationin demonstrations


May 7th, 2009

State of the Art: Survey and Assessment

• Rate and match interaction devices to use cases


May 7th, 2009

State of the Art: Survey and Assessment

• Result: Five main requirements

– more intuitive definition of directions and orientations in 3D

– more intuitive definition of points, trajectories, surfacesand objects in 3D

– definition of process parameters, such as physical values, e.g., velocity or force

– need for small, wireless and hand-held devices that could be operated with one hand, e.g. PDAs

– command a robot without using hands and fingers


May 7th, 2009

Evolution of Devices

IPA PDAIPA InTeach

Comau PDA

ADDF PDA

Comau SafeGamePad

Commercial HW

Reis Flybook

1 HandedKUKA inertial

jogging

KUKA 3D (teach wand)Visual servoing

KUKA 2DVisual servoing

KUKA Anoto Pen asteach wand

Anoto Pen ADDF Anoto Pen forCAD

IPA Anoto Pen forSketches

IPA/ADDF speech interfaces

Manual Guidance and Robot Handles (ABB, IPA, …)

KUKA newteach wand


May 7th, 2009

Overview: General Topics

Speech Interaction: Commanding the Robot

Manual Guidance: Jogging the Robot

Digital Pens: Understanding of Drawings

High Level Programming: Abstract Task Specifications

Ontologies: Understanding the Task


May 7th, 2009

General Topics: Manual Guidance

• Distinction

– Low Level Force Control (ITIA CNR, COMAU, DLR)

– Manual Guidance (KUKA, ITIA, COMAU, ABB, IPA, ADDF)

– Programming by Demonstration (ITIA, ABB, KUKA, IPA, AAF)

• Results:

– Set up of test beds and demonstrators

– Interaction with safety aspects e.g. in collision situations


May 7th, 2009

General Topics: Speech Interaction (1/2)

• What has to be done?

– Intuitive, hands-free communication

– Flexible set of commands that can be changedwithout hardware reconfiguration

• Main problems?

– Robustness, high recognition rate

– High noise level

– Direct effect – no long processing

• Approach?

– Use of existing speech recognizers

– Definition of a benchmarking process

– Integration of different microphone types toachieve better signal to noise ratio


May 7th, 2009

General Topics: Speech Interaction (2/2)

• Benchmark

– Tool has been developed

– Preparation of comparable test data

– Evaluation methods

• Throat microphone

– > 90% background noise cancellation

– Algorithms for mapping the frequencycontent of the throat microphone tostandard speech signals

• Demonstrator Integration

– D1 (based on Microsoft SAPI)

– D3 and D4 (based on Sphinx4 framework)


May 7th, 2009

Ontologies for Custom Programming

• Input: AppConf from ontology

• Intermediate: Transformations use an XSLT stylesheet, afterwards eps with interaction. Based on Apache FOP.

• Output:

– Visual layout and interaction as Anoto-Doc

– Web surface

– Voice XML (using TellMe)

Rules and

modality-specific

constraints Product data

Process data

Configura

ble part

Product data

and process

execution

E-form

Anoto PS

Nonconfig

urable

part

XML

appconf

Spoken dialogue

(VoiceXML/SALT)

Values

User

interaction

Gestures

Sensors

Ontology


May 7th, 2009

Overview: Success Stories

Robot Programming for Everyone

Intuitive Programming for Welding Tasks

Touch Screen Based Programming

in a Joiners Workshop

High Level Programming

Teach Wand: Camera based Programming


May 7th, 2009

Success Story: Robot Programming for Everyone

• Today’s robot programming techniques are to time consuming to enable robots to finish low quantities of castings

• A new robot programming concept was developed at ABB that

– enables robot programming by manual guiding

– enables speech communication

– offers an adjustable instruction set according to the tasks and personal needs

– enables integration of Plug’n’Produce* devices

Lead-Through

Programming

Server

*Plug’n’Produce will be abbreviated by PnP in the following


May 7th, 2009

• A simplified robot programming system has to include external devices

• Integration of XIRP-protocol for PnP functionality into Lead-Through Programming System– Automatic Device-discovery

– Server provides PnP services to operators automatically

– Direct integration of PnP into final robot program

• XIRP-protocol implemented on cheap microcontroller as server for simple external devices

External devices in robot applications

Grippers

SensorsCameras

Success Story: Robot Programming for Everyone - PnP


May 7th, 2009

• Based on a risk analysis a special handling device was developed for safe Lead-Through Programming

• Six integrated acknowledge switches enable continuous guiding with regrasping

Success Story: Robot Programming for Everyone - Safety


May 7th, 2009

Graphical visualization of instruction sequence

Lead-Through

Programming

System

Success Story: Robot Programming for Everyone

• Server generates robotprograms in native robotlanguage automatically

• No manual writing of codeneeded


May 7th, 2009

Demo set up at the Automatica 2008 fair

Candy feeder with

PnP functionality

Success Story: Robot Programming for Everyone - Demo


May 7th, 2009

Success Story: Intuitive Programming for Welding Tasks at the Treffler Workplace (1/4)

• Introduction– Application: MAG arc welding of

small lot size work pieces

– End User: Treffler Maschinenbau, a SME in the area of Augsburg, Germany. Contract manufacturing of welding products, approx. 70 workers.

• Devices and Methods:– Programming by Demonstration

– Guiding the robot using a Force Torque Sensor

– Path recording and post processing

– Guiding tool


May 7th, 2009


• Design Criteria

– Low interfering contour

– Guiding in a natural way

• Functionality

– Safety: emergency stop, three stage button, collision switch

– Functionality: force torque sensor, record button, laser scanner

• Evolution

– Three versions made of aluminium profiles and SLS, iterated with comments from Treffler


May 7th, 2009


• Guiding

– Admittance control for manual guidance

– Moving the table with the same interface

• Programming

– Trajectory compression using intelligent algorithms or simple interfaces

– Assigning welding parameters using simple interfaces or learning algorithms

– Adaption of process parameters using a 3-D interface

– Increasing the path accuracy using local sensor data (douglas-peucker, region-growth algorithms)


May 7th, 2009


Result 1:Reduction of

programming time for less experiencedpeople up to 50%

Result 2:Only small

reduction forexperienced robot

programmer

Result 3:Programming therobot for process

experts is possibleafter a short

training (<2 h)


May 7th, 2009

Success Story: KUKA Teach Wand

Camera

Teach

Wand

Enabling Switch

PDA

Teach Wand

Ball Joint

+ PDA integration

+ cheaper tracking

system

+ enabling switch

+ flexible mounting for

pointing device

+ lightweight pointing device

+ form of a welding torch

+ new design for LEDs


May 7th, 2009

Success Story: Smart worker interface for the teach wand

Definition of Motions for Welding

Seams, Approach Movements and

movement Parameters

Online Feedback of Robot Pose, online

correction of orientation and translation


May 7th, 2009


Some Screenshots of the PDA screen while programming a

weld seam

Programming Time for one trial run: Criteria

KCPTeach Wand

# of viapoints 28 28

# of welding points 42 46

# of welding seams 13 13

Seam length [mm] 2940 2940

Duration of the teaching [min] 100 26

Duration test and correction [min] 20 23

Duration complete [min] 120 49

# of points complete 70 74


May 7th, 2009



May 7th, 2009

Final status of D4 demonstrator cell (30 Jan 2009)

Success Story: Touchscreen based Programming in a Joiners Workshop (1/4)

• At Schreinerei

Som: Full

demonstrator

• At Fraunhofer IPA:

Reduced version to

demonstrate a

subset of

functionalities


May 7th, 2009

• Integration and evaluation of a graphics based user input device “Flybook”Results / future use:

– Flybook is used in the development phase of a new teach pendant generation

• Evaluation of a standard PDA as user input device “MDA compact”Results / future use:

– PDA not suitable as real input device for geometry adaption

– PDA can be used offline to generate sketches to be processed by the Sketch Editor



May 7th, 2009

• User interfaces for milling, drilling and spraying– Exemplary screen shots

Drilling

Milling Spraying



May 7th, 2009

• User interface of Sketch Editor– Exemplary screenshot


Time ComparisonMilling of a free form table:

• Manual: 30 min (includingtemplate preparation)

• Woodworking Ass.: 8 min


May 7th, 2009

Success Story: High Level Programming (1/3)

Automatic High Level Programming

• Workflow control

• High-levelprogramming(HLP) commands

• SOA services

• Model-based

• Automaticrobot programgeneration

• Collision avoidance


May 7th, 2009


Interactive High Level Programming

• Set up of robotcell

• Interactive defintion ofsurfaces

• Generation ofrobot path

• Interactive adjustment ofrobot path


May 7th, 2009



May 7th, 2009

Summary, Formal Aspects, Outlook

• Impact: Many devices and methods for human-like instructing of robots have been developed. They were integrated in the four Demonstration scenarios and also many other testbeds

• Improvements towards the SMErobot goals:

– Reduction of programming time: up to 58% when programming manual, >90% for automatic programming

– Simplicity of use: simple, easy to understand interfaces

– Safe operation: permitted

• Scientific Impact:

– Publications, conferences, fairs

• But, most important: Set ups at the end user sites!


May 7th, 2009

Thank you for your attention!

the robot capable of understanding human-like instructions · the robot capable of understanding...

Documents