advanced vision guided robotics - aia - vision online · advanced vision guided robotics steven...

Advanced Vision Guided Robotics

Steven Prehn Robotic Guidance, LLC

Traditional Vision vs. Vision based Robot Guidance

• Traditional Machine Vision – – Determine if a product Passes or Fails

• Assembly Verification • Find Defects • Gauging/Metrology

• Vision Guided Robots –

– It all about Location, • Locate and Pick Parts • Locate and move relative to Parts for Assembly • Locate parts and remove flash or apply epoxy

Robotic System Adaptability

• Basic Premise: Vision Guidance is needed

when the part is not always in the same position

J2

J3

J1

J4

J6

J5

TCP Uframe

Understanding Cameras

Digital Imager Array Megapixels Pixels

Imager Chip

Horizontal Pixels

Vertical Pixels 1/3"

CCD – Charged Coupled Device

Camera

Color Image Converted to Gray Scale

Continuous Image Converted to “Picture Elements” Or Pixels

Voltage = 1.0

Voltage = 0

Voltage = .5

CCD

CCD

CCD

Volts

Black = 0

Brightest = 255

Mid Gray = 128

Gray Scale

Gray Scale = Measured Light Level

Picture Elements – Pixels

255 255 255 255 255 255 255 255

255 255 255 255 255 255 128 128

255 255 128 255 255 255 128 128

255 255 128 128 255 255 255 128

255 255 128 128 128 255 255 128

255 255 128 255 255 255 255 255

255 255 128 255 255 255 255 255

255 255 255 255 255 255 255 255

1,1 1,2 1,3 1,4 1,5 1,6 1,7 1,8

2,1 2,2 2,3 2,4 2,5 2,6 2,7 2,8

3,1 3,2 3,3 3,4 3,5 3,6 3,7 3,8

4,1 4,2 4,3 4,4 4,5 4,6 4,7 4,8

5,1 5,2 5,3 5,4 5,5 5,6 5,7 5,8

6,1 6,2 6,3 6,4 6,5 6,6 6,7 6,8

7,1 7,2 7,3 7,4 7,5 7,6 7,7 7,8

8,1 8,2 8,3 8,4 8,5 8,6 8,7 8,8

grid location and gray scale value

Pixels have Two Properties:

An imager chip is a bunch of little light meters

Lens Mathematics

• Focal Length is the distance in millimeters from the optical center of a lens to the imaging sensor (when the lens is focused at infinity).

• Calculate Viewing Area – Working Distance from Camera to Object – Desired Field of View; Width of Height – Image Array Size: 1/3”, ½”, 2/3” Etc. – Lens Size – Focal Length

6mm 25mm

Lens Calculations

You can solve for any one of the missing parameters, if you know the other three

C-mount provides a fixed standard distance to imager

Joint to Cartesian Space

• Robot kinematics is the study of the motion (kinematics) of robots. In a kinematic analysis the position, velocity and acceleration of all the links are calculated without considering the forces that cause this motion. -1

• Joint angles (provided by encoders) and arm segment lengths are combined to render position and orientation

-1 Reference Wikipedia

Understanding Cartesian Coordinates

• Consider the Robots Face Plate Position with respect to a part (X,Y and Z)

• Now consider a plane pivoting around this point. Rotating around X,Y and Z.

• This is part positioning with 6 degrees of freedom.

• Our ultimate goal: How can vision be used to guide the

robot to the position of a part?

Position Relationships

• Start with a Cartesian coordinate system for rendering position (X,Y,Z)

• R is the position of the platter relative to the room.

• F is the position of the furniture in the World coordinate system.

• P is the position of the platter in the furniture frame or furniture coordinate system.

• Now consider a table where adjacent legs are shorter and a platter where one side is significantly tilted

R

F

P

X

Z

Y

Room

Furniture

Platter

World and Tool Frames

+X

+Z

+X

+Z

World frame

Tool frame

+Y

+Z

Top view

Tool frame

Two Key Contributors to the Robots Position User Frames

The Primary Robot position Coordinate system is referred to as the World Frame

Other frames can be created that are positioned relative to the world

Tool Frames

Using Kinematics the world frame, and Tool Frame, the robot can be used to create positions and establish planes

• World frame - default frame of the robot • User frame - user defined frame • Tool frame - user defined frame

User Frame

Robot

World Coordinate

System

$MNUFRAME[1,frame_num]

$MNUTOOL[1,tool_num]

• Tool Center Point

• TCP

• TOOL

• UTOOL

• UT

• User Frame

• USER

• UFRAME

• UF

Frames Important to Vision

• When a point is recorded, it references both the Tool Frame and the User Frame

User Frame

Robot

Positional Data

World Coordinate

System

Tool Frame

Tool Frame and Programming

• The Tool Frame is what the robot “looks at”

when you ask it to do a Linear or Circular movement.

• User Frames are based off of the World Frame

• They can be taught at any location and any orientation

• Positions are taught with respect to a User Frame (as well as, a Tool Frame)

User Frame

Robot

Positional Data

World Coordinate System

Tool Frame

User Frame

Robot and Tool Positions Relative to the Part

• The robots position and orientation is reported with six degrees of freedom – X,Y,Z, Pitch, Yaw, Roll

• The tool has a position and orientation, too.

• Robot and Tool frames are combined to find a position relative to the part. – This relationship is resolved

before moving the robot the part.

R T

P

Simple Camera Calibration

• Place a grid with known spacing at the same height as the top of the parts to translate Pixels to Units of Measure – (Grid to Camera)

• Tie in the orientation of the camera to the grid – Which direction representing X and Y

• Tie in the relationship of the grid to the robot – (Grid to Robot)

• Now we can determine the parts position relative to the robot

X

Z

X

Z

Part Height Variation Problem

ΔX

Vision Calibration and the User Frame

In order for visual offset data to be useful to the robot, both vision and the robot must recognize the same coordinate system.

This involves establishing a correspondence between the camera's coordinate system and a robot user frame.

To accomplish this a robot frame is

taught that corresponds to a frame used to calibrate vision.

Basic Cameras are 2D

• What aspects of a parts position relative to the robot can we determine solely from a 2D image?

• Variables that you may need to know: – Distance from the camera – Magnification of the lens – Size of the part – Calibration (EG: pixels/mm) – Orientation of the camera in space – Orientation of the robot in space

• How does the distance away from the camera effect the part position calculation?

Image to Robot Relationship

In two-dimensional applications, the XY plane of the user frame specified here should be parallel to the target work plane. How do you compensate when this is not the case?

Vision To Robot Transformations Considerations

• Camera mounting style – Fixed position or Robot mounted camera

• Cycle time • Size of part (FOV) vs. accuracy needed

• Part Presentation issues – In which axis's is the part likely move?

• X, Y, Rotation, Z, Pitch and Yaw – Is the part consistent or is its presentation consistent – Is it possible to correlate position from different

perspectives? – Can structured light be used to help identify location?

• Origin is called the Tool Center Point or TCP – Defines location of where work is done

• Default location is center of the faceplate

• Origin of the tool frame must be offset to a fixed point on the physical tool

• All offset measurements for the Tool Frame are relative to origin of the Face Plate

Tool Frame

Matrix Multiplication

If A is an n × m matrix and B is an m × p matrix,

the matrix product A:B (denoted without multiplication signs or dots) is defined to be the n × p matrix

* Reference Source – Wikipedia Matrix multiplication

Basic Matrix Math Continued • Inverse Matrix

– If A is a square matrix, there may be an inverse matrix A−1 of A such that

– where I is the identity matrix of the same order.

* Reference Source – Wikipedia Matrix multiplication

• To isolate R (the robot’s position) we performing algebraic manipulation of the frame matrices as follows:

where T-1 is the inverse of the tool.

• Since

Example of Basic Robot Math

R T

F

P

Standard Robot Equation: R : T = F : P

11 :::: −− = TPFTTR

ITT =−1:

1::: −= TPFIR1:: −= TPFR

Guidance Summary 1. Robot’s Knowledge of Calibration Frame (Used by the

camera) 2. Camera Calibration Assigns the translation to be Used. 3. Vision Process is Used to locate the Part Position

– Requires a Z height of the part relative to the frame of reference – Record a part reference position

4. Move the robot to pick the part and record this position.

The Camera is used to find the part and calculate its position. The robot is moved to a location relative to the part with

offsets appiled.

Visual Tracking

• 2D Line Tracking – X, Y location and angle orientation PLUS conveyor

position – Cue Management

Offset calculation

Camera

Pulsecoder

Conveyor

Random feeding

Basic 3D Robot Guidance Methods

• 2.5 D – For Rough Z approximation • 2.5 D – With Structured Light Reference • Single View 3D using Geometric Relationships • Multiple laser or structured light pattern

triangulation methods • Advanced 3D – (depth analysis)

2D Guidance with a Change in Z

• The image created by the camera is like looking through a Cone

• The ratio of Pixels to Units of Measure changes as you move within the cone

• If the part distance from the camera is not identical to when the camera was calibrated, finding a parts position accurately requires adjustment of the transformation.

• How do you know the part height? • What can be leveraged?

– Part scale, height sensors – Lens mathematics

2D Single Camera - 2.5 D Camera Image

Height change creates subtle apparent size change. Are you sure the part size is not different – creating the same affect?

• Apparent Part Size can be used to calculate relative height

• The height designates where in the calibration cone it is

• The transformation is adjusted relative to the frame of reference

Top Part

Bottom Part

Depalletizing

The world is not flat…

Traditional cameras see a flat world – 2D and flat

Robots work in the real world and must compensate for a parts position with 6 degrees of freedom

World Frame

• World frame has its origin at the intersection of axes 1 and 2

• Aid to Remember Coordinate Relationships: the Right Hand Rule

+Y

Origin

+X

-X

-Z

+Z

-Y

2D Robotic Assumptions • 2D imaging systems can be used if:

– The part always sits flat on a surface or fixture (no pitch or yaw changes)

– The part is consistent in its size and shape – The tool is designed to compensate for any variation in height

(and subsequent X, Y error) • 2D is not a good solution when:

– Parts are stacked and may be subject to tipping – Parts are randomly placed in a bin for picking – Parts enter the robot cell on a pallet that is damaged, or on a

conveyor that wobbles – high accuracy assembly process like hanging a door on an

automobile

Example 3D Robot Applications

• Racking and Deracking • Palletizing and Depalletizing • Welding uneven surfaces • Grinding and flash removal • Machine load • High accuracy assembly • Parts on Hangers • Picking Stacked parts • Picking parts randomly located in

bins

Laser

Light Plane

Light Stripe on work piece

Laser Line/Structured Light

Camera sees this view

Camera is at 90° and Laser at 45°

Lasers Vertical Position determines Z

Structured Light in The Real World

Hex wrenches with laser line Curved surface with laser line

A 2D Change of Perspective

• As part orientation changes in pitch and yaw, surface points converge or diverge.

Camera Image

Plane 1 Plane 2

Distance Moved

Grid Camera Camera

Calibration – Two Plane Method

• Requires either the robot to move the camera or the robot to move the grid.

• Renders greater vector accuracy. • Helps improve lens math.

Applying Geometric Relationships

• Identify fixed and reliable geometric features (corners or holes)

• Apply Geometric Position Relationships between features

• Compensate for Perspective

Geometric Relationships • Start with a known shape • Extract feature Point Position

with respect to calibrated cameras

• The part shape is assumed to be constant although position is not

• Combine camera position relationship with found feature to extract new position

Advanced 3D

• Several Companies are advancing 3D point cloud image generation

• Technologies include: – Time of Flight Sensors – Morei Interferometry – Structured Light – Stereo Projection – others…

• Processing renders part position by matching features in 3D Space

• Translation of the found position with respect to original position remains consistent to previous robot math examples

3D Sensors – From Wikipedia

FANUC 3D Area Sensor

"TOF Kamera 3D Gesicht" by Captaindistance - Own work. Licensed under CC BY 3.0 via Wikimedia Commons - http://commons.wikimedia.org/wiki/File:TOF_Kamera_3D_Gesicht.jpg#mediaviewer/File:TOF_Kamera_3D_Gesicht.jpg

• Bin – Define the size and location of the bin

• Robot – Model the EOAT and setup check to keep robot and EOAT from contacting bin

Break Down of Vision Tasks

Bin Avoidance

Bin Picking

• Locate the Bin • Locate Candidate Parts • Move to pick Candidates without collision • Remove parts without Collision

Summary

• Machine vision has progressed significantly in the last 10 to 15 years

• Advances in technology continue to provide new capabilities

• Pay close attention to 3D enhancements for Robotic Guidance Applications

Contact Information

Steven Prehn Robotic Guidance, LLC USA

Email: stevenprehn@roboticguidance.com