Modelling of Human Welder for
Intelligent Welding and Welder Training*
YuMing ZhangUniversity of Kentucky
Lee KvidahlIngalls Shipbuilding
NSRP Welding Panel MeetingBethesda, Maryland
May 4-5, 2016
For Unlimited Distribution. Research funded by the NSF under grant “NRI-Small: Virtualized Welding: A NewParadigm for Intelligent Welding Robots in Unstructured Environment,” IIS-1208420, Sept. 2012-August 31, 2016and grant “Machine-Human Cooperative Control of Welding Process” CMMI-0927707, October 2009-Spet. 2013
Contents
1. Sensing of 3D Arc Weld Pool Surface: motivation, method, real time
2. Characterization: from numerous points to three characteristic parameters
3. Control of 3D Weld Pool Surface: control theory method
4. Human Welder Response: modeling and analysis, control using welder model
5. Welder Motion Response: human-robot system, speed adjustment, 3D adjustment
6. Future Directions
1. Sensing of 3D Arc Weld Pool Surface
(Human Response Input)
Weld pool: where complex phenomena originate; but only the surface is
visible; the major feedback information available to human welders
Measurement of weld pool surface temperature distribution: needs the
emissivity to determine the temperature the infrared radiation but the
emissivity is slope dependent
Weld Penetration:
(1) Surface Specular: use laser reflection; (2) Arc Radiation: use laser
reflection and intercept at a distance low power continuous laser for
continuous measurement, no need for a special camera
Laser: 20 mW, 685 nm
Y.M. Zhang, H.S. Song, and G. Saeed. Observation of a dynamic specular weld pool surface. Measurement Science & Technology, 17(6), 2006.
Reflection law, surface constraint, error evaluation
-40 -30 -20 -10 0 10 20 30 400
20
40
60
80
100
120
140
160
180
X/mm
Y/m
m
Reflected dots from image processing and reconstructed surface
Image processed dots
Dots reflected by reconstructed surface
Hongsheng Song. Machine Vision Recognition of Three-Dimensional Specular Surface for
Gas Tungsten Arc Weld Pool. ECE Department, University of Kentucky, 2007.
XiaoJi Ma. Measurement of Dynamic Weld Pool Surface in Gas Metal Arc Welding Process.
Department of Electrical and Computer Engineering, University of Kentucky, Feb. 2012.
Analytical Solution
Key for Real Time Measurement and Control
W.J. Zhang, X.W. Wang, Y.M. Zhang, 2013. “Analytical Real-time
Measurement of Three-dimensional Weld Pool Surface,” Measurement
Science and Technology, 24(11), article Number 115011 (18pp),
doi:10.1088/0957-0233/24/11/115011
2. Characterization of 3D Weld Pool Surface
Characteristic parameters should be used rather than a large set of 3D
coordinates.
Should keep the fundamental information in the weld pool surface about
the weld joint penetration.
W.J. Zhang, Y.K. Liu, X. W. Wang, Y.M. Zhang. Characterization of three-dimensional weld
pool surface in gas tungsten arc welding. Welding Journal, vol. 91, 2012.
Left: Measured 3D weld pool surface parameters from 36 experiments; Right: Least squares model fitting with 3-parameter model using the width, length, and convexity.
1.7906 0.5657 10.8057 0.9868bw W L C
3. Control of 3D Weld Pool Surface
Modeling: how the characteristic parameters respond to the change in
current and travel speed – extract the model from experimental data
Control: Model predictive control algorithm
Yukang Liu, YuMing Zhang. Control of 3D Weld Pool Surface. Control Engineering
Practice, 21(11), 2013.
Welding Experiments
Speed Disturbance
0 20 40 60 80 100 1200
2
4
6
Distance (mm)
Wb (
mm
)
10 20 30 40 50 60 70 80 90 100 110 1200
1
2
3
4
5
6
7
8
Time (s)
Input
Para
mete
rs
Current/10 (A)
Voltage/3 (V)
Speed (mm/s)
10 20 30 40 50 60 70 80 90 100 110 1200
1
2
3
4
5
6
7
Time (s)
Weld
Pool P
ara
mete
rs
Width (mm)
Length (mm)
10*Convexity (mm)
4. Modeling and Analysis of Human Welder Response to 3D Weld Pool Surface
(mechanized welding, human adjusts the current)
Welding Parameters
Current/A Welding speed/mm/s Arc length/mmArgon flow
rate/L/min
57~81 1~2 3.5-4.5 11.8
Monitoring Parameters
Project
angle/°Laser to weld pool
distance/mmImaging plane to weld pool distance/mm
35.5 24.7 101
Camera Parameters
Shutter speed
/msFrame rate/ fps Camera to imaging plane distance/mm
4 30 57.8
Manual control system of GTAW process
Skilled human welder holds the currentregulator while observing the geometry ofweld pool;
Adjusts the welding current to controlthe process to full penetration.
Experiment Parameters
Y.K. Liu, Y.M. Zhang, L. Kvidahl. Skilled Human Welder Intelligence Modeling and Control.
Welding Journal, 93, 2014.
Linear modeling result.
In general, the human intelligent model can be written as:
0 200 400 600 800 1000 1200 1400 1600-4
-2
0
2
4
Sample Number
dC
urr
ent
Measured dCurrent
Linear Estimated dCurrent
( )= ( ( 3), ( 3), ( 3), ( 1))f f fI k f W k L k C k I k
Following linear model can be identified using standard least squares method:
( )= 0.16 ( 3) 0.082 ( 3)+1.81 ( 3)+0.26 ( 1)f f fI k W k L k C k I k
Model comparison between linear and ANFIS model.
Model Comparison between Neuro-Fuzzy Model and linear model
50 100 150
-2
0
2
4
Sample Number
dC
urr
ent
Measured dCurrent
Linear
ANFIS Estimated dCurrent
500 520 540 560 580 600
-2
-1
0
1
2
3
4
5
Sample Number
dC
urr
ent
Measured dCurrent
Linear
ANFIS Estimated dCurrent
1360 1380 1400 1420 1440 1460
-2
-1
0
1
2
3
Sample Number
dC
urr
ent
Measured dCurrent
Linear
ANFIS Estimated dCurrent
Average Model
Error /ARMSE /A
Maximum Model
Error /A
Linear Model 0.52 0.79 3.15
ANFIS Model 0.50 0.76 3.03
Y.K. Liu, W.J. Zhang, Y.M. Zhang. Dynamic neuro-fuzzy-based human intelligence
modeling and control in GTAW. IEEE Transactions on Automation Science and
Engineering, 12, 2015.
Model ComparisonLinear Models
( )= 0.049 ( 3) 0.0049 ( 3)+1.73 ( 3)+0.72 ( 1)I k W k L k C k I k
( )= 0.16 ( 3) 0.082 ( 3)+1.81 ( 3)+0.26 ( 1)f f fI k W k L k C k I k
Novice welder
Skilled welder
Nonlinear Model Comparison
In normal cases the skilled welder'sadjustments are minimal which can preventlarge oscillation and overshoot novice weldermodel suffers;
In other cases where the convexity iseither considerably small or large, theadjustment made by the skilled welder islarger than that of the novice welder, whichcan provide shorter settling time than novicewelder does.
The skilled welder model does providebetter adjustment than the novice welder.
Nonlinear model surface of the neuro-fuzzy human welder model (left: novice welder, right: skilled welder) for convexity = (a) 0.10 mm (b) 0.18mm (c) 0.26mm. Previous response is zero for all cases.
3
4
5
6
3
4
5
6
7
-2
0
2
W (mm)
Novice Welder
L (mm)
I
(A
)
-4
-3.5
-3
-2.5
-2
-1.5
-1
-0.5
0
3
4
5
6
3
4
5
6
7
-2
0
2
W (mm)
Skilled Welder
L (mm)
I
(A
)
-3
-2
-1
0
1
2
3
3
4
5
6
3
4
5
6
7
-2
0
2
W (mm)
Novice Welder
L (mm)
I
(A
)
-0.2
0
0.2
0.4
0.6
0.8
1
3
4
5
6
3
4
5
6
7
-2
0
2
W (mm)
Skilled Welder
L (mm)
I
(A
)
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
3
4
5
6
3
4
5
6
7
-2
0
2
W (mm)
Novice Welder
L (mm)
I
(A
)
0
0.2
0.4
0.6
0.8
1
3
4
5
6
3
4
5
6
7
-2
0
2
W (mm)
Skilled Welder
L (mm)
I
(A
)
-0.5
0
0.5
1
1.5
2
2.5
3
(a)
(b)
(c)
Control Experiments: Varying Initial Current
Control experiment result with different initial current (a) 52A; (b) 54A.
A B
A B
0 10 20 30 40 50 60 70 80 90 100 1100
1
2
3
4
5
6
7
8
Time (s)
Input
Para
mete
rs
Current/10 (A)
Voltage/3 (V)
Speed (mm/s)
BA
(a) (b)
A B
BA
10 20 30 40 50 60 70 800
1
2
3
4
5
6
7
8
Time (s)
Input
Para
mete
rs
Current/10 (A)
Voltage/3 (V)
Speed (mm/s)
BA
5. Welder Motion Response to Weld Pool Surface
(welding speed adjustment, 3D operation adjustment)
1. Equipment for experimental data: human-robot system
2. Extract good response from not-perfect performance of human welder
3. Quality evaluation model
4. Supervised learning using good data
Y.K. Liu, Y.M. Zhang, L. Kvidahl, 2014. “Skilled Human Welder Intelligence Modeling and
Control: Part I-Modeling,” Welding Journal, 93: 46s-52s.
Y.K. Liu, Y.M. Zhang, L. Kvidahl, 2014. “Skilled Human Welder Intelligence Modeling and
Control: Part II-Analysis and Control Applications,” Welding Journal, 93(5): 162s-170s.
Equipment - Virtualized Welding System
Illustration of virtualized welding operation.
Developed virtualized welding system.
In virtual station a humanwelder can view the mockup and moves the virtualwelding torch accordingly asif he/she is right in front ofthe work-piece;In welding station a robotarm (Universal Robot UR-5with six Degree of Freedom)equipped with the weldingtorch receives commands viaEthernet and performsGTAW.
Virtual Station and Welding Station
Virtual welding torch.
Equipment - Virtualized Welding System
Detailed view of the virtual station.
3D weld pool sensing system.
A low power laser (19 by 19 structure lightpattern) is projected to the weld pool surface;
Its reflection from the specular weld poolsurface is intercepted and imaged by Camera 1;
Weld pool image captured by Camera 2 (with orwithout virtual reality (VR) enhancement)
Major components in virtual station include aLeap motion tracking sensor, a mock up pipe, acomputer screen, and a projector.
3D scanning system
Visualization
system Motion sensor
Camera
Projector
Mockup
Screen
Detailed view of the visualization result.
Learning Experiments for Response Data
Teleoperation training experiments The welder observe the weld pool under
random welding current, and move the virtualwelding torch accordingly.
0 200 400 600 800 1000 1200 1400 1600 1800 20000
2
4
6
8
10
12
Sample Number
Input
and O
utp
ut
Speed*3 (mm/s) Current/5 (A) Speedf*3 (mm/s) Width (mm) Length (mm) Convexity*10 (mm)
Measured welding current, weld pool characteristic parameters and human arm
movement speed in thirteen training experiments.
Rating of Performance and Response Data
Welder Rating System To better distill the correct response of the human welder, the human
welder evaluates the measured data and corresponding back-side
weld penetration and assigns a rating (from 0 to 10) in each 5 s interval
(offline, thus less skill demanding).
0 200 400 600 800 1000 1200 1400 1600 1800 20000
5
10
Sample Number
Rating (
0-1
0)
Rating assigned by human welder in thirteen dynamic experiments.
Y.K. Liu, Y.M. Zhang, “Iterative local ANFIS based human welder intelligence modeling and
control in pipe GTAW process: A data-driven approach,” IEEE/ASME Transactions on
Mechatronics, DOI: 10.1109/TMECH.2014.2363050.
ANFIS Based Automated Rating of Data Quality
Welder Rating System
Both linear and ANFIS models are proposed to automate the
rating
Linear and ANFIS modeling of the rating (Welder Rating System)
0 200 400 600 800 1000 1200 1400 1600 1800 20000
2
4
6
8
10
12
Sample Number
Hum
an W
eld
er R
ating
Human Welder Rating
Linear Estimated Rating
ANFIS Estimated Rating
( ) ( ), ( ), ( ), ( )R k f W k L k C k S k
Supervised Modeling
Good Response data (with ratings greater than 8) are used to model how
human welder adjusts the speed per weld pool surface.
0 100 200 300 400 500 600 7000.6
0.8
1
1.2
Sample Number
Speedf
(mm
/s)
Measured Speedf Linear Estimated Speedf ANFIS Estimated Speedf
Linear and supervised ANFIS modeling of the welder response.
3D Operation Learning and Robot Implementation of Human Response
Experiment 1: Different Welding Current
Experiment 2: Speed Disturbance
40 50 60 70 80 90 100 1100
1
2
3
4
5
6
7
8
9
Time (s)
Weld
ing C
urr
ent
and P
ool P
ara
mete
rs
Current/10 (A)
Width (mm)
Length (mm)
Convexity*10 (mm)
40 50 60 70 80 90 100 110-0.5
0
0.5
1
Time (s)
Contr
ol In
puts
Speed (mm/s)
Z Adjustment (mm)
RX Adjustment/10 (deg)
RY Adjustment/10 (deg)
50 60 70 80 90 100 1100
2
4
6
8
10
Time (s)
Weld
ing C
urr
ent
and P
ool P
ara
mete
rs
Current/10 (A)
Width (mm)
Length (mm)
Convexity*10 (mm)
50 60 70 80 90 100 110-0.5
0
0.5
1
Time (s)
Contr
ol In
puts
Speed (mm/s)
Z Adjustment (mm)
RX Adjustment/10 (deg)
RY Adjustment/10 (deg)
6. Future Directions
o Varying Gap feedforward + feedback
o Operation inconsistence: automatic rating of operation quality
o Better Free Demonstration of Human Skills IMU sensor on torch, co-view
helmet
o What is an Co-View Helmet?
o Welder Operation Documentation: Heat Input/Welding Speed/Torch
Orientation/O’clock Position, Weld Pool Size and Shape
o Welder Operation Modeling and Diagnosis
o Improvement from Comparison with Skilled Welder Model
o Program Welding Robot for Intelligent Control
o Automatic Welding Parameters Adjustment Based on Speed and Torch
Orientation/Angle o
Key: Mobile Sensor
- Projective Torch : grid
pattern and dot matrix pattern
attached, as well as one IMU
- Shield glass (simulated by a
piece of glass covered by a paper)
- Sensory Helmet (camera
tripod to simulate the head
movement in 6 DOF)
- Weld pool (Convex spherical
mirror with known geometry)
W.J. Zhang, J. Xiao, Y.M. Zhang, 2016. “A mobile sensing system for three-dimensional weld
pool measurement in manual GTAW process," Measurement Science and Technology, 27 (2016)
045102 (24pp), doi:10.1088/0957-0233/27/4/045102.
Experimental system configuration
Simmer Wireless Inertia Measurement Unit includes:
1. tri-axial accelerometer (Freescale MMA7260Q)
2. tri-axial gyro sensor (InvenSense 500 series)
3. a magnetometer
4. a microprocessor (MSP430F1611)
5. a Bluetooth unit.
Key Component - Inertial Measurement Unit (IMU)
W.J. Zhang, J. Xiao, Y.M. Zhang, 2014. “Navigation of welding
torch for arc welding process,” Preprints of the 19th World
Congress of The International Federation of Automatic
Control, pp. 7158-7163, Cape Town, South Africa, August 24-
29, 2014.
Position experiment
The torch is smoothly moved
along the 3-D since curve. The results of torch trajectory position estimation
Measurement errors in Position Experiment 2
Tracking Accuracy Verification
W.J. Zhang, J. Xiao, Y.M. Zhang, 2014. “Navigation of welding torch for arc welding process,”
Preprints of the 19th World Congress of The International Federation of Automatic Control, pp.
7158-7163, Cape Town, South Africa, August 24-29, 2014.