Halden Project VR Workshop, 2-3 March 2005 1

Development of a Tracking Method for Augmented Reality Applied to

Nuclear Plant Maintenance Work

Presentation by Hirotake IshiiResearch Associate, Kyoto University, Japan

Guest Scientist, Institute for Energy Technology, Norway

Halden Project VR Workshop, 2-3 March 2005 2

Background

Serious situation of NPPBoth of improvement of safety and reduction of cost are required.

• Introduction of free electricity market.• Difficulties of maintenance for aged NPPs.• Decrease of expert maintenance workers.

Need further development of hardware and software for NPP operation

Augmented Reality (AR) is one of the promising technologies that can improve efficiency and safety.

Support for maintenance work in plant fieldThere are some rooms to improve its efficiency and safety by introducing state-of-the-art information technologies.

Halden Project VR Workshop, 2-3 March 2005 3

What is the AR?Augmented Reality(AR) expands the surrounding real world by superimposing computer-generated information on the user’s view.

By using Augmented Reality, it becomes possible for the workers to understand various information of the maintenance work intuitively.

Destination UPUPUP

Paper Manual AR

HMD

Superimposed information

Halden Project VR Workshop, 2-3 March 2005 4

Key Technology of the AR

TrackingMeasure the position and rotation of user’s view in real time to superimpose virtual object/information at correct position.

Many kinds of tracking methods are developedGPS (Differential GPS, Real Time Kinematics GPS)Ultrasonic/Magnetic/Infrared SensorInertial SensorArtificial Marker / Marker-lessHybrid of above (Combination)

can not be used in NPP

Halden Project VR Workshop, 2-3 March 2005 5

Requirement of Tracking Method applied to NPP

Limitations of NPP field from the viewpoint of trackingIndoorVarious size of equipment in wide areaLots of metal objects / obstacles / magnetic source

RequirementWide areaAccuracy and stabilityEasy / no installationLess expensive

• The artificial marker technique has a possibility to be used in a plant.

Halden Project VR Workshop, 2-3 March 2005 6

Artificial Marker Technique

Artificial Marker TechniqueCalculates the position and rotation of the camera from the position of markers pasted in the environment by image processing technique.Is applied to many AR applications.

StrengthsAccurate and stableLess expensiveHigh scalability

WeaknessesAvailable only when the distance between the marker and the camera is short (this means many markers need to be pasted in the environment) or the size of the markers is large.

Halden Project VR Workshop, 2-3 March 2005 7

Available Distance of Artificial Marker

Maximum distanceARToolKit (VGA, f=4mm)

1m : 8 cm3m : 25 cm5m : 40 cm

ProblemThere are many small objects like pipes in a plant.The surface of the objects is not flat.It is difficult to paste large markers.

It is necessary to make the markers smaller or make it easier to be pasted in a complicated environment

Halden Project VR Workshop, 2-3 March 2005 8

Development of a New Tracking System

Tracking system using barcode markerLong barcode markerEasy to paste on pipes

Tracking system using circular markerCircle shaped marker with code insideStable recognition in long distance

Halden Project VR Workshop, 2-3 March 2005 9

Tracking System using Barcode Marker

by Hiroshi Shimoda, Hirotake Ishii, Masayuki Maeshima,

Toshinori Nakai, Zhiqiang Bian and Hidekazu Yoshikawa, (Kyoto University)

Halden Project VR Workshop, 2-3 March 2005 10

Design of Barcode Marker

ID of marker (7 bits) Humming code (4 bits)

40mm 80mm

40mm

20mmCode”0” Code”1” Gap

11 bits data7 bits for ID (128 kinds of markers)4 bits for error correction (Humming code)

LengthFrom 640 mm to 1080 mm

Halden Project VR Workshop, 2-3 March 2005 11

Image of Tracking Method with Barcode Marker

　　 Field WorkerBarcode Marker

Check the crack of

the upper pipe.

Worker’s viewInstruction informationby AR

Small CameraHMD

Barcode markers are pasted on pipes3 position (both edge and middle point) of each marker are registered in advancePosition and rotation of workers are calculated by using 2 barcode markers(Solving P6P problem)

Halden Project VR Workshop, 2-3 March 2005 12

Marker Recognition Algorithm(1) Captured Image:

Halden Project VR Workshop, 2-3 March 2005 13

Marker Recognition Algorithm(2) Binarization:

Binarize the captured image with the camera at preset threshold level.

Halden Project VR Workshop, 2-3 March 2005 14

Marker Recognition Algorithm(3) Labelling:

Collect the connected pixels and mark a unique label on the connected part.

Halden Project VR Workshop, 2-3 March 2005 15

Marker Recognition Algorithm(4) Narrowing search area:

Exclude the parts which have no possibility as the part of the marker by its area and shape.

Halden Project VR Workshop, 2-3 March 2005 16

Marker Recognition Algorithm(5) Extraction parts of marker:

Pick up the 11 parts which are arranged in a line as a candidate of barcode marker.

Halden Project VR Workshop, 2-3 March 2005 17

Marker Recognition Algorithm(6) Decision of code:

Decide the code of barcode marker from the area of each part.

01

00

0

0

11

00

1

Halden Project VR Workshop, 2-3 March 2005 18

Marker Recognition Algorithm(7) Comparison with pre-registered barcode marker:

Correct the code of the marker with Humming code part and compare it with pre-registered barcode marker.

Halden Project VR Workshop, 2-3 March 2005 19

Marker Recognition Algorithm(8) Calculate position and rotation of camera

Extract 3 points from 2 barcode markers and solve P6P problem

Halden Project VR Workshop, 2-3 March 2005 20

Basic Evaluation of Marker Recognition

PurposeEvaluate basic ability of proposed tracking method

Experimental Method

Horizontal arrangement Vertical arrangement

Rotation

Distance

120 lux

Rotation

Pipe: 60φ×1100 mm

Pipe arrangement:Horizontal / Vertical

Rotation:0 / 20 / 40 / 60 / 80 degree

Distance:1 / 2 / 3 / 4 / 5 / 6 meters

Camera

Halden Project VR Workshop, 2-3 March 2005 21

Examples of Captured Images

Vertical, 0 degree, 6 meters

Vertical, 80 degree, 4 meters

Camera Resolution :H320xV240

Halden Project VR Workshop, 2-3 March 2005 22

Recognition Result

Distance between marker and camera Pipe Arrangement Rotation Angle 1m 2m 3m 4m 5m 6m

0 degree OK OK OK OK OK OK 20 degree OK OK OK OK OK - 40 degree OK OK OK OK - - 60 degree OK OK - - - -

Horizontal

80 degree - OK - - - - 0 degree OK OK OK OK OK OK

20 degree OK OK OK OK OK OK 40 degree OK OK OK OK - - 60 degree OK OK - - - -

Vertical

80 degree - - - - - -

“OK” means that the marker could be recognized. “-“ means that it could not be recognized.

Camera resolution: 320 x 240Viewing angle: 53 x 40 degree

Recognition rate: 10 – 30 fps

(Pentium Mobile 1.4GHz)

Halden Project VR Workshop, 2-3 March 2005 23

Trial Use in Fugen NPP

Trial Area

Water Purification Facility 10 barcode markers pasted in the area

All marker ID and positions were registered in advance.

Halden Project VR Workshop, 2-3 March 2005 24

Example

Recognized Markers

Halden Project VR Workshop, 2-3 March 2005 25

Result

Walked around the area with prototype system.1000 frame images were picked up, in which at least one marker was in the view.Recognition rate: 52.8% (47.2% failed)Erroneous recognition rate: 1.8%Cases of erroneous recognition

a marker image was captured against the light,a marker was too far from the camera, andthe angle of a marker against the camera direction was too large.

Halden Project VR Workshop, 2-3 March 2005 26

Example of Erroneous Recognition (Backlight)

Halden Project VR Workshop, 2-3 March 2005 27

Conclusion (Barcode Marker)

Proposed marker-based tracking method for AR support of NPP maintenance work.

Long barcode marker and simple image recognition.

Evaluated basic ability of proposed tracking method in a laboratory.

Long distance, enough fast and feasible.

Trial use in Fugen NPPRecognition rate: 52.8%, erroneous recognition rate: 1.8%

Halden Project VR Workshop, 2-3 March 2005 28

Tracking System using Circular Marker

byHirotake Ishii, Hidenori Fujino (Kyoto University)

Asgeir Droivoldsmo (Institute for Energy Technology)

Halden Project VR Workshop, 2-3 March 2005 29

Weakness of Square Markers

How to recognize square markers

HIRO

1. Detect edges 2. Detect 4 lines3. Calculate intersections

4. Calculate position and rotation

Before binarize

After binarize

　　　　　　　　　　　　　

　　　　　　　　　　　　●●　　　　　　　●●●●●　　●●　　　●●●●●　　　　　　　●　　●　　　　　　　　　　　　●　　　●　　　　　　　　　　　　●　　　●　　　　　　　　　　　　●　　　●●　　　　　　　　　　　●　　　　●　　　　　　　　　　　　●　　　　●　　　　　　　　　　　　●

●　　　　　　　　　　●●●　　　　　●●●●●　 ●●●●●

Real lineEstimatedline

Easily affected by jaggy shaped edges

Distance between features

Max half meters

Distance between features is short(Max to size of marker)

Accuracy depends on the distance between the features.

Halden Project VR Workshop, 2-3 March 2005 30

The center can be recognized accurately in any situation

In Case of Circular Marker

Well focused Badly focused Long distance

Distance between features can be long.Distance

between features

Max several meters

The center can be recognized accurately in any situation

Triangulation by plural markers can be used.

Halden Project VR Workshop, 2-3 March 2005 31

Design of Circular Marker

Outer black circle

Center white circle

Make as simple as possible

Middle code area circle which consists of black or white fans

Outer black circle and center white circle are used for calculating the threshold to analyze the code area.Diameter of the marker and the number of division of middle circle can be changed according to the situation.

Number of division

Number of unique markers

9 56

10 99

11 188

Halden Project VR Workshop, 2-3 March 2005 32

Marker Recognition Algorithm (1)

Captured Image

Halden Project VR Workshop, 2-3 March 2005 33

Marker Recognition Algorithm (2)

Calculation of logarithm

Halden Project VR Workshop, 2-3 March 2005 34

Marker Recognition Algorithm (3)

3x3 Sobel Filter

Halden Project VR Workshop, 2-3 March 2005 35

Marker Recognition Algorithm (4)

Labeling

Halden Project VR Workshop, 2-3 March 2005 36

Marker Recognition Algorithm (5)

Eliminate small area

Halden Project VR Workshop, 2-3 March 2005 37

Marker Recognition Algorithm (6)

Recognize ellipse and eliminate non-ellipse area

Halden Project VR Workshop, 2-3 March 2005 38

Marker Recognition Algorithm (7)

Recognize ID of marker and eliminate non-marker area

Details are in the paper

21

7

3

Halden Project VR Workshop, 2-3 March 2005 39

Marker Recognition Algorithm (8)

Result

21

7

3

Halden Project VR Workshop, 2-3 March 2005 40

Calculate position and rotation of camera

Calculate by using both of the solutions from PnP solver and the rough information of each circular marker’s rotation (Details are in the paper)

Marker position on the image Marker shape on the image

Accurate but plural solutionsSingle but unaccurate solution

Compare

Single and accurate solution

PnP solver

Halden Project VR Workshop, 2-3 March 2005 41

Developed Software

Camera Controllibrary (C++)

TCP/IP DLLfor Client

MarkerDesign

Printed Markers Printer Marker Maker

MarkerLocation

(XML)

Marker Visualize

Core library(C++)

TCP/IP libraryfor Server

Link

Link

Link

Camera DriverCMU

Camera DriverPGR

IEEE1394(IIDC)

DragonFlyFireFly

USB, DV

Direct ShowUnder development

TrackingResult

(Via TCP/IP)

TrackingServer

OpenGLDirectX

JAVA3DJNI

Display Software

Halden Project VR Workshop, 2-3 March 2005 42

Evaluation of the accuracy and stability (Single marker)

Experimental MethodOne circular marker (diameter is 40mm) was pasted on a small box.The number of the division of middle circle was 8,9,10.Distance was changed from 300cm to 580cm with 20cm step.Angle was changed from 0 degree to 75 degrees with 15 degrees step.For each condition, 100 images were captured and the position and rotation was calculated. The average and the variance were calculated.

Halden Project VR Workshop, 2-3 March 2005 43

Example of the image

(Distance:560cm, Angle:0 degree)

Camera resolution: H1024 x V768Focal Length: 8mm

Halden Project VR Workshop, 2-3 March 2005 44

Results (Maximum distance)

Number of

Division

Angle (deg.)0 15 30 45 60 75

8 520/560

520/560

500/560

440/520

380/440

X/300

9 520/560

520/560

500/560

440/520

380/440

X/X

10 520/560

520/560

500/560

400/500

380/420

X/X

Table : Maximum distance (diameter is 4cm)

Succeeded in all frames(cm) / Failed in some frames(cm)

Maximum distance of the circular marker is about 2 times of the square marker such as ARToolKit

Camera resolution: H1024 x V768Focal Length: 8mm

Halden Project VR Workshop, 2-3 March 2005 45

Results (single marker, depth)

The accuracy of the position is not good.

-300

-250

-200

-150

-100

-50

0

50

100

150

3000 3200 3400 3600 3800 4000 4200 4400 4600 4800 5000 5200 5400 5600 5800

Distance from marker[mm]

Erro

r[m

m] 0

15304560

Angle of marker[deg.]

Halden Project VR Workshop, 2-3 March 2005 46

Results (single marker, rotation)

Angle of marker[deg.]0

15

30

45

60

-15

0

15

30

45

60

75

3000 3200 3400 3600 3800 4000 4200 4400 4600 4800 5000 5200 5400 5600 5800

Distance from marker[mm]

Ang

le[d

eg.]

Halden Project VR Workshop, 2-3 March 2005 47

Evaluation of the accuracy and stability (plural markers on a helmet)

Experimental Method22 circular markers were pasted on a helmet.The number of the divisionof middle circle was 8.Distance was changed from 300cm to 560cm with 20cm step.Angle was changed from 0 degree to 180 degrees with 15 degrees step.For each condition, 100 images were captured and the position and rotation was calculated. The average and the variance were calculated.

1

2

3

45

67

8

9

10

11

12

13

14

1516

17

18

19

2021

22

Halden Project VR Workshop, 2-3 March 2005 48

Example of the image

(Distance:560cm, Angle:120 degree)

Camera resolution: H1024 x V768Focal Length: 8mm

Halden Project VR Workshop, 2-3 March 2005 49

Results (Position)

The accuracy is greatly improved

0153045607590

105120135150165180

Angle of helmet[deg.]

-300

-250

-200

-150

-100

-50

0

50

100

150

Erro

r[m

m]

3000 3200 3400 3600 3800 4000 4200 4400 4600 4800 5000 5200 5400 5600 5800

Distance from helmet[mm] Black : plural MarkersRed : Single Marker

Halden Project VR Workshop, 2-3 March 2005 50

Results (Rotation)

Tracked in wide angle. In the case of single marker, maximum angle is about 60 degrees.

-15

0

15

30

45

60

75

90

105

120

135

150

165

180

195

3000 3200 3400 3600 3800 4000 4200 4400 4600 4800 5000 5200 5400 5600 5800

Distance from helmet[mm]

Angl

e[de

g.]

0153045607590

105120135150165180

Angle of helmet[deg.]

Halden Project VR Workshop, 2-3 March 2005 51

Application Example : Tracking in the Office

Halden Project VR Workshop, 2-3 March 2005 52

Conclusion (Circular marker)

New circular marker has been designedTracking system that recognizes plural circular markers at one time and calculates the position and rotation of the camera by triangular method has been developed.Two experiments have been conducted in order to evaluate the accuracy and stability of the proposed method.It was confirmed that the accuracy can be greatly improved by using plural markers at one time and the distance between the marker and the camera can be long compared to the conventional method.

Halden Project VR Workshop, 2-3 March 2005 53

Future Works

Barcode MarkerImprovement of proposed method (Multi-camera, camera resolution, recognition algorithm, etc.)

Circular MarkerApply to real applications. (Visualization of radiation map in NPP, old church)

Development of hybrid trackingDeveloped tracking methods can be used with other tracking methods such as ARToolKit at the same time.Combine marker-less method (short distance), square marker (middle distance), barcode marker and circular marker (long distance)