the augmented mine worker - dxc...

ABSTRACTThe mining industry is faced with imperatives to improve worker safety and pro-ductivity, adapt to skills shortages, high worker turnover rates, and provide more effective maintenance to new and ever more complicated plant and equipment. While more effective use of data combined with advanced analytics offers oppor-tunities for improvement, providing this information to the worker in the field, in real-time, has proven expensive, cumbersome and ineffective. New capabilities are now emerging which address some of the limitations experienced to date.

Augmented reality (AR) provides a means to overlay interactive digital information on top of the physical world. Combining the latest visual display, mobile computing and ‘track and trace’ technologies, AR provides an experience similar to a Heads- Up Display in a fighter jet through small, personal devices, such as smartphones or wearable glasses. Applying AR would allow a mobile worker to ‘see’ all relevant information for a given task, in the context of their physical environment and at the time they require it.

This paper examines a number of use-cases of how AR can be applied to the min-ing industry and the benefits that can be realised.

INTRODUCTIONThe global mining industry is challenged with the need to produce ever increasing tonnages, safely and with a reducing number of skilled, experienced workers. The skills shortage is compounded by several factors as outlined by Johansson, Johans-son and Abrahamsson (2009). First, the retirement of the baby boomer genera-tion (a phenomenon sometimes referred to as ‘the big shift change’) is leading to a rapid loss of knowledge, the impact of which is aggravated by an historic loss of interest in mining and engineering in the late 80’s and early 90’s that saw fewer ‘generation X’ seeking careers in these areas. Second, the remoteness of most mine sites makes hiring and retaining staff to work at these locations difficult. Third, the average retention time of workers in the industry is dropping, partly due to a gen-eral trend across all industries, but also specifically contributed to by high demand for experienced mine workers creating wage opportunities and aforementioned challenges of working in remote locations.

In addition to all these factors, the level of automation on mine sites is increasing, with a commensurate increase in the complexity of the machinery being operated. The increase in equipment complexity would typically be accompanied by an in-crease in specialisation of operators and maintenance personnel, but this is difficult in a skills constrained environment. The increasing requirements for training and compliance also make it less practical to simply ‘fly-in’ specialists, as it is not un-common for workers to require several days of training and inductions in order to be able to conduct a few hours of work on a mine site.

Faced with all these challenges, the application of augmented reality in the mining industry would help overcome the challenges faced by maximising the safety and effectiveness of personnel on mine sites. The benefits would manifest themselves

AUThORSJ Bassan Senior Consultant, CSC [email protected]

V Srinivasan Innovation Program Manager, CSC [email protected]

A Tang Senior Analyst Programmer, CSC [email protected]

THE AUGMENTED MINE WORKERAPPlICATIoNS oF AUgMeNTeD ReAlITy IN MININg

THE AUGMENTED MINE WORKER – ApplIcATIONs Of AUGMENTED REAlITy IN MINING

through better maintenance outcomes, more effective training, better collaboration and knowledge transfer, and enhanced situational awareness for improved safety.

DESCRIPTION OF AUGMENTED REALITYTo augment is to add something. Augmented reality (AR) adds information and functionality to the real world. To understand how this might work and be useful, imagine looking up at the night-sky; suddenly the names appear next to all the stars and planets, and lines appear joining all the stars of the constellations. Per-haps, being interested in real estate, you are looking at a house in a street, and the information about the house’s last sale price is shown on top of the house. This is all possible today, with common, consumer level technology like iPhones and free or low cost applications such as ‘pocket universe’, the ‘Commonwealth Bank real estate’ app or the ‘Metro Paris Subway’ app (Broughall 2010) (Martin 2009) (Pres-selite, 2010).

Figure 1: Metro Paris Subway application for iPhone (Presselite, 2010)

AR is the technology behind such applications. In a few short years, AR has quickly progressed from high-cost research projects funded by governments and consor-tia of large enterprises, to a technology available on low cost consumer devices. Driven by consumerisation, the pace of technology advancement has accelerated in the last two to three years.

Unlike ‘Virtual Reality’ where an entire world is recreated, AR simply overlays com-puter generated images or information onto the real world. An example of see-through displays are the heads-up display (HUD) in a fighter plane that show target information and position of radar contacts to the pilot. Compact and wearable versions of the classic aircraft HUD are now available, and packaged into devices only fractionally larger and heavier than common safety glasses. Hand-held de-vices, such as smartphones and iPads can also create the effect of an ‘augmented’ window on the world. As the user holds up the device, images or text are super-im-posed on the camera image in real time. To the user, it provides the illusion that the device is see-through but with information overlayed on the surface of the display.

“…looking up at the night-sky; suddenly the names appear next to all the stars and planets, and lines appear joining all the stars of the constellations”

the augmented mine worker – applications of augmented reality in mining

AUGMENTED REALITY AS A MAINSTREAM TEChNOLOGYThe iPhone 3gs released in 2009 was one of the first mainstream consumer de-vices with the integrated gPS, compass and accelerometer sensors required for AR. Not long after the iPhone 3gs, google’s Android operating system made its entrance and is now available on quite a number of devices. The number of AR capable smartphones alone grew from eight million in 2009 to over 100 million in 2010 (Holden 2011).

There has also been an explosion of AR applications available on smartphones and tablets. This has been driven by the explosion of location-tagged data and ad-vances in object-recognition technologies. For example, Metaio (2011) have devel-oped an iPhone application (Figure 2) that recognises a printer and then displays repair instructions for the identified make and model.

Figure 2: iPhone AR Printer Application (Byrne 2011)

Fully transparent head-mounted displays are a much more practical solution for implementing augmented reality, and allow ‘hands-free’ operation. Until very re-cently, the cost of ‘see-through’ near-eye displays have been cost-prohibitive, but in 2012, google announced their ‘glass’ project – an integrated AR headset with see-through display (Figure 3). A limited release of 8000 units has been made available to selected public beta testers in 2013, at a price of only $1400 per unit.

Due to the explosion of consumer AR, led by companies like google and Apple, it is our belief that a broad range of industry and consumer market solutions will be-come commonplace in less than five years.

Figure 3: google glass project (www.techradar.com, 2013)

“Due to the explosion of consumer augmented reality, led by companies like Google and Apple, it is our belief that a broad range of industry and consumer market solutions will become commonplace in less than five years.”


APPLICATIONS OF AR IN MINING

DIgITAl INSTRUCTIoNSone of the most promising and exciting possibilities for AR is to assist in assembly or repair tasks. The concept is quite simple - step by step graphical instructions are overlayed and appear on the actual item being worked on, such that the worker is shown exactly where to fit the next part or insert the next screw. Using see-through head-mounted display technology allows the worker to use both hands for the task at hand rather then fumbling with paper instructions, laptops or personal digital assistants (PDAs).

The possibilities and benefits of AR for maintain, repair and overhaul (MRo) tasks can be grasped intuitively: the work instructions should be much easier to under-stand; an untrained worker is likely to perform the task as well as an experienced worker; and the task is likely to be completed faster and with fewer errors. experi-ments have shown that AR does indeed improve the outcome of maintenance and assembly tasks - principally by improving productivity and reducing errors (Rein-hart and Patron 2003).

As early as 1990, technology demonstrations were being developed to assist workers with maintenance or assembly tasks (Caudell and Mizell 1992). A surge of interest occurred in the late 1990’s and early 2000’s, with keen interest shown by automotive and aerospace manufacturers, and by the nuclear power industry who sponsored expensive prototype projects. The significance of these projects was that the organisations backing them recognised the benefits that could be realised if a workable AR system could be developed. While these projects all succeeded in producing functioning prototype systems, they were limited by the technology of the day that prevented widespread practical implementations in real-world envi-ronments.

Benefits For a mining operation, the presentation of digital instructions with AR technology provides benefits in three areas: 1) improvement to equipment availability; 2) re-ducing reliance on key personnel; and 3) improved health and safety outcomes.

equipment availability would be improved as a result of reduction in mean time to repair (MTTR) and by a reduction in human errors. AR would directly reduce both the time required to complete tasks and number of errors made. A reduction in reliance on key personnel would be achieved because workers could safely and reliably perform a wider range of tasks, possibly with less training. AR experiments have shown that personnel with less training or experience can satisfactorily com-plete tasks, in almost the same time and with almost the error rate as experienced personnel (ong, yuan, Nee 2008). This result becomes even more significant when considered in the light of factors such as the skill shortages in Australia, ‘the big shift change’ of the retiring baby-boomer generation, the need to maintain increas-ingly complex machinery and a tendency to shorter retention time for workers.

Case StudyThere are many maintenance repair and overhaul tasks undertaken on mine sites that are applicable to AR. These are typically related to fixed-plant maintenance or heavy vehicle maintenance. The specific case studies presented below demon-strate the benefits of AR instructions for heavy vehicle maintenance.

Henderson and Feiner (2009) developed and field tested an AR system for mainte-nance of armoured personnel carriers. The application, operational constraints and tasks supported were very similar to those required for the effective maintenance of heavy mining equipment. The prototype was developed to support United States Marine Corps mechanics operating on lAV-25 armoured personnel carriers, which is a light-wheeled military vehicle. like mining vehicles, the lAV-25 includes

“For a mining operation, the presentation of digital instructions with augmented reality technology provides benefits in three areas: 1) improvement to equipment availability; 2) reducing reliance on key personnel; and 3) improved health and safety outcomes.”


a large amount of electrical, mechanical, hydraulic and pneumatic infrastructure.

Wearing head-mounted displays (HMDs), the mechanics operated the system hands-free, with instructions appearing directly on the equipment being worked on. Instructions were provided as a combination of pointers (highlighting the loca-tion of components on the vehicle), text instructions, graphical instructions (e.g. showing direction of rotation of fixings or correct orientation of parts) and infor-mational displays (such as 3D views of sub-components). The system was com-pletely wearable and could be operated even within the restricted work area inside the turret of the vehicle.

The mechanics involved were able to complete a sequence of tasks using AR more quickly than with other instructional methods. The system reduced the head move-ments required and was favourably received by experienced mechanics who were positive about the approach in terms of intuitiveness and satisfaction.

ReMoTe CollABoRATIoNAR provides a breakthrough in the effectiveness of remote collaboration. By en-abling a shared real-time view of the world, AR allows a remote operator to see ex-actly what a worker is looking at, providing them with in-context guidance or even bringing themselves into the worker’s environment to demonstrate more complex actions.

While there have been major advances with collaboration technologies’ interaction through a hand-held device, laptop or fixed workstation, they still require a user to translate the information they receive into the context of the world around them. Furthermore, having workers alternately focus between a mobile device or desktop and their current environment increases cognitive load requirements (Alem, Tec-chia, Huang, 2011). Using AR, a worker or remote operator is no longer dependent or constrained by their peer’s observational and descriptive abilities. Rather than describing where to find a particular element in a schematic, a remote operator can simply highlight the area within the user’s view, decreasing margin for error and increasing productivity.

Although it has been found that shared voice, imagery and a shared pointer using a shoulder worn active camera laser may be sufficient for many remote collabora-tion tasks, ReMoTe (Remote Mobile Tele-assistance), a project by the Common-wealth Scientific and Industrial Research organisation (CSIRo) (Alem, Tecchia, Huang, 2011), found that a projected hand is more effective for remote guiding then a simple pointer as the hand could be used to convey more complex hand movements from “take this and put it here” to doing a “specific rocking movement with a spanner”. In addition to remote guidance, remote operators can find docu-ments and ‘show’ them to the field worker in an augmented workspace or allow multiple remote collaborators to interact within the worker’s world for collaborative product development as highlighted by ong, yuan, and Nee (2008).

In addition, ong, yuan, and Nee (2008) highlighted that “collaborative AR-based applications are being extensively explored due to their usefulness in manufactur-ing”. They highlighted a number of case studies including an AR tele-training sys-tem that enables collaboration on training tasks and an internet based AR system for product maintenance that can be leveraged by geographically dispersed users.

BenefitsWithin mining the attention on remote collaboration has been steadily increasing due to severe skill shortages, the increasing difficulty of attracting experienced staff to work at remote locations and increased costs of running 24x7 manned op-erations (Farrelly and Records 2007). It is expected that the future mining working environment will consist of a skeleton on-site workforce that remotely collaborates

“It is expected that the future mining working environment will consist of a skeleton on-site workforce that remotely collaborates with external specialists, partners and supervisory staff working out of remote operations centre in major urban centres.”


with external specialists, partners and supervisory staff working out of remote operations centre in major urban centres (Bassan et al 2008). In some cases the re-mote experts will be retired staff who are unable to work on-site anymore, but kept on retainer due to their deep expertise. Covering a larger breadth of duties, the skeleton on-site staff will be unable to have deep expert skills and will rely on re-mote collaboration and thus AR to maintain operations. Specifically, it is expected that AR will be of particular value in remote maintenance, emergency responses and for enhancing knowledge retention and management.

As mentioned, the use of AR can significantly improve the mean time to repair and reduce human errors. In such situations, AR can be leveraged to remotely guide a technician through the necessary steps for repair, investigation and operation. Not limited to simple pointers or verbal cues, the remote expert can highlight areas or even overlay previous work history on top of the worker’s current view. In the case of emergency responses AR can be used to rapidly engage with medical response teams. All mining workers undergo significant safety and health training. Unfortu-nately, findings from Patterson (2009) indicated that recalling information where a worker must apply previous knowledge “require high mental demand which can cause problems in unusual emergency situations”. AR allows a worker to receive medical support from officers en route to the incident. The medical team can high-light specific steps that a worker should take such as removing particular debris from the area. More importantly, the en route medical team can highlight areas they would like the worker to show them, allowing them to make a better, earlier assessment of the situation.

AR can be used to reduce the risk of losing operational and business knowledge due to having a smaller multi-skilled workforce. Bassan et al (2008) presented the approach of using a ’guild’ for high specialisation knowledge transfer areas. Using an example of geostatistical ore reserve estimation, they suggested that ap-prentices learning under a master is suitable for developing high level expertise. Using AR, previously retired staff can impart their deep expertise remotely to the level where they can even bring themselves into their apprentice’s environment to demonstrate more complex activities, not dissimilar to the embodied virtual agent (Figure 4), situated in an AR system developed by Thórisson et al (2004). During this process the remote experts can also be leveraged to codify the steps required to complete given tasks using the AR system to automatically capture task require-ments.

Figure 4: embodied virtual agent (Thórisson et al 2004)

Case StudyAs part of the CSIRo ‘Transforming the Future Mine’ theme, Alem, Tecchia, Huang (2011) developed ReMoTe (Remote Mobile Tele-assistance), a mobile AR system for remotely guiding workers. The project focused on specific industry maintenance

“…augmented reality will be of particular value in remote maintenance, emergency responses and for enhancing knowledge retention and management.”


challenges such as operator/technician reliance on remote experts due to the op-eration and maintenance of new complex pieces of equipment that are fully auto-mated and semi-automated.

ReMoTe projects the hand gestures of a remote helper onto a near-eye display worn by a worker. In addition to simple pointer functionality, the hand gestures allow the expert, often located in a major metropolitan city, to demonstrate how to perform specific complex manual procedures for workers located on site.

As part of the system the team created:

• A helper user interface used to guide the worker remotely via a touch screen device and an audio link, and

• A mobile worker system composed of a wearable computer, a camera mounted on a helmet and a near-eye display.

Design considerations included support for complex hand movements, worker mobility between tasks and a panoramic view providing the ability for the remote helper to point/gesture outside the worker’s field of view. Another valuable design addition was the four storage areas located on the sides of the helper user inter-face. The shared visual space allows short cuts for the helper, for example creating a “copy the current image of the stream” or taking “snapshots of locations/objects that are recurrent in the workspace” (Alem, Tecchia, Huang 2011).

A number of design iterations were performed while testing three maintenance and repair tasks: repairing a photocopy machine; removing a card from a computer motherboard; and assembling a lego toy. The user tests found the system intui-tive and easy to use with no discomfort on the near-eye display of the worker sys-tem reported.

Figure 5: ReMoTe by Alem, Tecchia, Huang (2011)

oPeRAToR gUIDANCeThe results of operating mining equipment poorly are breakdowns, downtime and lost productivity. Safety is also more likely to be compromised through unexpected equipment failure. If the operator’s attention can focus on the task at hand without having to assimilate masses of peripheral and marginally important information, the result will be better equipment productivity, higher utilisation, lower mainte-nance costs and most importantly, improved safety.

equipment operators are increasingly being provided with a multitude of systems to assist them in improving safety, maximising efficiency of operation and minimis-ing wear and tear on the equipment. In addition to the standard gauges and alerts necessary for operating the equipment, it is not uncommon for mining vehicles to be equipped with a bewildering array of systems and screens for purposes such

“If (an) operator’s attention can focus on the task at hand without having to assimilate masses of peripheral and marginally important information, the result will be better equipment productivity, higher utilisation, lower maintenance costs and most importantly, improved safety.”


as collision avoidance, dispatch systems delivering instructions, video cameras for situational awareness, fatigue monitoring of the driver, and guidance systems de-signed to avoid unnecessary mechanical stresses on the vehicle.

Figure 6: Cluttered operator cabin

Apart from aircraft (which have used heads-up displays for more than 50 years), automobile manufacturers have been incorporating AR into vehicles for more than 20 years. early in the development of military HUDs, the goal became to centralise critical information into the pilot’s field of vision to improve the pilot’s efficiency, reduce task saturation and information overload. Recently, HUDs on aircraft have been made obsolete, with the latest generation of aircraft like the F-35 using a full AR helmet mounted display (HMD) to replace the traditional fixed HUD.

In addition to ‘tried and true’ techniques used on aircraft HUDs to present infor-mation to the pilot, other innovative AR techniques have been investigated and trialled recently to test how best to display information to drivers, operators or mo-bile users in order to optimise the available cognitive capacity of the operator. All of these innovations combine to simplify the driver’s task by reducing their cogni-tive load, and draw their attention to hazards that are difficult to perceive.

Figure 7a: AR driver night vision (Bergmeier and lange 2008)

“…automobile manufacturers have been incorporating augmented reality into vehicles for more than 20 years.”


Figure 7b: AR driver night vision (Bergmeier and lange 2008)

Impact of tele-remote operationThe need to de-clutter an equipment operator’s cabin may soon become redun-dant due to the increase in automation and tele-remote operation of mine vehicles. However, far from reducing the need for AR, this development opens up a range of additional possibilities for integrating more meaningful information for the remote operator.

The tele-remote equipment operator is already, by definition, in a mixed reality environment (with real CCTV footage mixed with virtual dials). With clever user-in-terface design, additional information sources can be integrated and presented to the operator. The challenge then becomes to both integrate the additional sources of information and to intelligently ‘filter’ the information so that only the most rel-evant bits are displayed to the operator so they can make the best decisions about the vehicle operation at that moment, without suffering information overload.

BenefitsIntegrating AR into equipment operation can be expected to improve safety and increase productivity. The key features contributing to increased safety are im-proved situational awareness for the operator, reduced cognitive workload and the ability to effectively highlight potential hazards, such as a vehicle or person ap-proaching the vehicle in a blind spot. Rather then diverting their attention to mir-rors and CCTV systems regularly to check for the potential of a hazard, the driver’s attention can be immediately drawn to the hazard only when needed.

Case StudyThe operational life of mining shovels is commonly reduced due to poor operation. The root cause of the problem is the tendency for shovel operators to commence a swing while the bucket is still dug into the muck pile. This results in excessive stresses on the shovel (leading to long-term damage) but no perceptible move-ment (so the operator is frequently unaware of the overload). over time, the in-terval between overhauls is reduced and the time between failures is reduced. In economic terms, this means higher maintenance costs, and lower than expected equipment availability and lost production opportunity.

Many operations recognised the problem, but for those willing to tackle the situ-ation, the best method available was to gather onboard statistics, analyse them offline, and (if possible) correlate the statistics to the machine operator. At the very best, the operator might finally be presented with the statistics to help them im-prove their operating techniques many days later.

“The key features contributing to increased safety are improved situational awareness for the operator, reduced cognitive workload and the ability to effectively highlight potential hazards”


The CRC Mining developed a system that displays a graphic indication directly to the driver, in real-time, as they operate the equipment. Behind the display, the sys-tem is performing real-time analysis of data from sensors on the shovel. The imme-diate analysis and feedback loop allows the operator to take appropriate action to reduce stress on the equipment. Field trials generated reductions of ‘swing-during-dig’ of between 47% and 69%, as well as decreased dig cycle times (Hood 2004).

TRAININgAR presents interesting opportunities for making training more effective, realistic and cost effective. Virtual reality (VR) has been used effectively for training, both in the mining industry and other areas for many years; however, there has been an increased uptake in the last five years due to convergence of gaming technology and low cost computer graphics hardware. examples of the use of virtual reality for mine site training in main-stream use already include:

• Immersive Virtual reality ‘caves’, in which small groups of trainees can be fully immersed in a realistic, 3D environment, such as those constructed at the New-castle Mine Rescue Centre and UNSW (Stothard, galvin, Fowler 2004).

• PC-based 3D training programs which allow trainees to navigate and interact with a 3D environment (similar to a computer game) to achieve training objec-tives. examples include the NIoSH underground map reading and NIoSH con-veyor belt training (lucas, Thabet, Worlikar 2007).

• equipment simulators for training machine operators, such as haul truck, shov-els and dozers.

Benefits AR offers significant advantages over virtual reality as it allows the training to occur in a real-world environment, but can simulate virtual scenarios. As a com-parison, immersive virtual reality ‘caves’ are currently used by emergency response teams - the trainees enter the ‘cave’ wearing full fire fighting apparatus and carry-ing real tools and equipment. They respond to the training situation within the con-fines of the cave, which has several key limitations: the team must keep together in a close group while in the cave; realism is lost if they are not facing the outer wall of the cave; the entire environment is virtual; interaction with the environment is necessarily limited and only small handheld apparatus can be carried into the cave. Finally, the whole team must physically travel to the training facility, usually located in a major urban centre and therefore requiring significant travel time and cost for workers located on remote mine sites.

By contrast, consider an AR training simulation taking place on a mine site. The emergency response team would don AR headsets on the actual mine site, per-haps at the mine rescue shed. The team would drive real emergency response vehicles to a virtual fire which they can all see. Real world tools and equipment (including fire trucks, ambulances, fire apparatus, etc.) can be used in the training scenario. ‘Real’ water can be sprayed onto ‘virtual’ flames. The trainees can operate effectively as a team, using real communication tools like phones, radios and visual cues. The team can approach the emergency situation from all angles, including climbing ladders and walkways (as they are not confined to a virtual reality ‘cave’). The primary benefit would be the realism, accuracy and ability to make use of real tools and equipment in the training environment. Secondary benefits would be: reduced training costs due to no travel to a dedicated facility; repeatability of train-ing scenarios and reduced capital cost as no dedicated facility is required to house a VR ‘cave’.

Case StudyThe U.S. Navy Research laboratory, Washington, has developed an AR battle field training system (Brown et al, 2004). The system helps trainees in military

“Augmented reality offers significant advantages over virtual reality as it allows the training to occur in a real-world environment, but can simulate virtual scenarios.”


operations urban training scenarios, and allows a trainee, wearing full military gear and carrying real weapons, to participate in effective combat training. Inside their backpack, the user is carrying a small wearable computer, tracking devices, and wireless communications, which are all connected to a see-through head mounted display. Using AR, virtual enemy forces (or ‘Computer generated Forces’ - CgF) are realistically inserted on the head mounted display and trainees engage the virtual enemies in combat. By connecting to a central combat simulator, all the virtual en-emies behave intelligently to create a realistic scenario. Finally, a ‘weapons tracker’ allows the system to know where the weapons are being pointed and fired.

The use of AR provides several advantages to other training methods available. In traditional ‘blue force’/‘red force’ training games, twice the man-power is required in order to make up the ‘red’ team. The scenario is also not completely repeatable due to human factors, and there are issues using simulated munitions (including in-juries, set up of the environment and cleanup). Virtual reality training environments have in recent times been used for military training exercises. While the VR envi-ronment is safe, controlled and repeatable, it has a number of other drawbacks. For example, many of the real-world cues are not well simulated, special equipment cannot be easily moved into or operated in the immersive simulator, and the simu-lator does not allow for completely realistic navigation.

INCIDeNT INVeSTIgATIoNReconstructing the lead up to a major incident can be vital to correctly identify-ing the root causes and thereby preventing a recurrence. Physically inspecting and experiencing the environment is vital to understanding the decisions made in the lead up. Unfortunately, it is not always possible or practical to experience the situation as it was – for example in extreme cases, structures may be partially de-stroyed, or it may simply be that placing all the personnel and equipment in their required positions is more time consuming and expensive then the severity of the incident warrants.

Shea (2011) has proposed the development of an AR application for event recon-structions in incident investigations on mine sites. Incidents can be recreated and replayed to gain a better understanding of the contributing causes to incidents and near misses. For example, the location and movements of personnel and equip-ment could be recreated virtually. The investigator, present at the real scene, could recreate and play or pause the movements of virtual personnel and equipment im-mediately before the incident to gain a deeper understanding of the root causes. This could provide new perspectives as to the causes of major incidents and speed up incident investigations, thus both improving safety outcomes and allowing work to safely resume sooner.

The technology required to conduct an incident investigation with AR on a mine site is similar to that currently applied in manufacturing and engineering for mixed-reality prototyping. Mixed-reality prototyping is a commercialised and relatively mature technology where virtual components are designed and then modified around real-world environments. For example, using commercial products such as Mataio engineer, engineers can ‘see’ the proposed layout of pipes and cable trays within an existing facility.

“Incidents can be recreated and replayed to gain a better understanding of the contributing causes to incidents and near misses.”

https://twitter.com/CSCAustralia


Figure 8: example of mixed-reality engineering design (Metaio gmbH)

Case StudyMassachusetts Institute of Technology (MIT) have developed an AR system to aid in the teaching and understanding of historical events. In the system, participants walk around present-day lexington, Massachusetts, with the objective of trying to discover who fired the first shot at the 1775 battle of lexington. The participants are equipped with wireless hand-held devices that convey information from the perspective of one of four historical figures.

Participants discover and gather evidence while simultaneously experiencing the physical site of the battle. Information on the hand-held device is triggered by the location at which the participant is standing. The game helps participants to gather, evaluate, and interpret historical information, devise hypotheses and coun-ter-arguments, and draw informed conclusions about events that occurred in the past (Schrier 2005).

CONCLUSIONFaced with significant growth opportunities and challenges, the global mining industry is at an operational crossroad. While AR will not be a silver bullet, as we have demonstrated, it does provide a number of opportunities to present timely, in-context information that can improve productivity and spark insights. For ex-ample AR can be used to increase productivity in MRo activities and through enhanced remote collaboration experiences; and AR will mitigate knowledge loss within the workforce.

There are many approaches to augment the world with additional information. As such, it will be important to keep track of academic and commercial AR develop-ments and related research area such as Haptics which allows users to ‘touch’ vir-tual data. In the near future we may live in an almost completely augmented world. However, in the short term the challenge will be to target specific, extremely high value applications, some cases in collaboration with partners and industry, in oth-ers, kept confidential due to their highly strategic nature.

The mining industry is well positioned to become an early adopter of AR technol-ogy to address current operational challenges. As the mining industry transitions to the network centric mine of the future, typified by increased levels of automa-tion and remote operations, AR is likely to become a ‘business as usual’ technology required as a key enabler of the aspirational operating models being proposed.

ACKNOWLEDGEMENTSThis paper is reproduced in abridged form with the permission of the Austra-lian Institute of Mining and Metallurgy. It was first published in the Proceed-ings of the 2nd International Future Mining Conference, Sydney, NSW, 2011.

SYDNEY (national head office)

26 Talavera Road

Macquarie Park NSW 2113

Telephone: (02) 9034 3000

Facsimile: (02) 9034 3999

Copyright © 2013 CSC Australia Pty Limited. (ACN: 008 476 944) All rights reserved. 0121-14

https://www.facebook.com/csc

http://www.linkedin.com/company/csc

https://twitter.com/CSCAustralia

https://plus.google.com/+csc

http://www.youtube.com/cscpassion

http://pinterest.com/cscaustralia

http://www.csc.com/au


REFERENCESAlem, l, Tecchia, F, and Huang, W, 2011. Remote Tele-assistance System for Main-tenance operators in Mines. Underground Coal operators’ Conference, pp 171-177. University of Wollongong & the Australasian Institute of Mining and Metallurgy.

Apple Inc., 2011. Apple’s App Store Downloads Top 10 Billion [online]. Available from: <http://www.apple.com/pr/library/2011/01/22appstore.html> [Accessed: 16 June 2011].

Baird, K, Barfield, W, 1999. evaluating the effectiveness of augmented reality dis-plays for a manual assembly task, in Virtual Reality, Vol 4, Issue 4. pp 250-259. Springer, london.

Bassan, J, Srinivasan, V, Knights, P, Farrelly, C (2008). A day in the life of a mine worker in 2025, in First International Future Mining Conference and exhibition: 19-21 November 2008, pp 71 – 78. The Australasian Institute of Mining and Metal-lurgy.

Biocca, F, Tang, A, owen, C, Xiao, F, 2006. Attention Funnel: omnidirectional 3D Cursor for Mobile Augmented Reality Platforms, Proc. Conference for human-com-puter interaction (CHI 2006), Montreal, Quebec. pp 1115-1122.

Bergmeier, U, lange, C, 2008. Acceptance of Augmented Reality for driver assis-tance information. In Proceedings 2nd International Conference on Applied Human Factors and ergonomics, las Vegas, Nevada.

Broughall, N, 2010. Commonwealth Bank Real estate iPhone App Augments Awesomeness [online], gizmodo. Accessed from <http://www.gizmodo.com.au/2010/04/commonwealth-bank-real-estate-iphone-app-augments-awesome-ness> [Accessed: 16 June 2011].

Brown, D g, Baillot, y, Julier, S, Maassel, P, Armoza, D, livingston, M A, Rosenblum, l J, 2004. Building a Mobile Augmented Reality System for embedded Training: lessons learned, Proceedings of the Interservice / Industry Training, Simulation, & education Conference (I/ITSeC ’04), orlando, Florida.

Byrne, C, 2011. Augmented reality helps you fix your printer [online], VentureBeat

Available from: < http://venturebeat.com/2011/02/10/augmented-reality-printer-metaio/> [Accessed: 20 June 2011].

Caudell, T P and Mizell, D W 1992, Augmented reality: An application of heads-up display technology to manual manufacturing processes. Proc. Ieee Hawaii Interna-tional Conf. on Systems Sciences, 1992.

Defense Update, 2005. Millitary Application of oleD Micro-Display Technology, Defense Update, issue 5 [online]. Available from <http://defense-update.com/fea-tures/du-3-05/feature-oleD.htm> [Accessed 16 June 2011].

Didier, J-y, Roussel, D, Mallem, M, otmane, S, Naudet, S, Pham Q-C, Bourgeois, S, 2005. AMRA: Augmented Reality assistance in train maintenaince tasks, in Interna-tional Symposium on Mixed and Augmented Reality, 2005 (ISMAR 2005). Vienna, Austria.

Farrelly, C T and Records, l R, 2007, “Remote operations Centres – lessons from other

Industries”, Proc. Australian Mining Technology Conference, pp.105-110, 2-4 oct., The Vines, WA, Australia.

Friedrich, W, 2002. ARVIKA-augmented reality for development, production and service, in International Symposium on Mixed and Augmented Reality, 2002 (ISMAR 2002). pp 3-4. Darmstadt, germany.

Fuss, A, 2008. Digital Disruptions: Technology Innovations Powering 21st Century Business, Computer Sciences Corporation, leading edge Forum.


Henderson, S, J, Feiner, S, 2009. evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret, in Interna-tional Symposium on Mixed and Augmented Reality, 2009 (ISMAR 2009). pp 135-144. orlando, Florida.

Holden, W, 2011. Mobile Augmented Reality: opportunities, Forecasts & Strategic Analysis 2011-2015, Juniper Research.

Höllerer, T, Feiner, S, Terauchi, T, Rashid, g, Hallaway, D (1999). exploring MARS: De-veloping indoor and outdoor user interfaces to mobile augmented reality systems.

Computers & graphics, 23(6), 779–785.

Hood, M, 2004, Advances in Hard Rock Mining Technology, Proc. Mineral econom-ics and Management Society, Thirteenth Annual Conference, April 21 – 23, The on-tario Club, Toronto, Canada.

Johansson, J, Johansson, B, Abrahamsson, l, 2009. Future mining industry – Six-teen predictions, International Journal of Mining and Mineral engineering, 1(3): 304-312.

lens-Fitzgerald, M, 2011. layar’s News Round-up from its Anniversary event [on-line]. Available from: < http://site.layar.com/company/blog/layars-news-round-up-from-its-anniversary-event/> [Accessed: 18 June 2011].

lucas J, Thabet W, Worlikar P, 2007. Using Virtual Reality (VR) to Improve Con-veyor Belt Safety in Surface Mining in 24th W78 Conference 5th ITCeDU Workshop / 14th egICe Workshop, Bringing ITC knowledge to work, Maribor, Slovenia.

Madden, l, Samani, N, 2010. iPhone Augmented Reality Applications Report, Aug-mented Planet. Available from <http://www.augmentedplanet.com/wp-content/uploads/report/iPhoneApplicationReport_v1.pdf> [Accessed 16 June 2011].

Martin, M, 2009. Pocket Universe Ups the Astronomy App Ante, Tuaw. Accessed from <http://www.tuaw.com/2009/07/03/pocket-universe-ups-the-astronomy-app-ante/> [Accessed: 16 June 2011].

Metaio, 2011. Augmented Vision at Mobile World Congress 2011, Metaio Press Re-lease [online]. Available from <http://www.metaio.com/press/press-release/2011/augmented-vision-at-mobile-world-congress-2011/> [Accessed 16 June 2011].

Metaio gmbH, 2008. Whitepaper: Unifeye | Prototyping

Augmented Reality in Industrial Prototyping [online], Available

from: <http://old.metaio.com/fileadmin/homepage/Dokumente/english/Whitepa-per_Unifeye_Prototyping_01122008.pdf> [Accessed: 19 June 2011].

ong, S K, yuan, M l, and Nee, A y C, 2008. Augmented reality applications in manufacturing: a survey, International Journal of Production Research, 46: 10, pp 2707-2742.

o’Shea, D, Neal, D, 2010. 2010 Study Tour Report: Surf the Change or be Swept Away –Consumerization, Mobility and the Cloud, Computer Sciences Corporation, leading edge Forum executive Programme.

Patterson, J M, 2009. Human error in mining: a multivariable analysis of mining ac-cidents/incidents in Queensland, Australia and the United States of America using the human factors analysis and classification system framework, A Dissertation Pre-sented to the graduate School of Clemson University

Platanov, J, Heibel, H, Meier, P, grollmann, B, 2006. A mobile markerless AR system for maintenaince and repair, in International Symposium on Mixed and Augmented Reality, 2006 (ISMAR 2006). pp 105-108. Santa-Barbara, California.

Phillips, K, Piekarski, W. 2005. Techniques for Interaction in Real-time Strategy Augmented Reality games. ACM Int’l Conference on Advances in Computer enter-tainment Technology, Valencia, Spain.


Prentice, S and Fenn, J, 2010. Hype Cycle for Human-Computer Interaction, 2010, gartner, Inc, 4 August 2010. ID Number: g00200994.

Presselite, 2010. l’application Métro Paris sur iPhone et iPod Touch, Presselite web site [online]. Available from: <http://www.presselite.com/iphone/metroparis/> [Accessed on: 20/6/2011].

Reinhart, g and Patron, C, 2003 Integrating augmented reality in the assembly do-main –fundamentals, benefits and applications. Ann. CIRP, 52(1), 5–8.

Samsung, 2011. Samsung electronics Becomes the World’s First In Mass Production of the Transparent lCD Panel. Accessed from <http://www.samsung.com/us/news/newsRead.do?news_seq=19836> [Accessed 16 June 2011].

Schrier, K l, 2005. Revolutionizing History education: Using Augmented Reality games to Teach Histories, Master’s thesis (unpublished), Massachusetts Institute of Technology, Cambridge.

Schwald, B and de laval, B, 2003. An Augmented Reality System for Training and Assistance to Maintenance in the Industrial Context, WSCg’2003, February 3-7, 2003, Plzen, Czech Republic.

Shea, S, 2011. Next generation Personal Protective equipment. leF 2011 grant Sub-mission, CSC.

Stothard, P, galvin, J and Fowler J, 2004. Development, Demonstration and Implementation of a Virtual Reality Simulation Capability for Coal Mining opera-tions, School of Mine engineering, University of South Wales [online]. Available from <www.mining.unsw.edu.au/Publications/publications_staff/Paper_Fowler_ICCR_2004.pdf> (Accessed on February 26, 2007).

Szalavári, Z, Schmalstieg, D, Fuhrmann, A, gervautz, M, 1998. “Studierstube”: An an environment for collaboration in augmented reality. Virtual Reality, 3(1):37- 48

Tang, A, 2011. Augmented Reality with CSC omnilocation for the mobile worker. leF grant Report, CSC.

Tang, A, owen, C, Biocca, F, Mou, W M, 2003. Comparative effectiveness of aug-mented reality in object assembly, Proc. Conference on Human Factors in Comput-ing Systems (CHI 2003), Florida, USA. pp. 73–80.

Thórisson, K R, Benko, H, Abramov, D, Arnold, A, Maskey, S, Vaseekaran, A, 2004. Constructionist design methodology for interactive intelligences. A.I. Magazine 25(4), 77–90.

Tönnis, M and Klinker, g, 2006. effective Control of a Car Driver’s Attention for Vi-sual and Acoustic guidance Towards the Direction of Imminent Dangers, Proc. Int. Symp. on Mixed and Augmented Reality (ISMAR ‘06), 2006, pp. 13-22.

Vuzix Corporation, 2011. Press Release: Vuzix and NS Solutions Corporation Partner To Deliver Augmented Reality Solutions For Manufacturing [online]. Available from: <http://www.vuzix.com/site/_news/Press%20Release%20NS%20Sol%20Partner-ship%2002-24-2011%20-%20Final.pdf> [Accessed: 16 June 2011].

Vuzix, 2011. Product data sheet: Star 1200 Augmented Reality System, Vuzix Cor-poration [online]. Available from: <http://www.vuzix.com/ar/products_star1200.html> [Accessed: 20/6/2011].

Wiedenmaier, S, oehme, o, Schmidt, l, luczak, H, 2003. Augmented Reality (AR) for Assembly Processes Design and experimental evaluation in International Jour-nal of Human-Computer Interaction, Volume 16, Issue 3, 2003, pp 497-514. Taylor & Francis, london.

the augmented mine worker - dxc...

Documents