Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Frontiers in Computer Vision for Drilling
CoVarKenneth D. Morton PhD
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
CoVar• Our Goal:
– Provide cutting-edge machine learning solutions to help industries improve safety and efficiency, provide new insights and capabilities, while saving money
• This talk:– Video processing technology
demonstrations to improve rig safety and efficiency
VideoStructured
Information
2
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Why Video?• Video cameras:
– Inexpensive, widely deployed– Commercial-off-the-shelf technology
• Powerful tool for many problems where classical instrumentation is difficult or costly
• Video provides massive amount of unstructured information
• Computer Vision Algorithmsautomatically extract information from video data for safety or efficiency improvements on the rig– Note: Video is a complementary
source of information in addition to existing sensors; not a panacea
3
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Video Processing Technology Application Areas• Personnel Video Monitoring
• Muster Point Counting
• Shaker Table Fluid Front and Particle Tracking
• Rig Microstate Identification and Tracking
4
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Personnel Video Monitoring - Motivation• The drilling rig is a dynamic, rapidly
changing environment• Increased automation is key to
improving safety and efficiency– But automation comes with its own risks
• Many pieces of information required for safe automation are difficult to obtain
– Difficult or expensive to instrument, require user cooperation
– E.g., transponders – require user action• Lots of information from visual
interpretation of a scene– How to automate?
• Safety Alert 58:
Ensuring safety controls enable workers and drillers to confirm that the path of the iron roughneck is clear of personnel
5
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Personnel Video Monitoring (PVM)• Goal
– Prevent machine-human collisions
• Using pre-existing sensors
• No personnel actions required
• Technical challenges solved– Infer locations of
multiple people in a scene from a set of monocular cameras
6
Inferred Personnel &Equip MapPerson 1
Person 2
Person 3
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
PVM Under the Hood• Person detection
– Poor off-the-shelf person detection performance for some camera angles
– Developed classification techniques and rapid training procedures
• Person location– Via triangulation– Person detections in
images (image space)– Camera transformation
information (image world transformation)
– Multiple cameras
7
Person LocationPerson Detection
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
PVM Video
8
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
• Goal: Automate muster point roll-call with automated personnel counting and identification
Muster-Point Personnel ID
Example Video of Automatic Person MatchingExample video of non-duplicative personnel counting
9
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Shaker Table Analysis• Leverage video to track fluid front,
losses off table, fluid volume, solid/fluid ratio
• Perform discrete object detection & tracking with probabilistic filtering methods– Enables automatic tracking of hole
cleaning and cuttings volume (with calibration)
– Enables automatic localization of the fluid front, for feedback into automatic control mechanisms
10
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
• Must related pixel measurements to real-world measurements
• Estimate camera transform, map to world coordinates
• Note: example video is hand-held, shaky
11
Estimating Pixels vs. Estimating Distance
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved. 12
Fluid Front Tracking
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Solids Analysis
13
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Rig Microstate and Pipe Tally• Data collected on board an
active rig with 1hr of tripping-out-of-hole
• Camera relatively stable, has clear view of area around well-bore
• Goals:– Can we automate pipe-tallying
– automatically count number of pipes to enter/exit the hole?
– Can we measure other variables of interest? E.g., pipe velocity, pipe length (integral of velocity over time)
14
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Requires: Rigid Body Detection• Goal:
– Detect three types of rigid bodies in scene
– Pipe over hole, roughneck engaged, grabber
• Only grabber has significant motion within scene
• Three regions shown:– Blue: pipe detection region (tall, narrow)– Orange: grabber detection region– Yellow: Roughneck engaged detection
region
15
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Rigid Body Detection & Classification
• CoVar proprietary processing for automatic rigid body detection in videos
• Enables rapid re-training of new rigid body classification algorithms
16
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
• To make reliable inferences from observed data, need to use a logical formalism
• What enables a pipe to be removed from hole?
– 1) Grabber– 2) Pipe– 3) Roughneck
• As roughneck disengages, the pipe is still in scene – but this is the same pipe; we do not want to double-count the pipe
• FSM enables logical system:– After roughneck engaged and
disengaged, and pipe not detected, increment tally, do not increment tally further until grabber detected
Pipe Counting System: Finite State Machine
Initial State /
No State
Grabber Detected
Pipe In Scene
Pipe Disconnecting
Grabber detection flag On
Grabber detection flag Off, & Pipe detection flag OnTrack pipe velocity
Roughneck detection flag OnRoughneck flag
OffPipe detect flag OffIncrement pipe tally
17
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Example Video; Pipe Tally
18
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Automatic Aggregate Action Summary• Automatically• export information, with
pictures/clips, to arbitrary database format
• (E.g., SQL, EXCEL)
• Intelligent data down-sampling enables massive reduction in data storagerequirements
• Very fast, simple access to short video clips around interesting events
19
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
• Incorporate tool-joint detection
– Count # of pipes comprising pipe stand
• Various nomenclature fixes – Connector Grabber
• Detect other, rare events – e.g., Mud bucket in scene
• Automatic data compression – only store “interesting” events
– Can reduce data storage requirements by orders of magnitude
Moving Forward
Tool Joint Detected: Pipe Stand Component #2
19
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Conclusions• Video is an inexpensive sensing modality that can be leveraged
when classical instrumentation is difficult or costly but…• Video provides a massive amount of unstructured information• CoVar’s computer vision algorithms automatically extract
information from video data for safety or efficiency improvements on the rig
21
VideoStructured
Information
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Backup
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Simulated Rig PVM
23
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Leveraging Previous Experience• Previous work in
several DoD funded application areas– Real-time trip-wire
detection in first generation digital night-vision goggles
– Algorithms for road-cataloguing for potential IED detection
Tripwire Detection
24
Urban Change Detection
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Pipe Handling and Estimation• Goal: Automate pipe handling and
pipe tallies• CoVar has previously explored
various approaches to video processing for pipe management
– E.g., pipe matching, size/angle estimation; left
• Potential Applications:1. Measure pipe diameters and
lengths as they enter hole and automate drill string pipe tally
2. Count pipe joints during tripping into or out of hole
3. Measure tripping velocity into or out of hole
4. Automate pipe stabbing localization and rotation
Example Video of Automatic Pipe Localization
Example Video of Automatic Pipe Tracking
25
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Heave Measurement Real rig example measuring and tracking objects Example: Estimating relative location and angle of pipe, slip joint
Applied Technologies © 2015 CoVar Applied Technologies, Inc. All Rights Reserved.
Leveraging Commercial Technology
• Can we leverage monocular video data to automatically drive a car through a simulated environment?
• Technology: Map and speedometer information extraction, automated driving feedback control loops
27
Automated Driving