Leveraging Deep Learning and AI Applications in ManufacturingPeter Darragh & Stephen Welch

2

Why is a Microsoft AI/IoT Partner talking about visual inspection?

1.Why AI/IoT people are interested in industrial

visual inspection

2.How AI/IoT people are approaching visual

inspection problems

3.What customers value when asking AI/IoT people

to do visual inspection

3

”AI BANDWAGON”TRADITIONAL MACHINE VISION

• Industrial PCs or Smart cameras• Experts in lighting and image formation• Long term factory floor experience• Experienced in controls & automation• Solution often not internet connected• Data & images often not stored long term• Application/algorithm centric, not data centric

• Well versed in general AI solutions across many verticals (finance, banking, retail, healthcare, …)

• Cloud centric• Really good at enterprise scale, global solutions• Thousands of endpoints• Professional data science teams• Consumer service, not industrial, oriented• Limited factory floor experience

?

4

2 3 4

Which Images Show Defects?

1

GOOD GOODDEFECTIVE DEFECTIVE

5

Which Images Show Defects?

Traditional machine vision systems use a two-step process to make decisions

IMAGE CAPTURE

Human Machine Interface

Diversion Gates

Pick-and-Place Robots

…

SOFTWARE

COMPUTER VISION ALGORITHM

Feature Extraction Decisioning

MAX_CONTRAST > THRESHOLD1

AND

DEFECT_SIZE > THRESHOLD2?

COMPUTE(Integrated or Discrete)

6

TRADITIONAL MACHINE VISION

FEATURE EXTRACTION

MAX_CONTRAST > THRESHOLD1

AND

DEFECT_SIZE > THRESHOLD2?

IMAGE CAPTURE

PREDICTIONS/RESULTS

IMAGE CAPTURE

DEEP LEARNING MODEL

These algorithms are typically designed once by vision system manufacturer,

and “baked in” to production software.

DECISIONINGMay consist of many tunable

parameters, often difficult to find optimal configuration, even for experts.

PREDICTIONS/RESULTS

MODEL TRAINED ON YOUR DATA

Deep learning model trained using labeled examples from your experts, and updated as conditions change.

7

Alexnet Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. ResNet He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

8

ANSWER: Not Long

Why not?

How long have AI/IoT solution providers applied deep-learning to factory floor problems in general and visual inspection in particular?

9

Absence of AI on the Factory Floor AI - Exhibit A

10

Windows Azure General AvailabilityFeb 1, 2010 | Microsoft blog editor – Microsoft News Center Staff

Today marks a significant milestone. We are announcing the generalavailability of Windows Azure and SQL Azure in 21 countries. Startingtoday customers and partners across the globe will be able to launchtheir Windows Azure and SQL Azure productions applications andservice with the support of the full Service Level Agreements (SLAs).

Feb 1, 2010

11

Absence of AI on the Factory Floor AI. Exhibit B

AI Failing on the Factory Floor AI - Exhibit A

Abstract

There is a great disparity between the number of papers

which have been published about AI-based manufacturing

scheduling tools and the number of systems which are in

daily use by manufacturing engineers. It is argued that this

is not a reflection of inadequate AI technology, but is rather

Indicative of lack of a systems perspective by AI

practitioners and their manufacturing customers..

12

“At Foxconn we have met many AI companies interested in industrial scenarios. I’ve noticed

they first ask what data we have before

asking us what problems we want to

solve….In the past AI looked for hidden

relationships in data, whereas industry needs

to begin with problems and create value

by creating solutions”.

13

AI Failing on the Factory Floor AI - Exhibit B

Raja Shembekar, vice president of Denso’s North American Production Innovation Center, is the chief architect of their use of the Internet of Things (IOT). As a starting point Raja benchmarked other companies thought to be leaders in the technology. He found a lot of what he came to call “IOT wallpaper” with little real application. Cool-looking displays, but no real problem solving

14

When the Toyota Way Meets Industry 4.0:

AI Failing on the Factory Floor AI - Exhibit C

He built a small team, with about half IoT experts, and half shopfloorpeople like quality managers who were good at software. Together, they started to work on real problemsidentified at the gemba.

15

AI/IoT learning to succeed - Exhibit A

What are ‘Real-Problems’?

The application of AI needs to focus on solving problems that have not been solved in the past rather than creating new needs or looking for alternative solutions to problems that has been already solved.

Jay Lee

16

AI/IoT learning to succeed - Exhibit B

ANSWER:• Working together• Addressing previously unsolved problems• At the gemba

How do AI companies avoid making the same mistakes others have over the past 30 years?

17

AI/IoT learning to succeed - A conclusion

Purposeful innovation

“You need a direction, you need a very concrete challenge, you need to

understand your current condition, and you need to experiment step-by-step

towards shorter term targets and you don’t need to know how to get to the

challenge, but eventually you will find yourself there”

“you make a prediction and then you test the hypothesis”. PDCA. I

understand the problem and I have asked Why 5 times; I have an idea as to

what the problem is and here is the first experiment”.

18

AI/IoT learning to succeed - A conclusion

Principle #8:Use Only Reliable, Thoroughly Tested Technology That Serves Your People and Processes

19

“Any information technology must meet the acid test of supporting people and processes and prove it adds value before it is implemented broadly.”

AI/IoT learning to succeed - A conclusion

How Cloud AI Made a Difference at a Chemical Plant

“focus on multiple mid-size opportunities and stack them

on top of each other to really reach our return on

investment”

• Predictive models - critical equipment failures and forecasting

per pump and valve when CBM thresholds will be met.

• Findings leaks – to speed up maintenance response

• Finding energy anomalies – to identify degrading, motors,

pumps and sticking valves and where more insulation is needed.

20

Why the Cloud is Ideal for Deep Learning

The cloud is AI and Deep Learning friendly

• Elastic storage for large volumes of data

• Variety of storage options, e.g. NoSQL, SQL, logfiles, images

• Push-button and slider controls to provision, scale and remove

massive compute

• Services such as AutoML and no-code drag and drop interfaces

to create models, their workflows and integrations

21

Why the Cloud Might Not be Ideal for You

When the cloud isn’t helpful

• Against corporate policy

• Operating Expense versus Capital Expense

• Too much latency

• Connectivity concerns

22

Evaluating the consequences of lost connectivity

Monte-Carlo simulation to calculate the consequences requires:

• Frequency distribution of disconnect

• Duration distribution of disconnect

• Frequency distribution of AI inference (decision making)

• Cost Impact of decision consequences

23

Why the Cloud Might Not be Ideal for You

Dealing with Cloud Separation Anxiety

How long is too long of a disconnect?

• You need an inference before the next planned outage, e.g. inspection

work order suggestion.

Hours?

• You need an inference before the next change over, e.g. suggested set

points and recipes.

Minutes?

• You need an inference for the next part, e.g. visual inspection.

Seconds?

24

Chapter 6 Emergency Procedures

Section 4. Two-way Radio Communications

Failure6-4-1. Two-way Radio Communications Failure

.IFR conditions. If the failure occurs in IFR conditions, or if

subparagraph 2 above cannot be complied with, each pilot shall continue the

flight according to the following:

(a) Route.(1) By the route assigned in the last ATC clearance received;(2) If being radar vectored, by the direct route from the point of radio failure to the fix, route, or airway specified in the vector clearance;(3) In the absence of an assigned route, by the route that ATC has advised may be expected in a further clearance; or(4) In the absence of an assigned route or a route that ATC has advised may be expected in a further clearance by the route filed in the flight plan.

25

Dealing with Cloud Separation Anxiety

What do you do when you need AI all the time?

ANSWER: OPERATE AI/DL ON YOUR NETWORK AT YOUR PLACE

26

Dealing with Cloud Separation Anxiety

• Existing vision system captured very high quality images, but struggled to differentiate water, dust, and dirt from true defects

• High false positive rate (many non-defects categorized as defects)

• High false positive rates make automation infeasible, and introduces other production complexities

Case Study:Deep Learning on the edge improves quality and reduces costs at a glass manufacturer

27

Demonstrated deep learning performance through PoC, reducing false positive rate from ~25% to <1.0%

Current Vision System

Accuracy = ~75%False Positive Rate = ~25%

Accuracy = ~95%False Positive Rate = <1%

PREDICTED

Defect Good Water

AC

TUA

L

Defect 45 0 0

Good 0 50 0

Water 30 0 0

PREDICTED

New Defect Class

Defect Good Water

AC

TUA

L

New Defect Class 15 0 2 0

Defect 0 25 0 1

Good 1 0 50 0

Water 0 0 0 30

28

Real-time decisioning, configuration, and labeling performed on edge

29

INDUSTRIAL VISION

SYSTEM

Edge PC

Edge Container

FACTORY FLOOR CLOUD

ML Model +

Deployment Code

Human Machine

Interface

Local Data Storage

Optional

Storage

Optional

Scoring

FACTORY EQUIPMENT

Diversion Gates Pick-and-Place

Robots

Cloud Storage

Images + metadata (LAN)

ControlSignals

Containerization

Azure IoThub

Modbus/

Profibus/

Devicenet/

Ethernet/IP

Images & Data

Local Compute

Deployments & Updates

30

Intel Products and Technologies for IOT Edge

Compute AI/CV inferenceworkload accelerators

Connectivity

ETHERNETHARDWARE

TECHNOLOGYMANAGEABILITYREAL TIME SECURITY

COMPUTER VISION VIRTUALIZATION

AI in the Cloud

MOTORS

AND PUMPS

BOILERS AND

REACTORS

OVENS AND

KILNS

PACKAING AND

FOLDING

INDUSTRIAL

EQUIPMENT

DIR MFG

PLANT MANAGER

GATEWAY

31

AI on the Edge

Azure Stack EdgeCloud Managed Appliance

QUALITY MGR.

SHIFT LEAD

32

A Gateway + Edge Combination

MOTORS

AND PUMPS

BOILERS AND

REACTORS

OVENS AND

KILNS

PACKAING AND

FOLDING

INDUSTRIAL

EQUIPMENT

DIR MFG.

PLANT MANAGER

&

QUALITY MGR.

SHIFT LEAD

GATEWAY

33

Why Factory-floor AI and Deep Learning needs a hybrid edge/cloud to truly deliver 4.0 Smart Factory capabilities

&

INSPECTIONRESULTS

PROCESS TELEMETRY

LOT/BATCH/

SERIAL NO.

34

Like many machine vision systems, our customers’ systems struggle with a high false positive rate

35

Act

ual

97.7%Accuracy

30X

2-5X

False Rejects Reduction

Improvement Over Manual InspectionPredicted

Non-Defect 1

Non-Defect 2

Non-Defect 3

Defect 1

Defect 2

Defect 3

Defect 4

Defect 5

Defect 6

36

37

38

Spyglass delivers deep-learning, monitoring, and root cause analysis capabilities through Edge & Cloud together

INDUSTRIAL VISION

SYSTEM

SVI MACHINE

Edge Container

FACTORY FLOOR CLOUD

ML Model +

Deployment Code

Human Machine

Interface

Local Data Storage

Optional

Storage

Optional

Scoring

FACTORY EQUIPMENT

Diversion Gates Pick-and-Place

Robots

Cloud Storage

Azure SQL + Blob

Images + metadata (LAN)

ControlSignals

Monitoring + Alerting

Model Training

PyTorch

Quality

Analytics

Power BI

Modbus/

Profibus/

Devicenet/

Ethernet/IP

Reporting + Data (MQTT)

Local GPU Compute

Model Updates

Spyglass delivers real-time monitoring and alerting through local HMIs and Power BI

39

Deep Learning Drives Many Paths to Business Value

Precise, rapid, and

adaptable visual

inspection

Remove Defective Products

Early in Production

Improve Output Quality

Increase Production

Speed

Monitoring + Alerting to

Catch Quality Issues Early

Reduce Required Inspection Resources

Quality Issue Root

Cause Analysis

30X

2-5X

False Rejects Reduction

Improvement Over Manual Inspection

20% Increase in line speed

10X Reduction in missed defects

20+

<5 Minutes to detect quality problems

Varieties of defects detected

40

DEEP LEARNING

41

Define SuccessThe Spyglass team works

with you to define your

unique vision accuracy

requirements.

Supply ImagesProvide sets of images of

your products that

represent acceptable

quality as well as images of

each class of defect.

Prove it WorksUsing supplied images,

the Spyglass team

builds an AI model

demonstrating the

success criteria

How do we get started?

Q&A

Peter Darragh, EVP Product Developmentpeter.darragh@mariner-usa.com

Stephen Welch, VP Data Sciencestephen.welch@mariner-usa.com

Further Questions & Inquiries:info@mariner-usa.com

Visit Us atwww.mariner-usa.com