auditing risk management
TRANSCRIPT
-
8/11/2019 Auditing Risk Management
1/41
Company Confidential
Registration Management Committee (RMC)
1
How to Audit Risk Management
Atlanta, GAJuly 22 & 23, 2010
Kimberly MaggieRon Tarach
QUAL-TECH, INC.
Auditor Workshop
Atlanta, GA
July 22-23, 2010
-
8/11/2019 Auditing Risk Management
2/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 2
Agenda
What is Risk?
Risk Management Process
Examples Risk Management Criteria
Auditor perceptions of Risk Management
Risk Management Tools
Auditor knowledge of tools and actions
-
8/11/2019 Auditing Risk Management
3/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 3
Agenda (continued)
Audit Planning
Audit Planning Tools
Activity 1 - Brainstorming session using
Audit Planning Tool
Conducting the Audit of Risk ManagementProcess
Examples of areas to evaluate
Activity 2 - Brainstorming session using CaseStudy and Failure Modes and Effects Analysis(FMEA)
-
8/11/2019 Auditing Risk Management
4/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 4
Ice Breaker!
-
8/11/2019 Auditing Risk Management
5/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 5
What is Risk?
An undesirable situation or circumstance thathas both a likelihood of occurring and a
potentially negative consequence.
AS9100:2009, clause 3.1
-
8/11/2019 Auditing Risk Management
6/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 6
Risk is inherent in all processes. Unfortunately, we dontsee the results of ineffective risk management methods
until later.
-
8/11/2019 Auditing Risk Management
7/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 7
Risk Management Process
Most organizations spend a great deal of time andmanpower trying to document Risks but manytimes this data is decentralized and not easilyaccessible to the functions that need thisinformation.
Process manufacturing can be so complex thatRisks can be very subtle and if there is not astructured Risk Management Process that takesadvantage of corporate knowledge, lessons learnedan organizations exposure to Risk can remain
high.
-
8/11/2019 Auditing Risk Management
8/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 8
-
8/11/2019 Auditing Risk Management
9/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 9
Examples of Risk Management Criteria
Understanding the types of risk that could comeinto a company. They could be related to
Employees
Process
Design Manufacturing
Equipment
Environment
Project Security
-
8/11/2019 Auditing Risk Management
10/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 10
Examples of Risk Management Criteria
Understanding the types of risk that could comeinto a company cont.
External
Contractor
-
8/11/2019 Auditing Risk Management
11/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 11
Examples of Risk Management Criteria
(continued)
Employees the organizations need toensure the safety, training, andqualifications of employees.
Process managing process variation.
Design building quality into the productdesign from the start, including its affect
on planning. Manufacturing ensuring that
manufacturing is more efficient withstreamlined quality planning.
-
8/11/2019 Auditing Risk Management
12/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 12
Criteria for Risk Management Process
(continued) Equipment ensuring that equipment can
meet capabilities, current and future.
Environment ensuring that theoperations are not compromising theenvironment (adequate lighting,temperature control, noise, cleanliness,etc).
Security managing the security neededby the facility.
Project ensuring project risks areevaluated before beginning.
-
8/11/2019 Auditing Risk Management
13/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 13
Criteria for Risk Management Process
(continued) External developing plans to address the
potential impact of weather, issues withtransportation companies, city
infrastructure (relating to construction,road closures).
Contractor ensuring impact is consideredfor contractors working on the building,
equipment, or with employees.
-
8/11/2019 Auditing Risk Management
14/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 14
Auditor Perceptions of Risk Management Thats the way we identified and handled risk
when I worked at Aviation Anywhere, Inc.
When I audited a Original EquipmentManufacturer (OEM) last month they wereusing FMEAs.
This little company only uses tool XYZ they
cant be managing risk properly.
-
8/11/2019 Auditing Risk Management
15/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 15
Auditor Perceptions of Risk Management(continued)
Remember, the design and implementationof an organizations aerospace quality
management system is influenced by varyingneeds, particular objectives, the productsprovided, the processes employed and thesize and structure of the organization.
AS9100:2009 General
-
8/11/2019 Auditing Risk Management
16/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 16
Auditor Perceptions of Risk Management(continued)
Organizational application of Risk can varybased on situation, customer, product line.
Audit approach & interviewing will need to beappropriate to the organization.
Remember, what is Appropriate to the
organization.
-
8/11/2019 Auditing Risk Management
17/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 17
-
8/11/2019 Auditing Risk Management
18/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 18
Risk Management Tools
FMEAs e.g. dFMEA, pFMEA, etc.
Fault Tree Analysis (FTA)
Probabilistic Risk Assessment (PRA)
Event Tree Analysis (ETA)
Event Sequence Diagram (ESD)
Master Logic Diagrams (MLD)
Reliability Block Diagram (RBD)
-
8/11/2019 Auditing Risk Management
19/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 19
Risk Management Tools (continued)
Risk Assessment Matrix
Likeliness/Consequence Table
SWOT (Strength Weakness Opportunity
Threat) Business Continuity/Current Capability
Matrix
Risk Map and Control Scale
-
8/11/2019 Auditing Risk Management
20/41
-
8/11/2019 Auditing Risk Management
21/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 21
Risk controlled or Oh No?
-
8/11/2019 Auditing Risk Management
22/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 22
Risk Management Tools (FMEA)
-
8/11/2019 Auditing Risk Management
23/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 23
Risk Management Tools (Influencer Analysis)
-
8/11/2019 Auditing Risk Management
24/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 24
Risk Management Tools (Risk Consequence)
-
8/11/2019 Auditing Risk Management
25/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 25
Risk Management Tools
-
8/11/2019 Auditing Risk Management
26/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 26
Audit Planning
Selecting the right audit tool.
Identifying your audit criteria and anyreference documents.
Identifying your audit scope, includingidentification of the organizational andfunctional units and processes to beaudited.
Identifying an appropriate audit scope.
-
8/11/2019 Auditing Risk Management
27/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 27
Audit Planning Tools
Process (Turtle) Tool
Process Map Tool
Supplier Input Process Output Customer(SIPOC) Form
Process Based Management (PBM) ProcessFlow
-
8/11/2019 Auditing Risk Management
28/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 28
Process (Turtle) Tool
With What
(Materials, Equipment, Facilities)
Inputs (information and
material from other
processes)
How?
(Methods/Procedures/Techniques
With Who?
(Comp./Skills/Training)
Outputs (information
and Material to other
processes
How Effective/Efficient?
(Measurable Objective)
Process
-
8/11/2019 Auditing Risk Management
29/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 29
Process Map
-
8/11/2019 Auditing Risk Management
30/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 30
Supplier Input Process Output Customer(SIPOC) Form
-
8/11/2019 Auditing Risk Management
31/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 31
Process Based Management (PBM) ProcessFlow
-
8/11/2019 Auditing Risk Management
32/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 32
Activity 1 - Brainstorming session using
Audit Planning Tool
-
8/11/2019 Auditing Risk Management
33/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 33
Process (Turtle) Tool (Design)With What
Risk Management Software
Forms
Documents
Inputs
Customer, Internal Organization,Regulatory, Statutory
Special Requirements (e.g. product or
process complexity)
Critical Items (functions, parts, software,
characteristics, processes)
How?
AS9100, AS9110 and AS9120 Standards
Quality Manual
Standard Operating Procedure for Contracts
FMEA
Risk Assessment Matrix
With Who?
Sales
Engineering
Production
Quality
Outputs
Design
Planning
Production
Purchasing
Suppliers
Shipping
How Effective/Efficient?
Customer complaints
In process/final rejection
Design verification/validation
Process
Contract Review
- Risk Management
Outputs
Drawing/Spec
Travelers
Routers
Work Orders
Inspection Reports
-
8/11/2019 Auditing Risk Management
34/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 34
Process (Turtle) Tool (Design Excluded)With What
Risk Management Software
Forms
Documents
Inputs
Customer, Internal Organization,Regulatory, Statutory
Special Requirements (e.g. product or
process complexity)
Critical Items (functions, parts, software,
characteristics, processes)
How?
AS9100, AS9110 and AS9120 Standards
Quality Manual
Standard Operating Procedure for Contracts
FMEA
Risk Assessment Matrix
With Who?
Sales
Engineering
Production
Quality
Outputs
Planning
Production
Purchasing
Suppliers
Shipping
How Effective/Efficient?
Customer complaints
In process rejection
Final rejection
Process
Contract Review
- Risk Management
Outputs
Travelers
Routers
Work Orders
Inspection Reports
-
8/11/2019 Auditing Risk Management
35/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 35
Conducting the Audit of Risk ManagementProcess
Examples of areas to evaluate
Are all Risk identified during the RFQ and ContractReview Process e.g. special requirements, criticalrequirements.
Ensure Top management clearly understands whatRisks they have and what they are doing to ensurethey are mitigating those Risk.
Evaluate the selected Risk Management Tool foreffectiveness.
How are Risks communicated and managedthroughout the organization e.g. Design, Planning,Purchasing, Suppliers, Manufacturing, Inspection,Delivery and Post Delivery.
Design inputs, Design FMEAs, Design Verification andValidation.
-
8/11/2019 Auditing Risk Management
36/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 36
Conducting the Audit of Risk ManagementProcess
Examples of areas to evaluate continued
Critical characteristics across the quality lifecycle,ensuring the Process FMEAs and Control Plans arelinked.
Processes in place for capturing leading and laggingindicators related to Design Quality Performance.
Evaluate whether the organization has closed loopContinual Improvement Processes that captures andsustains Product and Process Quality.
Organization is using Lessons Learned and BestPractices.
-
8/11/2019 Auditing Risk Management
37/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 37
Conducting the Audit of Risk ManagementProcess
Examples of areas to evaluate continued
Ensure organizations Change Management Processinvolves the right people at the right time with theright process.
Ensure integration of Change Management withassessments to ensure correct consideration of Risk.
Ensure Risk Assessment tracked, recommendedcontrols to completion and ensured that Risk weremitigated as prescribed.
Ensure controls are in place for Risk that still remainafter mitigation actions.
-
8/11/2019 Auditing Risk Management
38/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 38
Activity 2 - Brainstorming session using
Case Study and FMEA
-
8/11/2019 Auditing Risk Management
39/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 39
Closing!
-
8/11/2019 Auditing Risk Management
40/41
Registration Management Committee (RMC)
Atlanta, GA
July 22-23, 2010 40
Questions!
-
8/11/2019 Auditing Risk Management
41/41
Registration Management Committee (RMC)
G
References
1.AS9100:2009
2.ISO 19011
3.FAA Risk Management Handbook 2009
4.NASA