Lifting the lid on the DOA/POA arrangement … Administrative chore or key document for Performance ?

Mark Barker 21st March 2017

Introduction

Mark Barker

Principal Airworthiness Surveyor Production, SAFA and Military Sector

EASA Production Organisation Approvals (POA) Manager CAA Focal for MAA Design Approved Organisation Scheme (DAOS)

2

Prior to CAA

3

CAA …

4

Subjects

What are we finding ?

The DOA/POA arrangement – Key requirements

Important Role of the Manufacturing/Production Engineer

Quality Escapes, Concessions and the Performance Conversation

Comparison of Findings to key risks from the Prime and SME Manufacturing Groups

5

8

2016

0

5

10

15

20

25

30

35

40

45

50

55

60

Subcon C

ontrol

Design Interface

Process C

ontrol

Quality Audit

Exposition

Facilities

Form 1 C

ompletion

Records/R

ecord Work

Personnel Q

ualification

Tooling

Facilities

Ident & Traceability

Issues Reports

Design Interface Findings

Always in the Top Three …

In 2016 alone, 51 findings directly allocated by Surveyors

Just outside the Top 12, but increasing …

20 findings of POAs unable to demonstrate manufacture to Approved Data

19 findings of lack of effective control of Concessions

Taken together, Link with Design is the subject generating the highest level of non-conformance across POA

9

Requirement Areas:-

Entry Criteria for POA

Demonstration of links to Approved Design Data

Direct Delivery Authority – Key for FAA, protection against suspect unapproved parts

No direct equivalent in other Industries

One of the Part 21 ‘deltas’ to ISO9000

10

Eligibility – 21.A.133

11

– The applicant shall:

a) justify that for a defined scope of work, an Approval under this subpart is appropriate for the purpose of showing conformity with a specific design; and

b) holds or has applied for an Approval of that specific design; or

c) have ensured, through an appropriate arrangement with the applicant for, or holder of, an Approval of that specific design, satisfactory co-ordination between production and design.

Links to Design – 21.A.133

12

• 133b/c Does the Applicant have suitably documented arrangements (see AMC No 2 to 21.A.133b&c) with a DOA to ensure satisfactory co-ordination including:

• 133b/c The timely transfer of all airworthiness and design data (Specifications, Drawings, Service Bulletins etc.)

• 133b/c The responsibilities & procedures of the applicant for developing and validating manufacturing data against design & airworthiness data supplied

Links to Design – 21.A.145

13

• 145b2 Does the applicant have an adequate and effective

procedure covering verification of production data with applicable airworthiness and design data . Where the POA develops its own manufacturing data from the design package, procedures are required to demonstrate correct transcription of the design data.

• 145b2 Is there an adequate and effective procedure to define traceability of such data to each product, part & appliance for the purpose of certifying safe operation and conformity

Receipt of Design Data is the start point

14

• Production often represented as ‘build to print’ – too simplistic.

• Design data is rarely sufficient to actually manufacture without generating and validating further data.

• In effect, design ‘activity’ without holding a design approval – Production/Manufacturing Engineering.

• May not be reviewed under external QMS approvals.

Process Works

Instructions CNC/CMM

Programmes Involvement in

FAIRs Bill of

Materials

Manufacturing Drawings

Routings/ Travellers

Manufacturing Sequences

Process Layouts

Involvement in Concessions

Purchase Order

Descriptions

Tooling Drawings Build Manuals

PRODUCTION/ MANUFACTURING

ENGINEERING

Welding Processes

Continuous Improvement

MRP Item Masters

Obsolescence Management

Production Methods

What are we looking for ?

16

• Regardless of whether the manufacture is conventional machining/assembly, CNC, composite layup or ALM, the question set is the same:-

• Who has approved the new/changed process/document ?

• Are they authorised in the Company procedures ?

• Is there evidence of validation of the change to the Design Data ?

• To what extent does the design organisation (i.e. the DOA) expect to be involved in this process ?

Links to Design – 21.A.133

17

• 133b/c Which persons or offices are responsible for

controlling the above arrangements and associated data

• 133b/c Acknowledgment by the holder of the TC/STC/repair or change approval/ETSO authorisation that the approved design data provided, controlled and modified in accordance with the arrangement are recognised as approved.

• AMC No. 1 references Intermediate Production Organisation (IPO) as allowed provided overall control is maintained.

• AMC No. 2 permits demonstration of interface via internal company procedures where the DOA and POA are the same legal entity but the arrangement must still be documented.

Intermediate Production Organisation

18

• Evidence that the Design Holder has visibility of the

lower-tier arrangement either directly or via flow-down

• POA awareness and access to the associated procedures, otherwise how does the organisation know how to work and where to report problems ?

• In case of FAA PMA, the PMA Holder may not own the design date - ‘licensing’. Has the ability to accept changes been delegated to the IPO or not ?

Why is all this important ?

At the very least, if an effective arrangement cannot be demonstrated the POA will be unable to make further EASA Form 1 releases of that part until addressed.

Immediate impact on business

Reduction of confidence in performance

If a systemic issue or if affecting parts with immediate airworthiness implications, then Level 1 Finding with wider effect on the approval – Limitation or Suspension

19

Safety Severity and Level 1

All findings are ranked for Safety Severity:- High, Moderate and Low.

Of the 594 Production Findings, 29 (4.8%) were ranked as High.

6 relating to Approved Data 3 relating to Design Links

CAA Project on consistent approach to Level 1

findings, results agreed by EASA.

20

Level 1 decision process

Is it a significant non-compliance with the requirement?

• Non-airworthy • Non-conformity • Fundamental • Systematic • Likely to happen

again

• Incompetence • Questionable character/culture • System/process failure • Resources • Inadvertent exposure of question

database

Level 1 decision process

Is it a significant non-compliance with the requirements?

Does it lower safety standards and hazards flight safety?

• Focus on the Non-conformance (not solely on the evidence of the non-conformance)

• Non-airworthy condition • Release of a part not

complying with design data • Compromised training

• A Hazard to flight safety will not always lead to a catastrophic event , consider critical parts

Level 1 decision process Discuss the potential level 1 with Sector Manager /

Principal (Immediate action required: revoke, limit or suspend)

• Advise org that a significant non-conformance has been identified • Discuss with manager prior to raising as a level 1 with org • Possible restrictions – approval/ratings/scope/locations/ • Corrective action plan timelines • Formal Closure process engaging with Sector Management

• The organisation cannot demonstrate that an item is in conformity with approved design data and is safe to fly • Design data not available

• Manufacturing processes not i.a.w. approved data • Incorrect/damaged tooling not i.a.w. approved data

• The organisation is not adequate to support the approval or part thereof due to: • Lack of Accountable Manager/nominated personnel • Competence of staff • Escalation of level 2 findings

• Falsification of records • Failure to respond to high safety severity findings within the agreed timescales • Denied access (after 1 written request) • Repeat significant finding

Consider the implications of the systemic breakdown of the quality system

Areas that could result in a Part 21G Level 1

• Both …

• Poor definition of who is responsible for what has been a feature of several recent reportable incidents, leading to uncontrolled introduction of changes into the field and subsequent restrictions of approvals.

• We will be sampling as part of our product audits, and we need a better view when judging POA performance.

So, DOA/POA Interface:- Key Document or Administrative Chore ?

25

Links to Design – 21.A.133

26

• 133b/c & 139b1 The arrangements to assist the DOA with airworthiness matters (i.e. traceability of parts & processes, retrofitting, technical information, deviations etc.)

• 133b/c Part 21 requirements such as 21.A.145b, 21.A.165c, f and g (compliance to design, DOA/POA interface requirements)

• 133b/c The arrangements to assist the DOA in showing compliance prior to type certification

• 133b/c & 139b1 The procedures to deal adequately with non-conforming parts (i.e. parts that do not conform to design data)

Escapes, Concessions and Occurrences

27

• Several recent incidents where NAAs (not just CAA) first became aware of production issues as a result of an EASA Airworthiness Direction with no prior communication from the POA.

• Addition to current NPA (2016-19) Rule Change to include need to advise NAAs on Occurrence Reports as well as the DOA/EASA, but this is a backstop to a good relationship between the National Authority and the POA.

• As the prime purpose of the POA is to manufacture to the approved Design Data, a Quality Escape or Concession is an important indicator of performance.

The Performance Conversation

28

• Specific meetings with Accountable Manager established with Entities as part of the CAA Performance Based Regulation (PBR) process.

• Includes specifics of company performance in complying to POA requirements, trends in Occurrence Reporting, Concessions, First Time Pass Rates, Scrap Costs etc.

• Entities concept currently restricted to larger organisations approvals with multiple approvals, but we need an understanding of overall POA performance in this area to support surveillance planning.

Amendment to Expositions

29

• Where an Occurrence Report to EASA/DOA is made as a result of production non-compliance, the National Authority should be advised.

• Using a similar approach to Leaflet C-180 on Suppliers, Exposition section on Procedures for Non-Conforming Product to include a trend statement for external escapes/concessions.

2014 – X 2015 – Y 2016 – Z

Amendment to Expositions

30

• Data expected to be available from Organisation’s internal Management Review.

• No specifics to maintain confidentiality.

• Does not include planned Production Permits or where change is initiated by Design Holder.

• Will be discussed during oversight audits (Root Cause and Prevention of Recurrence) and to inform surveillance planning.

Subjects

What are we finding ?

The DOA/POA arrangement – Key requirements

Important Role of the Manufacturing/Production Engineer

Quality Escapes, Concessions and the Performance Conversation

Comparison of Findings to key risks from the Prime and SME Manufacturing Groups

32

33

2016

0

5

10

15

20

25

30

35

40

45

50

55

60

Subcon C

ontrol

Design Interface

Process C

ontrol

Quality Audit

Exposition

Facilities

Form 1 C

ompletion

Records/R

ecord Work

Personnel Q

ualification

Tooling

Facilities

Ident & Traceability

Issues Reports

CAA Focus Areas

PAMG High Top 10

35

Risk Industry + CAA High Response

9. Tier 2/Tier 3 Supply 77 (1st) 106 (1st) 9 (1st)

23. Lack of Risk Management for Supply 70 (2nd) 95 (2nd) 7 (2nd =)

7. Emergent Technology incl. ALM 67 (3rd) 94 (3rd) 6 (3rd)

18. Incoming Competency levels 63 (4th) 86 (4th) 7 (2nd =)

8. Special processes, NADCAP, Crimp .. 62 (5th) 83 (5th) 4 (5th =)

19. NDT Skills, competence/retirement 59 (6th) 80 (7th) 5 (4th =)

12. Common approach on non-EU release 59 (6th) 70 (13th) 5 (4th =)

13.Policing of intent of design data/IPR 59 (6th) 77 (8th) 4 (5th =)

14. PMA +NAA training in acceptance 59 (6th) 72 (12th) 4 (5th =)

26. Lack of Lateral View of Escapes 58 (7th) 81 (6th) 4 (5th =)

SME Top 10 Areas

36

Risk Score

Raw Material Supply 74

Subcontractor Control 63

Capacity/Availability of Subcontractors 46

Production Competence 42

Supplier N/C to Purchase Order 41

Procurement not Involving Engineering 38

Flow down of treatment requirements 32

Legacy Support/Obsolescence 28

Single Source Supply 26

Competency Incoming 25

Proliferation of prime requirements 24

Correlation Areas

Several areas selected for Agenda Items today

Competency:-

22 direct findings on Competency 20 on Authorisations 13 on Certifying Staff 12 on Continuation Training …

Where do you think the challenge areas are ?

You all use other POAs – are we looking in the right

areas affecting you ?

37

Thank you for listening

Questions…