Audit Issues

Ian Paterson,

Vice-Chairman, Scottish Registration Board

The SER AuditI expect that by now most of you will have been audited – and that experience can be expected to include a wide range of results and part of my task this afternoon is to allay some of the myths, legends and misunderstandings surrounding the audit process

But if I could begin by quoting some words of wisdom, which in their original context had nothing whatsoever to do with audits but none the less are quite appropriate to this afternoon’s topic:

“Nothing in Life is to be feared.It is only to be understood”.

Marie Curie

Topics Covered

• Why do we audit ?

• What is an audit ?

• What does the audit cover ?

• What is the audit process ?

• Recent changes to the audit procedures and their implementation.

Why do We Audit ?• It is a requirement of SER’s appointment by the

Government that the activities of all certifiers are monitored to confirm compliance with the requirements of the Statutory Regulations and the agreed Scheme.

• The audit is the principal mechanism by which the government / public can have confidence in the performance/practice of certifiers without independent scrutiny of each of their projects.

• Certifiers are not alone in being audited under the legislation as BSD regularly audit the performance of SER and the Verifiers.

Certifiers Appointment• Before looking at the audit process it is worth

reminding ourselves of the basis of the appointment of certifiers.

• Certifier status is awarded to professional individuals who have demonstrated the requisite experience in the design of (some types of) building structures – generally “on trust” and on the basis of statements made in their application form.

• They are expected to understand the relevant regulatory framework, the requirements of scheme membership, and to carry out their work in a diligent, responsible and professional manner (certifying within their competence).

Certifier’s Appointment• There are no categories of certifier’s but it is

incumbent on all certifiers to ask themselves if their knowledge is sufficient to enable them to certify each project before doing so. We experience more problems with certifiers “trading down” to small domestic projects outwith their experience as we do with those “trading up” to larger more complex projects.

• The audit is generally the first real test of their understanding of, and compliance with, these requirements.

What is an Audit ?Wikipedia:• Audits are performed to ascertain the validity and

reliability of information; also to provide an assessment of a system's internal control. The goal of an audit is to express an opinion of the person / organization / system (etc.) in question, under evaluation based on work done on a test basis.

• An audit must adhere to generally accepted standards established by governing bodies. These standards assure third parties or external users that they can rely upon the auditor's opinion on the fairness of financial statements, or other subjects on which the auditor expresses an opinion.

What is an SER Audit ?

At a subjective level that depends very much on your point of view.

What is an SER Audit ?

We recognise that whilst larger practices with QA accreditation are used to being scrutinised at audit for smaller practices / sole practitioners this is unlikely to be the case.

So let us be clear that the purpose of the audit is to seek evidence of compliance in the form of an “audit trail” of contemporaneous actions and this is what the certifiers will be asked to demonstrate at audit.

What is an SER Audit ?• Our experience suggests that your opinion of the

audit will be formed more by your mindset going into it than on the process or outcome of the audit itself.

• But to paraphrase Madam Curie The audit should be understood – then it should hold no fears.

• There should be no unknowns as the audit process, and the scheme requirements are well documented within the various publications produced by SER.

• So lets start by trying to come up with a definition of the audit ……..

What is an SER Audit ?• An adversarial meeting where the auditors try their

hardest to catch the auditee out on trivial issues.

– NO. What we seek, and hope for, from the audit is a demonstration of the certifier’s compliance. The best outcome for SER is to be able to show BSD that the certification scheme is working to a high standard – not to have high levels of non conformances or suspensions which will call the effectiveness or reliability of the scheme into question.

What is an SER Audit ?

• A necessary, (or unnecessary), evil ?.

Let’s try another definition which I think is more appropriate:

• An objective review of the activities of the certifiers/bodies against a pre-published set of requirements/criteria to confirm compliance with the scheme criteria.

Or, to put a more positive spin on it …..

What is an SER Audit ?• An opportunity for certifiers to demonstrate

that their professional practice is of such a high standard it does not require to be subject to external checking/scrutiny.

• The principal face to face contact between certifier’s and SER.

• An opportunity to discuss the processes being used, share best practice and suggest improvements within the context of the work undertaken.

What is an SER Audit ?• But perhaps the definition which best puts the

audit into the appropriate context, and perhaps makes it more palatable is …

• The alternative to verifier / external engineering scrutiny of the design on EVERY project you undertake.

Objectivity – Consistency of Audit

• To a certain extent the audit process has evolved over the 3-4 years during which the audits have been carried out – largely in response to the experiences and situations encountered.

• SER make every effort to ensure the outcome of the audits are as fair and consistent as possible.

• Audit teams comprise two auditors - so the outcome/report should be a balanced view and not based on the opinions of a single individual.

Objectivity – Consistency of Audit• Audits are carried out and scored against a fixed set

of criteria (known both to the auditors and auditees in advance).

• The auditors are briefed to carry out the audit and scoring “to the book” – not to “take a view” or make a judgement which would skew the scoring to affect the outcome.

• EVERY audit is discussed in detail by SRB and the auditors opinions are frequently challenged – sometimes resulting in a change of score.

• That discussion is the stage at which any other influences, extenuating circumstances, and the auditees response are taken into consideration.

What Does the Audit Cover ?• A number of projects selected to cover the range

and scope of certification being undertaken in terms of project size, complexity, type and date of the certificates generated.

• A review of the available records for the project – we can only audit on the information presented to us if the audit process is to have any integrity. It is the auditee’s responsibility to provide access to contemporaneous records demonstrating compliance.

• An audit of the process and level of scrutiny undertaken before signing the certificate.

What does the audit cover ?• The audit is very much a review of process:• Has the certifier identified all of the building elements

which the certificate will be deemed to have covered ? – does he know what he is certifying ?.

• You must remember we are not certifying “the structure” but are certifying “the building” in respect of the structural requirements of the Regulations.

• So it is not just your won drawings you are certifying but all of the drawings on which the Warrant is granted.

• Failure to recognise this often gives rise to issues at audit.

What does the audit cover ?• Has the certifier produced, or obtained, the

necessary level of information for warrant (Blue Book) ?

• Have they specified CDE’s in sufficient detail (TB1) ?

• Have they obtained/scrutinised the information accompanying the warrant application ?.

• Has the design been checked to the required standard ?

• Was the design completed before the certificate was signed ?

One Size Fits All ?• We have frequently been asked why we can’t have

a simpler audit process for smaller projects.• We did initially try a smaller projects check list on

the early audits – but it proved inconsistent and ineffective.

• In reality the use of the same check list on both small and large projects invariably means that there is more to scrutinise in larger projects because of their content and complexity.

• There are no “simpler regulations” for smaller projects – every building is expected to comply with the requirements of the regulations.

What is Required for the Audit ?• The certifier is required to demonstrate compliance with

the scheme requirements through reference to contemporaneous project records (audit trail).

• All information used or consulted in the certification process.

• Certification plan.• Drawings submitted with the warrant application

(architect’s and engineering).• Calculations (own or suppliers).• SI reports, survey reports, specifications etc.• Hard copy, or accessible electronic copy – onus is on

certifier to provide access to the relevant information.

What does the Audit NOT cover ?• We DO NOT carry out a detailed engineering or

arithmetic check of the drawings or calculations – we look for evidence that that has been carried out by others (as required under the scheme guidance) - though on some occasions deficiencies in the design may become apparent during review of the project documentation.

• We DO NOT look into commercial arrangements and appointments (other than Management of Risk and Protection of Certifiers – GN6).

Audit Process• Notification and arrangement of audit, including

projects to be audited.

• Opening meeting to explain audit procedure.

• Audit carried out by 2 experienced engineers, able to relate their own experience to “real life” certification situations, against a pre-published set of check lists.

• Closing meeting to discuss findings and advise on remainder of process.

• Preparation of audit report

Audit Process• Certifier (or Certification Coordinator) responds to

audit report – opportunity to dispute or ask for clarification if necessary but we look for recognition of issues involved and statement as to how these will be avoided or dealt with on ongoing certification activities.

BUT

• There is no point in disputing the facts of the audit at this stage (e.g. availability of info) as the issues will all have been discussed as part of the close out meeting.

Audit Process

• Full report, including response, discussed in detail at SRB (again for consistency) at which stage circumstances of audit will be taken into consideration before arriving at a final recommendation.

• SER Board consider SRB recommendations and make final determination of audit outcome.

• So it is quite a cumbersome process but has been made deliberately so to maximise the standardisation and protect the certifiers.

Jekyll and Hyde ?• The role of the certifier is independent of, and

different to, that of the engineer – even if undertaken by the same individual – and that requires a change in mindset.

• As and engineer your scope / remit can be set or amended through you appointment.

• As a certifier you are responsible for confirming the structural adequacy/performance of the entire building – not just “the structure”.

• When acting as a certifier YOU are Building Control and you must make adequate enquiry to determine/demonstrate compliance of elements you have not designed.

Common Issues from Audit• Inadequate scoping of certificate coverage.• Poor management of process/project to reflect

certification requirements.• Poor records.• Do you know what you are certifying ? – and

have you a record of this ?.• Inadequate review – or evidence of review (GN

3 certification options 3 and 4).• Inadequate detail on warrant plans.• Further details in the audit feedback reports

available on the website.

Audit Procedures• Many certifiers don’t recognise

the full significance of this document.

• Only read it when they know they are going to be audited (if even then).

• Document sets in detail what we are looking for/at in an audit.

• Excellent source of guidance on what you should be producing or looking for before signing a certificate.

Revised Audit Procedures• Revised Audit procedures published on 24/10/11.• Intended to reflect/ address experience over first 3

or so years of auditing, in particular where some aspects of the scheme requirements were not being interpreted correctly by certifiers.

• Very few material “changes” in the requirements.• More clarification or further guidance.• Brings together and formalises practice already

discussed at SRB and implemented during audits (how situations should be recorded/scored).

Revised Audit Procedures

• As only minor or process changes revised procedures will be implemented immediately on all future audits.

• Auditors will consider any issues where retrospective application would impact on audit outcome and these will be discussed at SRB before a final decision is made.

Principal Changes - Timescales• Affect auditors and auditees.• Audits must be undertaken within 8 weeks of

receipt of notification.• Projects to be audited will be confirmed after

audit has been arranged – 10 working days prior to audit.

• Auditors to upload reports within 5 days of audit.• Auditee to respond within 10 working days.• Warning issued then if no response within a

further 20 working days automatic temporary suspension.

Principal Changes - Bodies

• Grade of body to reflect performance of certifiers (large bodies expected to support certifiers and promote common practices and standards).

• Body obligation to provide training (certifiers CPD outcomes reflected in body scores).

• Protection of Certifiers emphasised.

Principal Changes – Project Audits• Evidence standards clarified – and harmonised

with BSD “Blue Book” and other recently issued or revised documents (TB1 & 6, GN 3 and 9).

• Certifier and Project Audits merged creating new category P9A to check certifiers acting within limits of competence.

• Some revisions to categorisation of non conformances.

• Scoring system amended to reflect changes (scoring is a measure of performance but is not the only determinant of the audit outcome).

In Conclusion• Hopefully you now have a greater understanding

of not just of what is involved in the audit – but perhaps more importantly also why.

• Hopefully that new understanding will serve to allay some of the misconceptions and fears surrounding the audit process.

Please remember:

• The audit is the alternative to verifier / external engineering scrutiny of the design on EVERY project you undertake.