w g e m 25 26 f 2015 fvo, gaudit-network.wikispaces.com/file/view/draft audit... · ·...

Version 1 rev.3 – 25-26/02/2015 (with PT/SP additional comments)

NAS Audit Evidence subgroup - working document

REFERENCE DOCUMENT FOR NAS NETWORK

WORKING GROUP - AUDIT EVIDENCE

MEETING 25 AND 26 FEBRUARY 2015

FVO, GRANGE



The National Audit Systems (NAS) Network

The NAS network is a network of officials (auditors) from national competent authorities,

responsible for the performance of audits of official control systems as provided for by article

4(6) of Regulation (EC) No 882/20041. The networks meet regularly, under the chairmanship

of, and facilitated by, the FVO to exchange experiences in implementing national audit

systems on official control activities. During the course of these exchanges; discussions,

workshops etc. good principles and practices are identified and agreed by the network.

To enable dissemination of information the network, working in plenary session and through

sub-groups, facilitated by the FVO, consolidate agreed principles and good practices on

specific topics into reference documents. These reference documents may be used as

guidance documents, however, they do not constitute an audit standard and are not legally

binding.

Audit Evidence

OBJECTIVES

The objective of this document is to guide and support Competent Authorities (CA) and audit

bodies in managing audit evidence.

The aim is:

To provide principles and definitions regarding audit evidence

To identify characteristics, types, and sources of audit evidence

To discuss evidence collection planning

To give principles/discuss verification of audit evidence

This document is intended to assist in the implementation of Section 6 of the Annex to

Commission Decision 2006/677/EC.

SCOPE AND INTENDED AUDIENCE

This guidance applies to planning and performing of audits as required by Article 4(6) of

Regulation (EC) No 882/2004.

It is intended for use by CAs / audit bodies that carry out audits on official control (systems)

according to the requirements of Article 4(6) of Regulation (EC) No 882/2004.

It supports the development of good practice in audit evidence collection and verification in the

area of official control activities e.g. feed, food, animal health and welfare and plant health.

1 OJ L 191, 28.5.2004



I. BACKGROUND AND CONTEXT

{where does evidence fit in the audit cycle – evidence and evidence collection plan}

{The objective of collecting evidences : exemplify, measure, compare, evaluate, demonstrate

to meet the objective in order to support audit conclusions}

{audit criteria and how it links with audit objectives}

The collection of audit evidence is a common (familiar) but important step in the audit

process. The quality of the evidence collected has a direct and significant effect on the audit

findings and conclusions.

The audit team should, at the audit planning stage of an audit, consider what audit evidence

should be required. During the audit process, the audit team should verify the audit evidence

collected and ensure it is appropriate and sufficient to achieve the audit objectives.

Sometimes, other evidences may be necessaries, in order to cover specific findings. Audit

evidence needs to be compared to the audit criteria and the audit objectives to allow the audit

team produce audit findings and present persuasive audit conclusions.

Only audit evidence that is appropriate and sufficient will effectively support audit findings

and conclusions which are capable of withstanding challenge and satisfy internal and external

scrutiny.

{Audit objectives: (IIA) 2210.A1- Internal auditors must conduct a preliminary assessment of

the risks relevant to the activity under review. Engagement objectives must reflect the results

of this assessment}{Audit criteria: means the set of policies, procedures or requirements used

as a reference against which audit evidence is compared, i.e. the standard against which the

auditee’s activities are assessed.}

. (ISO) – (IIA) 2210.A3- Adequate criteria are needed to evaluate controls. Internal auditors

must ascertain the extent to which management has established adequate criteria to

determine whether objectives and goals have been accomplished. If adequate, internal

auditors must use such criteria in their evaluation. If inadequate, internal auditors must work

with management to develop appropriate evaluation criteria. For me the approach seems

different, the criteria are linked for the IIA to the measure of the organisation’s goals, the

criteria used for the appreciation of the evidences come from the key internal control points

of the procedures (in the risk matrix of the audit team)}{Different terminology may be used

by different MS for the same processes}

{reference to ISO to help with language versions}

{Evaluation of Evidence: Is this within the scope of this document or does it belong to the

next phase ? Possibly dealt with in a separate reference document on “Recommendations”}

Deleted: ,

Deleted: ¶¶

Comment [KN1]: Other than the map there was little to organise our thoughts in this section of the paper, so I have put in some text describing audit evidence in relation to the audit process map and the context.

Comment [H2]: I do agree, that a preliminary risk assessment could be carried out. But this is not a definition of an “audit objective”.

Deleted: ¶

Comment [KN3]: Should the definitions for audit objectives and criteria be here or below in the definitions section?

Deleted: ¶



II. DEFINITION(S)

This document should be read in conjunction with the definitions contained in Regulation (EC)

No 882/2004 and Commission Decision 2006/677/EC bearing in mind that the definitions of

those documents apply.

Audit Evidence: Observations, records, statement of facts or any other information, obtained

directly by the auditors or provided by the auditee or third parties, which are sufficient, relevant,

reliable and useful in order to be able to analyze the root causes and always verifiable.

Effectiveness: is the extent to which official controls produce an (intended) effect / achieve an

objective2. In this particular context the objectives are those of Regulation (EC) 882/2004.

Effectiveness is not to be confused with efficiency, which is normally used when we want to refer

to input-output ratio i.e. cost and/or resources required to produce an output. (“Auditing

effectiveness of official control systems” NAS network document)

Findings: results of the evaluation of the evidence collected during the audit against the

applicable standard (e.g. legislation), described in an objective manner. (FVO SOP Audit

Performance)

Conclusions: statements made by the audit team concerning the outcome of the audit which are

based on and after consideration of all the findings and the audit objectives but which do not

propose any course of action. (FVO SOP Audit Performance)

III. AUDIT EVIDENCE

Include statement on usefulness of evidence, i.e. when it helps to reach goals of the audit "An

effective audit has persuasive findings and conclusions. The quality of audit findings and

conclusions relies on the judgements the auditor makes and these judgements are directly

dependent on the quality of the audit evidence collected and the competence of the auditor

collecting and analysing it”

Audit Evidence: information used by the auditor in arriving at the conclusion on which the

auditor's opinion is based…."(international Standard on Auditing (UK and Ireland)

Audit Evidence: Audit evidence is the information internal auditors obtain through observing

conditions, interviewing people, and examining records. Audit evidence should provide a factual

basis for audit opinions, conclusions, and recommendations. (IIA - SAWYER) / Audit evidence is

the information that supports or refutes an audit objective (IIA – David O’Regan)”.

The nature of audit evidence in systems audits

{particularities of systems audits vs financial or compliance audits}

For financial or compliance audits, evidence only needs to be collected to demonstrate activities

are being carried out to planned arrangements. For systems audits, evidence needs to be collected

to verify the effective implementation of planned arrangements

Quantitative versus qualitative

2 Objectives may be at a strategic or operational level.

Deleted: ¶

Deleted: Audit Evidence: records, statements of fact or other information which are relevant to the

audit criteria and verifiable. (ISO 19011:2011 from ISO 9000:2005)¶

Comment [ST4]: Adapt these definitions to the NAS context.

Deleted: {Note: characteristics not mentioned in this section are better described in the next chapter,

less prescriptive and explaining the concept}¶



A. Characteristics of audit evidence

{Some text?}

Description

Persuasive

The persuasiveness of evidence is linked to its appropriateness (relevant

and reliable) and sufficiency.

(Linked also to target audience and findings)

Appropriateness

/ Usefulness

The appropriateness of the evidence is the measure of the quality of the

evidence determined by its reliability and relevance.

Sufficient

When there is enough evidence to persuade a reasonable person that the audit

findings and conclusions are valid, and that the recommendations are

appropriate. [IIA]

Amount of evidence considered enough: [Scoping paper]

i) for the auditor to form a reasonable opinion (sample size,

representativeness)

ii) to convince interested parties/stakeholders of validity of auditors

opinions (persuasive)

Relevant

When the evidence is clearly and logically related to the audit questions, audit

criteria and audit findings. [IIA]

Extent to which the information bears a clear and logical relationship to the

audit findings (and audit objectives). [Scoping paper]

Reliable

When evidence is obtained through the use of appropriate techniques. When

the same findings arise when alternative techniques are used or when

information is obtained from different sources.

The best obtainable information through the use of appropriate engagement

techniques. [IIA]

The degree to which evidence can be considered trustworthy (accurate and

credible), the likelihood of coming up with the same answers if audit test is

repeated or information is obtained from a different source or test. [Scoping

paper]

Continuity and integrity of evidence. e.g. in a laboratory the reliability of

results could be in question if sample identification, documentation and/or

security is suspect.

Verifiable

Archived in a proper and available support. If applicable, provided with

appropriate references, linked to original information or documents.

MAMA

Objective Collected with open mind, free from subjective impressions, responds

directly to the findings. MAMA

Representative Adequately expresses or demonstrates the extent of the findings.

Representative of the audit universe and … time… {UK’s example}

Logical/

Rational/

Reasonable/

Linked to persuasiveness MA

Comment [ST5]: JM To clarify: The characteristic “Reliable” should include that evidence has not been, nor is likely to have been, interfered with/altered/amended in an unauthorised or un-endorsed manner. This is particularly important in the laboratory environment as suggested, but in other areas also for example in the context of the traceability of animals and animal products. This is what I meant by the “Continuity and integrity of evidence”.



Description

Sound

Reference to Annex I – Audit Evidence Mind Map

B. Types

Type Description Examples /Techniques Considerations

Observed

(or Physical)

Information

gathered by the

auditor through

personal

observation of

people, events

and physical.

Examples:

Visual control

?

Sample

Techniques:

Direct inspection or

observation of people,

property or events.

Listening, smelling?

On-site verification

Shadow inspection /

Witness audit

Review audit3

Whilst usually the most persuasive

evidence, the auditor must be aware

that a risk exists that his/her presence

may distort or prejudice what would

normally occur, thus reducing the

quality of the evidence.*

Ways to record this type of evidence –

photo, notes, checklists, samplings??

Cross reference with Section V.

Verification of Evidence.

Documentary

Information

prepared by

others than the

auditor.

Documentary

information can

exist both in

paper and

electronic form.

Exemples :

Documents containing

routines website

information, etc.

Photos

Internal/external

Paper/electronic

Legal/work

ISO definition of

“document”?

Techniques:

Review of mandatory

rules or laws,

documents, reports,

manuals, literature,

external and internal

websites, postal or

web-based surveys,

Data-base

information..

This evidence may be in electronic or

hardcopy format. *

However, useful information may not

always be documented, thus

necessitating the use of other

approaches also.*

Be sure to record the date on which the

information was gathered as the

information may change later on.

3 Audis of the FBO without the presence of the inspector

Deleted: of evidence

Comment [agesp6]: Or replay inspection with or without inspectors

Deleted: Photo

Comment [KN7]: We have agreed that this is documentary evidence. It was given as an example in an international definition given by Maura.

Comment [agesp8]: Electronic documents (invoices, administrative documents, reports, etc.) are more and more common every day and there is a need that their electronic signatures are properly guaranteed to prevent manipulations/fakes (through electronic reference sites, links to government agencies where to check the data, etc.). Documents that make up the evidence homogeneity is becoming a need (same types of records, reports, invoices, etc.). There are serious problems in many cases with the file and manipulation of evidence, starting by which make the audited and the Auditors. Traceability of evidence is their ultimate guarantee of reliability.



Type Description Examples /Techniques Considerations

Oral /

Inquiry

Information

gathered from

people through

interviews and

focus groups.

Such

information may

take the form of

written or oral

statements.

Examples:

Oral / written

interview

Single / group

Techniques:

Interviews

Presentations

Questionnaires?

Knowledge/facts?

Oral evidence is generally important in

performance audits, as information

obtained in this manner is up-to-date

and may not be available elsewhere.*

However, information should be

corroborated and statements confirmed

if they are being used as evidence.*

Analytical

Indirect or

derived

evidence /

information

constructed by

the auditor

combining

information

from different

sources and

analysing that

information to

reach a

conclusion.

Examples:

Comparison

Computation

Ratio

Crossing

Techniques

Analysis through

reasoning,

reclassification,

computation and

comparison

Such evidence is obtained by using

professional judgement to evaluate

physical, documentary and oral

evidence. *

Be aware of importance of Audit

experience and skills

Based on page 96 of “Internal Audit Practice”, chapter on “Gathering and analysing information”

* based on page 60 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase

Note: Types are not related to description of ways to record evidence as this aspect may be

covered by internal procedures.

C. Sources

Source Type of Evidence Examples / Techniques Considerations

Obtained

directly

by the

auditors

Observed (and

Physical)

Oral / Inquiry

Analytical

Direct inspection,

On-site verification

Observation

Interviews,

Preparation of

questionnaires

Previous audit reports

from the audit bodies,

Analysis

The auditors can determine the

methods that will provide the best

quality of evidence for the particular

audit. However, their skills in

designing and applying the methods

will determine the quality of the

evidence. *

Provided

by the

auditee

Documentary

Information from

databases, documents,

activity statements and

files (e.g. procedures,

instructions, legal acts,

inspection reports,

Auditors must determine the reliability

of data that is significant to the audit

questions by review and corroboration,

and by testing the auditee's internal

controls over information, including

general and application controls over

Deleted: of evidence

Comment [agesp9]: Or check lists…

Deleted: ¶



Source Type of Evidence Examples / Techniques Considerations

Oral / Inquiry

management reviews,

organisational and

planning documents,

certifications).*

Answers to questionnaires

Oral replies during

interviews

computer-processed data. *

Provided

by third

parties

Documentary

Oral / Inquiry

Information which may

have been verified by

others or whose quality is

well known, e.g. national

statistical data.*

Information belonging to

third parties (Business

Operators, Customs,

Stakeholder

representatives, other

CAs, etc.)

Third parties audit reports

Websites

Answers to questionnaires

Oral replies during

interviews

The degree to which such information

can be used as audit evidence depends

on the extent to which its quality can

be established and its significance in

relation to the audit findings. *


{Maybe we could add a point for the common deficiencies: failing to scrutinize

important point, failing to maintain auditor independence, failing to supervise work}

IV. EVIDENCE COLLECTION PLANNING

Why do we need it?

Main purpose is to allow a targeted evidence gathering to support the audit findings.

This should focus on the audit objective and scope.

Reference to Annex II – Diagram of Audit Process

What is the benefit?

To gather enough evidence and not more than needed.

To anticipate the difficulties of gathering audit evidences

To save time in obtain it from the auditees.

Plan the audit so that enough (sufficient) evidence can be obtained to be able to draw

conclusions that have a bearing on the object of the audit.

(RdH - link evidence sufficiency to the audit objectives)

How do we do it?

Comment [j10]: I suggest it would, if possible, be useful to consider some examples of best practice as regards the actual documentation and cross-referencing of audit evidence.

Comment [H11]: This is a rather challenging part of the document. I would say; it’s not only a matter of adequate planning. But this is about the process in which you jump to conclusions on the basis of audit findings compared to the audit objective. For example: audit objective is to assess the effectiveness of official controls. You do have some audit findings, some are positive, some are negative. But how and on what basis can you conclude about effectiveness?



Which methodologies are used? Is there a “good practice” that we can identify?

Iterative approach? Use of external experts (e.g. in data analysis)?

Knowledge of, and information available to, internal auditors vs. external auditors.

Bias of auditors?

Importance of on-site

Note: important to link with characteristics of evidence (how to ensure we get useful

information)

Factors to consider when judging the quality and quantity of audit evidence:

the purpose for which the evidence will

be used

a higher standard is required for evidence supporting

audit findings than for background information provided

in the audit report

the level of the significance of the audit

finding

in general, the higher the level of significance, the higher

the standard of evidence that is required

the degree of independence of the source

of the evidence

greater reliance can be placed on evidence which

emanates from independent sources

the cost (money or time) of obtaining

additional evidence relative to likely

benefits in terms of supporting findings

and conclusions

at some point, the cost of obtaining more evidence will

outweigh the improved persuasiveness of the total body

of evidence

the risk involved in making incorrect

findings or reaching invalid conclusions

the greater the risk of legal action, controversy or

surprise from reporting an audit finding, the higher the

standard of evidence needed

the care taken in collecting and analysing

the data Including the extent of the auditors' skills in these areas


Reference to Annex III – example of evidence matrix

When does it take place?

Create a timeline with audit steps and where evidence collection planning takes place:

Planning - Audit Objectives + scope - Phase 1 (desk study) – Risk analysis/desk study

results – Phase 2 (test – on-site activities) – Audit report? Diagram?

Planning – preparation – execution – reporting? Diagram?

Evidence collection planning may take place at different stages, depending on the

audit planning approach (Desk-based / on-site). Refined, adapted and developed along

the audit process. On-site evidence collection is particularly important where the audit

is being used to confirm/verify the effective implementation of planned arrangements.

Retention of Audit Evidence.

This would have particular significance in respect of independent scrutiny, evaluation

of or challenge to an audit system and /or its findings.

Should be kept during a period described by the audit body or national rules.



Link to Section III, table “A. Types”, type “Observed (and Physical)”, ways to record

this type of evidence.

V. VERIFICATION OF AUDIT EVIDENCE

A. Verification: (ISO 9000 definition?)

Is the evidence really “evidence” The information collected is not audit evidence until it has

been verified.

(meeting its characteristics – described above in Section III.A)? Root-cause-analysis

– link with the evidence, runs along with the collection and verification of

evidence.4Reference to Annex IV – to be developed.

Who does it?

Auditors and their managers.

When to do it?

Along with evidence gathering. Importance of on-site (do we need to emphasise here ;

need additional text, refer to the document on effectiveness); at specific stages)

How to verify?

Cross-checking / Review of auditor’s work (own review or supervision) / Quality

checks/ Peer review.

[in BTSF CB-D3-P04]

4 Reference to “Root-cause analysis of non-compliance – outcome of the workshop (MANCP WG-meeting 21-

22/11/2012)”

Deleted: (

Deleted: ?)

Deleted:

Deleted: ;

Deleted: );

Deleted: ;

Comment [agesp12]: It is highly recommendable to insist on follow-up activity as a revision of audit along its whole development (e.g.:follow-up person/unit functions are explicitly covered in some audit units manuals). The aim would be to improve follow-up quality and focus its ongoing recommendations. The work of an "external" follow-up unit in the audit process provides a great added value. The idea that evidence of audit must be a certain type, size, quality, etc could be reinforced too. It must be replicable and a third party can reach the same conclusions through the evidence collected by the original team



B. Validation: (ISO 9000 definition?)

Who validates?

Auditors (and their managers?).

Importance of competence and roles/responsibilities when validating evidence.

When to validate?

At specific stages. In revising the audit’s draft report.

How to validate?

Supervision of auditor’s work; Peer review



Annex I - Example of a mind map on Audit Evidence (to be adapted)

Deleted: ¶



Annex II – Diagram of Audit Process

NAS Audit Evidence subgroup - working document 14

Annex III - Example of an evidence matrix5:

1 - From the risks cartography of the competent authority showing a high level of criticality on the

subject of “species substitution”, the audit team planned a mission

2 – The audit team analyses the process of the CA to deal with that subject. The audit team elaborates

a risk matrix of the process to identify the key points and the criteria showing they are under control.

The audit team also reduces the scope of the mission to the horse meat, because it appears to be, one

of the easiest products to substitute with a cheaper one, hard to detect and enables quick and strong

profits.

3 – The process of providing audit evidences audit can be summarized with the following matrix

Audit

Objective

Steps and criteria Audit evidence

Type of evidence Level of evidence

document

ary

observat

ion

testimonia

l

Analytical enough Too

weak

Too

much

Is the CA

efficient in

horse meat

species

substitution

controls?

Plannin

g

All the

country is

covered, all

year long

The

annual

control

plan

The Y-1

synthesis

Interview

of the

meat

board

manager

Data

registered

in the

informatio

n system

related to

the meat

control

plan

Yes

From the

production

to the

distribution

chain

Yes

the orders

are

efficiently

transmitted

to the agents

Message

and

instruction

s given to

the agent

through

the

country

Local

interview

of meat

agents

Yes

Executi

on

The control

plan is

respected

(quantity,

quality, data

recorded)

Local

planning

declining

the

national

instruction

s

Interviews

of

managem

ent and

agents

Local

results vs

local

objectives

in the

informatio

n system

Intervi

ew of

agents

is

useles

s

The agent

knows how

to make a

sampling

Training

records of

agents on

the subject

Agent

evaluation

Interviews

of agents

Need

to add

an on-

site

observ

ation

to

conclu

5 This matrix is linked to a specific kind of audit and can be adapted to other cases.


Audit

Objective

Steps and criteria Audit evidence

Type of evidence Level of evidence

document

ary

observat

ion

testimonia

l

Analytical enough Too

weak

Too

much

s de

The agent

knows the

product, the

law, the

internal

procedures,

Quality of

local

records

Yes

Analyse The labs

used a the

right

equipment

The analysts

have the

competencie

s

The lab is

referenced

Prosecut

ion

The level

correspond

to the level

of the fraud

The rate of

prosecution

is

homogeneo

us on the

territory

The rate of

validation

by the court

is high

We can also add a column to the matrix to write the findings and another one for conclusions


Annex IV – Audit map and root-cause analysis

Alternative diagram

Comment [ST13]: HU we may put into „the planning to conclusion” scheme the cross check box too, because cross check can generate new audit questions, like the root cause analysis.