electronic data interchange (edi): key audit issues
TRANSCRIPT
ELECTRONIC DATA INTERCHANGE (EDI): KEY AUDIT ISSUES
by
TANOM SURAVONGTRAKUL
B.Acc., Chulalongkorn University, 1983 M.B.A. , Michigan State University, 1986
A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF
THE REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE
(BUSINESS ADMINISTRATION)
in
THE FACULTY OF GRADUATE STUDIES
The Faculty of Commerce and Business Administration Department of Management Information Systems
We accept this thesis as conforming to the required standard
THE UNIVERSITY OF BRITISH COLUMBIA
August 1992
® Tanom Suravongtrakul, 1992
In presenting this thesis in partial fulfilment of the requirements for an advanced
degree at the University of British Columbia, I agree that the Library shall make it
freely available for reference and study. I further agree that permission for extensive
copying of this thesis for scholarly purposes may be granted by the head of my
department or by his or her representatives. It is understood that copying or
publication of this thesis for financial gain shall not be allowed without my written
permission.
Department of ÔGYY\f?\eXŒ
The University of British Columbia Vancouver, Canada
DE-6 (2/88)
ABSTRACT
The development of EDI technology has created many concerns and challenges for
the auditing profession. Along with its many suggested benefits, the technology brings an
important potential to change business information systems and the way businesses operate.
As a consequence, it may put auditors in a new audit environment and may thus force
significant modification to the established methods of auditing.
This study identifies important EDI audit issues as viewed by information systems
auditors in the greater Vancouver area. A three round Delphi methodology was used to
solicit opinions from a group of IS audit experts. The expert respondents predominantly
held managerial positions in internal audit functions while their organizations comprised
a wide cross section of sizes and industries.
The research findings reveal a consensus set of eleven most important issues.
Among these EDI audit issues, "Controls Over EDI Network", "Backup, Disaster
Recovery and Contingency Plans", "Auditability and Audit Trail", "Audit Involvement
during the System Development", and "Legal and Audit Evidence" are rated in the top five
ranks.
i i
TABLE OF CONTENTS
Abstract i i
Table of Contents i i i
List of Tables vi
Acknowledgements vii
Chapter 1. INTRODUCTION 1 1.1 Background and Motivation 1 1.2 EDI Concepts 2
1.2.1 The Definition of EDI 2 1.2.2 The Benefits of EDI 3 1.2.3 The Growth of EDI 4
1.3 The Implications of EDI on Auditing 5 1.4 Statement of Problem and Need for the Research 6 1.5 Research Objectives 8 1.6 Application of Proposed Research 8 1.7 Overview of the thesis 9
Chapter 2. LITERATURE REVIEW 10 2.1 Previous Empirical Research 10
2.1.1 West [1988] 10 2.1.2 Holstrum et al. [1988] 12
2.2 Selected Professional Publications 14 2.2.1 Staats [1981] 14 2.2.2 Hinge [1988] 15 2.2.3 Hansen and Hill [1989] 17 2.2.4 Sadhwani et al. [1989] 20
Chapter 3. RESEARCH FRAMEWORK 22 3.1 Introduction 22 3.2 EDI'S Key Audit Issues 22
3.2.1 Audit Evidence 23 3.2.2 Audit Trail 24 3.2.3 Audit Involvement during the System Development 25 3.2.4 Timing of Audit Tests 26 3.2.5 Audit Reporting (Periodic versus On-Line) 26 3.2.6 Audit Focus (Substantive versus Compliance Testing) 27 3.2.7 Pre-determination of Audit Scope (Boundary of Audit) 29 3.2.8 Audit Tools 30
iii
3.2.9 Audit Techniques 31 3.2.10 Audit Risk Assessment 32 3.2.11 The Changing Role of Auditors 33 3.2.12 Audit Responsibility in Evaluating Controls 34 3.2.13 Relationship Among Company's Auditors 36 3.2.14 Collaboration Among Auditors of EDI Parties 38 3.2.15 Auditor Skills (Skills required of auditors) 39 3.2.16 Auditor education and training 40
Chapter 4. RESEARCH DESIGN 42 4.1 Research Questions 42 4.2 Selection of Research Methodology 42 4.3 The Delphi Process - An Overview 44 4.4 Instrument Development 46
4.4.1 Round 1 Questionnaire 46 4.4.2 Round 2 Questionnaire 47 4.4.3 Round 3 Questionnaire 49
4.5 Participant Recruitment 50 4.6 Data Collection Procedures 52
4.6.1 Round 1 53 4.6.2 Round 2 53 4.6.3 Round 3 54
Chapter 5. ANALYSIS AND DISCUSSION OF RESULTS 56 5.1 Introduction 56 5.2 Round 1 Results 56
5.2.1 Controls Over EDI Networks 57 5.2.2 EDI Contracts (Trading Partner Agreements) 58 5.2.3 Backup, Recovery and Contingency Plans 58 5.2.4 Third Party EDI Services 59 5.2.5 EDI Records Retention 59
5.3 Round 2 Results 60 5.3.1 The Rating of Round 2 Issues 60 5.3.2 Additional Issues Identified in Round 2 62
5.4 The Comparison of Round 1 and Round 2 Results 62 5.5 Round 3 Results 63
5.5.1 Rating of the Original 21 Issues 66 5.5.2 Rating of the Final 25 Issues 66
5.6 The Interpretation of the Results 67 5.7 The Comparison of Round 2 and Round 3 Results 68 5.8 Movement Towards Consensus 70 5.9 Study Participants 73
5.9.1 Organizational Category 73 5.9.2 Position and Primary Area of Responsibility 74
iv
5.9.3 Professional Designations 74 5.9.4 Areas of Audit Expertise 75 5.9.5 Level of Audit Experience 75 5.9.6 Background in EDI Technology 75
5.9.6.1 Engagement in an EDI Project 75 5.9.6.2 Self-report Level of knowledge and Understanding
of the EDI Technology 76 5.9.6.3 Primary Source(s) of knowledge and
Understanding of the E D I Technology 76
5.8.6.4 EDI audit manual or guideline 77
Chapter 6. CONCLUSIONS 78 6.1 Summary of Findings and Conclusions 78 6.2 Generalizeability of Results 79 6.3 Limitations of Research Study 80 6.4 Directions for Future Research 81
BIBLIOGRAPHY 82
APPENDIX A - ROUND 1 QUESTIONNAIRE 87
APPENDIX B - ROUND 2 QUESTIONNAIRE 91
APPENDIX C - ROUND 3 QUESTIONNAIRE 98
APPENDIX D - ROUND 1 RESULTS . 108
APPENDIX E - ROUND 2 RESULTS 115
APPENDIX F - ROUND 3 RESULTS (21 Issues) 119
APPENDIX G - ROUND 3 RESULTS (25 Issues) 123
v
LIST OF TABLES
TABLE 1 - ISSUES IDENTIFIED IN ROUND 1 57
TABLE 2 - RATING OF ROUND 2 ISSUES 61
TABLE 3 - ISSUES IDENTIFIED IN ROUND 2 62
TABLE 4 - ROUND 3 RATING OF 21 ORIGINAL ISSUES 64
TABLE 5 - ROUND 3 RATING OF 25 FINAL ISSUES 65
TABLE 6 - THE TOP ELEVEN ISSUES IN ROUND 2 AND ROUND 3 . . . . 68
TABLE 7 - RESEARCH SUBJECTS: PARTICIPATION PATTERN 73
TABLE 8 - RESEARCH SUBJECTS: ORGANIZATIONAL CATEGORY . . . . 73
TABLE 9 - RESEARCH SUBJECTS: TYPE OF POSITION 74
TABLE 10 - RESEARCH SUBJECTS: PROFESSIONAL DESIGNATION . . . 74
TABLE 11 - RESEARCH SUBJECTS: AREAS OF AUDIT EXPERTISE . . . . 75
TABLE 12 - RESEARCH SUBJECTS: LEVEL OF EXPERIENCE 75
TABLE 13 - RESEARCH SUBJECTS: ENGAGEMENT IN A N EDI PROJECT 75
TABLE 14 - RESEARCH SUBJECTS: LEVEL OF KNOWLEDGE OF EDI TECHNOLOGY 76
TABLE 15 - RESEARCH SUBJECTS: PRIMARY SOURCE(S) OF KNOWLEDGE OF EDI TECHNOLOGY 76
TABLE 16 - THE TOP ELEVEN ISSUES IN VANCOUVER 78
vi
ACKNOWLEDGEMENTS
This thesis is dedicated to my parents, Lieang Tang and Kimchang Lee, who
always have confidence in me and support me in my endeavour for personal growth and
professional development.
Sincere thanks go to my thesis advisor, Professor Albert S. Dexter, and the
members of my thesis committee, Professor Dan A. Simunic and Andrew W. Trice, for
their advice and encouragement. I would also like to thank Ms. Khim Seow at Commerce
General Office for her assistance with the final copy of the thesis.
Special appreciation is extended to Mr. Alan R. Drinkwater, Membership Director
of the EDPAA-Vancouver, Mr. James W. Topham, President of the EDPAA-Vancouver,
and Ms. Angela M . Louie, President of the IIA-Vancouver, for their kind assistance which
contributes greatly to the achievement of this research study.
Finally, I gratefully acknowledge the Thai-Canada Rattanakosin Scholarship which
have provided financial support throughout my study in Canada.
vii
Chapter 1. Introduction
1.1 Background and Motivation
The recent development in information technology in the form of Electronic Data
Interchange (EDI) has created many concerns and challenges for the auditing profession.
Along with its many suggested benefits, this technology brings the important potential to
change business information systems and the way businesses operate and, consequently,
to create a new audit environment and to force significant changes in the established
methods of auditing.
While the global business community has increasingly paid attention to EDI, and
corporations in North America and Europe have been adopting this technology at a
noticeable rate, the auditing profession has not been as prompt in its approach to EDI audit
concerns. At present there are no auditing standards or specific guidelines regarding
EDI/EFT [Sadhwani et al. 1989; Cowan, 1990]. Studies indicate that EDI systems will
prevail over paper-based systems [Hinge 1988; Schatz 1988; West 1988; Holstrum 1988;
Tsay 1989], and that indication, plus the potential direct impact of EDI on auditing,
suggest that EDI is a technology that deserves immediate attention from auditors.
To date, academic research has provided little insight into this domain. Although
EDI is expected to have a profound impact on many aspects of auditing, the nature and
extent of such an impact are not specifically known. It is the intent of this project to
conduct an exploratory research into EDI to identify and assess key impact issues of audit
concerns. The knowledge of audit concerns contributes to the overall success of EDI
adoption and technological improvement in the business world.
1
1.2 EDI Concepts
1.2.1 The Definition of EDI
Although there are variations in the definition of EDI, Hinge's [1988] definition
is adopted for the purposes of this project:
Electronic data interchange (EDI) is the intercompany, computer-to-computer exchange of business documents in standard formats. Through EDI, such common business forms as invoices, bills of lading, and purchase orders are transformed to a standard data format and electronically transferred between trading partners. [Hinge, p. 9]
This definition is chosen because it captures the essence of an EDI system and it
has the relevant meanings in the context of auditing and of this project. The above
definition includes the EDI essential terms "intercompany", "computer-to-computer", and
"standard formats" which also meet the criteria for an adequate definition of an EDI
system suggested by Powell [1991]1. The first two terms, "Intercompany" and "computer-
to-computer", imply that there must be at least two different computer systems involved
in the electronic transfers of business data. Furthermore, The term "intercompany" is
appropriate for the purpose of this project because although EDI systems can be
implemented by non-business organizations for purposes other than trading, and EDI
transmissions can occur between different computers of the same company (e.g., between
the administrative office and manufacturing plants), only the EDI systems for trading
activities among different business enterprises are of interest in this audit issues project2.
1 Based upon his extensive reviews of the literature, Powell [1991] suggests that although EDI can be defined in many different ways, an adequate definition of EDI must indicate a transmission of data/information between at least two different computers using a standard format, [p.4]
2 Most of the reviews in the auditing literature express concerns on the audit of open EDI network systems among business enterprises rather than the closed or non-business oriented EDI network systems.
2
In addition, the term "computer-to-computer" signifies the automation of business functions
and the reduction of paperwork, which have important meanings for auditing activities,
because it means that information can flow directly from the sender's application to the
receiver's application without paper and without human intervention. Moreover, the term
"standard formats" helps differentiate EDI from electronic mail and facsimile transmission.
Because EDI standard formats3 are in coded, machine-readable forms, EDI messages can
be created and interpreted by computers. Electronic mail and facsimile transmission, on
the other hand, do not have such standard formats and their messages are in free text form
which must be created and interpreted by humans.
1.2.2 The Benefits of EDI
EDI offers many attractive benefits. Hinge [1988], Gardner [1989], and Wright
[1990b] discuss the direct benefits of EDI in terms of speed, accuracy, and savings.
Hansen and Hill [1989], basing their findings on a survey by EDI Research, Inc., cite
speed, accessibility of information and improved customer services as the most frequently
mentioned benefits of EDI. In his doctoral dissertation, Kavan [1991], using Porter's
definition of competitive advantage, states that EDI contributes to both cost effectiveness
and product differentiation strategies. He also mentions that enterprises are adopting EDI
to increase productivity, reduce financial exposure, and gain a competitive advantage in
the market place. Others view EDI as an increasingly essential technology for business
survival. For instance, Schatz [1988], Emmelhainz [1990], and Powell [1991] point out
3 Examples of EDI standards currently available are—ANSI ASC X12 for cross industry, AIAG for automotive, TDCC for transportation, USC for retail, and EDIFACT for international. For detailed discussions of EDI standards see Emmelhainz [1990, pp.63-87], Kimberley [1991, pp.97-124].
3
that the reason businesses adopt EDI technology is not solely for competitive advantage
but for survival. They note that companies are being forced by both their suppliers and
customers to implement the EDI systems, and Schatz cites as an example General Motors'
1984 letter that gave its suppliers until 1987 to get on-line with EDI or go off-line with
GM. Furthermore, Tsay [1989] reinforces the point by predicting that those who resist
EDI technology could eventually run the risk of losing their business to competitors.
1.2.3 The Growth of EDI
Since the emergence of the EDI concept in the late 1960s, the adoption of EDI
systems has continued to grow. During the 1970s significant progress was made towards
the development of EDI standards, and by the mid-1980s, there was a noticeable expansion
of EDI use. With relatively inexpensive supporting software and hardware, EDI links
between customers and suppliers became more feasible for many industries. Current
information suggests a bright future for EDI growth. For example, based upon her
research findings, Emmelhainz [1986] indicates that the use of EDI is likely to become the
norm in the purchasing community in the relatively near future and that third party
network services will play an important role in the continued growth of EDI. Hinge
[1988], an EDI expert, claims that EDI has become a prerequisite for doing business, and
she predicts that by 1993 an estimated 70 percent of U.S. businesses will make significant
use of EDI. She also notes that the international EDI market is growing, particularly in
Canada and Great Britain. In his recent review, Damyanoff [1991] affirms the trend
toward the international use of EDI and reports that U.S. Customs has made the
combination of EDI link (among parties involved in importing-exporting transactions) and
4
EDIFACT 4 the basis of its proposed Customs Modernization Act of 1990. Because the US
has a number of trading partners all over the world, it is reasonable to expect a pervasive
use of EDI both domestically, not only in the US but also in its trading partners' countries,
and internationally in the near future.
1.3 The Implications of EDI on Auditing
In spite of their benefits, the unique characteristics of EDI introduce additional
complexities into business transaction processing and the audit environment. Academic
and professional studies consider the ramifications of EDI adoption from many
perspectives. For example, Hansen and Hill [1989] conclude that EDI necessitates new
control and audit considerations and that there are methods and procedures to respond to
the changes. They discuss the impact of EDI on internal controls in terms of the absence
of source documents, bridging applications, and direct interaction with trading partners.
As a computer audit specialist, Wright [1990b] agrees that the greatest direct effect of EDI
will be on corporate accountants and internal and external auditors. She suggests three
main areas to be considered: controls, contracts, and paper elimination. Cowan [1990]
considers information flows and boundaries to data ownership as important audit and legal
issues. He concludes:
Although the use of EFT/EDI does not alter the essence of audit objectives, it creates new issues and it has changed the information flow that the auditor needs to understand... The problem with EFT/EDI is the sheer pace of transactions and their integration with the accounting functions of an
4 EDI for Administration, Commerce and Transport is the acronym for standards developed within WP4 (Working Party 4 on the facilitation of International Trade Procedures of the Economic Commission for Europe, a commission of the United Nations) [Hinge 1988,p. 76,86].
5
organisation. [Cowan, p. 30]
As a lawyer who specializes in electronic trading, Wright [1991, p.38] confirms
that companies' adoption of EDI systems will have a profound effect on auditors and states
the following points as the major audit concerns:
- auditors' responsibility for a system that lacks adequate controls; - auditors' obligation to electronic legal issues; - auditors' duty to provide advice regarding the establishment of necessary controls.
The extensive movement toward "paperless" electronic data processing will
eliminate much of the traditional audit trail and will radically change the nature of audit
control, audit evidence, audit techniques, and the timing of audit tests. As a result, many
aspects of the conventional audit process will have to change to suit the new environment.
1.4 Statement of Problem and Need for the Research
As discussed in the preceding sections, EDI technology will alter business conduct
and, as a consequence, will force the practice of auditing to change. The major concern
is to indicate to auditors the importance of this technology. The impact of EDI technology
has been sudden but its effect is extensive and widespread. As implied in Wise's article
[1989], the challenges of the electronic system apply not only to EDP 5 or information
systems auditors but also to all types of auditors. He elaborates that point by noting that
when there was always a paper trail of documents, non-EDP auditors could avoid "auditing
through" the computer by "auditing around" the computer. However, as businesses
5 Electronic Data Processing
6
progress toward a "paperless office" there will be fewer documents to enable "auditing
around" the computer. Thus, inevitably, all types of auditors are under pressure to adapt
themselves to new circumstances. A similar concern applies at the academic and research
levels. As educators of future auditors, both instructors and researchers will need to
conceive and respond suitably to changes brought about by this new trend in technology.
There are many indicators of a need to respond promptly to the EDI challenges.
For instance, as stated by Cowan [1990] and Wright [1991]:
There are no auditing standards or specific guidelines (regarding EDI/EFT) at present. What the auditors have achieved to date is the adaptation of existing professional standards to track new developments....[TJhe professional and regulatory bodies will need to adapt more quickly to changes in technology, though this should not be at the expense of rigorous assessment of the precise impact of those changes. [Cowan, p. 31]
Electronic transaction technologies present the accounting profession with a daunting task. The audit and control of electronic systems require new methods drawn from the principles of past practices. Accountants must fast educate themselves in these new ways. They will otherwise be swept under the avalanche of electronic data that industry is generating. [Wright, p. 39]
Furthermore, Kavan [1991] shows the urgency of the issue in that while auditors
have not yet established firm guidelines, the situation has been made more serious because
business enterprises have implemented the technology without awareness of the legal and
accounting problems:
Many organizations, eager to implement EDI, overlook critical controls and safeguards. Because this technology is so new, user documentation, standards, conventions, guidelines, and the law are either not developed or are inadequately implemented. [Kavan, p. 14]
The auditing profession needs to have available extensive new research that will
7
provide an insight into the EDI audit issues. At present, very little academic research has
been accomplished in the area of EDI [Kavan 1991]. My own review indicates that there
has been even less research in the specific area of EDI and auditing. Although a
formalized study, which is closely related to this project, was done by West in 1988, and
a more general study, a part of which is applicable to this thesis, was done by Holstrum,
Mock, and West in 1988, prior to the initiation of this project, there has been no published
study of this type.
1.5 Research Objectives
In response to the stated problems and needs for research, this study is designed
to achieve the following objectives:
1.5.1 To identify and gain consensus on key issues of concern to auditors
when auditing in the EDI environment.
1.5.2 To rank the priorities of these issues.
1.6 Application of Proposed Research
Knowledge of important EDI audit issues will be useful in helping accountants,
managers, information system consultants, and vendors better to understand and build EDI
systems that satisfy audit needs and concerns. Systems that are satisfactory from the audit
perspective have well built-in controls. Such secure systems would contribute to the overall
success of EDI adoption in the business world. In addition, by knowing the significance
and the priorities of the EDI audit issues, educators and researchers can direct their efforts
8
toward the most critical areas and, as a consequence, can better satisfy professional needs.
1.7 Overview of the thesis
This thesis proceeds as follows. Chapter two reviews the previous empirical
studies and the selected professional publications which are related to the control and audit
of the "paperless" EDI systems. Chapter three outlines the critical issue frameworks used
in the research. Chapter four formulates the specific research questions and
methodological details of the study. Chapter five analyzes and discusses the research's
findings and the final chapter presents the conclusion.
9
Chapter 2. Literature Review
2.1 Previous Empirical Research
2.1.1 West [1988]
In completing his doctoral dissertation, West conducted a study comprised of two
distinct phases:
1) A general investigation, using the Delphi technique, of potential technological
changes that may have an impact on the audit environment in the year 2000.
2) A detailed investigation, through a case study, of the impact of one specific
technological change upon auditing.
The first phase of the study included a panel of 31 highly experienced accounting
professionals. He found that the trend toward a "paperless" accounting system was the
area of primary concern to the majority (20) of the respondents. As a result, the
"paperless" or EDI system was chosen to be studied in more detail in the second phase.
In the second phase of the study, eight internal auditors (seven were computer
auditors), and six external audit managers (all general auditors) evaluated a case study
detailing a "paperless" purchasing, accounts payable, and inventory control system. They
were asked to evaluate the exposures, key controls, and reliability of the system. The
results of this in-depth analysis showed that:
1) An adequate audit trail could be obtained from the paperless system.
2) There is a high level of consensus on which exposures and controls were considered
most critical. The three most important controls were:
10
- Controls (separation of responsibilities) over computer program changes
- Dual access controls
- Computerized matching of invoice, purchase order, and receiving report prior to
recording the liability.
The auditors also suggested the following as important additional controls:
- managerial reviews
- programmed controls (e.g., range and limit checks)
- frequent (e.g., monthly) testing of perpetual records.
3) There was significant concern about fraud and unauthorized transactions (especially
fraudulent payment of accounts payable and unauthorized access to data, programs, and
inventory).
4) There was a lack of consensus concerning the adequacy of internal controls.
5) There was an increased emphasis on audit tests of the system, less emphasis on detailed
tests of balance, and little use of analytical procedures.
West's research is important because it marked the first attempt to identify EDI
audit concerns. Although the research is not built upon any existing theories, the resulting
predictions of technological trends/events seem to have a high degree of accuracy. Present
EDI literature contains evidence that suggests the trend toward a "paperless" system has
become a reality. Electronic trading systems are now being used in many major industries
in North America, Europe, Asia, and Australia [Emmelhainz 1990; Wright 1990; Baker
1991(b); Damyanoff 1991]. Further, although the case study is simplified in comparison
to a real-world company, and the tasks assigned to the subjects are restricted to the
11
evaluation of internal accounting and computer controls, the resulting analysis provides a
useful insight into the implications of electronic trading systems on auditing. In addition,
the selection of the "paperless" revenue cycle for in-depth analysis is appropriate. Today,
purchasing is one of the prime areas for EDI applications in most pioneering organizations.
However, the prediction about the shift of audit focus toward system testing remains
controversial. While some current professional literature confirms this shift, others
indicate a move towards more substantive testing. This issue will be addressed in more
detail in this study6.
2.1.2 Holstrum et al. [1988]
This study is built upon the findings of West's Delphi survey. It provides a
detailed analysis of the impact of technological change on audit evidence and control
structures and also examines the impact of social, legal, and economic changes on
auditing. Among other issues, the study predicts that:
By the year 2000, most computers will be able to communicate with one another...the volume of paper documents will be reduced, but the volume of available information will increase significantly, [p.xxi]
This projection is supported by the more recent professional reviews which indicate the
increased use of automated EDI networks in current businesses [Emmelhainz 1990; Wright
1990; Baker 1991(b); Damyanoff 1991].
In addition, audit software embedded in the audited entity's operating system and
interconnections with mainframes and large databases are predicted as likely to become
vital audit tools, and audit of the systems development process is viewed as increasingly
6 Please see Chapter 3 under the issue "Audit Focus".
12
important. Further, high-level systems review and evaluation software, database access
and modification controls, computer monitoring, and examination of controls over
paperless intercompany information network such as Electronic Data Interchanges (EDI)
are mentioned as the focus of future control testing, whereas the imbedded audit software
for continuous on-line monitoring (auditing) of the system is included as a major
substantive testing.
The study also predicts that blurred boundaries of the audited entity, continuous on
line auditing, and expanded responsibility for evaluating the integrity of internal and
external databases will influence the changing role of auditing. Moreover, the authors cite
"the key overriding skill of being able to readily adapt to rapidly changing information
technology, including computer adeptness and interfacing effectively with expert systems"
[Holstrum et al., p. 179] as the skills required of future auditors. Supplementarily, they
suggest that auditors receive extended education and training and that the auditing
curriculum be modified to emphasize computer familiarity, computer modelling, and the
behavioral impact of information technology.
Finally, the authors note that, although the expert panellists in this study believed
that paperless transactions (computer-to-computer input) are technically feasible, and that
by the year 2000, more than half of the most common types of business transactions
(e.g.payment, invoicing, ordering, payroll time-cards) in large companies would be
completed without paper, some experts felt that the "paperless trading" may have
difficulty in gaining public acceptance, and that auditors would have problems with the
significant disappearance of the paper audit trail.
13
In general, rapid advances in computer technology integrated with
telecommunication technology show these predictions are being actualized. Today, the
technical ability to conduct business without paper is already available. Further,
substantial improvements in hardware and software technology can facilitate the
development of sophisticated systems and tools. In addition, the current environmental
movement to reduce the amount of paper used together with other motives such as speed,
accuracy, and saving may help improve social acceptance of the "paperless" concept and
lead to the common use of EDI systems.
2.2 Selected Professional Publications
2.2.1 Staats [1981]
Staats, retired Comptroller of the United States, cites the following duties as the
critical challenges confronting the auditing profession in the year 2000:
- Auditing paperless transactions,
- Auditing to prevent and detect fraud,
- Reporting on the adequacy of internal controls.
In addition, he predicts that,
"Paper transactions will be virtually eliminated, and auditors will have to review transactions as they occur. Moreover they will concentrate more on tests of systems than on testing individual transactions.'' [Staats 1981, p . l l ] .
His predictions of "paperless" transactions and the review of transactions as they
occur are well supported in the studies by West [1988], Holstrum et al. [1988], and
Hansen and Hill [1989]. The prediction regarding the shift towards system testing, which
14
will be examined in this study, remains debatable. Although the shift is confirmed by the
results from West's case study of the "paperless" purchasing system, it is challenged by
such reviews as Jancura et al. [1986], ICAEW [1989], and Brown [1991].7
2.2.2 [Hinge 1988]
This report states that a company's use of EDI will have a profound effect on
auditing activities. Besides giving the formal definition of EDI which will be adopted in
this thesis project, Hinge suggests the following auditing issues to be considered when
designing the information system (p.43-45):
1) Payment Validation
Audit Concern : reconciliation of invoice, purchase order,
and receiving documents to assure the correct
payment amount.
Effect of EDI : all these documents are computerized, and the validation process
is changed.
Strategy : automate the validation process to get time saving benefits.
2) Audit TraU of Activity
Audit Concern : tracking data flow within the company; recording authorizations.
Effect of EDI : information security procedures are altered; paper documents and
paper backup files are missing; EDI data flow can now be
documented internally, between company and EDI V A N , and
between company and trading partner.
7 See Chapter 3, under the "Focus of Auditing" issue.
15
Strategy : replace signatures with codes and IDs; electronic signatures are also an
option.
: date/time stamp all activities and all attempts to access the information
system;
: maintain a specific audit trail database;
: require identification of the terminal/pc to track the point of access.
3) Order/payment control
Audit Concern : ensuring only authorized sources can place orders and initiate
payments.
Effect of EDI : no authorization "sign-offs"; less human intervention means less
control.
Strategy : create safeguards parallel to those of paper systems;
: require password access to the system;
: incorporate "reasonableness checks" into the system;
: emphasize user training to reduce system errors.
4) Accounting/transaction correspondence
Audit Concern : insuring that internal company data reflects actual inventory and
dollar figures.
Effect of EDI : all files are computerized; no paper backup to verify records.
Strategy : spot check actual transactions versus system files;
: verify assets with different (that is, non-EDI) data.
Hinge advises that EDI data within a company be used to generate accounting
16
reports to facilitate the audit process, and a "control reporting" service offered by EDI
VANs be used in tracking data flow between companies.
2.2.3 Hansen and Hill [1989]
Hansen and Hill believe that EDI does change the control and audit environment,
but methods and procedures exist that are supportive to those changes. The authors
address EDI's impact on internal controls in terms of:
- the absence of source documents (authorization signature)
- bridging applications (automatic initiation of transactions)
- direct interaction with trading partners (direct initiation of transactions by outsiders implies that system access control is very critical).
In addition, the authors cite that EDI has a dramatic impact on control evidence
because it is in machine-readable format (electronic documents) and it is distributed at
locations that transcend traditional corporate boundaries. Also, the authors note that
controls must be exercised beyond the traditional system periphery, and this changes the
auditor's evaluation of general controls. Where third-party VANs are used, auditors are
urged to evaluate network application features either directly or through the VAN's
auditor.
The following issues are addressed as important EDI control concerns. The
corresponding control strategies were also suggested:
1) Validation of payments
Concern : both (source) document and signature may be missing
(cannot be matched for verification).
control strategy : programmed routines that match control documents before
17
allowing the next transaction process to begin; codes and IDs to
replace signatures
2) Audit trail
Concern : a trail of documents that allows tracking of the transaction activities is not
necessary to process transactions in an EDI system.
control strategy : depends on the method of EDI data entry
2.1) source documents are batched, then entered via direct entry terminals - a batch
number serves as a batch reference.
2.2) source documents are entered as received - a programmed routine assigns
electronic documents to batches that are automatically numbered while computer-
created source documents are batched and filed by entry station.
2.3) transactions are entered directly without preparation of source documents -
surrogate documents (computer-generated substitutes for source documents) indicate
the person preparing or authorizing the transactions.
3) Order and Payment Control
Concern : signatory authority is removed; opportunities for unauthorized access may
increase
control strategies : a file to hold transactions that require managerial approval; levels
of password control to restrict access to applications and data files;
encryption may be used to prevent data or password pirating;
computerized checks to emulate human judgement in detecting
fraudulent activity.
18
4) System Boundaries and Flow of Transactions
Concern : VANs have shown some reluctance to allow auditors other than their own
to access their facilities.
Strategy : the client's auditor may benefit from a control evaluation performed by
professional computer-audit specialists.
In closing, Hansen and Hill propose the concept of "Continuous Auditing" for
audit considerations. They address the followings points as the key characteristics of
continuous process auditing:
1) On-line monitoring of the major modules of EDI processing
-a supervisory program (programmed control)
-Integrated Test Facility (ITF)
2) Systems metrics for key processing functions
- software monitors to collect performance measurement data
3) System alarms to call attention to system problems
-embedded audit modules to monitor all transaction activity and to notify
the auditor of any activities having special audit significance (typically the
modules write such information on a file called "the audit log".
4) Functional acknowledgements to capture data flows and errors within moments of
their occurrence
-some firms consider EDI orders as authentic if there is a record of
subsequent payments
-If production lead times or payment terms render confirmation of
19
subsequent payment difficult, the firm may consider confirming the
existence of such EDI transmitted orders through "independent
confirmations".
2.2.4 Sadhwani et al. [1989]
Sadhwani, Kim, and Helmerci claim that although traditional controls no longer
apply to EDI integrated systems (in which a computer-based network enables transactions
to be initiated, recorded, approved, and executed electronically), it is viable to maintain
adequate control and auditability. This paper provides an outline of some of the methods
and procedures that managers and system designers must comprehend and implement in
such a system.
The authors emphasize that EDI internal controls must be designed to:
-promote auditability of data
-provide assurance that information is completely and correctly posted
-ensure that transactions are authorized and posted on a timely basis.
The article suggest that the evaluation of internal controls in a typical EDI network
should involve the following three parties, and all parties must provide assurance that the
proper controls exist within their individual systems:
-the originator of the transactions and documents
-the processor (e.g. a third-party network or a bank)
-the receiver of the data and documents.
The authors urged the auditor to get involved early:
"... the auditor must play a significant role during the design and development of EDI systems and must assure management that secure,
20
auditable, and properly controlled systems are developed and that adequately designed programmed procedures are effectively implemented. " [Sadhwani et al. 1989, p. 24].
The authors comment that although SAS No. 48, "The Effect of Computer
Processing on Examination of Financial Statement", provides a broad framework for the
internal control issue, "Current auditing standards do not provide specific guidelines that
pertain to EDI systems. " [p.27]. They recommend that auditors consider the following
issues when evaluating internal controls of an EDI system [p.27]:
-control boundaries
-processor's overall general control environment
-data transmission controls
-data access controls
-audit objectives (how they could be redefined)
-restructuring of internal controls to reduce control gaps
-new risk exposures when using third-party networks.
21
Chapter 3. Research Framework
3.1 Introduction
The purpose of this chapter is to provide a general EDI issue framework upon
which research questions explored in this study are based. Because there is no established
theory at present, an extensive literature review will identify potential issues of important
audit concern regarding businesses' adoption of EDI. The literature consulted consists of
previous empirical works, EDP audit professional publications, seminars and conferences,
information technology (particularly EDI) sources, and relevant works from auditing
publications.
The second phase of West's study [1988] provides a useful insight into and a partial
framework for EDI audit issues. However, because the analysis is confined to the
reliability of internal controls and the auditability of the paperless purchasing systems, the
issues raised are neither exhaustive nor conclusive. Current professional reviews suggest
many other important issues and unanswered questions which need to be considered. This
study analyzes and verifies additional issues to those discussed by West.
3.2 EDI'S KEY AUDIT ISSUES
The unique attributes of EDI that make it advantageous—e.g. reduction of
paperwork and human handling, and direct connection with suppliers and customers-create
a number of concerns among auditors. For the purpose of this study, these concerns will
be classified and discussed in terms of their consequences on the following aspects of
22
auditing:
• Audit Evidence
• Audit Trail
• Audit Involvement during the System Development
• Timing of Audit Tests
• Audit Reporting (Periodic versus On-Line)
• Audit Focus (Substantive versus Compliance Testing)
• Pre-determination of Audit Scope (Boundary of Audit)
• Audit Tools
• Audit Techniques
• Audit Risk Assessment
• The Changing Role of Auditing
• Audit Responsibility in Evaluating Controls
• Relationship Among Company's Auditors
• Collaboration Among Auditors of EDI Parties
• Auditor Skills (Skills required of auditors)
• Auditor education and training
3.2.1 Audit Evidence
Rationale: The absence of paper documents and signatures in EDI systems implies the absence of important audit evidence such as proof of authorization and legal documentation in paper form. Auditors must assure that equivalent and acceptable forms of audit evidence are established and properly incorporated into EDI systems.
23
In the highly automated EDI environment (application-to-application processing),
it is not always necessary to create paper source documents or they may need to be
available only for a short time. In the absence of paper documents, signatures and other
information which usually appear on the documents as "evidence of authorization" are
also missing. The lack of both paper documents and signatures causes important concerns
regarding payment validation and order/payment control [Hinge 1988; West 1988; Hansen
and Hill 1989; Baker 1991]. This concern may be serious if most of the traditional
authorizing procedures and controls are removed when the electronic systems are
implemented. From a control and audit viewpoint, equivalent forms of audit evidence
must be developed to substitute paper documents and hand-written signatures. Some
experts recommend using electronic signature and electronic authorization (EA) processes,
but the question of the acceptability of such alternatives remains unsettled [Lewis 1989].
Whatever substitute forms are used, they should be identified, agreed upon by the auditor,
and incorporated into the system from the early phases of the system development.
3.2.2 Audit Trail
Rationale: Because substantial reduction of paperwork means a possible loss of the audit trail and the consequent inability to conduct an audit, auditors must take actions to ensure the auditability of the EDI system and the availability of audit trail in proper forms.
In the EDI environment, business transactions are processed in an invisible,
electronic form that is heavily coded and almost impossible to monitor, and a trail of paper
documents that allows tracking of the transaction activities also no longer exists. Auditors
have habitually relied on the paper audit trail to test the reliability of a system. Without
24
paper documents, auditors may have difficulty, or in some cases they may find it
impossible to conduct an audit. The potential loss of audit trail and the inability to audit
in a paperless environment are the main concerns expressed in many reviews [Hinge 1988;
West 1988; Hansen and Hill 1989; Baker 1991]. Actions must be taken to ensure the
auditability of the system and the availability of an audit trail in proper forms.
3.2.3 Audit Involvement during the System Development
Rationale: Auditors must get involved early in EDI projects to ensure that proper controls and auditability features are designed and incorporated into the systems. For this reason, guidance is needed for auditors to accomplish competently this important task.
In order to handle audits of a complex information system, auditors are strongly
urged to take a proactive approach and become involved early in the project, especially
during the system design and development stage [Bieber 1987; Rhodes 1987; Holstrum et
al. 1988; Kothari 1988; Craig-Bourdin 1989; ICAW 1989; Sadhwani et al. 1989; Wise
1990]. Although the concern of "audit independence" keeps the roles and the extent of
audit participation indeterminate, there is agreement that auditors perform a function that
would be useful at the system design stage, and that audit involvement in the development,
testing, and installation of computer-application systems can substantially add value to the
process.
That claim applies to such a complicated system as EDI. From an audit
perspective, audit resources can be used productively at this stage because it is the best
opportunity for the auditor to assure that proper controls and auditability features (e.g. the
creation of audit trails) are designed and incorporated into the system. Further, the auditor
25
can acquire background knowledge and solid understanding of the system which will prove
valuable in the subsequent audits. Therefore, sound guidance on audit participation in EDI
projects is needed to assist the auditor to accomplish capably this important task.
3.2.4 Timing of Audit Tests
Rationale: Because of the high volume of transactions and the velocity of electronic processing, EDI transactions may have to be reviewed as they occur. Consequently, the audit process must be modified and specific audit standards established.
In contrast with a conventional paper-based system where audit testing is performed
periodically, the high volume of transactions and the velocity of electronic data processing
may force auditors to review EDI transactions as they occur. In such a case, the concepts
of "Concurrent Accounting" and "Continuous Process Auditing", widely discussed in both
academic and professional reviews8, become relevant [Staats 1981; West 1988; Holstrum
et al. 1988; Kothari 1988; Hansen and Hill 1989; Baker(a) 1991]. Upon the availability
of such a supporting tool as audit software embedded in the audited entity's operating
system, the concept can be feasible9. In such a circumstance, the audit process must be
modified significantly and specific standards must be established to guide the practice.
3.2.5 Audit Reporting (Periodic versus On-Line)
Rationale: Because the high speed of EDI transactions makes information obsolete within a very short time, there is a need for more frequent
8 See chapter 2, Hansen and Hill [1989] for the discussion of key characteristics of continuous process auditing.
9 Holstrum et al. [1988] predicted that such an embedded audit software would be available for audit use by the year 2000.
26
accounting disclosure. Auditors will have to adjust their reporting procedures to meet the need for timely and accurate information.
In concord with continuous process auditing, the reviews indicate a propensity to
move away from formal periodic financial reporting towards more frequent accounting
disclosure10 [Holstrum 1988; ICAW 1989; Yang 1990]. The claim is that the high
volume and speed of EDI transaction processing make information obsolete within a very
short time. Hence, audit reporting procedures need to be adjusted to reflect that problem
and to accommodate the need for timely and accurate information. According to Yang
[1990], the Securities and Exchange Commission (SEC) had marked the movement toward
this direction by introducing EDGAR in May 198411 . He noted that since the start of the
pilot project, the idea of EDGAR has gained the enthusiastic attention of filing firms,
securities analysts, and the general public. When such an electronic reporting concept is
put into practical use, interested parties should be able to promptly access information they
desire. However, in such a circumstance, auditors will be subject to greater demands-
from management, investors, institutions, and the general public-to certify the reliability,
security, and integrity of crucial databases on a continuous basis.
3.2.6 Audit Focus (Substantive versus CompUance Testing)
1 0 According to the ICAEW's report [1989], although the speed of the move towards this trend is unclear, it is clear that the technology is already available to enable reporting "at a frequency that could, in theory, be 'up to the minute" [p.4].
" Electronic Data Gathering Analysis and Retrieval (EDGAR) system is "An electronic data processing system that is capable of receiving companies' financial reports electronically, allowing for their review by the SEC staff in a similar manner, and permitting computerized dissemination of information to investors, analysts and others capable of receiving information in this way." [Yang 1990, p.49].
27
Rationale: Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied. Therefore, efforts are needed to establish appropriate audit approaches.
Although there is a general agreement that the focus and the types of audit tests
need to change to suit highly automated, paperless intercompany EDI networks, there is
no consensus on the direction or specific procedures that should be applied to attain the
audit objectives. Staats [1981] and West [1988] believe that the shift should be away from
testing individual transactions towards tests of the system and system security. They also
note that auditors should be more concerned with fraudulent or intentional manipulations
of the records in such a paperless system. Jancura et al. [1986] suggest (for audit tasks
in general) that the possibility of added calculations and statistical analyses without
excessive additional costs should enable auditors to perform more analytical reviews. The
Information Technology Group of the Institute of Chartered Accountants in England and
Wales [ICAEW 1989] observes the discordance between clients' needs and audit focus and
notes that:
Users with high volume, rapid response transaction processing systems need well controlled computer systems or they run the real risk of going out of business. They have to understand their systems and control them. They also expect their auditors to understand such systems and to be able to provide critical comment thereon. Yet many auditors, faced with increasing complexity of clients' systems, look to substantive testing techniques rather than control-based compliance testing techniques. This response is frequently driven by cost-effectiveness considerations. It can also reflect the inherent difficulties of compliance testing in respect of complex systems. [ICAW, p.3]
28
During a seminar on "Implications of Emerging Technology to Auditors"12, a
speaker who is a partner responsible for the Computer Audit Support Group of a big six
accounting firm in Vancouver B.C. , commented that considering the availability of
advanced software to ease the task, auditors should be able to perform more substantiation
of data. Brown [1991] quotes Hugh Parkes, general manager of group audit at National
Australia Bank, as stating that:
/ think it will require major changes to audit methodologies, and to the sacred cows of auditing-particularly the issues of substantive and compliance auditing. I have serious questions as to the validity of some of these with advanced large-scale systems, where there is virtually no paper, very little to substantiate it, and it's necessary to have a good understanding of how the transactions are arrived at... these are (issues) of international significance,... [Brown, p. 12].
These questions remain unresolved. Research efforts are needed to establish
appropriate audit approaches. Different audit functions, e.g. internal and external audits
may be called to have different audit focuses.
3.2.7 Pre-determination of Audit Scope (Boundary of Audit)
Rationale: Paperless intercompany transactions create a "boundaryless" information system environment, and auditors may be required to audit beyond the traditional boundaries of clients' systems. Therefore, audit responsibilities need to be pre-determined and agreed upon by the parties involved.
Paperless intercompany transactions create a "boundaryless" information system
1 2 The lecture was a part of a one-day seminar on "Emerging Information Technologies—An Auditor's Perspective" organized by the EDP Auditors Association, Inc. (Vancouver Chapter) at Simon Fraser University (Harbour Center) on November 8, 1991.
29
environment, and auditors may be required to audit beyond the traditional boundaries13
of clients' systems. In order for auditors to prevent a "boundaryless" audit responsibility
from occurring while they continue providing an adequate audit service, the scope and
extent of audit task and responsibility need to be pre-determined and agreed upon by the
parties involved. For example, in a fully automated EDI system, where electronic data
originating at one company is transmitted to a receiving company and incorporated directly
into that company's application system (it is to be hoped after some forms of review or
computerized edit checks), the auditor of the receiving company may choose to take the
responsibility of evaluating the reliability of either or both the network over which the data
was transmitted and the quality controls of the sending company [Holstrum et al. 1988,
p. 173].
3.2.8 Audit Tools
Rationale: The increased complexity of intercorporate automated paperless transactions make it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. More powerful tools must be developed to match the growth in sophistication of clients' systems.
The increased complexity of intercorporate automated paperless transactions make
it more difficult, or in some cases impossible, for auditors to test and evaluate network
systems with existing audit tools. It is therefore critical that more powerful audit tools be
developed to match the growth in sophistication of clients' systems. Auditors need to be
1 3 "Boundary defines a system in terms of the degree of control it can exercise, and with EDI, this control is somewhat extended beyond the original focal organization." [Kavan 1991, p.41].
30
equipped with appropriate tools adequately to achieve audit objectives and to satisfy
clients' and the general public's expectations. The literature review suggests that such
tools as integrated audit networks, portable workstations, interconnections with large
databases, audit software embedded in the operating systems, expert systems, multiple
input modes, evaluation software, and natural language programming are indispensable
future audit tools [West 1988; Holstrum et al. 1988].
3.2.9 Audit Techniques
Rationale: Traditional audit techniques may no longer apply in an EDI environment. Auditors need to develop effective techniques to enable them to describe, evaluate, and test an intercompany information network that includes very few paper documents.
EDI technology introduces a new era to business transaction processing. The
significant cutback on paper may put the end to auditing "around" the computer. The
complex interconnections of EDI networks imply that such traditional audit techniques as
test data and Integrated Test Facility (ITF) have to be significantly modified. The volume
and the sheer pace of transaction processing may make continuous on-line computer
monitoring more effective than human observation. Programmed edit checks and
programmed monitoring systems may have to be used either to prevent or to detect unusual
activity for near-immediate follow-up or to do both. The flowcharting technique, which
is document oriented, may have to be replaced by another technique which enables
auditors to describe, document and evaluate an accounting system that does not include
paper documents [West 1988; Wise 1989]. In summary, to demonstrate competence in
dealing with EDI systems, auditors need to find effective audit techniques that are capable
31
of describing, evaluating, and testing a "paperless" intercompany information network.
3.2.10 Audit Risk Assessment
Rationale: Audit risks in an automated open EDI network are significantly different from those in a closed system because EDI involves more parties and more diverse exposures. A distinctive approach of audit risk assessment is required, and sound guidance must be developed to guide the practice.
Certain attributes of EDI technology make an EDI system more secure while others
make it more risky. The automation of system functions means high consistency of
transaction processing and increased system reliability. Further, because errors that may
be generated by such a system are typically systematic (non-random error), they are easier
to be detected and corrected. Thus, if there is an assurance that the system functions
properly and reliably, business and audit risks may decrease substantially. On the other
hand, the EDI notion involves many exposures over which users have little or no control.
For instance, the security of an EDI network depends on such a considerable amount of
trust among all the involved parties that adequate controls must be maintained over each
individual system, and each party must follow the agreed-upon rules. Further, the use of
Value Added Network (VAN) 1 4 creates new risk exposures because of the company's
increased reliance on a third-party to provide acceptable controls and services. In addition
to that, the company auditor's inability to evaluate directly third party (VAN) performance
increases audit risks. Moreover, open intercompany systems are naturally more vulnerable
1 4 A communication network over which a third-party vendor performs EDI services beyond transmission of data—for instance, translation, training, encryption, etc. These services add significant value to the basic function of message switching and enable different computers to communicate to each other [Emmelhainz 1990; Baker 1991(b); Kimberley 1991].
32
than the closed singular systems, and the dial-in lines used in the transmission of EDI
transactions are more prone to be attacked. This vulnerability coupled with the
acceleration of transactions, increases business and audit risk substantially because a single
failure, if it occurs, can be widespread within a short time. The reviews recommend
advanced planning and up-front risk analyses as a vital part of an EDI audit. By
recognizing the risks associated with the technology, auditors can help improve the
security of the system by establishing an integrated program of risk assessment and
monitoring of early warning indicators. [West 1988; Sadhwani et al. 1989; Eckerson 1990;
Chalmers 1990; Burns and Sorkin 1991]. However, because EDI involves more parties
and different exposures, risk assessment in an EDI environment require consideration of
a different set of elements from that required in the conventional system. Thus, sound
guidance should be established to assist auditors in performing this task.
3.2.11 The Changing Role of Auditors
Rationale: In an EDI environment, the role of auditing will have to change to meet both the new demands of clients and the needs of the profession. Failure by auditors to assume suitable new roles adequate to meet such demands may lead to the decline of the profession.
In an EDI environment, the role of auditing will have to change to meet both the
new demands of client needs and the needs of its own practices. Like other advances in
information technology, EDI should be viewed less as a threat than as an opportunity.
Auditors are in a good position to assume many new roles which businesses and the
general public expect them to, and incidentally, these roles can be both financially and
professionally rewarding. As suggested in the professional literature, auditors are uniquely
33
suited to serve as "IT15 advisor[s]" or consultants on technology decisions. Auditors can
help assure the integrity of internal controls and also control future audit fees.
Furthermore, as independent experts, auditors can mediate different technological views
that may exist among managers, technologists, and senior executives, and help an
organization reach quality decisions. Moreover, by working together, external and internal
auditors can advise and assist an organization's executives with the technology's strategic
implications [Hogarth 1986; Willits 1990; Brown 1991]. Failure by auditors to develop
a strategy to meet clients' changing needs and their technology demands may lead to the
decline of the auditing profession. Thus, auditors must continue to live up to the
expectations of the corporations and the public.
3.2.12 Audit Responsibility in Evaluating Controls
Rationale: Auditors, especially internal auditors, may be held responsible for the review and evaluation of external control(s)16 in open EDI network systems. In such circumstances, it is necessary that guidance be developed and standards be established to assure that the task is performed adequately.
Both academic and professional literature indicate that the responsibility of auditors
for the review and evaluation of control structures17 in open automated EDI network
systems may have to be expanded. For instance, Holstrum et al. [1988] suggest that
1 5 Information Technology.
1 6 External control is defined as "a domain of factors that operate outside of an organization and may affect the way the organization and management operate. " [Barrett 1990, p. 63].
1 7 The term "Control Structures" is used here to cover both internal and external controls.
34
"blurred boundaries of the audited entity" demand the expanded responsibility of the
auditors to evaluate the integrity of both internal and external databases. Hansen and Hill
[1989] recommend that controls be exercised beyond the traditional system periphery and
that, where a third-party V A N is used, auditors evaluate network application features
either directly or through the VAN's auditor. Sadhwani et al. [1989] have a similar
concern, and they assert that the evaluation of controls in a typical EDI network involve
the following three parties, each of which must provide assurance that proper controls are
maintained within each individual system:
--the originator of the transactions and documents;
—the processor (i.e. a third-party network or a bank);
—the receiver of the data and documents.
Moreover, Barrett [1990] believes that "Only a global notion of control is realistic
and relevant in the fast-paced global economic environment..." [p.68]. He then proposes
that auditors recognize the importance of the concept and the audit of external control18.
In addition, if the audit responsibility in reviewing controls is to be expanded,
authorities seem to suggest that internal auditors will have increased responsibilities. For
instance, Staats [1981] states that the "paperless" transaction processing and the increased
reliance on the adequacy of the system controls demand more commitments from the
internal auditors. He claims "internal accounting control is the area where the corporate
internal auditor's depth and breadth of knowledge is superior to everyone else's—inside or
1 8 External control was defined as "a domain of factors that operate outside of an organization and may affect the way the organization and management operate. " [Barrett 1990, p. 63].
35
outside the company." [p.7]. Further, West [1988] advises that internal auditors analyze
proactively data security and data communication. Moreover, Barrett [1990], who
underscores the importance of external control, insists that it is internal auditors who
should take responsibility for understanding and auditing external control. He cites the
following activities as examples of sources of external control that deserve attention from
internal auditors:
—assessing the effectiveness of regulation;
—evaluating external audit performance;
—evaluating material acquisition in a just-in-time setting and EDI
transactions;
—auditing customers.
The auditing profession needs to take this issue of external control into serious
account. If auditors are to be held responsible for external control, the task of reviewing,
testing, and evaluating this type of control must be performed properly. The profession
needs not only to develop solid guidance for but also to establish standards of practice on
this added activity.
3.2.13 Relationship Among Company's Auditors
Rationale: As business information systems grow in complexity, external auditors may have to rely on internal auditors, and information systems auditors will be requisite members of audit teams. It is crucial to develop suitable audit approaches to promote and make the best use of this interrelationship.
Along with the tendency towards increased demands EDI makes on the internal
36
audit function, the reviews encourage close cooperation between internal and external
auditors. One reason for that cooperation is that, as business information systems become
increasingly complex, external auditors may have to rely more on internal auditors.
Brown [1991] illustrates this point by stating, with a quotation from Hugh Parkes, general
manager of group audit at the National Australian Bank, that:
The technology gap is beginning to blur the distinction between the roles of internal and external audit Junctions.... fit is found that] even the most experienced external auditor required additional expert advice [from internal auditors] in order to perform an adequate audit... 'the reality is that internal audit environments of banks are so much bigger than the external audit commitment that it is the people who are there all the time, who increasingly have the balance of knowledge about the systems. This does pose questions and some challenges to the relationship between internal and external audit. Knowledge of the client is very important, and external auditors should make sure they have enough of this. '. [Brown, p. 12]
The foregoing statement illustrates a typical situation in current businesses. As
outsiders, external auditors are forced with a limited time to gain sufficient knowledge and
understanding of the client's systems to conduct an adequate audit. By teaming with
internal auditors, they can be more efficient and more effective in accessing client
computer resources and in identifying the strengths and weaknesses of the systems.
Moreover, the same generalized audit software used for internal audits can be useful for
preparing and performing the annual external audit. Furthermore, in some cases audit
coverage can be expanded and overall audit fees can be reduced because much of the
external auditors' substantive testing can be replaced with less expensive compliance
testing [Boughton 1987; Brown 1991].
37
In addition, because of the increasing complexity of the "paperless" EDI networks,
teamwork between computer auditors and general auditors is strongly recommended. The
stake here is that telecommunications is an area requiring the expertise of a technical
computer audit expert and technical audit software. Further, it is recommended that to
create a good working relationship between financial and information systems auditors, the
differences between the two types of auditors must be both communicated and
deemphasized [Moeller 1986; West 1988; Dunmore 1989; Utter and Bertram 1989]. It
is crucial for the auditing profession to develop audit approaches suitable to promote the
workability and to make best use of this inter-relationship.
3.2.14 Collaboration Among Auditors of EDI Parties
Rationale: An EDI network involves not only auditors of a company but also auditors of its trading partners and of third parties. Because the tasks of these auditors are inter-dependent, it is vital that the roles of each party's auditors be determined and the rules of collaboration be established.
Interconnections of different organization's systems into one large system is one
of EDI's key characteristics. A typical EDI network comprises of not only a company's
system but also outsiders' systems. These outsiders can be trading partners (i.e.,
customers, suppliers), service bureaus, VANs, and third-party telecommunication
networks. As a result, auditors of an EDI network consist of auditors (internal and
external) of a company under consideration as well as auditors (internal and external) of
the network participating parties. These groups of auditors are inevitably inter-dependent
on each other's work because the controls and security of a system depends heavily on
those of the connecting systems. Therefore, close collaboration among auditors of all
38
parties in an EDI network is desirable, and each group of auditors can gain substantial
benefits from other group's work. For example, as recommended by Hansen and Hill
[1989] and by Sadhwani et al. [1989], because most third parties are reluctant to allow
auditors other than their own to access their facilities, auditors of a company that uses
third-party services may evaluate network application features through the work that has
been performed by the third-party's auditors. Further, while urging a company to seek
the auditor's reports from third parties, Hansen and Hill [1989] remark that "This practice
is not yet commonplace, but should become so as EDI becomes more pervasive." [p.412].
Today, businesses make an extensive use of EDI technology and a large proportion
of EDI networks of trading partners uses a third-party service to serve as an intermediate
processing agent. It is therefore important that the roles and responsibilities of each
party's auditors be determined, and the rules for collaboration among these auditors be
established.
3.2.15 Auditor Skills (Skills required of auditors)
Rationale: Because of rapid expansion in the extent, scope, and types of information to be audited in an EDI system, auditors need to acquire certain skills that enable them to maintain a high standard of practice.
Because of rapid expansions in the extent, scope, and types of information to be
audited in an EDI system, auditors need to acquire a specific set of skills that enables them
to overcome difficulties and maintain a high standard of practice. Some essential skills
cited in the reviews are the computer skills, the ability to adapt readily to rapidly changing
39
information technology19, and the high analytical skills to accommodate the shifting
composition of audit tasks to high-level analysis [West 1988; Holstrum et al. 1988]. It is
important that a comprehensive set of required skills be researched and incorporated into
skill development programs for auditors.
3.2.16 Auditor education and training
Rationale: Auditors trained today practice in a significantly different environment from that in the past. The educational curriculum and training requirements of auditors must be updated to reflect technological change and to embody the types of knowledge and proficiency required of auditors to maintain the profession.
Auditors trained today practice in a significantly different audit environment from
that in the past. Therefore, the educational curriculum and training requirements must
be updated to reflect technological change and to embody the types of knowledge and
proficiency required of auditors to maintain the profession. Certain education and training
needs required of information systems auditors in general may also apply to EDI auditors.
The reviews suggest the following subjects as important:
- continuing education for auditors to gain expertise in data processing and fraud prevention and detection [Staats 1981, p. 11]
- proficiency in auditing and data processing, as well as a solid business background [Helms 1986]
- knowledge of methods being developed in the study of human information processing and artificial intelligence in addition to a broad base of skills and a high degree of professional commitment [Elliott 1986].
1 9 Adaptability which was specified includes: computer adeptness; ability to interface effectively with expert systems; and knowledge of and adeptness with modelling concepts and applications.
40
- more use of case studies methods in the educational process to train auditors in sharing audit expertise with organizations adopting new technologies [Gilhooley 1987]
- sound understanding of the system (accounting/management) implemented by audited companies to process their business information [West 1988, p. 187]
- thorough understanding of computerized information systems (IS) and learning of technical data processing concepts, as well as the traditional computer audit skills [Glynn and Lemieux 1990] In addition, the following ongoing training for audit staff is recommended:
- training for computer security auditing with focus on: (1) general security, (2) specific applications, and (3) technological trends. [Gallegos 1987]
- "individualized training of staff members" for a small EDP audit department. Audit performance should be evaluated and the needs for staff training be determined in accordance with that evaluation. [Goldner 1987]
The reeducation and retraining of auditors is an important issue because failure to
educate and train auditors capably to keep pace with business advances in the use of
information technology threatens the existence of the auditing profession.
41
Chapter 4. Research Design
4.1 Research Questions
This research project is designed to answer the following questions:
1) What are the most important EDI audit issues as seen by leading audit
professionals?
2) What is the order of importance of these issues?
4.2 Selection of Research Methodology
In the search for an appropriate research methodology to investigate the
foregoing research questions, a variety of technology assessment techniques20 were
explored. The techniques which seem to be applicable are Delphi technique, survey
technique, expert panels/workshops, and compilation and analysis of all available
information. In this study, the technique of compilation and analysis of existing literature
is used to form the primary research issue framework in chapter 3. A Delphi survey,
however, was used to solicit opinions and consensus of leading information systems
auditors in the greater Vancouver area regarding the importance of EDI audit issues.
The Delphi methodology is a cost-effective way systematically to solicit and
combine the individual judgments of experts in the field and thus to obtain a reasoned
consensus. It is capable of yielding answers to the research questions being addressed in
2 0 By forecasting what the consequences might be if a complex emerging technology is in fact adopted, technology assessment techniques enable us to study complex technologies and their potential impacts upon society. [Fowles i97g; p. 146] This type of assessment, therefore, can be well applied to an emerging technology such as EDI and is suitable for the purpose of this study.
42
this research project, and it has been, and continues to be, widely and successfully used
in similar issue researches in the area of Management Information Systems (MIS).
Further, in comparison with the survey technique, a research finding shows that the Delphi
technique, even when only two rounds are conducted, generates a more reliable and valid
result than the survey technique. [Martino 1983]
As opposed to the expert panels/workshops or committee meeting approach, the
Delphi technique replaces direct debate by a carefully designed program of sequential
individual interrogations conducted through written questionnaires. Information and opinion
feedback from the earlier parts of the program are used in the later stages. The
respondents may, for instance, be asked the reasons for previously expressed opinions, and
a compilation of these reasons may then be presented to each respondent with an invitation
to reconsider and possibly revise his or her earlier estimates. This inquiry and feedback
may stimulate the experts to consider factors they might have inadvertently overlooked or
disregarded on first thought.
The Delphi technique offers many benefits. It eliminates committee activity and
reduces the pitfalls of face-to-face discussion.21 Because each Delphi panellist is allowed
to give opinions independently and anonymously in the written questionnaires, the
influence of undesirable psychological factors, such as specious persuasion, unwillingness
to abandon publicly expressed opinions, and the "bandwagon" effect of majority opinion,
can be greatly reduced. Moreover, considering the relatively small Vancouver IS auditors
2 1 Lanford [1972] states that "face-to-face discussion tends to make the group less accurate, whereas the controlled-feedback procedure [as used in the Delphi approach] makes group estimates more accurate. " [p. 22]
43
community, the possibility of personal conflict and social pressures that might occur can
be prevented. Also, in spite of the capability to avoid the important drawbacks of open
face-to-face discussion, the Delphi approach can still obtain the benefit of such discussion.
By assimilating comments from Delphi panel members into the questionnaires in the
subsequent rounds, a desirable interaction among several participants expressing their
opinions can be facilitated22. Furthermore, another benefit is that the Delphi procedures
create a well-defined process that can be described quantitatively. Finally, because the
Delphi findings reflect reasoned, self-aware opinions, expressed in the light of the opinions
of associate experts, they are claimed to provide a sounder basis for long-range decision
making than do unarticulated intuitive judgments. [Lanford 1972; Martino 1983]
4.3 The Delphi Process - An Overview
The Delphi approach requires several iterations of questionnaire completions by
the expert participants. In general the Delphi process in this specific study proceeds as
follows:
(1) Potential Delphi panel members are identified from the group of Vancouver's leading
IS auditors, and 33 experts are selected23. The reason for using this sample size is
discussed in section 4.5 of this chapter.
(2) The selected experts are contacted and requested for an agreement to participate in the
2 2 According to Martino [1983, p.23], the experiment by Salancik [1973] showed that Delphi panels do assimilate the comments from panel members into their aggregate estimates and group interaction does occur.
2 3 For purposes of conducting the research within a limited budget, Vancouver, a major city in Canada, was chosen as the study site.
44
study. Among these, only 12 are asked to participate in the unaided first round survey.
The rationale for using only a subset of the entire sample in the first round is discussed
in the later section dealing with participant recruitment.
(3) An open ended first round questionnaire and a cover letter explaining the research are
sent to the 12 selected experts. The purpose is to solicit unbiased views and identification
of important issues from the highly experienced IS auditors in the greater Vancouver area.
(4) The results from the first round questionnaire are analyzed and integrated with EDI
audit issues previously identified in the existing literature. The purpose is to facilitate the
rating of the issues and the identification of the most important audit concerns. This
integration can also provide information on the perspectives of the Vancouver IS auditors
on important EDI audit issues addressed in the North American literature.
(5) The second round questionnaire and a cover letter are sent to all 33 Delphi panel
members to rate the importance of EDI audit issues. The panellists are asked to rate,
rather than rank, each issue because rating is a less stressful and time-consuming
procedure. In this round the panellists are also provided with an opportunity to add new
issues.
(6) The results are analyzed and feedback is incorporated into the third round
questionnaire. The incorporation of this feedback is aimed at facilitating the reflection of
opinions and the movement toward consensus. A greater depth of insight is expedited by
multiple rounds.
(7) The third round questionnaire is sent to all 33 panel members regardless of whether
or not they responded in the second round. The purpose of still including the panellists
45
who fail to respond in the second round is to improve the response rates.
(8) The analysis and feedback procedures will be repeated until a convergence of opinions
or the stability of ideas is obtained.
4.4 Instrument Development
This study consists of multiple rounds, requiring the development of a separate
questionnaire for each round. The following sections outline the procedures in developing
these instruments:
4.4.1 Round 1 Questionnaire
The primary purpose of the first round survey is to generate an initial list of
issues to be rated in subsequent rounds. The format of the questionnaire developed for this
first round is therefore open-ended, asking the participants to state objectively what they
regard as the key EDI audit concerns. The unaided format is used here to minimize the
risk of the researchers biasing the participants' responses and to give the audit experts an
opportunity to identify important issues without the distraction of considering issues from
other sources.
The first-round questionnaire and the cover letter can be found in Appendix A.
The instrument consists of three parts:
- a cover letter
- notes to the respondents (to define terms used in the core questionnaire)
- a core questionnaire (a participant identification section (optional), instruction, and space for issue identification)
46
In addition to the instructions for answering the questionnaire, the notes to
respondents are added to clarify and define the terms "auditors" and "EDI" used in the
core questionnaire. The notes also differentiate "EDI" from "EFT (Electronic Fund
Transfer)". Because each term may have different meanings and interpretations for each
individual audit expert, this procedure is performed to provide common definitions and
thus create a common frame of mind when participants respond to the questions.
4.4.2 Round 2 Questionnaire
In a conventional Delphi study, the round 2 issue list would be generated solely
from the round 1 resulting issues and would therefore reflect only the opinions of the
participating Delphi panelists. However, the use of only a subset of the total panelists in
the first round and the pressure to complete the research within a reasonable amount of
time justify the use of a supplementary technique to generate additional issues for the
round 2 survey. The primary purpose of this supplementary issue generation is to assure
that, to the greatest possible extent, all the relevant issues are identified and included in
the study. The issue list for the second round survey is thus generated using the following
two techniques:
I. Analysis of results from round 1 survey
II. Compilation and analysis of existing literature
Each of these techniques, together with the issues discovered by the first round survey,
are discussed in more detail below.
I. Analysis of Results from Round 1 Survey
Each of the completed questionnaires was carefully examined by the researcher.
The issues identified by the first round experts (as shown in table 1) were analyzed and
47
classified. The following is a list of five new issues, i.e. previously not well articulated
in the existing literature, which were identified and clarified by the researcher based upon
the analysis of the results of the round 1 questionnaires:
• Controls Over EDI Network • EDI Contracts • Backup, Disaster Recovery and Contingency Plans • Third Party EDI Services • EDI Records Retention
n. Compilation and Analysis of Existing Literature
As discussed in Chapter 3 on the EDI audit issue framework, the reasonably
current information on important EDI audit concerns is available, and this information
should be used, when applicable, to provide a broader consideration of the subject. The
16 issues outlined in chapter 3 are thus incorporated into the initial issue list resulting
from round 1 survey. These issues are:
• Audit Evidence • Audit Trail • Audit Involvement during the System Development • Timing of Audit Tests • Audit Reporting (Periodic versus On-Line) • Audit Focus (Substantive versus Compliance Testing) • Pre-determination of Audit Scope (Boundary of Audit) • Audit Tools • Audit Techniques • Audit Risk Assessment • The Changing Role of Auditing • Audit Responsibility in Evaluating Controls • Relationship Among Company's Auditors • Collaboration Among Auditors of EDI Parties • Auditor Skills (Skills required of auditors) • Auditor education and training
The foregoing issues and the results from round 1 survey are then put into
48
random order24 and combined into a single round 2 questionnaire. As shown in
Appendix B, the resulting instrument consists of three parts:
- a cover letter
- a core questionnaire (a participant identification section, instruction, and 21 issues to be rated)
- an open-ended section for adding new issues (instruction and space for issue identification)
The style and format of the main body of the questionnaire closely imitates the
layout established in previous Delphi studies on MIS issues. The open-ended section is
included as a control to ensure that the two techniques described above have been
sufficient to generate all of the potentially important issues.
4.4.3 Round 3 Questionnaire
The round 3 issue list is generated from the twenty-one issues in round 2,
together with the four new issues which were added. This final questionnaire provides an
opportunity for participants to reflect on their answers in round 2 by supplying, for each
original issue, information on both the group's mean response and that particular
individual's response. This instrument is similar to the round 2 instrument and is aimed
at obtaining the final rating and ranking of each issue.
The round 3 instrument can be found in Appendix C. The instrument consists
of four parts:
- a cover letter
- a core questionnaire (a participant identification section, instruction,
2 4 The Rand Corporation, A Million Random digits with 100,000 Normal Deviates, The Free Press, 1955.
49
21 issues to be rated, the group's mean and the individual's original rating, and open spaces for final rating and rationale if individual final rating is significantly different from the group)
- a questionnaire on additional issues (instruction and 4 new issues to be rated)
- a questionnaire on background information
It should be noted that the 21 original issues to be rated are now presented to
the participants in a decreasing order of importance, as determined by the group mean
response from round 2. In addition, each participant's questionnaire shows his or her
previous individual response as compared to the group mean response for each issue.
The analysis of the open-ended section from round 2 reveals four additional
important issues. These new issues and their rationals are incorporated into the round 3
questionnaire but are listed separately and without any previous rating. No further
opportunity to add new issues is provided in this final round.
The questionnaire on respondent's background information is added to the round
3 questionnaire so that data can be gathered for analysis and categorization purposes. In
general, this section asks for area(s) of audit expertise, level of experience both in
information systems auditing and EDI system auditing, and an indication of the firm's
preparation for EDI technology. Further, as a way to motivate response, each participant
is asked to indicate whether or not he or she would like to receive a copy of the final
research findings.
4.5 Participant Recruitment
According to Lanford [1972], research by Norman Dalkey of the RAND
50
Corporation shows that most of the limitations of using the Delphi technique can be
overcome by working with groups of at least twenty. Further, as cited by Dexter et al.
[1992],
"In fact, Dalkey (1969) found Delphi studies produce quite accurate results with a group size of approximately thirty individuals. Furthermore, he found that increasing the number of participants does not markedly enhance the accuracy of the findings. " [p. 7]
In order to minimize the study's time and costs without sacrificing the values
of its results and to take into account the possibility of unexpected drop-outs, this project
seeks to identify and obtain cooperation from 33 of the Vancouver leading IS auditors.
The participant recruitment procedures began by the researcher's identifying and
seeking cooperation from two contact persons from two local accounting professional
organizations. The Director Membership of the EDPAA (Mr. Alan Drinkwater) and the
President of the IIA (Ms. Angela Louie) agreed to assist in this research project. The
researchers personally met with both of the contact persons to promote commitment and
to create a better understanding of the nature of the study and of the instruments. Then,
with their assistance, the membership databases of the EDP Auditor Association (EDPAA)
and of the Institute of Internal Auditors (IIA), Vancouver Chapter, were used to identify
and recruit potential participants. Besides their involvement and experience with EDI
projects, participants were recruited on the basis of professional qualifications, audit
specializations, and peer recommendations. Once appropriate individuals were selected,
they were contacted in person and asked for their agreement to participate in this multiple-
round Delphi study. Care was taken to ensure that this sample group represented various
types of organizations (public accounting firm, private or limited company, crown
51
corporation, government, and academic) and included both internal and external IS audit
experts. Because the majority of this group of experts is from the major firms in
Vancouver which are either utilizing or pioneering the EDI technology, this group can be
regarded as providing a representative sampling group of the Vancouver EDI audit
community.
It should be noted that only a subset (12) of the total number of the panelists
(33) were selected to participate in the unaided first round survey. The reason is that the
unstructured nature of the first round survey would likely make this the most difficult and
time consuming iteration in the study. Therefore, in order to reduce turnaround time and
complete the research project within a reasonable time frame, the size of the study group
was reduced to include only participants who have the most extensive experience in
auditing EDI systems.
This sampling approach generates a group of qualified and motivated
participants, and it is hoped that they could collectively reach a reasonable level of
consensus in judging key EDI audit issues.
4.6 Data Collection Procedures
The data collection of this research project was conducted in Vancouver, Canada
between March 16 and July 10, 1992. The questionnaires for each round were mailed to
participants outside the downtown area and hand-delivered to those located downtown. A
stamped, self-addressed return envelope was always provided with each questionnaire,
although the respondents were instructed either to mail or fax the completed questionnaires
52
to the primary researcher at the University of British Columbia. A l l follow-up on late
respondents was done by telephone calls.
4.6.1 Round 1
The first round questionnaires were forwarded to a subset of the final survey
sample. As explained in the participant recruitment section, only 12 experts were asked
to complete this open-ended first round questionnaire.
The contact persons from the two professional organizations had helped to make
initial contact with each participant to increase understanding and commitment. In
addition, the cover letter of the questionnaire explained in detail the initial goal, ultimate
objective, and nature of the study. Also, participants were encouraged to contact the
primary researcher directly should they need any clarification. The participants were
requested to reply within two weeks. After three weeks, which is a reasonable time period
for all the mail to reach the researcher, non-respondents were contacted by telephone to
encourage responses.
Once the first round questionnaires were returned, the responses were analyzed
and classified to isolate the primary EDI audit concerns of Vancouver IS auditors. These
issues were then combined with those identified in the literature to prepare a list of issues
to be rated in round 2.
4.6.2 Round 2
The analysis of the first iteration results and the inclusion of issues generated
by using supplementary techniques led to the creation of a new questionnaire for the
second round. The questionnaires were then sent to all 33 participants. As in the first
53
round, the cover letter of the questionnaire explained the goal, ultimate objective, and
nature of the study as well as encouraged participants to contact the primary researcher
directly should they need any clarification.
In this iteration, participants were asked to rate a list of issues in terms of their
importance on a scale from 1 to 10. In this manner, the most important issues could be
quantitatively identified and the appropriate analysis could be made. In addition,
participants were given a final opportunity to add new issues to the list to ensure that all
the major EDI concerns had been identified. These issues were then analyzed and
classified before being incorporated into the final round questionnaire.
In this round, the participants were requested to reply within two weeks. After
three weeks, follow-up telephone calls were made to non-respondents to encourage
responses. It was found that these follow-up calls greatly increased the response rate.
Some panelists were out of town during the period of two weeks when the questionnaires
were sent out. Having thought that it was too late, they did not respond. However, after
the follow-up calls they were willing to complete the questionnaires and return them by
facsimile so that the researcher received the responses in a timely manner.
4.6.3 Round 3
As discussed in the instrument development section, the construction of the core
questionnaire in round 3 was based on the results of round 2, with a separate page to
gather respondents' background information. In this round the questionnaires were sent
to the same group of thirty three participants as in round 2.
As in the first two rounds, the cover letter of the questionnaire encouraged
54
participants to contact the primary researcher directly should they need any clarification.
The participants were requested to reply within two weeks. Although the response rate
improved from the second round, follow-up telephone calls were still necessary to motivate
responses.
In this iteration, participants were asked, after considering the feedback in the
form of the mean group response, to give a final rating for each issue, using the same
scale as in round 2. No further opportunity was provided to add new issues. As suggested
by proponents of the Delphi technique [Delbecq 1972; Martino 1983], two iterations are
generally enough to establish the list of issues and their relative ordering. The third
iteration serves primarily to confirm the ordering and promote a consensus among
participants. In this study, the analysis of round 3 results indicates that the Delphi process
moved the group toward a consensus on the eleven most important EDI audit issues.
55
Chapter 5. Analysis and Discussion of Results
5.1 Introduction
This chapter discusses the results from the three-round Delphi survey. Although
all key EDI audit issues identified by the respondents in round 1 [Appendix D] are
included in the round 1 results, only those issues which are not addressed in chapter 3 are
discussed in detail in this chapter. The rating results from round 2 are stated and analyzed
along with the additional important issues revealed in this round. Then, comparison is
made of round 2 and round 1 results. Next, the results from round 3 which are the final
rating of all the issues are then discussed and the comparison is made with the results from
round 2. The movement towards consensus and the background information on the
respondents are outlined in the last two sections of the chapter.
5.2 Round 1 Results
In the first round, open-ended questionnaires were sent out to 12 selected audit
experts. However, only 9 questionnaires from 10 experts were returned. This happened
because two of the respondents worked together and submitted a single copy of the
questionnaire. Two non-respondents were away; however, they participated in the second
and the third round.
Based on the narrative content and the labelling of issues [Appendix D], the
responses from the first round were analyzed and classified into seven major issues. Table
1 shows these key EDI audit issues identified by this group of experts.
56
T A B L E 1 - ISSUES IDENTIFIED IN ROUND 1
Frequency Issue
12 New Controls Over EDI Networks
6 New EDI Contracts (Trading Partner Agreement)
4 Auditability and Audit Trail
4 New Backup, Recovery and Contingency Plans
3 New Third Party EDI Services
3 Legal and Audit Evidence
2 New EDI Records Retention
The foregoing issues are presented in order of the frequency of citations given
by the respondents. Of these seven issues, only two were previously addressed in the
compilation and analysis of existing literature in Chapter 3. The issues which were not
discussed in the issue framework in Chapter 3 but were identified during this round are
designated as 'new' issues. These issues are discussed in more detail in the following
section.
5.2.1 Controls Over EDI Networks
Rationale: Because weak controls can cause significant financial loss to EDI trading partners, auditors must assure that controls over the EDI networks such as access controls, authentication controls, transmission controls, and controls over mailboxes are effective.
As shown in Appendix D, there are 12 responses in round 1 that can be
classified under the control issue. These responses include audit concerns for controls over
(third party) mailbox service, data confidentiality, system security, integrity of data,
authentication of trading partners, trading partner's security, access controls to the EDI
(network) environment, financial controls, communication (transmission) controls with
57
suppliers, and accurate and complete transmission of data.
5.2.2 EDI Contracts (Trading Partner Agreements)
Rationale: Because EDI contracts are the basis for the company's future dealings with and liabilities to EDI partners, auditors must ensure that the terms, services, conditions, and responsibilities of each EDI party are clearly defined and the contracts are inclusive and enforceable.
As shown in Appendix D, there are 6 responses in round 1 that can be classified
under the EDI contracts issue. The respondents feel that EDI contracts or trading partner
agreements are important because they believe these contracts or agreements are the basis
for the company's future dealings with its EDI partners. Therefore, they suggest that in
order for the agreements to be enforceable all agreements/contracts must be in written
form and cover all significant issues. In addition, terms, services, and responsibilities of
each party must be clearly defined and agreed upon by all parties involved.
5.2.3 Backup, Recovery and Contingency Plans
Rationale: Because of a company's increasing reliance on the EDI network for operational and financial services, auditors must assure that control strategies exist for backup and recovery in case the network fails. These strategies form an indicator of the company's ability to continue as a going-concern.
As shown in Appendix D, there are 4 responses in round 1 that can be
classified under the issue of backup, recovery and contingency plans. The respondents
express concerns over this issue because, in their view, in an EDI environment a company
relies heavily on the EDI networks. In case of network failure where a company does not
have adequate plans and strategies for backup and recovery, the company's assets in the
form of valuable data may be lost, and the company's ability to continue as a going-
58
concern may be in jeopardy.
5.2.4 Third Party EDI Services
Rationale: Different types of EDI networks have different implications for the participating companies and their auditors. Auditors must evaluate their clients' third party EDI service firms in terms of responsibilities, resources, and abilities to provide, on an ongoing basis, reliable and secure services per contract terms.
As shown in Appendix D, there are 3 responses in round 1 that can be classified
under the issue of third party EDI services. In case where a company uses third party EDI
VAN(s), the respondents think that the types of networks and the reliability, availability,
and security of EDI services need to be evaluated. As certain of the respondents stated,
such an evaluation may be based on the service firm's "financial resources to provide
service per contract terms on an ongoing basis" and "adequate provision for trouble
shooting, client communication, system upgrading capabilities". The respondent auditors
believe such an evaluation of EDI service qualities should be done because different types
of EDI networks have different implications for the network participating companies and
their auditors.
5.2.5 EDI Records Retention
Rationale: Because EDI uses electronic source documents, guidance must be established to ensure that EDI records are maintained properly and securely for an appropriate amount of time to suit tax, audit, backup, and management purposes. Ineffective records management can lead to exposures such as the loss of critical data files and major litigation costs and penalties.
Although this issue is mentioned together with the responses under the issue of
auditability and audit trail, it has different implications and is important enough to be an
issue on its own. As shown in Appendix D, there are 2 responses that can be directly
59
classified under this issue. The respondents affirm that the system access logs and
complete records of all EDI transactions in the form of electronic data files need to be
maintained for the complete fiscal period to satisfy audit purposes. The professional
literature confirms that an electronic record management system needs to be established
to satisfy legal, audit, and management requirements. As Decker [1991] warns, ineffective
records management can lead to such risks as loss of critical data files and major litigation
costs and penalties.
5.3 Round 2 Results
In the second round, a twenty-one item questionnaire was sent to 33
participants. Each participant was asked to rate each issue in terms of its importance using
the scale from 1 to 10. In this round, 32 questionnaires were returned, resulting in the
response rate of 97 percent.
It should be noted that both the 2 non-respondents and the 10 respondents in
round 1 responded in round 2 (i.e. all selected participants in round 1 responded in round
2). The only one non-respondent in the second round was not selected to participate in the
first round. This particular participant had to go overseas before having the opportunity
to answer the round 2 questionnaire and was not scheduled to return until after the round
2 cut-off date.
5.3.1 The Rating of Round 2 Issues
Whereas Appendix E shows detailed round 2 results, Table 2 summarizes the
rating of key EDI audit issues by this group of experts. The group's mean responses of
60
each issue shown in the second column are used in ranking the importance of the issues
in the first column. Moreover, because the spreads or standard deviations (stdev) of the
mean scores should also be taken into account when considering the importance of the
issues, they are provided in the third column.
T A B L E 2-RATING OF ROUND 2 ISSUES
Rank Mean Stdev Issue
1 9.34 0.96 Controls Over EDI Network
2 8.72 1.35 Backup, Disaster Recovery and Contingency Plans
3 8.47 1.66 Auditability and Audit Trail
4 8.06 1.69 Audit Involvement during the System Development
5 7.81 1.70 Legal and Audit Evidence
6 7.72 1.55 EDI Contracts
7 7.69 1.67 EDI Records Retention
8 7.41 1.77 Third Party EDI Services
9 7.31 1.47 Auditor Education and Training
10 7.09 1.76 Audit Techniques
11 6.94 1.66 Auditor Skills (Skills required of auditors)
12 6.63 2.03 Audit Focus
13 6.53 2.34 Audit Scope (Boundary of Audit)
14 6.50 2.06 Audit Risk Assessment
15 6.44 1.78 Audit Responsibility in Evaluating Controls
16 6.31 2.05 Collaboration Among Auditors of EDI Parties
17 6.28 1.89 Changing Roles of Auditors
18 6.25 2.19 Timing of Audit Tests
19 6.22 1.34 Audit Tools
20 5.50 1.82 Relationship Among Company's Auditors
21 4.28 2.00 Audit Reporting (Periodic Versus On-Line)
61
5.3.2 Additional Issues Identified in Round 2
As a procedure to obtain as complete an issue list as possible, the questionnaire
in round 2 provided a final opportunity for participants to contribute additional issues of
importance. Four additional issues were identified at this stage. These issues and their
rationale are stated in Table 3.
TABLE 3 - ISSUES IDENTIFIED IN ROUND 2
Issue Rationale
Form of Audit Assurance
The closer interrelationships established between trading partners in an EDI network will affect business and financial risks. Therefore, auditors should reevaluate the types of assurance required by the public and formulate suitable audit procedures and related opinions to satisfy these needs.
Professional Support for Practising Auditors
The auditor has to face many new issues when auditing an EDI system. Thus, professional organizations such as EDPAA, IIA, and CICA should take a proactive approach to providing reference materials and training opportunities to help practising auditors understand and deal effectively with the EDI environment, its risks and control measures.
Inconsistent EDI Approaches
Inconsistent EDI approaches (used by various EDI trading partners) can lead to operational and administrative problems resulting in missed business opportunities, additional costs, and weakened internal controls. Therefore, auditors must be aware of inconsistencies and provide direction to management.
The Network and Ownership of Data
As EDI systems develop, the sharing of common data/programs will increase and the information flow that the auditor needs to understand will change. Auditors must take part in defining information flows and boundaries to data ownership. (This will also help draw legal boundaries among parties in a large integrated EDI system).
5.4 The Comparison of Round 1 and Round 2 Results
Note that all seven issues identified by the respondents in the first round are in
the top eight rank in the second round [Table 2]. For example, "Controls Over EDI
62
Network", the issue stated most often by the respondents in the first round, was the most
important issue in the second round. This result indicates a strong consistency between
the first expert group and the larger panel in reporting the issues of importance. Also, four
issues from the literature review, "Audit Involvement during the System Development",
"Auditor Education and Training", "Audit Techniques", and "Auditor Skills" round out
the top eleven issues in the second round. Therefore, the first round can be counted as
being successful in revealing most of the potentially important EDI audit issues in the
Vancouver area, and the supplementary issue generation technique is useful in making the
list more inclusive.
5.5 Round 3 Results
In the third round, the questionnaires with a list of 21 original issues and 4
additional issues from the second round were sent to the sample of 33 participants. As in
the second round, each participant was asked to rate each issue in terms of its importance
using the scale from 1 to 10. No further opportunity was provided for respondents to add
new issues. In this round, 33 questionnaires were returned resulting in the response rate
of 100 percent.
It should be noted that the only one participant who was not selected to
participate in the first round but was selected and did not respond in the second round, did
respond in the third round. As a result, this participant rated the issues only once in the
final round without rating in the previous two rounds. In summary, among 33 respondents
in the third round, 10 had responded in all three rounds, 22 had responded in both the
second and the third round, and 1 responded only in the final round.
63
T A B L E 4 - ROUND 3 RATING OF 21 ORIGINAL ISSUES
Rank Mean Mean diff
Stdev Issue
1 9.44 - 0.65 Controls Over EDI Network
2 8.82 0.62 1.09 Backup, Disaster Recovery and Contingency Plans
3 8.70 0.12 1.09 Auditability and Audit Trail
4 8.15 0.55 1.52 Audit Involvement during the System Development
5 7.83 0.32 1.30 Legal and Audit Evidence
6 7.50 0.33 1.44 EDI Records Retention
7 7.42 0.08 1.33 EDI Contracts
8 7.30 0.12 1.49 Audit Techniques
9 7.26 0.04 1.57 Third Party EDI Services
10 7.21 0.05 1.32 Auditor Education and Training
11 7.02 0.19 1.31 Auditor Skills (Skills required of auditors)
12 6.45 0.57 1.71 Audit Risk Assessment
12 6.45 0.00 1.42 Audit Responsibility in Evaluating Controls
14 6.39 0.06 2.00 Audit Scope (Boundary of Audit)
15 6.38 0.01 1.46 Audit Focus
16 6.24 0.14 0.99 Audit Tools
17 6.14 0.10 1.80 Timing of Audit Tests
18 6.09 0.05 1.82 Collaboration Among Auditors of EDI Parties
19 6.05 0.04 1.56 The Changing Roles of Auditors
20 5.27 0.78 1.50 Relationship Among A Company's Auditors
21 3.91 1.36 1.40 Audit Reporting (Periodic Versus On-Line)
64
T A B L E 5 - ROUND 3 RATING OF 25 FINAL ISSUES
Rank Mean Mean
diff Stdev Issue
1 9.44 - 0.65 Controls Over EDI Network
2 8.82 0.62 1.09 Backup, Disaster Recovery and Contingency Plans
3 8.70 0.12 1.09 Auditability and Audit Trail
4 8.15 0.55 1.52 Audit Involvement during the System Development
5 7.83 0.32 1.30 Legal and Audit Evidence
6 7.50 0.33 1.44 EDI Records Retention
7 7.42 0.08 1.33 EDI Contracts
8 7.30 0.12 1.49 Audit Techniques
9 7.26 0.04 1.57 Third Party EDI Services
10 7.21 0.05 1.32 Auditor Education and Training
11 7.02 0.19 1.31 Auditor Skills (Skills required of auditors)
12 6.55 0.47 1.92 Professional Support for Practising Auditors
13 6.45 0.10 1.71 Audit Risk Assessment
13 6.45 0.00 1.42 Audit Responsibility in Evaluating Controls
15 6.39 0.06 2.00 Audit Scope (Boundary of Audit)
16 6.379 0.011 1.46 Audit Focus
17 6.375 0.004 1.76 Inconsistent EDI Approaches
18 6.28 0.095 1.98 Network and Ownership of Data
19 6.24 0.04 0.99 Audit Tools
20 6.14 0.10 1.80 Timing of Audit Tests
21 6.09 0.05 1.82 Collaboration Among Auditors of EDI Parties
22 6.05 0.04 1.56 The Changing Roles of Auditors
23 5.69 0.36 2.18 Form of Audit Assurance
24 5.27 0.42 1.50 Relationship Among Company's Auditors
25 3.91 1.36 1.40 Audit Reporting (Periodic Versus On-Line)
65
5.5.1 Rating of the Original 21 Issues
Whereas Appendix F shows detailed round 3 rating results of the original 21 EDI
audit issues, Table 4 summarizes the final rating and ranking of these issues. The group's
mean responses of each issue are used to determine the ranking of importance of the issues
in the first column. These mean responses are shown in the second column while their
corresponding standard deviations (stdev) are shown in the fourth column. The mean
differences between adjacent pairs of issues are shown in the third column. It should be
noted that a tie occurs in the final ranking of the 12th issue.
5.5.2 Rating of the Final 25 Issues
While Appendix G shows detailed round 3 results for the final 25 EDI audit issues,
Table 5 summarizes the final rating and ranking of these issues. As in the previous table,
the group's mean responses of each issue are used to determine the ranking of importance
of the issues in the first column. These mean responses are shown in the second column
while their corresponding standard deviations (stdev) are shown in the fourth column. The
mean differences between adjacent pairs of issues are shown in the third column. The 4
issues added by the respondents in the second round are shown in bold type. It should
also be noted that a tie occurs in the final ranking of the 13th issue.
The rating results in Table 4 and Table 5 show that the four issues which were
added in the second round have no effect on the judgment of the eleven most important
EDI audit issues. The reason is that none of the four issues takes place in the top eleven
rank in the final round. However, these issues do have an observable effect on the rating
and ranking of the 12th to 20th issues. The effect of these additional issues is discussed
66
in more details in Section 5.7 of this Chapter.
5.6 The Interpretation of the Results
Caution should be exercised in interpreting the absolute order of issues obtained
by the Delphi approach. As presented in both round 2 and round 3 of this study [Table
2,4, and 5], differences in mean ratings are negligible in many cases, given the size of the
corresponding standard deviations (stdev) or the spreads of individual scores from the
group mean. Consequently, additional revisions of the ordering can be anticipated if the
study is repeated or the sample size is enlarged. Therefore, it would be prudent not to
have absolute confidence in the current ordering of the issues. In a traditional method,
statistical tests on the significance of the ordering would be useful. However, in the Delphi
approach the statistical assumptions of normality and independence are violated making
such an analysis inappropriate. As argued by Kiudorf [1991],
"Despite the lack of an appropriate statistical test, it is possible to make the general statement that 'as the distance between issues increases, the likelihood of incorrect ordering decreases. '" [p. 70].
Using this argumentation and the mean group responses of the issues, we can be
quite certain that issue 1 through 11 are in the group of highest priorities25 [Table 5].
Although we can not be absolutely certain about the ordering of issues 7 through 10, we
can be confident that issue 1 should be placed first and issue 2 and issue 3 should be
placed before issue 4, 5, and 6. On the other hand, with comparable sizes of standard
deviations, the mean scores of issues 23 through 25 are observably lower than those of the
2 5 The fact that the top eleven issues remain the same in round 2 and round 3 also affirms the importance of these eleven issues.
67
preceding issues, making us quite confident that these are issues of lower priorities.
In short, although it must be acknowledged that the exact ordering of the issues
may lack strong statistical support, the eleven most important issues as determined by the
group mean ratings are acceptable because they are resulted from the study which follows
the well-established protocol in issues research.
5.7 The Comparison of Round 2 and Round 3 Results
The round 3 results show a high degree of consistency with the findings from round
2. In both rounds, the eleven most important issues remain the same. Although the ranking
orders of issues 6 through 10 change slightly, the five most important issues remain
identical. In addition, the two issues of least importance, "Audit Reporting" and
"Relationship Among A company's Auditors", remain the same in both rounds.
T A B L E 6 - THE TOP E L E V E N ISSUES IN ROUND 2 AND ROUND 3
EDI Key Audit Issues Rank in Round 2
Group Mean
Rank in Round 3
Group Mean
Controls Over EDI Network 1 9.34 1 9.44 Backup, Disaster Recovery and Contingency Plans
2 8.72 2 8.82
Auditability and Audit Trail 3 8.47 3 8.70 Audit Involvement during System Development
4 8.06 4 8.15
Legal and Audit Evidence 5 7.81 5 7.83 EDI Records Retention 7 7.69 6 7.50 EDI Contracts 6 7.72 7 7.42 Audit Techniques 10 7.09 8 7.30 Third Party EDI Services 8 7.41 9 7.26 Auditor Education and Training 9 7.31 10 7.21 Auditor Skills (Skills required of auditors)
11 6.94 11 7.02
68
Table 6 illustrates the change pattern of issue ranking and rating for the eleven
most important issues in round 2 and round 3. It can be observed from Table 6 that an
opportunity to reflect on round 2 group ratings results in minor changes to the rating and
ranking of the eleven most important issues in round 3. For all top five issues, the mean
group responses increased while the standard deviations decrease. The reason may be
that, being confirmed by peer ratings, the respondents felt more confident of the
importance of these issues and gave higher ratings in the third round.
The more noticeable change occurs in the rating and ranking of the top 20 issues.
In round 3, three of the four issues which are added in round 2 are rated in the 12th to
18th rank. As shown in Table 5, "Professional Support for Practising Auditors" is ranked
12th, "Inconsistent EDI Approaches" is ranked 17th, and "The Network and Ownership
of Data" is ranked 18th, surpassing the importance of some of the original issues in round
2. These indicate that opinions and concerns expressed by expert peers do have significant
effects on the rating of the top 20 issues in the subsequent round. The respondents had
the opportunity to be reminded of certain important issues which they did not think of in
the first place. Further, these results may also indicate that, in addition to its own
advantages, the Delphi process utilized in this study captures some benefits similar to those
of an open discussion or committee approach. Also, because none of the four issues which
were added in round 2 was ranked in the top eleven, it may be concluded that the first
round open-ended questionnaire and the compilation and analysis of existing literature have
been successful in identifying the eleven most important issues.
69
5.8 Movement Towards Consensus
One of the advantages of using the Delphi technique is that it encourages
participants to reach a consensus on the issues of greatest importance. Measuring the
change in the standard deviations of mean ratings between subsequent rounds is an
appropriate method for showing movement towards consensus. A declining mean standard
deviation indicates that participants are reflecting on the issues and revising their ratings
to correspond more closely with their colleagues. For the 21 original issues measured for
importance, the mean standard deviation in round 2 is 1.75 whereas that in round 3 is
1.42. This decline in spread means the scores are clustered more closely about the center
and thus indicates a movement towards consensus.
A further indicator of the trend towards consensus is illustrated in figure 1 - the
statistical summary and graphical display of the participants' rating of the eleven most
important EDI audit issues in round two and round three. The box plot is used because it
is useful for identifying quickly the median, hinges, and outside values of the issue rating
in each round and it makes enables comparison of rating of issues in two different rounds
on the same scale. The median and interquartile range (IQR) are not distorted by extreme
scores like the mean and the standard deviation and thus are suited for summarizing spotty
numbers.
The plot shows the median and the IQR of the first, fourth, and eleventh pair of
issues were unchanged between round two and round three. Moreover, the unusual
responses (*) which appeared in round two of the first and fourth issues disappeared in
70
round three, indicating that the respondents which gave unusual low scores in
relation to the group norm in round two increased their scores to meet the group norm in
round three. In addition, although some unusual responses remained, the IQR and the
whisker length of the third, fifth, seventh, eighth, and tenth pair of issues became
noticeably shorter in round three. Collectively, these incidents indicate a movement
towards group consensus on the importance of EDI audit issues.
While it is possible that additional rounds might have improved the degree of
consensus, it is highly unlikely that perfect agreement would ever be attained because the
study respondents would continue to maintain certain independent views. As happened
in the third round, two respondents maintained exactly the same rating as they did in the
second round for every original issue. In addition, such factors as their position, type of
audit (internal, external), level of experience, industry, and other organizational aspects
may influence the respondents' views of the importance of a given issue.
71
ZL
5.9 Study Participants
This section discusses the various characteristics of the 33 information systems
audit experts recruited to participate in this study. Among the 33 participants, 12 were
requested to participate in all three rounds and 21 were requested to participate in round
2 and round 3 only. Table 7 displays the actual participation of these 33 individuals.
T A B L E 7 - RESEARCH SUBJECTS: PARTICIPATION PATTERN
Number of Rounds Responded Round Responded Number of Respondents 3 Round 1 2 3 10 2 Round 1 2 0
Round 1 3 0 Round 2 3 22
1 Round 1 0 Round 2 0 Round 3 1
TOTAL 33
During the final round, an effort was made to collect some background information
and descriptive data about the individual respondents and organizations reflected in this
research project. The questionnaire for this purpose is exhibited at the end of Appendix
C. Because the round 3 findings are the final research results, it is appropriate to use the
information obtained from the respondents in this round as a basis in providing a better
understanding of selected characteristics of the research subjects.
5.9.1 Organizational Category
T A B L E 8 - RESEARCH SUBJECTS: ORGANIZATIONAL CATEGORY
Category Number of Respondents Percent of Total Public Accounting Firm 7 21.21 Audit Services Bureau 1 3.03 Incorporated Company 13 39.40 Crown Corporation 6 18.18 Government Agency 3 9.09 Academic Institutions 3 9.09
Total 33 100.00
73
The 7 respondents in the accounting firm category are from 6 firms which are
ranked in the eight largest accounting firms in Greater Vancouver26. The respondents in
the incorporated company category are from companies in variety of industries (financial,
food, insurance, manufacturing, and retail).
5.9.2 Position and Primary Area of Responsibility
T A B L E 9-RESEARCH SUBJECTS: TYPE OF POSITION
Type of Position Number of Respondents
Percent of Total
Internal Auditor 24 72.73
External Auditor 9 27.27
Total 33 100.00
5.9.3 Professional Designations
T A B L E 10 - R E S E A R C H SUBJECTS: PROFESSIONAL DESIGNATIONS
Designations Number of Respondents
Percent of 33 Total
Respondents27
Chartered Accountant (CA) 20 60.61 Certificate in Data Processing (CDP) 1 3.03 Certified General Accountant (CGA) 7 21.21 Certified Internal Auditor (CIA) 6 18.18 Certified Information Systems Auditor (CISA)
14 42.42
Certified Management Accountant (CMA) 4 12.12 Others (CIPS's ISP and F C C A L 2 6.06
2 6 As reported by Peter Brow in Business in Vancouver as of June 4, 1991 (p. 29). Firms were ranked on number of chartered accountants employed by the firms.
2 7 Because an individual respondent may possess more than one designation, the sum of the percent numbers in this column may be greater than 100.
74
5.9.4 Areas of Audit Expertise
T A B L E 11 - RESEARCH SUBJECTS: AREAS OF AUDIT EXPERTISE
Area of Expertise Number of Respondents
Percent of 33 Total Respondents
External Audit 20 60.61 Internal Audit 26 78.79 Information Systems Audit 20 60.61 General Audit 11 33.33 Others (Systems Analyst, IS Security, V F M and Fraud Audit)
3 9.09
5.9.5 Level of Audit Experience
T A B L E 12 - RESEARCH SUBJECTS: L E V E L OF EXPERIENCE
Years of Experience Auditing: No. of Respondents
Information Systems Auditing :
No. of Respondents None 2 6 1-3 2 6 4-5 2 7 6-10 8 7
11-15 5 3 16-20 10 3
Over 20 4 1
5.9.6 Background in EDI Technology
5.9.6.1 Engagement in an EDI Project
T A B L E 13 - RESEARCH SUBJECTS: ENGAGEMENT IN A N EDI PROJECT
Engagement in EDI Project
Number of Respondents Percent of 33 Total Respondents
Yes 11 33.33 No 21 63.64
No answer 1 3.03 Total 33 100.00
2 8 Because an individual respondent may specialize in more than one area of auditing, the sum of the percent numbers in this column may be greater than 100.
75
One of the respondents who report as have never been engaged in an EDI project
states that he will be involved in his first EDI project in the next few months. Further,
the respondents who have the experience of being engaged in EDI projects report their
capacities as "(being involved in) system development", "(is responsible for) information
forum", "review of a pilot project in purchasing department", "preliminary discussion with
vendors to streamline account payable process", "organize and monitor the progress (of
the EDI project)", "part of information system strategic plan", "presently involved in
planning stages with a few clients", "auditor", "general control review for financial
statement", "general review of completed work by internal audit", and "internal auditor".
5.9.6.2 Self-report Level of knowledge and Understanding of the EDI Technology
T A B L E 14 - RESEARCH SUBJECTS: L E V E L OF KNOWLEDGE OF EDI TECHNOLOGY
Level Number of Respondents Percent of 33 Total Respondents
Good Working Knowledge 2 6.06 Average Knowledge 24 72.73 Little Knowledge 6 18.18 No answer 1 3.03
Total 33 100.00
5.9.6.3 Primary Source(s) of knowledge and Understanding of the EDI Technology
T A B L E 15 - RESEARCH SUBJECTS: PRIMARY SOURCE(S) OF KNOWLEDGE OF EDI TECHNOLOGY
Source Number of Respondents First hand experience 7 Professional Literature 30 Oral Communication/Discussion Group on EDI
19
Course and Seminar on EDI 8 Others (EDI Software Vendors) 2
76
5.8.6.4 EDI audit manual or guideline
In the background information section, a question was asked to obtain information
on the use and development of EDI audit manual or guideline. As can be expected from
the number of organizations in Vancouver which have EDI systems in operation at this
point of time, 3 firms have the EDI audit manuals in use, 4 firms are in process of
developing ones, and the rest neither has one in use nor is in the process of development.
This information is useful in understanding the progress of the respondents' firms in
standardizing audit procedures for EDI systems.
77
Chapter 6. Conclusions
6.1 Summary of Findings and Conclusions
The primary goal of this research is to identify the most important EDI audit issues
as viewed by the Vancouver IS audit community. The resulting eleven most important
issues are summarized in Table 16.
Table 16 - The Top Eleven Issues in Vancouver
Final Rank Issue of Importance
1 Controls Over EDI Network
2 Backup, Disaster Recovery and Contingency Plans
3 Auditability and Audit Trail
4 Audit Involvement during the System Development
5 Legal and Audit Evidence
6 EDI Records Retention
7 EDI Contracts
8 Audit Techniques
9 Third Party EDI Services
10 Auditor Education and Training
11 Auditor Skills (Skills required of auditors)
The most important issues for this group of audit experts concern controls and
security of EDI systems. This is not unexpected because by nature the fundamental goal
of auditing is to evaluate and ensure that clients'systems are secure and adequately
controlled. These security and control issues are significant for auditors in many aspects.
78
They play an important part in determining a company's ability to continue as a going-
concern and as a consequence, they greatly affect business and audit risks. Also, they
form the main basis for planning the entire audit work in each engagement.
The researcher would also like to make a personal observation here that this theme
of audit concerns seems to be in concordance with general public concerns. Having
opportunities to discuss the EDI technology with people in different occupations, the
researcher noticed that the first and major concern expressed by these people are the
security and reliability of EDI systems. The researcher thus views the resulting audit
concerns as being in the right direction and regards this circumstance as an opportunity
for auditors to keep up with the shifting trends and to continually dignify their profession
by capably fulfilling their important roles as the 'public watchdog' on this relatively new
information technology.
6.2 Generalizeability of Results
Whereas the results of this study provide an indication of the EDI audit priorities
of the Vancouver's IS auditors, the findings may be applicable in other North American
cities. The rationale is that there is no significant diversity in this continent on both
general dimensions such as language, politics, economy, educational systems, and specific
dimension such as organizational culture, industrial activities, advancements in computer
and information technology, and accounting and auditing systems. Therefore, one may
induce that the concerns of Vancouver IS auditors may be comparable with those of their
counterparts in other Canadian and American cities. In addition, as suggested by Kiudorf
79
[1991], in determining whether the most important issues in one specific setting apply in
other settings, it is useful to consider the issues themselves. Of all the EDI audit issues
measured for importance in this study, none appears to be rigidly unique to the Vancouver
audit environment. Most of the issues tend to be universal in nature and can be seen as
concerns for IS auditors in many communities of the auditing profession. These
considerations provide some measures of face validity for the claim that the results may
be generalizeable.
In summary, this study identified the most important EDI concerns in a single
Canadian city. The nature of the findings suggests that these concerns probably apply to
other industrialized environments, but additional research would be required to establish
the actual validity of generalizing the results.
6.3 Limitations of Research Study
Although attempts were made to preserve the rigour of the Delphi research, several
constraints did exist. The limitations of this research are summarized below:
M Non-Random Participant Recruitment: there were no pre-existing lists or databases of qualified candidates for random sampling. Therefore, participant recruitment was accomplished largely through peer recommendations and personal contacts.
iii/ The number of auditors who have first-hand experience with EDI systems: the limited number of organizations in Vancouver which already have EDI systems in operation make it difficult to obtain audit experts with high levels of specific experience in auditing EDI systems.
iv/ Statistical Significance of Issue Ordering: as is true in previous Delphi issues studies, the specific ordering of consecutive issues of importance may not have statistical significance.
80
y I Comparisons with other studies: this project is the first attempt to conduct a Delphi survey research of this topic and nature. Therefore, data from previous studies is generally not available for comparative purposes.
6.4 Directions for Future Research
Whereas this study is a first attempt to identify the priorities and concerns of IS
auditors in a Canadian context, there are a number of additional research projects that
would be useful in improving our understanding of the topic. In particular, it would be
useful to solicit views from IS auditors in industrialized settings other than Vancouver; this
would clarify whether auditors in Canada and other industrialized nations face a common
set of concerns. EDI technology has been largely directed in companies in North
America, Europe, and Australia. Therefore, these areas would be of particular interest in
a supplementary study.
A repetition of the Vancouver study in a few years would also be useful. The
continuing advancements of computer and telecommunication technologies will change the
face of the EDI technology. Also, when EDI systems are more common in Vancouver, and
IS auditors have more experience with EDI systems, it would be of interest to track the
evolution of EDI audit priorities over time. This will be useful in maintaining a current
understanding of important issues and trends. Also, future researchers should be
encouraged to utilize the Delphi methodology in order to preserve the comparability of the
results.
81
BIBLIOGRAPHY
Baker, Carol, (a) "EDI in Business." Accountancy (UK) 107 (Apr 1991): 121-124.
Baker, H . Richard, (b) EDI: What Managers Need to Know about the Revolution in Business Communications. PA: TAB Professional and Reference Books, 1991.
Barrett, Michael J. "External Control." Internal Auditing 6 (Summer 1990): 62-68.
Boughton, Charles M . "Combining the Efforts of Internal and Independent Auditors: Using EDP to Maximize Audit Resources." Woman CPA 49 (January 1987): 26-27.
Brown, Janet. "Alarmed by Inadequate IT Training." The EDP Auditor Journal 1 (1991): 11-12.
Burns, David C. and Sorkin, Horton Lee. "EDI Security and Controls." Bank Management 67 (Feb 1991): 27-31.
Chalmers, Leslie S. "New Technology Introduces New Risks." Journal of Accounting & EDP 5 (Winter 1990): 28-30.
Cowan David, "EFT/EDI-Electronic Age Poses New Legal Problems.", Euromoney (UK). Corporate Finance Supplement. July 1990, pp.27-30.
Craig-Bourdin, Margaret. "The Here and Wow!" CA Magazine (Canada) 122 (Aug 1989): 20-30.
Damyanoff Dan, "EDI and EDIFACT: The Future's Cornerstones.", Global Trade, vol: 111, Issue 6, June 1991, pp. 35,41.
Decker, David L . "Record Retention - A Critical Internal Control." The EDP Auditor Journal 1 (1991): 61-68.
Delbecq Andre L . , Van de Ven Andrew H. , and Gustafson David H. Group Techniques for Program Planning: A guide to Nominal Group and Delphi Processes. Glenview, Illinois: Scott, Foresman and Company, 1975.
Dexter Albert S., Marius Janson A . , Kiudorf Enn, and Laast-Laas Juri. "Key Information Technology Issues in Estonia: Definition and Measurement." Working Paper 92-MIS-001, University of British Columbia. March 1992.
82
Dunmore, David B. "Farewell to the Information Systems Audit Profession." Internal Auditor 46 (Feb 1989): 42-48.
Eckerson, Wayne. "EDI Susceptible to Costly Order Errors. " Network World 7 (Sep 17, 1990): 23-24.
Elliott, Robert K. "Auditing in the 1990s: Implications for Educational and Research." California Management Review 28 (Summer 1986): 89-97.
Emmelhainz, Margaret A . The Impact of Electronic Data Interchange on the Purchasing Process. Ph.D. diss., The Ohio State University, 1986.
Emmelhainz, Margaret A . Electronic Data Interchange: A Total Management Guide. New York: Van Nostrand Reinhold, 1990.
Fowles Jib. Handbook of Futures Research. Westport, Connecticut: Greenwood Press, 1978.
Gallegos, Frederick and Bieber, Douglas W. "Emerging Technology and Information Systems Auditing." Journal of Accounting & EDP 3 (Summer 1987): 47-56.
Gardner Elizabeth, "A direct line Between Buyer and Supplier.", Modern Healthcare, vol: 19, Issue 11, Mar 17, 1989, pp: 26-28.
Gilhooley, Ian. "Emerging Technologies and Auditing: IIA's Advanced Technology Forum." Internal Auditor 44 (Feb 1987): 50-54.
Goldner, Gary. "EDP Auditing with a Small Staff." Journal of Accounting & EDP 2 (Winter 1987): 36-42.
Hansen James V. and Hill Ned C , "Control and Audit of Electronic Data Interchange.", MIS Quarterly. December 1989, pp. 403-413.
Helms, Glenn L . "Career Opportunities for Information Systems Auditors." Journal of Accounting & EDP 2 (Fall 1986): 9-12.
Hinge Kathleen C. Electronic Data Interchange. New York: American Management Association, 1988.
83
Holstrum Gary L . , Mock Theodore J., and West Robert N . The Impact of Technology on Auditing—Moving Into the 21st Century. Altamonte Springs, Florida: The Institute of Internal Auditors Research Foundation, 1988.
Holstrum Gary L . , Mock Theodore J., and West Robert N . , "Information Systems in the 1990s.", Internal Auditor, vol 47, February 1990, pp. 32-37.
Hogarth, Dennis. "How Does IT Al l Tie Together?" CA Magazine (Canada) 119 (Dec 1986): 75-77.
ICAEW (Institute of Chartered Accountants in England and Wales). IT and the Future of the Audit. London: Progress Fine Art Graphic Services Limited, 1989.
Jancura, Elise G.; Lehman, John; Baab, John G.; Gilges, Robert D. ; Kinard, James C ; Overbey, John T.; Robins, Richard S.; Stewart, Trevor R.; Wasserman, Arnold. "Widespread Computerization and Automation of Business Operations-Part II- Impact of New Development on the Profession." Woman CPA 48 (Oct 1986): 26-31.
Kavan Bruce C. The Adoption of Inter-organizational Systems: The Example of Electronic Data Interchange. Ph.D. diss., University of Georgia, 1991.
Kimberley, Paul. Electronic Data Interchange. New York: McGraw-Hill, Inc, 1991.
KiudorfEnn. Key MIS Issues for Management: An Eastern European Perspective. M.Sc. Thesis, University of British Columbia, 1991.
Kothari, Nick. "Auditing's Role in Systems Development." CA Magazine (Canada) 121 (Oct 1988): 55-60.
Lanford H . W. Technological Forecasting Methodologies. N.p.: American Management Association, Inc, 1972.
Lewis, Barry. "Electronic Authorization- The Next Wave In Automation." Journal of Systems Management (March 1989): 28-32.
Lipsett Carol, "EDI implications for security and audit controls.", CIPS Review. August/september 1989, pp.20-21.
Martino Joseph P. Technological Forecasting For Decision Making. New York: Elsevier Science Publishing Co., Inc, 1983.
84
McDonald, Hal. "EDI Implementation Consideration." The EDP Auditor Journal 1 (1990): 43-46.
Moeller, Robert R. "Using a CPA Firm as an Internal Auditor." Journal of Accounting & EDP 2 (Fall 1986): 20-24.
Norris Daniel M . and Waples Elaine, "Control of Electronic Data Interchange Systems.", Journal of Systems Management, vol: 40, Issue 3, March 1989, pp.21-25.
Powell Kevin D. Security and Control of Electronic Data Interchange Systems. M.Sc. Research Project: California State Polytechnic University, Pomona, June 1991.
Rhodes, Wayne. "The Audit Experience." Infosystems 34 (July 1987): 18-20.
Sadhwani Arjan T, Kim Ill-Woon, and Helmerci John, "The Impact of Electronic Data Interchange on Internal Controls.", Journal of Accounting and EDP. Fall 1989, pp. 23-31.
Sadhwani, Arjan T, Ill-Woon Kim, and John Helmerci. "EDI's Effect On Internal Controls." EDPACS XVII no. 1 (July 1989): 1-11.
Schatz Willie. "EDI: Putting the Muscle in Commerce & Industry. " Datamation. 15 March 1988: 56-64.
Staats, E. "Auditing as We Enter the 21st Century-What New Challenges Will Have to be Met." Auditing: A Journal of Practice and Theory 1 (Summer 1981): 1-11.
Tsay Bor-Yi, "System Controls for Electronic Data Interchange.", CPA Journal, vol: 59, Issue 6, June 1989, pp. 70-73.
Utter, Allen C. and Bertram, Timothy R. "Revisiting "A Farewell to the Systems Audit Profession." Internal Auditor 46 (Jun 1989): 70-72.
West Robert N . The Impact of Paperless Systems and Other Technological Changes Upon Auditing. Ph.D. diss., University of Southern California, 1988.
Willits, Stephen D. "Information Technology: Decisions, Decisions, Decisions." CA Magazine (Canada) 123 (Aug 1990): 51-54.
85
Wise Timothy M . , "EDI: Progressing Toward the Paperless Office.", Internal Auditing, vol: 5, Issue 1, Summer 1989, pp. 75-81.
Wise, Timothy M . "Looking at the Systems Development Audit." Internal Auditing 6 (Summer 1990): 69-74.
Wright, J. Benjamin, (a) EDI and American Law : A Practical Guide. Alexandria, Verginia: TDCC:The Electronic Data Interchange Association, 1990.
Wright, J. Benjamin. The Law of Electronic Commerce. Boston: Little, Brown and Company, 1991.
Wright Margaret, (b) "Accounting in a Paperless Office.", Australian Accountant (Australia). vol:60, Iss:7, August 1990, pp. 44-48.
Yang David C , "The Effect of EDGAR on Auditing Practice.", Ohio CPA Journal. Vol: 49, Issue 4, Winter 1990, pp.49-50.
86
APPENDIX A
ROUND 1
DELPHI QUESTIONNAIRE
FOR
AUDIT EXPERTS
87
March 16, 1992
tacuity of Commerce & Business Administration
James Topham Peat Marwick Thome P.O. Box 10426 Pacific Center 777 Dunsmuir Street Vancouver, B.C. V7Y 1K3
The University of British Columbia 2053 Main Mall Vancouver, B.C. Canada V6T 1Z2 Tel: (60-;) 822-8500 Tax: (604) 822-8489
Dear Mr. Topham:
UBC, in conjunction with the Vancouver Chapters of the EDPAA and the IIA, is conducting a Delphi study to investigate the impact of Electronic Data Interchange (EDI) on the audit process. The goal of our project at this point is to identify on the basis of expert opinions the key issues of EDI audit impact. You have been selected to participate in this study because of your considerable expertise in information systems auditing and interest in EDI control and audit issues.
The ultimate objective of this Delphi study is to obtain consensus (or response stability) from the panel of experts on important EDI audit issues. The issues indicated by you and your peers will be combined with those stated in the literature to form the list of important issues to be rated by a group of Vancouver's IS auditors in the next questionnaire. Attaining the stated objective normally requires two or three iterations. Your participation therefore will be limited to two or three questionnaires.
We believe you will benefit from participation in this research project. As you are aware, EDI is an increasingly popular technology which has the potential to significantly change business information systems and the practice of auditing. This potential impact makes this study worthy of your time and attention. Furthermore, you will benefit by being able to compare and contrast your opinions with those of other experts in your field. In addition, the knowledge of important EDI audit issues will assist you and your firm in directing efforts and resources to the most critical areas. Therefore, we ask your assistance in identifying important EDI audit issues by completing the attached open-ended questionnaire.
May we have your reply by March 27? Please be assured that your individual responses will be kept strictly CONFIDENTIAL. You will of course be able to receive the eventual results of the study which will not identify specific participants. If you have any questions, please feel free to contact Professor AI Dexter at 822-8380. We thank you very much for your cooperation.
Sincerely, Sincerely, Sincerely,
James W. Topham President, EDPAA Vancouver Chapter
Angela M. Louie President, IIA Vancouver Chapter
Albert S. Dexter Associate Professor UBC Faculty of Commerce
88
NOTES TO THE RESPONDENTS:
1) The term "auditors", on its own, is used to signify all types of auditors. References to internal, external, and specific types of auditors, are made explicitly in the questionnaire.
2) Although there are variations in the definition of EDI, the following definition is adopted for the purposes of this research project:
Electronic data interchange (EDI) is the intercompany, computer-to-computer exchange of business documents in standard formats. Through EDI, such common business forms as invoices, bills of lading, and purchase orders are transformed to a standard data format and electronically transferred between trading partners.
3) EDI should be differentiated from Electronic Fund Transfer (EFT) which refers to the transfer of value electronically from buyer to seller as assisted by a financial intermediary, usually a bank. Because of the complementary objectives of EDI and EFT (the elimination of paper in business transactions), many organizations involved in electronic payments are proponents of EDI.
89
1992 DELPHI STUDY OF K E Y EDI AUDIT ISSUES Round 1
Your Top Five EDI Audit Issues
Your Name (Optional)
Please list what you feel are the five most important issues in auditing EDI systems. Please list the audit issues in order of their relative importance and kindly give a brief rationale/description of each issue. The importance of the issue should be considered in terms of its impact on different aspects of the audit process.
1) Issue:
Rationale:
2) Issue:
Rationale:
3) Issue:
Rationale:
4) Issue:
Rationale:
5) Issue:
Rationale:
Thank you very much for your cooperation.
90
APPENDIX B
ROUND 2
DELPHI QUESTIONNAIRE
FOR
AUDIT EXPERTS
91
April 15, 1992
Andy Campbell Internal Audit MacMillan Bloedel 925 West Georgia Street Vancouver, B.C. V6C 3L2
Dear Mr. Campbell:
h acuity of Commerce & Business Administration
The Universily of British Columbia 2053 Main Mall Vancouver, BC. Canada V6T 1Z2 Tel. (604) 822-8500 Fax: (604) 822-8489
UBC, in conjunction with the Vancouver Chapters of the EDPAA and the HA, is conducting a Delphi study to investigate the impact of Electronic Data Interchange (EDI) on the audit process. You have been selected to participate in this study because of your considerable expertise in information systems auditing.
The objective of this Delphi study is to obtain consensus from the panel of experts on the key issues of EDI audit impact. Attaining this objective normally requires two or three iterations. Your participation therefore will be limited to two or three questionnaires. The next questionnaire will incorporate the results from this questionnaire, and will be sent to you in about one week after the results of this round have been received and analyzed. In order to have your responses included in the upcoming round, please have your responses mailed or fax (822-8489 attn. Al Dexter) to us by April 30.
We believe you will benefit from your participation in this research project. As you are aware, EDI is an increasingly popular technology which has the potential to significantly change business information systems and the practice of auditing. This potential impact makes this study worthy of your time and attention. Furthermore, you will benefit by being able to compare and contrast your opinions with those of other leading experts in your field. In addition, the knowledge of important EDI audit issues will assist you and your firm in directing efforts and resources to the most critical areas.
Please be assured that your individual responses will be kept strictly CONFIDENTIAL. You will of course be able to receive the eventual results of the study which will not identify specific participants. If you have any questions, please feel free to contact professor AI Dexter at Tel. 822-8380. We thank you very much for your cooperation.
Sincerely, Sincerely, Sincerely,
James W. Topham President, EDPAA Vancouver Chapter
Angela M. Louie President, HA Vancouver Chapter
Albert S. Dexter Associate Professor UBC - Commerce
9Z
1992 DELPHI STUDY OF KEY EDI AUDIT ISSUES Round 2
Your Rating of EDI Audit Issues
Your Name
The issues listed below were obtained by combining the results from round 1 survey with the issues frequently discussed in previous studies and professional reviews. We are interested in determining the degree of importance of these issues based on your opinion. Please indicate your opinion by rating each issue on a scale of 1 to 10, where 10 indicates that the issue deserves the highest priority from the auditing profession and 1 indicates that the issue has the lowest priority.
Please use the space provided in front of the statement of each issue to assign your ratings. The more important the issue, the higher the rating. You can assign the same number to more than one issue.
RATING SCALE: Not Moderately Critically Important Important Important 1 2 3 4 5 6 7 8 9 10
Your Rating Key EDI Audit Issues and their Rationale:
Backup, Disaster Recovery and Contingency Plans Rationale: With a company's increasing reliance on EDI network for operational and financial services, auditors must assure that adequate measures for backup and disaster recovery exist. These measures form an indicator of the company's ability to continue as a going-concern.
Controls Over EDI Network Rationale: Because weak controls can cause significant financial loss to EDI trading partners, auditors must assure that controls over the EDI network such as access controls, authentication controls, transmission controls, and controls over mailbox are in place.
Relationship Among A Company's Auditors Rationale: In an EDI environment, information systems grow in complexity, external auditors may have to rely more on internal auditors, and information systems auditors will be requisite members of audit teams. Suitable audit approaches must be developed to promote and make the best use of this inter-relationships.
93
Your Rating Key EDI Audit Issues and their Rationale:
Audit Tools " Rationale: The increased complexity of intercompany automated paperless
EDI transactions makes it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. More powerful tools must be developed to match the growth in sophistication of clients' systems.
Auditor education and training " Rationale: Auditors practice today in a significantly different environment
from that in the past. The educational curriculum and training requirements for auditors must be updated to reflect technological change and to embody the types of knowledge and the proficiency required of auditors to maintain the profession.
Audit Reporting (Periodic versus On-Line) Rationale: Because the high speed of EDI transactions makes information obsolete within a very short time, there is a need for more frequent accounting disclosure. Auditors will have to adjust their reporting procedures to meet the need for timely and accurate information.
Legal and Audit Evidence " Rationale: The absence of paper documents and signatures in EDI systems
implies the absence of important legal and audit evidence such as proof of authorization and other documentation in paper form. Auditors must assure that equivalent and legally acceptable forms of evidence are established and properly incorporated into EDI systems.
The Changing Roles of Auditors Rationale: In an EDI environment, the role of auditing will have to change to meet both the new demands of clients and the needs of the profession. Failure by auditors to assume suitable new roles adequate to meet such demands may lead to the decline of the profession.
Audit Focus (Substantive versus Compliance Testing) Rationale: Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied. Therefore, efforts are needed to establish appropriate audit approach.
Auditor Skills (Skills required of auditors) Rationale: Because of rapid expansion in the extent, scope, and types of information to be audited in an EDI system, auditors need to be specially trained to acquire certain skills that enable them to maintain a high standard of practice in such an environment.
94
Your Rating Key EDI Audit Issues and their Rationale:
Audit Scope (Boundary of Audit) Rationale: Paperless intercompany transactions create a "boundaryless" information system environment, and auditors may be required to audit beyond the traditional boundaries of clients' systems. Therefore, audit scope and responsibilities need to be pre-determined and agreed upon by the parties involved.
Collaboration Among Auditors of EDI Parties Rationale: An EDI network involves not only auditors of the company but also auditors of its trading partners and of third parties. Because the security of an EDI system depends on those of others in the network, these auditors are inter-dependent. It is vital that the roles of each party's auditors be determined and the rules of collaboration be established.
Audit Involvement during the System Development Rationale: Auditors must take a proactive approach and get involved early in EDI projects to ensure that proper controls and auditability features are designed and incorporated into the systems. Guidance must be established to assist auditors to accomplish this task.
Third Party EDI Services Rationale: Different types of EDI networks have different implications for the participating companies and their auditors. Therefore, auditors must evaluate their clients' third party EDI service firms in terms of responsibilities, resources, and abilities to provide, on an ongoing basis, reliable and secure services per contract terms.
Audit Risk Assessment Rationale: Audit risks in an automated open EDI network are significantly different from those in a closed system because EDI involves more parties and more diverse exposures. A distinctive approach of audit risk assessment is required, and guidance must be established.
Audit Responsibility in Evaluating Controls Rationale: In open EDI networks, auditors, especially internal auditors, may be held responsible for the review and evaluation of external controls (in addition to internal controls). In such circumstances, it is necessary that guidance be developed and standards be established to lead the practice.
95
Your Rating Key EDI Audit Issues and their Rationale:
Auditability and Audit Trail Rationale: Because substantial reduction of paperwork means a possible loss of the audit trail and the consequent inability to conduct an audit, auditors must take actions to ensure the auditability of the EDI system and the availability and the adequacy of an audit trail in proper form.
Audit Techniques Rationale: Traditional audit techniques may no longer apply in an EDI environment. Auditors need to develop effective techniques to enable them to describe, evaluate, and test an intercompany information network that includes very few paper documents.
Timing of Audit Tests Rationale: Because of the high volume of transactions and the velocity of electronic processing, EDI transactions may have to be reviewed as they occur. Consequently, the audit process must be modified and specific audit standards established.
EDI Contracts Rationale: Because EDI contracts are the basis for the company's future dealings with and liabilities to EDI partners, auditors must ensure that terms, services, and responsibilities of each EDI party are clearly defined and the contracts are inclusive and enforceable.
EDI Records Retention Rationale: Ineffective records management practices can lead to exposures such as the loss of critical data files; therefore, possible litigation costs and penalties could result. Guidance on EDI records retention must be established and organizational and individual accountability must be clearly defined.
96
Additional Issues
In the space provided, please feel free to indicate any additional EDI audit issues which you think important and deserve consideration from the study group. This will help us ensure that the list of issues which we will send to you in the next round is most comprehensive.
Your Rating Additional EDI Audit Issues and their Rationale:
Issue: Rationale:
_ Issue: Rationale:
_ Issue: Rationale:
Thank you very much for your cooperation.
97
APPENDIX C
ROUND 3
DELPHI QUESTIONNAIRE
FOR
AUDIT EXPERTS
98
, 1992
Faculty of Commerce & Business Administration
Gait Arthur
The University of British Columbia 2053 Main Mall Vancouver, B.C. Canada V6T 1Z2 Tel: (604)822-8316 Fax:(604) 822-8521
Partner, Deloitte & Touche PO Box 49279, Four Bentall Centre 200-1055 Dunsmuir Street Vancouver, B.C. V7X lp4
Dear Mr. Arthur:
Thank you very much for your participation in the previous rounds of the Delphi study on EDI Key Audit Issues. This is the THIRD AND FINAL ROUND of the study. We have now established a preliminary list of the most important EDI audit issues based upon the responses from you and your audit expert peers. In order to complete this study, we need your assistance one final time so that we can determine the final ranking of the issues. Detailed instructions for this round are provided in the attached questionnaire.
Could you please have your responses mailed or faxed (822-8489 attn. A l Dexter) to us by4he" J22fltT33f"M«y'. We would like to reassure that all individual responses will be kept strictly CONFIDENTIAL. If you have any questions, please feel free to contact Professor Al Dexter at Tel. 822-8380. We sincerely wish to thank you in advance for your support and cooperation.
We hope that you have found your participation in this research project to be a meaningful experience. If you are interested in having a copy of the final results of the study, please so indicate on the questionnaire. We will be pleased to forward it to you upon completion. Again, thank you very much for making this research project feasible.
Sincerely,
President, EDPAA Vancouver Chapter
Angela M . Louie President, IIA Vancouver Chapter
Albert S. Dexter Associate Professor UBC
99
1992 DELPHI STUDY OF K E Y EDI AUDIT ISSUES Round 3
Your Rating of EDI Audit Issues
Your Name
The following list of EDI audit issues is presented in the order of importance as determined from the previous round of Delphi questionnaires. The average group rating and your original rating are also provided. Please review these ratings and the accompanying rationale for each issue. Make a final rating decision and record it in the blank space. Please remember that each issue is to be rated on a scale from 1 to 10, where 10 indicates that the issue deserves the highest priority from the auditing profession and 1 indicates that the issue has the lowest priority (the more important the issue, the higher the rating).
RATING S C A L E : Not Important 1 2 3
Moderately Critically Important Important
4 5 6 7 8 9 10
Average Group Rating
9.3
Your Original Rating
Your Final Rating
Key EDI Audit Issues and their Rationale:
Controls Over EDI Network Rationale: Because weak controls can cause significant financial loss to EDI trading partners, auditors must assure that controls over the EDI network such as access controls, authentication controls, transmission controls, and controls over mailbox are in place.
8.7 Backup, Disaster Recovery and Contingency Plans Rationale: With a company's increasing reliance on EDI network for operational and financial services, auditors must assure that adequate measures for backup and disaster recovery exist. These measures form an indicator of the company's ability to continue as a going-concern.
100
Average Your Your Key EDI Audit Issues and their Rationale: Group Original Final Rating Rating Rating
8.5 Auditability and Audit Trail Rationale: Because substantial reduction of paperwork means a possible loss of the audit trail and the consequent inability to conduct an audit, management and auditors must take actions to ensure the auditability of the EDI system and the availability and the adequacy of an audit trail in proper form.
8.1 Audit Involvement during the System Development Rationale: Auditors must take a proactive approach and get involved early in EDI projects to ensure that proper controls and auditability features are designed and incorporated into the systems. Guidance must be established to assist auditors to accomplish this task.
7.8 Legal and Audit Evidence Rationale: The absence of paper documents and signatures in EDI systems implies the absence of important legal and audit evidence such as proof of authorization and other documentation in paper form. Auditors must assure that equivalent and legally acceptable forms of evidence are established and properly incorporated into EDI systems.
7.7 EDI Contracts Rationale: Because EDI contracts are the basis for the company's future dealings with and liabilities to EDI partners, auditors (especially internal) should be involved in negotiation process to help ensure that terms, services, and responsibilities of each EDI party are clearly defined and the contracts are inclusive and enforceable.
7.7 EDI Records Retention Rationale: Ineffective records management practices can lead to exposures such as the loss of critical data files; therefore, possible litigation costs and penalties could result. Guidance on EDI records retention must be established and organizational and individual accountability must be clearly defined.
101
Average Your Your Key EDI Audit Issues and their Rationale: Group Original Final Rating Rating Rating
7.4 Third Party EDI Services Rationale: Different types of EDI networks have different implications for the participating companies and their auditors. Therefore, auditors must evaluate their clients' third party EDI service firms in terms of responsibilities, resources, and abilities to provide, on an ongoing basis, reliable and secure services per contract terms.
7.3 Auditor education and training Rationale: Auditors practice today in a significantly different environment from that in the past. The college and university educational curricula and training requirements for auditors must be updated to reflect technological change and to embody the types of knowledge and the proficiency required of auditors to maintain the profession.
7.1 Audit Techniques Rationale: Traditional audit techniques may no longer apply in an EDI environment. Auditors need to develop effective techniques to enable them to describe, evaluate, and test an intercompany information network that includes very few paper documents.
6.9 Auditor Skills (Skills required of auditors) . . Rationale: Because of rapid expansion in the extent, scope, and types of information to be audited in an EDI system, auditors need to be specially trained to acquire certain skills that enable them to maintain a high standard of practice in such an environment.
6.6 Audit Focus (Substantive versus Compliance Testing) Rationale: Although there is a general agreement that the focus and the types of audit tests need to change to suit highly automated, paperless intercompany EDI networks, there is no consensus on the direction or specific procedures that should be applied. Therefore, efforts are needed to establish appropriate audit approach.
102
Average Your Your Key EDI Audit Issues and their Rationale: Group Original Final Rating Rating Rating
6.5 Audit Scope (Boundary of Audit) Rationale: Paperless intercompany transactions create a "boundaryless" information system environment, and auditors may be required to audit beyond the traditional boundaries of clients' systems. Therefore, audit scope and responsibilities need to be pre-determined and agreed upon by the parties involved.
6.5 Audit Risk Assessment Rationale: Audit risks in an automated open EDI network are significantly different from those in a closed system because EDI involves more parties and more diverse exposures. A distinctive approach of audit risk assessment is required, and guidance must be established.
6.4 Audit Responsibility in Evaluating Controls Rationale: In open EDI networks, auditors, especially internal auditors, may be held responsible for the review and evaluation of external controls (in addition to internal controls). In such circumstances, it is necessary that guidance be developed and standards be established to lead the practice.
6.3 Collaboration Among Auditors of EDI Parties Rationale: An EDI network involves not only auditors of the company but also auditors of its trading partners and of third parties. Because the security of an EDI system depends on those of others in the network, these auditors are interdependent. It is vital that the roles of each party's auditors be determined and the rules of collaboration be established.
6.3 The Changing Roles of Auditors Rationale: In an EDI environment, the role of auditing will have to change to meet both the new demands of clients and the needs of the profession. Failure by auditors to assume suitable new roles adequate to meet such demands may lead to the decline of the profession.
103
Average Your Your Key EDI Audit Issues and their Rationale: Group Original Final Rating Rating Rating
6.3 Timing of Audit Tests Rationale: Because of the high volume of transactions and the velocity of electronic processing, EDI transactions may have to be reviewed as they occur. Consequently, the audit process must be modified and specific audit standards established.
6.2 Audit Tools Rationale: The increased complexity of intercompany automated paperless EDI transactions makes it more difficult, or in some cases impossible, for auditors to test and evaluate network systems with existing audit tools. More powerful tools must be developed to match the growth in sophistication of clients' systems. . . .
5.5 Relationship Among A Company's Auditors Rationale: In an EDI environment, information systems grow in complexity, external auditors may have to rely more on internal auditors, and information systems auditors will be requisite members of audit teams. Suitable audit approaches must be developed to promote and make the best use of this inter-relationships.
4.3 Audit Reporting (Periodic versus On-Line) Rationale: Because the high speed of EDI transactions makes information obsolete within a very short time, there is a need for more frequent accounting disclosure. Auditors will have to adjust their reporting procedures to meet the need for timely and accurate information.
Finally, if your final rating for a specific issue is significantly (5 to 9 scores) different from the group average, please briefly describe your rationale for the final rating on a separate sheet and return with the questionnaire. For example, suppose the group average on a particular question was 8.4, but your rating of the question was 2.5, then this would be significantly different from the average.
104
ADDITIONAL ISSUES
The following are four additional issues from the previous Delphi round. Please rate their importance by using the same scale as the above issues.
RATING S C A L E : Not Moderately Critically Important Important Important 1 2 3 4 5 6 7 8 9 10
Form of Audit Assurance Rationale: The closer interrelationships established between trading partners in an EDI network will affect business and financial risks. Therefore, auditors should reevaluate the types of assurance required by the public and formulate suitable audit procedures and related opinions to satisfy these needs.
Professional Support Rationale: The auditor have to face many new issues when auditing an EDI system. Thus, professional organizations such as EDPAA, IIA, and CICA should take a proactive approach to providing reference materials and training opportunities to help practising auditors understand and deal effectively with the EDI environment, its risks and control measures.
Inconsistent EDI Approaches Rationale: Inconsistent EDI approaches (used by various EDI parties) can lead to operational and administrative problems resulting in missed business opportunities, additional costs, and weaken internal controls. Therefore, auditors must be aware of inconsistencies and provide direction to management.
The Network and Ownership of Data Rationale: As EDI systems develop, the sharing of common data/programs will increase and the information flow that the auditor needs to understand will change. Auditors must take part in defining information flows and boundaries to data ownership. (This will also help draw legal boundaries among parties in a large integrated EDI system).
105
1992 DELPHI STUDY OF KEY EDI AUDIT ISSUES
Background Information
Your Name (Optional)
The following information is needed to help us with the statistical analysis of the data you will provide us in the questionnaires and in making comparisons among different groups of auditors.
1) What is your present job t i t le and primary area of responsibility?
2) Are you certified for the following professional designations?
CA CDP CGA CIA CISA CMA
Other(please specify)
3) Your area(s) of audit expertise: External audit Internal audit Information systems audit General audit Others (please specify)
4) Years of experience in
auditing
information systems auditing
Other computer related positions
5) Have you ever been engaged in an EDI project? No Yes. Please specify in what capacity:
106
6) How do you judge your level of knowledge and understanding of the EDI Technology? Good working knowledge Average knowledge Little knowledge
7) Please indicate the primary source(s) of your knowledge and understanding of the EDI Technology:
First hand experience Professional literature Popular literature Oral communication
_Others(please specify)
8) Does your firm or audit department have an EDI audit manual or guideline in use, or is it in the process of developing one?
EDI audit manual or guideline in use Yes No In process of developing Yes No
9) Would you like to have a copy of the results?
Yes No
Thank you for your help in completing this study!
107
APPENDIX D
ROUND 1 RESULTS
108
Table 1.1 - Controls Over EDI Networks
Frequency Issue Rationale 12 Control over
mailbox "If just one-to-one relationship with partner, must be certain that 3rd party mailbox service is controlled. Once you start dealing with many suppliers, how do you know all the different mailboxes are secure. Audit reports issued on security/control of sources are limited by their very nature, i.e., if a report is issued to-day, the same controls may not exist or be overridden tomorrow. Can you ever be sure of control- a matter of trust?"
Data confidentiality
"Need to be sure data are sent (only) to correct destination/transmitted timely and data held in service firm backup file are secure. "
Financial Controls
"Weak controls can cause financial loss to participants. "
System security not compromised?
"Need assurance that the transmission channel cannot be used as a backdoor into (company) system."
Assess (access) controls to the EDI (network) environment
"Controls over such areas as approval of payment and receipt of goods will be dependent on access to the EDI systems. For example, transactions for receipt of goods could be fraudulently approved if the access controls to the system are weak."
"Controls should ensure adequate password control to prevent unauthorized access to the system (purchasing, receiving)."
Communications security
Concern over competition reviewing transactions. If you have a value added network, and communication lines go down, who is Liable, if purchase order not filled, or data lost? Who is liable if business loses result from unauthorized changes to data? Can unauthorized access be identified?
109
Frequency Issue Rationale Integrity of data "Transmission errors-Data or transactions may
be lost, duplicated, inaccurately transmitted or altered during transmission.
Application errors-our EDI partner may omit, duplicate or inaccurately send or receive data or transactions. "
How do we confirm that orders received are valid, complete and accurate?
"How do we know that the order was originated by a legitimate and authorized person/entity? How do we know transaction details are complete and accurate (i.e. shipped to legitimate location)."
Authentication of trading partners
"Errors in authentication could result in misappropriation of funds through transfers to fraudulent partners. Goods shipped to fraudulent trading partners could be misappropriated in the same manner."
Trading Partner's security
"The degree to which a trading partner secures his end of the network has implications regarding confidentiality, accuracy and completeness of data. Our clients rely on their trading partners' security."
Adequate communication controls with supplier
"The controls should ensure that the transmission of the description, quantity and price are correct."
Accurate, complete transmission of accurate, complete data
"Essential that only accurate, complete data are put into the pipeline and equally essential that those data are transmitted accurately and completely. "
110
Table 1.2 - EDI Contracts (Trading Partner Agreement)
Frequency Issue Rationale
6 Contract with supplier /partner
"The basis for all future dealings with your EDI partner. The contract must be all inclusive and allow for technology changes. "
Clarity of the trading agreement
"Who is responsible for what and when, what standards are to be followed; recourse available to any partner-will identify the exposures to my clients and my exposures as an auditor."
Organizational responsibility
"Responsibility for controls has to be defined and agreed to by the trading partners and the network supplier. "
Partnership Agreements
"Procedures agreed as to cohort constitutes an offer, acceptance, receipt and acknowledgement of documents. "
Agreement on contractual arrangements
"Disagreement on responsibility for loss or assumption of risks may impose unfair difficulties on one or more of the partners if there is no agreement in place. "
Written agreement to cover all significant issues
"To be enforceable, terms, service, responsibilities must be defined."
Ill
Table 1.3 - Auditability and Audit Trail
Frequency Issue Rationale 4 Auditability "An issue within your company, your partner and
with the third party service. Must be absolutely certain documents/data are sent and received accurately. "
Adequacy of audit trail
"If an adequate audit trail does not exist it is difficult to determine if the application controls are working and it may expose the Railway to potential legal liability (no backup to support a transaction received or sent)."
Audit evidence and records
"The maintenance of a complete record of all transactions and an access log of who has been on the system.
Access to and retention of EDI data files
"External auditors will require access to EDI data files for attest purposes. Access to the data may be limited if the files are maintained on a third party network (VAN or WAN).
As paper forms and hardcopy printouts are phased out the electronic data files will be the only form of supporting documentation. The transactions in these files should be retained for the complete fiscal period."
112
Table 1.4 - Backup, Disaster Recovery and Contingency Plans
Frequency Issue Rationale 4 Backup, Disaster
Recovery and contingency planning
"With the Railway's increasing reliance on EDI for operational and financial services, operations will be greatly hindered without adequate backup for the network. "
"Assets in the form of account receivable data may be lost if the data is not backed-up on a regular basis and stored in a secure location."
"A company's ability to continue as a going-concern may be in jeopardy if the EDI network is lost. It will be imperative to have an alternate hot-site for recovery of the system and data to allow continuation of the normal business operations."
"Contingency plans if EDI not available, or computer not available. "
Table 1.5 - Third Party EDI Services
Frequency Issue Rationale 3 Third party EDI
services "Third party EDI companies need to provide a secure, reliable and available service. "
Reliability of service "Need to know service firm has the financial resources to provide service per contract terms on an ongoing basis and has made adequate provision for trouble shooting, client communication, system upgrading capabilities. "
Type of EDI network "Different types of EDI networks have different implications for an auditor. Some are more secure than others, some involve more intermediaries than others."
113
Table 1.6 - Legal and Audit Evidence
Frequency Issue Rationale 3 Court acceptance/Dispute
mechanism "Will the electronic signatures be accepted in the courts? Only time will tell! A dispute mechanism must be set up to deal with issues that come up."
Legal implications of transmitting waybills, bills of lading, and purchase orders electronically. Are these documents enforceable?
"Current contract law does not address EDI transactions."
Do electronic contracts impose the same rights and obligations as a written contract.?
"In the absence of the normal contract process which involves offer and acceptance, how will disputes over amounts, quantities, terms etc. be resolved (i.e. no signatures). When is a contract formed?
Table 1.7 - EDI Records Retention
Frequency Issue Rationale 2 Access to and retention
of EDI data files "External auditors will require access to EDI data files for attest purposes. Access to the data may be limited if the files are maintained on a third party network (VAN or WAN).
As paper forms and hardcopy printouts are phased out the electronic data files will be the only form of supporting documentation. The transactions in these files should be retained for the complete fiscal period."
Retention of electronic or hard copy information
"What will stand up in a court of law (evidence)? What are legal and regulatory requirements (e.g. tax department)? Who should keep this information (sender or receiver) and for how long?"
114
APPENDIX E
ROUND 2 RESULTS (THE ORIGINAL 21 ISSUES)
115
( s a -Backup Trail Involve Evidence Contract Record
8 10 8 9 9 8 9 10 9 9 8 5 6 8 9 9 6 9 7 5 6 10 10 10 10 6 10 5 9 8 10 8 9 8 9 10 9 10 7 9 7 7 10 10 10 10 9 10 10 10 9 8 9 7 8 9 9 9 7 6 6 5 4 10 10 10 8 10 10 10 10 8 8 8 6 8 6 8 10 10 9 9 5 10 8 9 9 7 6 8 . 9 6 8 8 7 8 8 7 10 9 9 9 9 8 8 9 6 7 8 6 8 7 10 10 10 9 8 9 7 10 9 9 8 8 7 8 10 7 8 9 7 6 7 10 7 10 1 10 5 5 8 10 9 10 10 6 10 10 10 9 6 8 7 7 10 10 10 10 10 10 10 10 9 10 8 10 8 8 10 9 8 8 7 8 7 9 9 8 9 8 8 8 9 6 8 8 9 9 9 10 10 5 6 6 6 5 10 8 9 8 8 10 10 10 10 10 10 10 8 7 9 7 6 8 3 9 8 8 5 3 8 7 9 6
Rank 1 2 3 4 5 6 7 Total Score 299 279 271 258 250 247 246 Respondent 32 32 32 32 32 32 32 Mean Score 9.34 8.72 8.47 8.06 7.81 7.72 7.69
Stdev. 0.96 1.35 1.66 1.69 1.70 1.55 1.67
116
partes Educate Technique Skill Focus Scope Risk
7 7 7 7 6 6 2 8 8 8 7 6 6 6 6 6 6 8 7 6 6 10 7 5 7 5 5 5 7 7 9 7 8 8 7 8 6 4 6 3 4 2 10 9 10 7 9 8 10 7 6 8 7 3 8 4 5 5 6 5 3 4 6 8 7 3 3 5 2 5 6 9 7 10 9 10 3 8 8 6 5 5 8 5 7 8 9 8 9 7 7 4 8 8 7 5 6 7 9 8 8 8 8 9 9 8 5 6 5 6 3 8 7 6 7 8 8 10 6 8 8 8 8 8 5 7 9 6 7 6 7 2 7 7 7 7 6 5 3 7 3 8 10 7 4 9 6 10 8 7 8 9 8 7 10 10 10 10 10 10 10 8 8 9 8 10 9 9 7 6 7 8 7 5 8 8 8 7 8 5 7 6 6 8 8 8 8 7 8 4 7 8 5 4 7 3 8 8 7 5 8 5 8 8 10 5 8 8 8 9 6 3 3 3 6 4 7 10 9 7 9 8 10 8
Rank 8 9 10 11 12 13 14 Total Score 237 234 227 222 212 209 208 Respondent 32 32 32 32 32 32 32 Mean Score 7.41 7.31 7.09 6.94 6.63 6.53 6.50
Stdev. 1.77 1.47 1.76 1.66 2.03 2.34 2.06
117
Response Colla Roles Timing Tools Relation Report
4 5 7 7 7 3 3 5 8 5 7 7 6 5 5 7 5 6 6 8 3 7 5 5 5 5 5 5 5 8 7 4 5 6 3 5 3 2 2 7 2 2 10 8 5 10 8 8 8 7 2 5 1 6 3 1 5 4 5 6 5 4 4 3 4 3 3 3 5 1 5 8 9 8 7 9 3 5 7 7 5 8 5 3 7 7 7 8 8 8 6 8 6 7 8 7 6 6 8 9 8 8 9 8 7 8 8 5 6 5 3 2 6 8 7 6 6 4 7 7 8 7 7 6 4 4 8 4 5 8 6 3 2 5 4 5 5 5 5 2 4 8 8 9 6 8 6 7 7 8 7 7 8 6 10 6 4 10 6 6 6 8 6 7 9 9 6 8 8 7 4 6 7 5 3 6 6 6 7 5 4 6 5 6 9 5 6 6 6 5 6 8 7 6 5 5 9 8 8 5 5 5 5 9 8 10 5 7 7 5 5 1 4 2 4 4 2 7 10 9 8 5 7 2
Rank 15 16 17 18 19 20 21 Total Score 206 202 201 200 199 176 137 Respondent 32 32 32 32 32 32 32 Mean Score 6.44 6.31 6.28 6.25 6.22 5.50 4.28
Stdev. 1.78 2.05 1.89 2.19 1.34 1.82 2.00
118
APPENDIX F ROUND 3 RESULTS
(THE ORIGINAL 21 ISSUES)
119
ÇioT Backup Trail Involve Evidence Record tract 9 9 8 9 9 8 8 10 9 9 8 8 8 7 9 9 6 9 7 6 5 10 10 10 10 7 6 9 9 9 10 8 8 8 8 10 9 10 7 9 7 7 10 10 10 10 9 9 8 10 9 9 9 7 8 8 9 9 8 8 6 4 5 10 10 10 8 10 9 8 9 8 8 8 7 6 8 9 10 10 8 8 9 6 8 9 9 7 7 9 8 8 8 8 8 8 8 8 10 9 9 9 8.5 8 8 9 6 7 8 7 7 7 9 9 9 9 8 8 9 10 9 9 8 8 8 7 10 8 8 9 7 7 6 10 7 9 1 9 5 5 8 10 10 10 10 10 5 10 10 10 7 8 7 8 10 9 9 10 10 10 8 10 9 9 8 8 8 6 10 9 8 8 8 8 8 9 9 8 8 9 8 8 10 10 6 7 7 6 6 10 9 9 8 8 10 10 10 9 9 9 8 7 8 9 8 7 8 3 8 9 9 5 8 8 7 6 9 10 9 9 9 8 7 6 9 9 9 8 8 5 9
Rank 1 2 3 4 5 6 7 Total Score 311.5 291 287 269 258.5 247.5 245 Respondent 33 33 33 33 33 33 33 Mean Score 9.44 8.82 8.70 8.15 7.83 7.50 7.42
Stdev. 0.65 1.09 1.09 1.52 1.30 1.44 1.33
120
Technique par̂ yS Educate Skill Risk Response Scope
7 7 7 7 2 5 6 8 8 8 7 6 6 7 6 6 6 8 6 5 6 6 8 T
s 7 5 7 6 9 7 7 7 7 5 7 4 8 6 6 3 5 4 10 10 9 7 9 9 8 8 7 6 6 4 6 7 6 5 5 5 6 5 4 6 8 7 5 5 4 3 7 6 8 9 5 5 9 7 8 8 6 5 6 6 9 7 8 8 7 7 7 8 5 8 7 7 8 6 8 8 8 7.5 8 7.5 9 6 8 6 6 8 8 3 7 7 7 8 6 7 9 8 8 8 8 7 7 5 7 8 6 6 7 8 2 7 7 7 6 7 5 3 10 4 8 6 7 5 9 7 10 7 7 7 7 7 10 10 10 10 10 10 10 8 7 7 8 8 7 7 7 7 6 7 7 8 5 8 3 8 8 7 6 7 8 5 7 6 3 5 7 8 8 8 8 8 9 6 6 8 9 8 8 6 7 3 7 3 3 7 5 5 7 9 9 8 8 7 10 8 9 6 8 7 7 7 7 7 8 8 6 6 7
Rank 8 9 10 11 12 12 14 Total Score 241 239.5 238 231.5 213 213 211 Respondent 33 33 33 33 33 33 33 Mean Score 7.30 7.26 7.21 7.02 6.45 6.45 6.39
Stdev. 1.49 1.57 1.32 1.31 1.71 1.42 2.00
121
1
Focus Tools Timing Colla Roles Relation Report 6 7 7 5 5 3 3 6 7 7 7 5 6 5 7 6 6 7 5 8 3 5 5 5 5 6 5 5 6 6 5 6 6 6 3 3 7 3 3 3 2 2 7 7 8 7 5 6 5 5 5 3 3 5 4 2 3 5 6 4 5 4 4 5 5 3 4 3 4 2 8 7 7 7 8 8 3 6 7 5 6 6 5 3 8 8 8 7 7 7 6 6 7 8 6 7 6 6 8 8 7.5 8 7.5 7 5 6 6 5 8 6 3 2 7 7 7 8 7 4 6 8 6 7 8 7 4 4 7 6 8 4 5 3 2 5 5 5 4 5 5 2 4 8 8 8 8 8 5 7 7 7 7 7 7 6 10 6 10 6 4 6 6 7 7 7 6 7 6 5 7 7 6 7 4 5 3 8 6 6 6 8 7 4 5 6 7 6 8 5 5 7 6 7 8 8 5 5 7 6 5 6 8 5 4 6 4 1 1 4 4 3 7 5 7 10 9 6 2 8 6 6 7 6 5 5 6 5 5 6 5 5 3
Rank 15 16 17 18 19 20 21 Total Score 211 206 202.5 201 199.5 174 129 Respondent 33 33 33 33 33 33 33 Mean Score 6.38 6.24 6.14 6.09 6.05 5.27 3.91
Stdev. 1.46 0.99 1.80 1.82 1.56 1.50 1.40
122
APPENDIX G ROUND 3 RESULTS
(THE FINAL 25 ISSUES)
123
Backup Trail Involve Evidence Record Sacf 9 9 8 9 9 8 8 10 9 9 8 8 8 7 9 9 6 9 7 6 5 10 10 10 10 7 6 9 9 9 10 8 8 8 8 10 9 10 7 9 7 7 10 10 10 10 9 9 8 10 9 9 9 7 8 8 9 9 8 8 6 4 5 10 10 10 8 10 9 8 9 8 8 8 7 6 8 9 10 10 8 8 9 6 8 9 9 7 7 9 8 8 8 8 8 8 8 8 10 9 9 9 8.5 8 8 9 6 7 8 7 7 7 9 9 9 9 8 8 9 10 9 9 8 8 8 7 10 8 8 9 7 7 6 10 7 9 1 9 5 5 8 10 10 10 10 10 5 10 10 10 7 8 7 8 10 9 9 10 10 10 8 10 9 9 8 8 8 6 10 9 8 8 8 8 8 9 9 8 8 9 8 8 10 10 6 7 7 6 6 10 9 9 8 8 10 10 10 9 9 9 8 7 8 9 8 7 8 3 8 9 9 5 8 8 7 6 9 10 9 9 9 8 7 6 9 9 9 8 8 5 9
Rank 1 2 3 4 5 6 7 Total Score 311.5 291 287 269 258.5 247.5 245 Respondent 33 33 33 33 33 33 33 Mean Score 9.44 8.82 8.70 8.15 7.83 7.50 7.42
Stdev. 0.65 1.09 1.09 1.52 1.30 1.44 1.33
124
Technique par^3 Educate Skill Support Risk Response 7 7 7 7 6 2 5 8 8 8 7 7 6 6 6 6 6 8 7 6 5 6 8 7 7 8 5 7 9 7 7 7 6 7 5 4 8 6 6 2 3 5 10 10 9 7 8 9 9 8 7 6 6 6 4 6 6 5 5 5 na 6 5 6 8 7 5 4 5 4 7 6 8 9 7 5 5 7 8 8 6 7 5 6 9 7 8 8 8 7 7 8 5 8 7 7 7 8 8 8 8 7.5 6 8 7.5 6 8 6 6 5 8 8 7 7 7 8 9 6 7 8 8 8 8 7 7 7 7 8 6 6 4 7 8 7 7 7 6 6 7 5 10 4 8 6 10 7 5 7 10 7 7 6 7 7 10 10 10 10 6 10 10 8 7 7 8 7 8 7 7 7 6 7 6 7 8 8 3 8 8 8 7 6 8 5 7 6 7 3 5 8 8 8 8 8 8 9 6 8 9 8 9 8 6 3 7 3 3 6 7 5 7 9 9 8 8 8 7 8 9 6 8 8 7 7 7 7 8 8 7 6 6
Rank 8 9 10 11 12 13 13 Total Score 241 239.5 238 231.5 216 213 213 Respondent 33 33 33 33 33 33 33 Mean Score 7.30 7.26 7.21 7.02 6.55 6.45 6.45
Stdev. 1.49 1.57 1.32 1.31 1.92 1.71 1.42
125
Scope Focus Owner Tools Timing Colla
6 6 6 8 7 7 5 7 6 5 7 7 7 7 6 7 8 8 6 6 7 6 5 6 7 5 5 5 7 6 7 6 6 5 6 4 3 3 2 7 3 3 8 7 8 5 7 8 7 7 5 8 7 5 3 3 4 3 na 5 6 4 3 5 4 4 5 3 4 9 8 8 7 7 7 7 6 6 6 7 7 5 6 7 8 7 7 8 8 7 6 6 8 8 7 8 6 9 8 8 8 8 7.5 8 3 6 8 8 6 5 8 9 7 9 7 7 7 8 5 8 6 8 6 7 8 2 7 3 4 6 8 4 3 5 3 3 5 5 4 9 4 9 8 8 8 8 7 7 7 7 7 7 7 10 10 6 5 6 10 6 7 7 7 7 7 7 6 5 7 8 8 7 6 7 7 8 4 5 6 6 6 7 5 7 5 6 7 6 6 7 9 9 6 7 8 7 7 5 5 6 5 6 5 6 5 6 4 1 1 10 7 6 6 5 7 10 7 8 7 4 6 6 7 7 6 4 5 5 5 6
Rank 15 16 17 18 19 20 21 Total Score 211 211 204 201 206 202.5 201 Respondent 33 33 32 32 33 33 33 Mean Score 6.39 6.38 6.38 6.28 6.24 6.14 6.09
Stdev. 2.00 1.46 1.76 1.98 0.99 1.80 1.82
126
Roles Assure Relation Report 5 na 3 3 5 6 6 5 5 8 8 3 6 7 5 5 6 6 6 3 3 3 2 2 5 4 6 5 5 4 4 2 5 na 4 4 3 5 4 2 8 5 8 3 6 7 5 3 7 6 7 6 7 7 6 6
7.5 6 7 5 6 3 3 2 7 6 4 6 7 6 4 4 5 4 3 2 5 5 5 2 8 9 8 5 7 7 7 6 4 8 6 6 7 7 6 5 4 5 5 3 8 2 7 4 8 5 5 5 8 9 5 5 8 8 5 4 4 3 4 3 9 7 6 2 6 5 5 5 5 4 5 3
Rank 22 23 24 25 Total Score 199.5 176.5 174 129 Respondent 33 31 33 33 Mean Score 6.05 5.69 5.27 3.91
Stdev. 1.56 2.18 1.50 1.40
127