Expect the UnexpectedPlanning the Scope of an IT Performance

Audit

Robin Garity, C.P.A., C.I.S.A.October 2014

StandardsImportance Audit Assignment #1 – Michigan Business One

Stop SystemAudit Assignment #2 – Branch Office System

Agenda

Generally Accepted Governmental Auditing Standards (GAGAS) states 6.07 Auditors must plan the audit to reduce audit

risk to an appropriate level for the auditors to obtain reasonable assurance that the evidence is sufficient and appropriate to support the auditors’ findings and conclusions.

6.09 The scope defines the subject matter that the auditors will assess and report on, such as a particular program or aspect of a program, the necessary documents or records, the period of time review, and the locations that will be included.

What do the standards say about Performance Audit Planning?

Determines direction of audit (many possibilities) Security Accurate processing Efficiency of system Governance

Determines audit value What will change if the conclusion is that the auditee/system

is not effective? Will recommendations be useful?

Why is planning the audit scope important in a performance audit?

Ensures that all significant risks are identified and addressed during the audit

Poor scope planning can result in a stressful audit Inadequate resources

Inefficient testing

No pressure…But don’t mess up when planning the audit scope!

Why is planning the audit scope important in a performance audit?

(continued)

Assignment based on criticality to audit entity

System mission - Create a one-stop shop for individuals or businesses doing business with the State of Michigan

No prior audits

Implemented in 2009

Known costs of $21.3 million to date for development and maintenance

Audit Assignment Example #1Michigan Business One Stop System (MBOS)

Confidential and critical licensing information in the system. Operating System Access and Configurations Database Access and Configurations Application Access Monitoring Processes

Scope Planning Ideas

Interviewed project manager, DBA, and system administrators

Reviewed system documentation Data dictionary Network diagram Development contracts

Reviewed policies and procedures for managing the system

Interviewed users/stakeholders

Scope Planning Procedures

Very few customers liked or used MBOS

Process was much more complex for customers

Applicant data must be reentered into secondary systems

New development projects on hold because of uncertainty regarding MBOS’s future

Departments unsure of what license information is available in the system

What We Heard

FROM:Operating System Access and ConfigurationsDatabase Access and ConfigurationsApplication Access

TO:Project Planning - Is there a plan for making the system more

effective? Governance - Is there leadership to make decisions on the future of

the system?Updating of System - If departments are unsure of licenses in the

system, are license applications really up to date in MBOS?

Scope U-Turn

Always interview users of the system during planning.

Keep in mind the future impact.

Be flexible.

What We Learned About Planning the Audit Scope

Findings No strategic plan for continued development and use of the

system. No post-implementation review to determine if expected benefits

were realized. Lack of an effective governance structure. No process to periodically review and update the content (out-of-

date fees, applications, etc.)

Latest update – DTMB is shutting down the system because it is not providing the expected benefits.

Outcome

System used in branch offices for vehicle registrations, driver licensing, etc.

The Department of State collects approximately $2.2 billion per year through the various systems that process driver and vehicle related transactions.

Audit assignment based on revenue and criticality of system

Audit Assignment Example #2Branch Office System

Branch Office System Application controls

Access/segregation of Duties Proper input of licensing and registration data Change management

Scope Planning Ideas

Interviewed project managers, DBA, and system administrators.

Reviewed system documentation Data dictionary Network diagram Development contracts System flows

Reviewed policies and procedures for managing the system.

Interviewed system users.

Scope Planning Procedures

Branch Office System scheduled for replacement.

Many systems process driver and vehicle related data on the back end and store confidential data. The Branch Office System is primarily data input.

Complex flow of information between departments for use in processing driver and vehicle-related data.

Prior non-IT audit of fee calculations (audited around systems) but no actual IT audits.

What We Found Out

FROM:Branch Office System Application controls

Access/Segregation of duties

Proper input of licensing, registration data

TO:Excluding Branch Office System (being replaced)Security for other driver and vehicle related systems that store confidential data

Operating System

Database

Reviewing actual processing of data outside of Branch Office System Are matches and input of information proper to ensure no registrations to suspended licenses,

deceased, stolen vehicles, etc.

Excluding fee calculations

A New Focus

Consider new development projects

Consider entire process

Understand in detail what has already been audited

What We Learned About Planning the Audit Scope

Security weaknesses Access issues Data processing inconsistencies

Potential Audit Conclusions

Be sure to: Spend sufficient time in planning Obtain complete understanding of business processes and

flow of system data Listen to what auditee and users think are the problems Evolve your scope

To ensure: Audit value Impact on future processes An efficient audit

Final Suggestions For Planning the Audit Scope