1Using the Balanced ScorecardPlanning and Conducting Performance AuditsSoutheastern and Southwest Intergovernmental Audit Forum Joint ConferenceAugust 30 September 1, 2006

Using BSC for Performance Audits

Shared understanding of scorecard Why bother? Knowing-doing gap: why you arent

more effective

How to use a scorecard to structure audit

How to use scorecard to manage and evaluate audit

Every picture tells a story, dont it*

* apologies to Rod Stewart

2Mission story

Customer Focus Story

Internal Process Story

3Learning and Knowledge Story

Financial Story

Organizational story

4Truth is truthTo the end of reckoning.

William ShakespeareMeasure for Measure, V.i.45

SAO Organizational ScorecardPerspective Objective Measure Target

Mission - be a world leader in professional services - competitive with any private firm in quality of services and efficiency OPI - Useful Information Rating Rating of 2.50: scale from 1 to 3- identify fiscal impact that equals SAO budget, that is, deliver recommendations that help the state use its limited resources more effectively and efficiently Org-Fiscal Impact 10 x Annual Budget = $160,000,000 per year

- produce results that add value to the operations of the legislature and the client agency- provide legislators and their staff with information that they find useful in policy deliverations and decision making- spend taxpayer money wisely to acquire the knowledge, equipment, and other

resources needed to achieve the SAO mission and deliver high quality services

Customer Focus - build respectful relationships based on trust, cooperation, and open communication with our clients OPI-Customer Feedback Rating Rating of 4.0: scale from 1 to 5

- create deliverables that are easy to read, understand, and to use

- maintain effective working relationships with those we audit- provide customers with information that helps them improve their operations and their accountability

Internal Processes - continuously identify better approaches to our work OPI-Service Delivery Status 85% of budget - deliver high quality products and services within the fastest time frame

- focus resources on service delivery

- provide information and services when we say we will

Learning & Knowledge - create a positive work environment that encourages creativity, innovation, high quality results, and low turnover Org-Employee Feedback Rating Rating of 3.75; scale from 1 to 5

- give employees the opportunity to fulfill their potential Org-Workforce Capacity 85% of needs identified

- promote growth of knowledge and learning, risk-taking, innovation, and change

Financial - deliver support services that help the SAO strategies to achieve their mission Org- AAS Productive Workforce 98% to 102% of budget- maintain the right number of people with the right knowledge at the right time to deliver SAO services Org-Audit Support Workforce 98% to 102% of budget

Org-Budget Compliance - Dollars 98% to 100% of budget

Org-EES-Cost Recovery 100%

Org-MAS-Cost Recovery 100%

Actively provide government leaders with useful information that improves accountabilityBalanced Scorecard - Organizational

Washington State Scorecard

5North Carolina DOTSafety

Goal 1: Develop and deliver services, products and projects that ensure a safe and secure environment for employees, stakeholders /partners, and customers Objective A: Reduce employee accident and injury/illness rates

1. Reduce number of DOT vehicle and equipment accidents by 5% from 2004 benchmark (Sanderson)

2. Achieve an overall DOT incident rate of 5.5 or less (Sanderson) 3. No work place fatalities (Sanderson)

Objective B: Reduce accidents, injuries, and fatalities among stakeholders/partners 1. No work place fatalities (Sanderson) 2. Provide real-time information to employers regarding their

employees driving records (Tatum) 3. Increase the number of employers with access to the

Transportation (Tatum)

BSC Causal Relationships

Financial

Learning &Knowledge

InternalProcesses

CustomerFocus

Mission

Satisfied customers

Efficient, high quality internal processesSkilled employees

Allocated resources

Mission achievement

Research on Scorecard Use in Public Sector Comprehensive range of measures covering

operations Improved strategic planning Helped translate strategic objectives into

improvement in operations at service level. European Quality Award Model (similar to TQM)

compared to BSC BSC linked better to strategy BSC easier to adapt to customer measures BSC easier to involve people at all levels and

to relate to specific job BSC good link between process measures

and business strategy

6The Art of Getting Things Done

About 10% of all managers take decisive, purposeful action.

Managers know what to do. Managers have the knowledge. Managers are very busy. Managers permanently face requests.

Baruch and Ghoshal

Do not mistake activity for achievement.

Mabel Newcomber20th century academic and writer

Detached Managers

20%

Detached Managers

20%

Purposeful Managers

10%

Purposeful Managers

10%

Procrastinators

30%

Procrastinators

30%Busy

Managers

40%

Busy Managers

40%

LowLow

LowLow

HighHigh

HighHigh

Focus

Energy

Manager Actions

7Making choices

Willpower, not motivation, drives purposeful action

intention formation

engaging willpower

intention protection

Scorecards and Focus

Scorecards tell a story

Scorecards focus attention on critical activity and results

Goal setting and management lead to focus

Reticular Formation

Reticular Activating System - RAS

Brains attention center

Filter between conscious and subconscious

Real events and thoughts are the same

8Characteristics of a scorecard Strategies align with mission

Relationships among limited number of measures are clear, provide relevant information

Scorecards tell a story about what is important and about accountability to everyone in the organization

Scorecards balance financial / non-financial information short term / long term targets

lead / lag measures

Why don't you write books people can read?

Nora Joyce to her husband James 1882-1941

Scorecards and Auditing

Scorecard as business model Key activities in scorecard Measures reflect key activities Audit verifies outputs / outcomes

9Using BSC to Structure an AuditBalanced Scorecard: Audit Measures Worksheet

Perspective Objective/Goal/Outcome Measure TargetMissionWhy do we exist?Why are we doing this?

Customer Focus

Internal Processes

Learning & Knowledge

Financial

How will we acquire resources?

What must we do to develop employee skills and technology to continue adding value and

To satisfy our stakeholders and to meet our mission, how will we allocate our appropriated

To achieve our mission, what customer needs must we meet What must we do to satisfy, retain and attract new

To satisfy customer needs and to achieve our mission, in what internal business processes

To achieve our goals, what must the organization know and learn, and how must we

@work: Auditing ScorecardsCity of Bedrock Balanced Scorecard

Provide responsible leadership and the services and infrastructure necessary for a safe environment and a high quality of life for our citizens

Perspective Objective Measure TargetMission

Average score 85%

Citizens believe they have a high quality of life

City leaders are responsive to citizen needs and desires Average score 85%

Average score 80%

Customer Focus Telephone courtesy of employee - overall citizen satisfaction Average score 80%

Provide police services perceived as satisfactory by citizens

Phone calls directed accurately

Satisfaction with ambulance service Average score 80%

Average score 80%

Number of misdirected calls per month

Internal Processes Staff members are courteous while providing services Phone call monitoring by supervisors

Quick response to service calls

Effectiveness in maintenance of city buildings Average time to respond to service calls

Accuracy of response to requests

Acquisition excellence

Learning & Knowledge Employees received adequate training to perform efficiently Hours of job-related training per employee annually

Employees are treated as valued customers

Financial Operates within budget Budget variance measured monthly Variance from budget is 2%

Finance reports are prepared and delivered to users timely Number of late reports per month No. reports delivered after due date

Finance reports are accurate and reliable Number of errors in standard finance reports / month No. of errors monthly

Payroll is processed accurately Number of paychecks with errors per pay period 2 paycheck errors per month

Payables are processed timely % of late payments per month

There are enough employees on staff to perform servicesNumber of vendor payments made more than 30 days after invoice is received

Target to be determined after baseline established

Response to items on citizen survey about courtesy of call center employees

Response to items on citizen survey about satisfaction with police services

Response to items on citizen survey about satisfaction with ambulance services

Fewer than 5 instances of discourteous responses by agents in 1 month

Fewer than 5 instances of misdirected calls in 1 month

12 hours of job related training / yr

Employee engagement indicates overal satisfaction with the city as an employer

Provide services that result in overall citizen satisfaction with community safety Response to items on citizen survey about feelings of safety

Response to items on citizen survey about satisfaction with quality of life

Response to items on citizen survey about satisfaction with responsiveness of leaders

@work: Auditing ScorecardsCity of Bedrock Balanced Scorecard

Provide responsible leadership and the services and infrastructure necessary for a safe environment and a high quality of life for our citizens

Perspective Audit Notes Measure TargetMission

Average score 85%

Average score 85%

Average score 80%

Customer Focus Average score 80%

Average score 80%Average score 80%

Number of misdirected calls per monthInternal Processes Phone call monitoring by supervisors

Average time to respond to service calls

Learning & Knowledge Hours of job-related training per employee annually

Financial Budget variance measured monthly Variance from budget is 2% Number of late reports per month No. reports delivered after due dateNumber of errors in standard finance reports / month No. of errors monthlyNumber of paychecks with errors per pay period 2 paycheck errors per month

% of late payments per monthNumber of vendor payments made more than 30 days after invoice is received

Fewer than 5 instances of discourteous responses by agents in 1 Target to be determined after baseline established

12 hours of job related training / yr

Employee engagement indicates overal satisfaction with the city as an employer

Response to items on citizen survey about courtesy of call center employees

Response to items on citizen survey about satisfaction with police services

Response to items on citizen survey about satisfaction with ambulance services Fewer than 5 instances of misdirected calls in 1 month

Response to items on citizen survey about feelings of safety

Response to items on citizen survey about satisfaction with quality of life

Response to items on citizen survey about satisfaction with responsiveness of leaders

10

Auditing Scorecards

Review architecture and security of e-systems

Review relevant policies and plans Review goals and measures Identify unintended consequences Share information with management Result: ensure that scorecard

processes function as intended, emphasizing the quality and strength of process controls and identifying ineffective metrics

Evaluating Audit Processes & Results Mission

Customer Focus

Internal Processes

Learning & Knowledge

Financial

Did client/auditee receive useful information?

Did client find the information easy to understand? Timely? Project surprise free?

Was the audit work and project management high quality and effective?

Did audit team members learn and grow on this audit? Was employee climate positive?

Did we adhere to our time and dollar budgets?

Evaluating Audit Processes & Results

Balanced Scorecard Measures - Audit & Review

Perspective Objective Measure TargetMission - Provide various levels of assurance A&R-Audit Plan Completion Status 85% of budget

about that: compliance, information OPI-Useful Information Rating Office Wide Target = 2.50, scale from 1 to 3; Project Level Target = 3.00, scale from 1 to 3 accuracy, performance achieves Org-Fiscal Impact Audit & Review Target = $160,000,000.00 the expected results, and systems PL-% of Recommendations Action by Client 90% of budget effectiveness PL-Reports with Reviews of IT Systems 60% of budget effectiveness PL-Useful Information Rating Office Wide Target = 2.50, scale from 1 to 3; Project Level Target = 3.00, scale from 1 to 3

Customer Focus - Balanced reporting A&R-Reports Delivered On-Time 90% of budget- Credible OPI-Customer Feedback Rating Rating of 4.0: scale from 1 to 5- Independence PL-Customer Feedback Rating Average Rating of 4.00; scale from 1 to 5- Objectivity- Recognized expertise- Timely service

Internal Processes - Audit tools and technology A&R-Hours Audit Staff-Assurance Services 74% of the budget- Auditing A&R-% Projects with Complete BSC Data 90% of budget- Packaging information/deliverables OPI_A&R-Audit Duration 85% of budget (i.e., production) PL-Projects Addressing at least 2 of 3 SAO IT Objectives 60% of budget- Project management PL-Projects Meeting Fieldwork Checkpoint Meeting Date 80% of budget- Quality control and compliance with standards- Risk assessment (statewide and project level)

Learning & Knowledge - Positive employee climate A&R-% of Highly Qual PMs 30% of senior staff- Technical expertise A&R-Attendance at Core Courses 93% of budget- Training A&R-Compliance w/ Training Standards - CPE Standards 100% of eligible A&R staff for the quarter

A&R-Workforce Capacity 85% of needs identifiedOrg-Employee Feedback Rating Rating of 3.75; scale from 1 to 5

Financial - Budget allocation (size of efforts) A&R-% Audit Staff with > 100 Hours Comp Time 5% or less- Recover cost for billable services A&R-Audit Resources < 3000 hours

A&R-Productive Workforce # of productive employees (for the month) established by Direct of Audit at the beginning of each Fiscal YearF_Budget Summary 85.00% to 100.00% of budget

The State Auditor's Office provides services that improve and assure the quality and accuracy of information for business decision making.

11

References Used McAdam, Rodney and Edel ONeill. 1999. Taking a critical perspective

to the European Business Excellence Model using a balanced scorecard approach: a case study in the service sector. Managing Service Quality 9 (3): 191-197.

McAdam, Rodney and Timothy Walker. 2003. An Inquiry into Balanced Scorecards Within Best Value Implementation in UK LocalGovernment. Public Administration 81 (4): 873892.

Nohria, Nitin, William Joyce and Bruce Roberson. 2003. What Really Works. Harvard Business Review 81 (7).

Phillips, Jason Keith. 2004. An Application of the Balanced Scorecard to Public Transit System Performance Assessment. Transportation Journal 43 (1): 26-55.

Schmidt, Chris. 2005. The Drivers View. Internal Auditing 62(3): 29-32.

Talbot, C. 1999. Public performance towards a new model, in Special Issue: Quality issues in the public service. Public Policy and Administration 14 (3): 1535.

Wong, Jeff. 2000. The Role of the Balanced Scorecard in Operational Auditing. Internal Auditing 15 (4): 33-36.

Contact

Deborah L. Kerr, Ph.D.deborah.kerr@heart.org512.433.7106

/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (None) /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /False

/Description >>> setdistillerparams> setpagedevice