performance measurement summit

LCDR Bill ButlerU.S. Coast Guard Headquarters

Commandant (CG-512)Office Performance Management & Assessment

Office of Boating SafetyPerformance Measurement Performance Measurement

Improvements cannot be made without measures

Today’s discussion…1. Why we’re doing performance Measurement

2. The Logic Model

3. CG-541 Waterways Management

4. Reviewing Performance Data

5. Campaign Plan Requirements

Public Law 103-62

1.1 Government Performance & Results Act (GPRA)

1.2 OMB Guidance & Requirements – Circular A-11

1.3 OMB’s Program Assessment Rating Tool (PART)

• Strategic Plans (Submission Year + 5 years forward)

– Mission Statement– Goals & Objectives– Intent– Key Factors

• Performance Plans (Annual)

– Verifiable Output, Service Level, & Outcome Measures– Tangible, measurable performance targets

• Performance Reports (Annual)

– Actual performance, and remediation intentions

1.4 GPRA & OMB Requirements

1.5 GAO Assessments of USCG

When we're finished, program managers will be able to use the information from the dashboard measures to assess performance and goals across program

1.6 Purpose and Vision

Vision: Build program dashboard measures that will enable program managers to ensure the effective and efficient use of Coast Guard resources to achieve results.

• Data Meets OMB, GAO & DHS Requirements• Data Measurement Card for each Measurement• Data Measurement Card captures…

– Measurement Name– Measurement Description– Measurement Algorithm– Measurement Type (Outcome, Output, Activity)

– Frequency of Collection– Data Source (Where the data is extracted from)

– Data Scope & Limitations

1.7 Data Specification Requirements

True or False?

1. The Coast Guard will use Performance Measurement Data to change billets.

FALSE!

The information acquired by our office is used only for the development of your unit’s scorecards and dashboard measurements.

True or False?

2. Some of the activities we do you really can’t measure.

FALSE!

If we do then we measure it!

True or False?

3. You can’t measure this because we have no data collected on it.FALSE!

When we do our data collection, we will look for where data may come from, even if it comes from a non-government source.

True or False?

4. Our office will be critiquing your performance information.FALSE!

Our goal is to help facilitate the development of your program’s scorecard. Program managers will be the ones reviewing actual performance.

We do request performance data for OPG, RAP, BBD, and Campaign Plans.

True or False?

5. Using performance measurement data makes better arguments for additional resources.

True!

How can you legitimately request more resources when you have nothing to back up your request?

True or False?

6. Performance measurement gives you the opportunity to make a compelling argument for what you do.

True!

A good performance measurement communicates worth!

Capabilities Activities Outputs Outcomes

Activity Measures

Output Measures

Outcome Measures

Mission Purpos

e & Goals

Capability Measures

12345

Logic Model Purpose: To show how your day-to-day activities effect the organization Outcomes.

4. Performance Logic Model (skeletal)2.1 The Logic Model

Readiness Management System

Performance Logic Model

PerformanceConsumesCapability

Functional Requirements

Implementing Capabilities(Assessed by Readiness Facet)

SupportActivities

SupportOutputs

Repl

enish

men

tM

etric

s

Status Metrics

Support GoalsPeopleTraining

EquipmentSupply

InfrastructureInformation

RightCapabilities

& Mix

Maintained&

Modernized

Trained&

Capable

StrategicSupportGoals

Support Logic Model

SupportConsumesCapability

MaritimeSafety

MaritimeSecurity

NationalDefense

MaritimeMobility

Protectionof NaturalResources

StrategicGoals

ConsumptionMetrics

Operational Activities

Operational Outputs

Operational Outcomes

Capability Module

SupportReplenishes Capability

RiskRiskManagementManagementSystemSystem

2.2 The Original Logic Model

1. Helps us explain to GAO, OMB and DHS the stem-to-stern logic of our world of work.

2. It’s a process for getting to our measurements and measurement development.

3. It gets people to see how their work affects missions and outcomes

4. It shows the interdependency of all our missions.

2.3 Why the Logic Model?

12345

4. Performance Logic Model (skeletal)2.4 Logic Model – Purpose


Activity Measures

Output Measures

Outcome Measures

1

Mission Purpose & Goals

Capability Measures

The Public benefits we aim to influence.

• What does the public expect from us?• What Strategic Goals are we aiming to achieve?• Helps us explain to GAO, OMB and DHS the logic

of our world of work.

• Conduct a situation analysis: – Identifying your strengths and weaknesses – identifying trends that might impact our business in the

next 5-7 years

• Define key success factors and business fundamentals

– What success factors are or are NOT

• And, if I hear…“We facilitate commerce…”

2.4b Logic Model – Mission & Purpose?

12345

4. Performance Logic Model (skeletal)2.5 Logic Model – Outcomes


What is the greater

“societal” Outcome?

Activity Measures

Output Measures

Outcome Measures

Mission Purpos

e & Goals

Capability Measures

Benefits provided as a direct result of USCG products & services

1. What is the “greater good” to society?

2. How are you going to measure this “greater good?”

3. If you use “Economic Impact” or other monetary figures or numbers be prepared for some in-depth research.

4. Many Outcome measurements presented to our office are NOT Outcome measurements.

5. Give The Outcome Measurements Test to your measurement… Does the public care about this?”

2.5b Logic Model – Outcomes

Capabilities Activities Outputs

What is the impactof this

Output?

Outcomes

Activity Measures

Output Measures

Outcome Measures

Mission Purpos

e & Goals

Capability Measures

12345

4. Performance Logic Model (skeletal)2.6 The Logic Model – Outputs

Products & services provided as a direct result of USCG work.

• The End Result of the goods & services we provide!– COI– CG License– Cleared Channel– BNTM– A working ATON– A successfully transited vessel

• Ask: What impact is our Outputs having on the desired Outcome?

• Assess: The effectiveness of this work. – Do we need all deployed ATON in a specific channel?

2.6b Logic Model – Outputs

Capabilities Activities

Why do we do this

Activity?

Outputs Outcomes

Activity Measures

Output Measures

Outcome Measures

Mission Purpos

e & Goals

Capability Measures

12345

4. Performance Logic Model (skeletal)2.7 The Logic Model – Activities

The work performed by USCG resources.

• Usually the easier to do• Activities are the events that go into

producing the Output. For example, to get a COI– Inspect Life Saving Equipment– Inspect Fire Fighting Equipment– Inspect Navigation systems– Inspect stability and subdivision

• Ask: What impact is our activities having on the desired Output?

• Assess: The effectiveness of this work. – Do we really need to be inspecting lifejackets?

2.7b Logic Model – Activities

12345

4. Performance Logic Model (skeletal)2.7 The Logic Model – Capabilities


Activity Measures

Output Measures

Outcome Measures

Mission Purpos

e & Goals

Capability Measures

The state of Readiness and the Capabilities needed to perform USCG work.

• Our office does NOT facilitate “Capability” and Readiness

• Separate program in CGHQs does (CG-7)

• Owners of Mission Outcome (MS, MEP, WM, SAR, DR, MLE, PWCS) are responsible for conveying their needs.

2.7b The Logic Model – Capabilities

Make sure Data Measurement Card captures… – Measurement Name– Measurement Description & Definition– Measurement Algorithm– Measurement Type (Outcome, Output, Activity)

– Frequency of Collection– Data Source (Where the data is extracted from)

– Data Scope & Limitations

2.8 Logic Model – Measurements

3.1 Waterways Management Logic Model

VTS

Organizational Outcomes

3.2 CG-541 Waterways Management Logic Model

Maritime Safety

Maritime Security

Environmental Stewardship

Primary

Collateral

Waterways Response & Restoration

Efficient & Available

Waterways

Readiness

Equipment

Infrastructure

Information

People

Training

Mission Activities

Traffic Control / Navigation Assistance

Bridge Administration

Aids To Navigation

Marine Information

Waterway Access & Usage

Domestic Ice Breaking

Mission Outputs

Directed Navigation

Management

Bridge Actions & Permits

Marked Channel

Information for the Mariner

Waterway Access Controls

Cleared Channel

Mission Outputs

Directed Navigation

Management

Bridge Actions & Permits

Marked Channel

Information for the Mariner

Waterway Access Controls

Cleared Channel

3.3 CG-541 Waterways Management Logic Model

PRIMARY OUTCOMES

People Marine Information Information for the Mariner Efficient & Available Waterways

401. Percent of PAL Filled 301. Percent of LAKI Accuracy 201. Number of BNTM Discrepancy Notifications 101A. Percentage of VTS transits delayed - Coastal Ports

302. Number of Asset Hours Devoted to International Ice Patrols 202. LAKI Report (Accuracy of LAKIs) 101B. Percentage of VTS transits delayed - Western RiversTraining 303. Percentage of LAAs published within standards 203. E-Nav: Number of International Standards approved 101C. Percentage of Transits delayed - Great Lakes

402. Percent of Quals Attained 304. E-Nav: Percentage of Signal Availability 214. Number of Advisory Committee & Interagency meetings 102A. Economic Impact of disruptions-Coast Ports

315. Number of Hours for Outreach/Stakeholder Efforts 215. Number of Harbor Safety Meetings 102B. Economic Impact of disruptions-Western RiversEquipment 316. Number of Hours for HSC Efforts 102C. Economic Impact of disruptions-Great Lakes

403. SORTS / CASREPS 103. Waterways Utilization Measurement

404. % of resources hours expended Waterways Access & Usage Waterways Access Controls 108A. Number of Collisions

314. Number of Hours Enforcing COTP Waterways Access Controls 204. Number of Limited Access Areas 108B. Number of AllisionsInfrastructure 213. Number of COTP Waterway Access Controls & Permits 108C. Number of Groundings

405. Available / Secure 108D. Number of CAGs

406. Status of CGDN+ Traffic Control / Navigation Assistance Directed Navigation Assistance 108Da. CAGs from ATON Discrepancies

305. Number of VTS Transits 205. Number of Transits needing Traffic Control/Nav Assistance 108Db. CAGs from Bridges Descrepancies/Allisions withInformation 306. % of Transits needing Traffic Control/Nav Assistance 206. Number of COTP Operational Traffic Controls (& Permits) 108Dc. CAGs in GLs or from GL Pilot Errors

407. Accurate & timely 307. Number of Hours Enforcing COTP Operational Traffic Controls 108Dd. CAGs in VTS controlled areas

108De. Collisions with IcebergsSupply Bridge Administration Bridge Actions & Permits 109. WWs Navigation Safety Risk Index (PAWSA)

408. Funding/Resource hours 308. 60% Completed w/I 10 Months 207. Number of Final Agency Bridge Actions (Total) 110. Number of VTS Reported Incidents

308A. 90% Drawbridge Changes w/I 12 Months 207A. Number of Changes to Drawbridge Operating Schedules 111. Number of Incidents per Marine EventsCapabilities Requested 308B. Number of Resources Hours per Bridge Action 207B. Number of T-H Alterations

A. Trained Personnel 308C. Number of T-H Bridges on Backlog List 207C. Number of Prescribed Lighting COLLATERAL OUTCOMES

B. Cutter w/Icebreaking capability Safety

C. Cutter w/buoy tending capability Aids to Navigation Marked Channels 104. Number of Maritime-related Deaths and Injuries from CAGs

D. Communications Equipment 309. Number of AtoN Asset Hours 208. Percent of Aid Availability 105. Percentage of Persons in Distress Saved

E. Airframe (long range) 310. Number of Hours (on-scene) to Service Aid 209. Number of Aids Serviced

F. Airframe (medium range) Stewardship (Environmental)

G. Partnerships/MOUs/MOAs Domestic Icebreaking Cleared Channels 106. Number of Spills >100G / 100 M Tons ShippedH. Informational Technology 311. Number of Asset Hours Devoted to Domestic Icebreaking 210. Percentage of Critical Waterways Opened

I. VTS Facilities Security

J. Authorities (policy, jurisdiction) Great Lake Pilotage Safety Great Lake Pilotage Safety Services 107. Maritime Security Risk Index

K. Bridge Database 312. Number of hours conducting Pilot rate development 211. Number of Enforcement Actions

L. SEA-21 Funding 313. Number of hours conducting Pilotage oversight 212. Number of Investigations

INDEPENDENT ACTIVITIES INDEPENDENT SERVICES INDEPENDENT OUTCOMES

Polar Icebreaking Polar Icebreaking Safe Passage and Power & Influence in Polar Regions

317. Number of Asset Hours Devoted to Polar Icebreaking 216. Number of Polar Operations Completed 112. No. of hours N of Arctic Circle/S of Antarctic Circle

113. Percentage of Planned Polar Operations Completed

OUTCOMESCAPABILITIES

Outputs - 200 SeriesActivities - 300 SeriesCapabilities - 400 Series Outcomes - 100 Series

ACTIVITIES OUTPUTS

Current Measurement:

Number of days Waterways closed

3.3 Current Domestic Ice Measurements

Number of Critical Waterway Closure Days on the Great Lakes

0

1

2

3

4

5

6

7

8

9

Closure Days - Normal Winter 0 0 7 0 4 0 0 0

Closure Days - Severe Winter 0 7 0 7 0 0 0 0

Target - Normal Winter 2 2 2 2 2 2 2 2 2 2 2 2 2 2

Target - Severe Winter 8 8 8 8 8 8 8 8 8 8 8 8 8 8

Closure Days 0 7 7 7 4 0 0 0

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

1. Great Lake Pilotage • They produce one regulation per year.

2. Polar Ice Breaking • They make one trip to the South Pole per year

3. International Ice Patrol• No Icebergs reported in shipping lanes.

4. Vessel Traffic Service• They moved the office out of Waterways and into CG-7

(Capabilities)

5. Bridge Admin some permits take years to develop

3.4 Challenges to Waterways Measurements

1. Proper Representation during workshops• Over-represented on Bridges• Under-represented on Ice

2. Developing the master WWs Logic Models required concurrent individual program development

3. It was easy to test the viability of measures.

4. Measurement definition caused lots of work• What’s a “transit”• What’s a “vessel”• What’s a “delay”

5. Merging of several cultures into one Directorate6. And, if I hear…“We facilitate commerce…”

3.5 Lessons Learn from Waterways Management

• Requires that measurements are Reported• Don’t be afraid of the results…

– Having bad results is better than no system of measurement at all

• Constantly Assess your Measurements– Is the measurement giving me what I need to

know?– Are there other sources of data?

• Be prepared to explain your results• Determine the Frequency of Reporting

4.1 Reviewing Performance Data

Annual Number of Distinct Allision Events by District

0

100

200

300

400

500

600

700

800

900

D17 13 15 17 9 8 5 4 11 7D14 16 14 7 3 5 7 11 4 7D13 25 21 43 42 32 27 34 23 23D11 32 32 35 40 29 37 30 28 37D9 39 36 27 73 47 27 43 49 68D8 535 527 390 444 438 463 413 341 365D7 63 56 71 57 46 49 58 75 57D5 48 30 35 41 44 56 46 57 64D1 53 42 56 69 77 74 68 74 56Total Allisions 824 773 681 778 726 745 707 662 684UCL 878.9 878.9 878.9 878.9 822.7 822.7 822.7 822.7 822.7 822.7 822.7 822.7 822.7 822.7Average 740.6 740.6 740.6 740.6 684.3 684.3 684.3 684.3 684.3 684.3 684.3 684.3 684.3 684.3LCL 602.3 602.3 602.3 602.3 546.0 546.0 546.0 546.0 546.0 546.0 546.0 546.0 546.0 546.0Annual Target 890 855 821 756 744 729 710 685 649 624 679 673 668 664 660

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

4.2 Program Results – Reporting Outcomes

How many delays / district / period ? Annual Number of Distinct Allision Events by District

0

100

200

300

400

500

600

700

800

900

D17 13 15 17 9 8 5 4 11 7D14 16 14 7 3 5 7 11 4 7D13 25 21 43 42 32 27 34 23 23D11 32 32 35 40 29 37 30 28 37D9 39 36 27 73 47 27 43 49 68D8 535 527 390 444 438 463 413 341 365D7 63 56 71 57 46 49 58 75 57D5 48 30 35 41 44 56 46 57 64D1 53 42 56 69 77 74 68 74 56Total A llisions 824 773 681 778 726 745 707 662 684UCL 878.9 878.9 878.9 878.9 822.7 822.7 822.7 822.7 822.7 822.7 822.7 822.7 822.7 822.7Average 740.6 740.6 740.6 740.6 684.3 684.3 684.3 684.3 684.3 684.3 684.3 684.3 684.3 684.3LCL 602.3 602.3 602.3 602.3 546.0 546.0 546.0 546.0 546.0 546.0 546.0 546.0 546.0 546.0Annual Target 890 855 821 756 744 729 710 685 649 624 679 673 668 664 660

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

What waterways were affected?

What were the results of these delays?

PRIMARY OUTCOMES

People Marine Information Information for the Mariner Efficient & Available Waterways

401. Percent of PAL Filled 301. Percent of LAKI Accuracy 201. Number of BNTM Discrepancy Notifications 101A. Percentage of VTS transits delayed - Coastal Ports

302. Number of Asset Hours Devoted to International Ice Patrols 202. LAKI Report (Accuracy of LAKIs) 101B. Percentage of VTS transits delayed - Western RiversTraining 303. Percentage of LAAs published within standards 203. E-Nav: Number of International Standards approved 101C. Percentage of Transits delayed - Great Lakes

402. Percent of Quals Attained 304. E-Nav: Percentage of Signal Availability 214. Number of Advisory Committee & Interagency meetings 102A. Economic Impact of disruptions-Coast Ports

315. Number of Hours for Outreach/Stakeholder Efforts 215. Number of Harbor Safety Meetings 102B. Economic Impact of disruptions-Western RiversEquipment 316. Number of Hours for HSC Efforts 102C. Economic Impact of disruptions-Great Lakes

403. SORTS / CASREPS 103. Waterways Utilizationy Measurement

404. % of resources hours expended Waterways Acess & Usage Waterways Acess Controls 108A. Number of Collisions

314. Number of Hours Enforcing COTP Waterways Access Controls 204. Number of Limited Access Areas 108B. Number of AllisionsInfrastructure 213. Number of COTP Waterway Access Controls & Permits 108C. Number of Groundings

405. Available / Secure 108D. Number of CAGs

406. Status of CGDN+ Traffic Control / Navigation Assistance Directed Navigation Assistance 108Da. CAGs from ATON Discrepancies

305. Number of VTS Transits 205. Number of Transits needing Traffic Control/Nav Assistance 108Db. CAGs from Bridges Descrepancies/Allisions withInformation 306. % of Transits needing Traffic Control/Nav Assistance 206. Number of COTP Operational Traffic Controls (& Permits) 108Dc. CAGs in GLs or from GL Pilot Errors

407. Accurate & timely 307. Number of Hours Enforcing COTP Operational Traffic Controls 108Dd. CAGs in VTS controlled areas

108De. Collisions with Icebergs

Support Bridge Administration Bridge Actions & Permits 109. WWs Navigation Safety Risk Index (PAWSA)

408. Funding/Resource hours 308. 60% Completed w/I 10 Months 207. Number of Final Agency Bridge Actions (Total) 110. Number of VTS Reported Incidents

308A. 90% Drawbridge Changes w/I 12 Months 207A. Number of Changes to Drawbridge Operating Schedules 111. Number of Incidents per Marine EventsCapabilities Requested 308B. Number of Resources Hours per Bridge Action 207B. Number of T-H Alterations

A. Trained Personnel 308C. Number of T-H Bridges on Backlog List 207C. Number of Prescribed Lighting COLLATERAL OUTCOMES

B. Cutter w/Icebreaking capability Safety

C. Cutter w/bouy tending capability Aids to Navigation Marked Channels 104. Number of Maritime-related Deaths and Injuries from CAGs

D. Communications Equipment 309. Number of AtoN Asset Hours 208. Percent of Aid Availibility 105. Percentage of Persons in Distress Saved

E. Airframe (long range) 310. Number of Hours (on-scene) to Service Aid 209. Number of Aids Serviced

F. Airframe (medium range) Stewardship (Environmental)

G. Partnerships/MOUs/MOAs Domestic Icebreaking Cleared Channels 106. Number of Spills >100G / 100 M Tons Shipped

H. Informational Technology 311. Number of Asset Hours Devoted to Domestic Icebreaking 210. Percentage of Critical Waterways Opened

I. VTS Facilities Security

J. Authorities (policy, jurisdiction) Great Lake Pilotage Safety Great Lake Pilotage Safety Services 107. Maritime Security Risk Index

K. Bridge Database 312. Number of hours conducting Pilot rate development 211. Number of Enforcement Actions

L. SEA-21 Funding 313. Number of hours conducting Pilotage oversight 212. Number of Investigations

INDEPENDENT ACTIVITIES INDEPENDENT SERVICES INDEPENDENT OUTCOMES

Polar Icebreaking Polar Icebreaking Safe Passage and Power & Influence in Polar Regions

317. Number of Asset Hours Devoted to Polar Icebreaking 216. Number of Polar Operations Completed 112. No. of hours N of Arctic Circle/S of Antarctic Circle

113. Percentage of Planned Polar Operations Completed

OUTCOMESCAPABILITIES

Outputs - 200 SeriesActivities - 300 SeriesCapabilities - 400 Series Outcomes - 100 Series

ACTIVITIES OUTPUTS

4.3 Program Results – “Drill Down”

Name of Critical Waterway Date Section Effected Time of delay Cause

1

2

3

4

5

6

Comments:

Last Four Ice Seasons This Year

03 04 05 06 07 Hrs

Comments:

4.4 Program Results – Report “Causes”

No. of channels cleared

No. of Assists

SAMPLE Report

Where do we go from here?• Continue refining & defining measures • Develop Targets • Develop Efficiency Measures• Continue assessing Risk & Risk “buy down” • Develop Sensitivity Analysis• Develop costs and costing models

We can’t do any of these unless we’re measuring!

4.4 The Way Ahead

5. EXECUTION: Transforming direction into results; providing risk-based performance management; and ensuring continuous improvement and adaptation to changing conditions.

3. PLANNING & PROGRAMMING: Developing alternatives for legislative change, investment, reinvestment, and allocation of resources; determining cost/benefits; risk factors, and outcome probabilities.

2. INTENT: Establishing goals, areas of emphasis, and priorities for minimizing risk and closing performance gaps; providing planning and budgeting guidance.

4. DIRECTION & BUDGETING: Providing strategic plans and policies; providing performance plans to promulgate initiatives and establish budgets, milestones, and targets; providing CONOPS, OPORDERS, and Operational Planning Guidance.

AssessmentIntent

DirectionExecution

Planning

1. ASSESSMENT: Identifying cross-outcome risks & performance results; needs and preferences; outcome drivers; key trends; resulting gaps and issues; and returns on investment, reinvestment, and resource allocation alternatives.

AUG-DEC

AUG, OCT, DEC

JAN-MAR

APR-JUL Simplified Planning Process

5.1 Campaign Planning Process

Seven Program Missions• Marine Safety• Marine Environmental Protection• Waterways Management• Ports, Waterways, & Coastal Security• Search and Rescue• Maritime Law Enforcement• Defense Readiness

5.2 Campaign Plans Required for…

PART 1: Program Mission• Introduction• Mission and Vision• Guiding Principles• Roles, Responsibilities, and Mission

Business Case• Linkage with Departmental and Executive

directives

5.3 Components of a Campaign Plan

PART 2: Strategic Assessment• Key Outcome Measurements• Current Performance• Targeting • Objectives and Strategies

– Opportunities and Challenges– Performance Gaps and Issues – ACCCP Gaps and Issues


PART 3: Action Plans• Initiative • Synopsis of the Initiative

– A“1-minute Elevator summary • Situation-Needs Assessment

– The “necessity” of the situation• Proposed Course of Action• Benefits, Costs and Risks• Illustrations
