compliance performance measure proposals and preliminary trends
DESCRIPTION
Compliance Performance Measure Proposals and Preliminary Trends. Greg Pierce – Chair, Performance Measures Task Force Compliance and Certification Committee Meeting March 10-11, 2010. Topics. PMTF Tasks and Approach Performance Measure Proposals Preliminary Trends Outreach Effort - PowerPoint PPT PresentationTRANSCRIPT
Compliance Performance Measure Compliance Performance Measure Proposals and Preliminary TrendsProposals and Preliminary Trends
Greg Pierce – Chair, Performance Measures Task Force
Compliance and Certification Committee Meeting
March 10-11, 2010
2TopicsTopics
PMTF Tasks and Approach
Performance Measure Proposals
Preliminary Trends
Outreach Effort
Next Steps
3PMTF TasksPMTF Tasks
Establish an open, consistent and repeatable metric development process to identify and evaluate metrics
Develop compliance improvement measures at the highest levels to provide insight in terms of a set of compliance principles
Coordinate and work with Regional Entities, RMWG, OC, PC and CIPC to ensure agreement
Recommend how to benefit from the compliance improvement measures
4ApproachApproach
Adopt RMWG’s open metric development process
Review data source on compliance and other key input factors
Propose a set of performance measures at high levels
Gather data and validate trends
Solicit feedback and other ideas from stakeholders
Prepare a draft report starting April 2010
5Proposed Performance MeasuresProposed Performance Measures
Compliance Culture
Timeliness of CVI/CIQ
Average Duration of Mitigation Plan Completion
Violation Risk Index
Average Duration of Audit Report Completion w/ No Violations
Average Duration of Audit Report Completion w/ Potential Violations
Availability/timeliness of 'lessons-learned’ guidance from audits, operating experience, violations and disturbances
Feedback to standards development
The detailed individual metric definitions and specifications are available at http://www.nerc.com/filez/pmtf.html.
6Compliance Culture MeasureCompliance Culture Measure
Metric – Compare Self-Disclosed Violations to the Total Number of Discovered Violations
Self-Disclosed = Self-report + Self-certification
Indicates the level of trust and cooperation within the “NERC Enterprise” as well as the culture of compliance of the stakeholder community
7Trend By Discovery MethodTrend By Discovery Method
Reported Violation Trend By Discovery Method(June 18, 2007 - December 31, 2009)
0
100
200
300
400
500
600
2007 142 71 118 0 0 4 2
2008 574 331 211 15 4 2 3
2009 579 297 214 127 23 2 0
Self-ReportCompliance
AuditSelf-
CertificationSpot Check Investigation
Exception Reporting
Periodic Data Submittal
8Compliance Culture TrendCompliance Culture Trend
Self Disclosed Violations (June 18, 2007 - December 31, 2009)
46.6%50.4%42.1%
35.0% 18.5% 17.2%
0%
20%
40%
60%
80%
Per
cent
age
Self-Reported Self-Certification
Self-Certification 35.0% 18.5% 17.2%
Self-Reported 42.1% 50.4% 46.6%
Total
2007 2008 2009
77.1% 68.9% 63.8%
9Timeliness of CVI/CIQTimeliness of CVI/CIQ
Average Duration – for CVI/CIQ with Open status
Total number of CVIs/CIQs for Open and Closed status
Number of lessons learned and trends issued to industry
10CVI/CIQ Timeliness MeasureCVI/CIQ Timeliness Measure
Duration (Days) CVI CIQ
Oldest Average Oldest Average
NERC 644 317 361 242
Region 823 444 126 107
Status of NERC/Regional CIQs and CVIs(February 2008 - February 2010)
0
10
20
30
40
Co
un
t
Open 6 11 4 8
Closed 25 4 15 4
NERC CIQs NERC CVIs Regional CIQs Regional CVIs
CIQ - Compliance Inquiry CVI - Compliance Violation Investigation
11Triggers for NERC/Regional CVIs/CIQsTriggers for NERC/Regional CVIs/CIQs
Triggers for NERC/Regional CVIs/CIQs(February 2008 - February 2010)
0
10
20
30
40
50
60
Co
un
t
Regional CVIs 0 1 8 2 1 0
Regional CIQs 0 1 15 3 0 0
NERC CVIs 1 3 10 1 0 0
NERC CIQs 0 4 24 1 2 0
Audit Complaint EventNERC Report
Media Report
Other
CIQ - Compliance Inquiry CVI - Compliance Violation Investigation
12
Average Duration of Mitigation Plan Average Duration of Mitigation Plan CompletionCompletion
Duration – from Date plan received to Mitigation close date
Sum of all durations divided by total number of violations discovered in a particular quarter
• Remove zero duration days
13
Average Number of Days Average Number of Days for Completing Mitigation Plansfor Completing Mitigation Plans
Average Number of Days for Completing Mitigation by Quarter*(2007 - 2009)
0
50
100
150
200
250
3Q07 4Q07 1Q08 2Q08 3Q08 4Q08 1Q09 2Q09 3Q09 4Q09
Number of Days
* Note that a total of 607 mitigation completions are not included in this chart. Their completion dates are the same as or earlier than their receiving dates.
14Violation Risk Index (VRI)Violation Risk Index (VRI)
Violation Risk Index is defined as a risk factor and severity level weighted average based on unmitigated violations
• Assess potential consequences of a particular unmitigated violation.
• Unmitigated violations of higher risk factor requirements have more weighting value than of lower risk factors.
• Determine the change in reliability levels due to Standard requirement violations.
Weighting factors and an example are included in Appendix A.
15
NERC Unmitigated ViolationsNERC Unmitigated Violations(6/18/07 to 12/31/2009)(6/18/07 to 12/31/2009)
Unmitigated Violations by Quarter(3Q 2007 - 4Q 2009)
939 937
11011044 1084 1092
990909 883
837
0
200
400
600
800
1000
1200
3Q2007 4Q2007 1Q2008 2Q2008 3Q2008 4Q2008 1Q2009 2Q2009 3Q2009 4Q2009
Co
un
ts
16Violation Risk TrendViolation Risk Trend
Violation Risk Index Normalized on Violation Risk Factor and Severity Level
100 98105 102
9183
7667
0
20
40
60
80
100
120
1Q2008 2Q2008 3Q2008 4Q2008 1Q2009 2Q2009 3Q2009 4Q2009
Violation Risk Index
17
Average Duration of Audit Report Average Duration of Audit Report Completion w/ No ViolationsCompletion w/ No Violations
Measure audit process efficiency and ability to
promptly advise industry of audit results These reports do not have potential violations
Starting point: audit ending point
Ending point: report finalized with entity and received by NERC
18
Average Duration of Audit Report Average Duration of Audit Report Completion w/ No ViolationsCompletion w/ No Violations
Average Duration of No-Violation Audit Report Completion(2Q 2008 - 3Q 2009)
0
30
60
90
120
150
3Q08 4Q08 1Q09 2Q09 3Q09
Nu
mb
er o
f D
ays
19
Average Duration of Audit Report Average Duration of Audit Report Completion w/ Potential ViolationsCompletion w/ Potential Violations
Measure enforcement process efficiency and ability to promptly advise industry of audit
results These reports have potential violations
Starting point: audit ending point
Ending point: report finalized with entity and received by NERC
Data is not available until second quarter of 2010
20
Timeliness of Lessons-LearnedTimeliness of Lessons-Learned
Establish a common and central platform to share lessons-learned
Timeliness of actionable lesson-learned guidance from operating experience, violations and disturbance events
• Lessons-Learned guidance
• Recommendations from event analysis
• Compliance analysis
• Advisories
21
Feedback to StandardsFeedback to Standards
Timeliness of compliance results/feedback to standards
• # of SARs (e.g., refine a particular requirement)
• # of suggestion and comment submittals
The Standards input form is available at http://www.nerc.com/files/Standards_Input_Form_Final_2008June30.doc
Direct input to “results-based” standards initiative
• Focus on risk-based standards, using first assessment triage sheet
22Outreach EffortOutreach Effort
Chair met with and presented to the RMWG on February 18, 2010
Chair presented to the NERC/Regional Entity Compliance Group at its conference call on February 26, 2010
Chair presented to the OC/PC/CIPC at their joint WebEx conference call on March 9, 2010
23Next StepsNext Steps
Continue reaching out to seek guidance
• Members will meet with OC and PC on March 16-17, 2010
Recommend the proposals in June, desiring CCC approval in September 2010
Comments and suggestions are due April 2, 2010
• Send to PMTF Chair, Greg Pierce at [email protected] and NERC at [email protected]
24
Question & Answer
25Appendix A Appendix A
Violation Weighting Factor Table
Note: 1. The weighting values are derived by applying similar ratios developed in the Base Penalty Amount Table described in section 4 of the ERO Sanction Guidelines, Appendix 4B to the NERC Rules of Procedure. 2. Reference materials are available in a NERC white paper “Toward Ensuring Reliability: Reliability Performance Metrics”. It can be viewed at:http://www.nerc.com/docs/pc/rmwg/Reliability_Metrics_white_paper.pdf.
26VRI ExampleVRI Example
Violations # Weighted Averagein a quarter for the quarter Weighting Factor
L-L 47 0.6 0.01299L-M 16 0.4 0.02597L-H 17 0.7 0.03896L-S 52 3.4 0.06494M-L 92 2.4 0.02597M-M 25 1.3 0.05195M-H 71 5.5 0.07792M-S 137 17.8 0.12987H-L 46 2.4 0.05195H-M 39 4.1 0.10390H-H 51 7.9 0.15584H-S 135 35.1 0.25974
total 728 81.5 1.00000
VRI = 81.5 for the quarter