A Federal R&D Evaluation Framework for Influencing Safety Culture Change

in the U.S. Rail Industry

Strengthening Safety Culture Symposium

October 1-2, 2014 Halifax, Nova Scotia

MICHAEL COPLEN Senior Evaluator Office of Research and Development Office of Railroad Policy and Development Federal Railroad Administration

JOYCE RANNEY Safety Culture, Senior Evaluator

Surface Transportation Human Factors Volpe National Transportation Systems Center

1999 Study: Compliance with Railroad Operating Rules and Corporate Culture Influences

3

• Identify, develop, and implement innovative safety culture pilot projects in U.S. railroad industry

• Develop safety culture interventions applicable across different organizations and environments

• Evaluate utilization, impact, and effectiveness of pilot projects

• Where successful, support broad-scale adoption and implementation across industry

Develop a “business case” for safety culture in the railroad industry

Safety Culture in U.S. Railroad Industry Research and Evaluation Strategy, 2001

Approach Carriers Start Date Functions Outcomes

Participative Safety Rules Revision

ACBL, CSXT, KCS, CN-IC

1999 All Operating 30% reduction in reportable injuries Drop in liability claims

Root-Cause Analysis Problem Solving Canadian Pacific 2003 Mechanical

50% drop in injury rates (all injuries)

Clear Signal for Action (CSA)

• Peer-to-Peer Feedback

• Continuous Improvement

• Safety Leadership

Amtrak 2001 Station Services 76% drop in injury rates 71% drop in reportable injuries

Union Pacific 2005 Road Crews 79% drop in L.E. decertification rates 81% drop in derailments

Union Pacific 2006 Yard Crews

65% drop in yard-derailment rates

Confidential Close Call Reporting System (C³RS)

Union Pacific Canadian Pacific New Jersey Transit Amtrak

2007 2008 2009 2011

Road & Yard Crews

31% reduction in derailments at 1 site 90% drop in discipline cases 48% drop in excess-speed reports

Safety Culture Demonstration Pilot Impact Evaluations in U.S. Railroad Industry

4 4

Safety Culture Rail Industry Policy Influences

Organization Policy Changes Union Pacific “Total Safety Culture” Program

Toronto Transit System-wide safety culture change

Canadian Pacific Re-committed to ISROP

RSIA of 2008 Requires “Risk Reduction Programs” Relies on pilot programs for promulgate regulations

Amtrak Began implementing “Safe-2-Safer” Program; Joined C3RS

New Jersey Joined C3RS

BNSF Began major safety culture change initiative

*Body of evidence suggests R&D pilots strongly influenced industry wide

changes.

5

From Pilots to Carrier-wide and Industry-wide Change

6

• Union Pacific - TSC • BNSF - AO • Amtrak – S2S • Norfolk Southern

• Passenger Rail CSA • C3RS • Short Line Safety Institute

Responses

CSA • Yard • Road • Mechanical

C3RS • Yard • Road • Mechanical • Engineering

Pilots Carrier-wide

Industry-wide

FRA’s Confidential Close Call Program (C3RS)

7

8

FRA’s PAX Industry Clear Signal for Action (CSA) Program • No-cost customizable CSA software and training

materials • Low-cost implementation support

• Objective - Enhance Safety and Security in entire company by: • Safety leadership • Peer-to-peer feedback • Continuous improvement

• Simultaneous implementation • All crafts, all locations nationwide

• System Safety Organization

Restructuring

• Safe Align® (Safety Leadership Development)

• BAPP® (Behavioral Accident Prevention Process -- continuous improvement)

• “Approaching Others” (Employee-led training)

- Identifying exposure and risk - Cues for pausing work - providing effective feedback)

Responses

BNSF’s Strategic Safety Culture Initiatives

• Align Field and Safety Functions

• Provide managers with safety leadership skills

• Strengthen . . .

• Engaging workforce

Needs Responses

10

HOW CAN AN EVALUATION FRAMEWORK INFLUENCE SAFETY CULTURE?

11

CIPP Evaluation Model: (Context, Input, Process, Product)

• Context • Input

• Implementation • Impact

Daniel L. Stufflebeam's adaptation of his CIPP Evaluation Model framework for use in guiding program evaluations of the Federal Railroad Administration's Office of Research and Development. For additional information, see Stufflebeam, D.L. (2000). The CIPP model for evaluation. In D.L. Stufflebeam, G. F. Madaus, & T. Kellaghan, (Eds.), in Evaluation models (2nd ed.). (Chapter 16). Boston: Kluwer Academic Publishers.

Stakeholder engagement is key

Types of Evaluation

12

FORMATIVE SUMMATIVE When: • Before or during R&D

projects/programs • After R&D projects/programs

Purpose: To guide: • Program planning • Program design • Implementation strategies

To assess: • Completed projects or project

lifecycles • Accomplishments • Impacts To meet accountability requirements

Primary Focus:

• To improve programs • To prove program merit or worth

13

Evaluation Framework: Roles of Evaluation

Context Inputs Implementation Impact

Formative Evaluation

Identifies:

- Needs - Problems - Assets

Helps set:

- Goals - Priorities

Assesses:

- Alternative approaches

Develops:

- Program plans

- Designs - Budgets

Monitors:

- Implementation - Documents issues

Guides:

- Execution

Assesses: +/- outcomes

Reassess: - Project/program plans

Informs: - Performance metrics - Strategic planning - Policy development

Summative Evaluation

Assesses:

Original program goals and priorities

Assesses:

Original procedural plans and budget

Assesses:

- Execution

Assesses:

- Outcomes - Impacts - Side effects - Cost-effectiveness

Roles and Types of Evaluation

14

• Utility (useful): to ensure evaluations serve the information needs of the intended users.

• Feasibility (practical): to ensure evaluations are realistic, prudent, diplomatic, and frugal.

• Propriety (ethical): to ensure evaluations will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results.

• Accuracy (valid): to ensure that an evaluation will reveal and convey valid and reliable information about all important features of the subject program.

• Accountability (professional): to ensure that those responsible for conducting the evaluation document and make available for inspection all aspects of the evaluation that are needed for independent assessments of its utility, feasibility, propriety, accuracy, and accountability.

* The Program Evaluation Standards were developed by the Joint Committee on Standards for Educational Evaluation and have been accredited by the American National Standards Institute (ANSI).

15

Evaluation Standards* Guiding principles for conducting evaluations

Evaluative Framework – Safety Culture Illustrative Questions

Context Inputs Implementation Impact

Formative Evaluation

What are the highest priority needs for improving safety and safety culture? What is the existing culture and the context of that culture?

Given the need for safety culture change, what are the most promising alternatives? How do they compare (potential success, costs, etc.)? How can this strategy be most effectively implemented? What are some potential barriers to implementation?

To what extent is the program proceeding on time, within budget, and effectively? Is the program being implemented as designed? If needed, how can the design be improved?

To what extent are intended users (states, organizations, public) using the program? What other indicators of use, if any, have emerged that indicate the program is being used and behavioral change is occurring? What are some emerging outcomes (positive or negative)? How can the implementation be modified to maintain and measure long-term success?

Summative Evaluation

To what extent did the program address this high priority need?

What strategy was chosen and why, compared to other viable strategies (re. prospects for success, feasibility, costs)?

To what extent was the program carried out as planned, or modified with an improved plan?

To what extent did this program effectively address the need to improve safety and safety culture? Were there any unanticipated negative or positive side effects? What conclusions and lessons learned can be reached (i.e., cost effectiveness, stakeholder engagement, program effectiveness)?

16

Context Inputs Implementation Impact

Formative Evaluation

What is the context leading ASLRRA to commit to the SLSI? What are the industry’s high priority safety needs?

What are the possible tools and alternative approaches for improving safety culture and safety compliance?

What aspects of the original SLSI are being implemented as planned? To what extent are the safety culture and safety compliance assessment tools being used as designed? How can they be improved?

What are the emerging effects and impacts of the SLSI? Are the safety culture and safety compliance assessments being used? Why or why not?

Summative Evaluation

To what extent did the SLSI effort address ASLRRA’s high priority safety needs and goals?

What were the actual activities and outcomes delivered as part of the SLSI? How well did the assessment tools and processes map to the industry safety needs?

To what extent was the SLSI carried out as planned? How strong was the implementation, in collaboration, confidentiality, facilitating use, etc.?

To what extent has the SLSI been successful? How well did the safety culture and safety compliance assessment tools work? What were the outcomes?

Example Core Evaluation Questions: Short Line Safety Institute

Mission: to enhance safety culture and safety compliance of short line and regional railroads through voluntary, non-punitive partnerships

17

Context

Short Line Safety Institute Evaluation Framework

Inputs Outputs

• Short line and regional railroads

• Employees/labor unions

• Management • FRA RRS • Other railroads • DOT at-large • Congress • Citizenry

FRA R&D Team Assessment Tools: • Safety culture • Safety compliance • Interview protocols

Educational Materials: • Employee • Manager

Organizational Plan

Outcomes

• Fully functioning Safety Institute

• Safety action plans • Ongoing assessments • Site-based training

programs • Improved safety culture • Increased safety

compliance • Use of Institute

repository resources • Reduced accidents and

injuries

Priority Improve crude-by-rail transportation safety

Mission: Enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships.

Situation Rapid increase in crude oil production and related incidents

Activities • Needs

assessments • Job analyses • Literature

reviews • Stakeholder

engagement strategies

• Organizational planning

What we invest What we get For whom What we see

FRA R&D Funding • ASLRRA grant and

matching funds • UCONN grant • Volpe IAA

ASLRRA Team • Onsite assessments • Assessment reports • Participant feedback • Buy-in

Context, Input, Implementation, and Impact Evaluation 18

FRA Safety Culture Selected Bibliography • Zuschlag, M., Ranney, J., Coplen, M. (2012). Impact Evaluation of an Organizational Change Program for Union Pacific Road and Yard

Operations Shows Improved Safety Performance and Safety Culture. Safety Science (submitted for publication). • Zuschlag, M., Ranney, J., Coplen, M., Harnar, M. (2012, November).Transformation of Safety Culture on the San Antonio Service Unit

of Union Pacific Railroad. U.S. DOT Federal Railroad Administration. DOT-FRA-ORD-12-16. http://www.fra.dot.gov/eLib/details/L04121.

• Kath, L., Marks, K. & Ranney, J. (2010). Safety climate dimensions, leader–member exchange, and organizational support as predictors of upward safety communication in a sample of rail industry workers. Safety Science, doi:10.1016/j.ssci.2010.01.016 http://www.fra.dot.gov/eLib/details/L03035

• Morrow, S. L., McGonagle, A. K., Dove-Steinkamp, M. L., Walker, C. T. Jr., Marmet, M., and Barnes-Farrell, J. L. (2010). Relationships between psychological safety climate facets and safety behavior in the rail industry: A dominance analysis. Accident Analysis and Prevention, 42(5), 1460-1467. http://www.fra.dot.gov/eLib/Details/L04247

• Coplen, M., Ranney, J. and Zuschlag, M. (2009, September). Decreases in Collision Risk and Derailments Attributed to Changing At-Risk Behavior Process at Union Pacific, U.S. DOT Federal Railroad Administration [RR09-20]. http://www.fra.dot.gov/eLib/details/L01342

• Coplen, M., Ranney, J. and Zuschlag, M. (2009, September). Improved Safety Culture and Labor-Management Relations Attributed to Changing At-Risk Behavior Process at Union Pacific, U.S. DOT Federal Railroad Administration [RR09-19]. http://www.fra.dot.gov/eLib/details/L01462

• Coplen, M. and Ranney, J. (2009, May). Safe Practices, Operating Rule Compliance, and Derailment Rates Improve at Union Pacific Yards with STEEL Process, U.S. DOT Federal Railroad Administration [RR09-08]. http://www.fra.dot.gov/eLib/Details/L04248

• Raslear, T., Ranney, J. and Multer, J. (2008, December). Confidential Close Call Reporting System: Preliminary Evaluation Findings, U.S. DOT Federal Railroad Administration, December 2008 [RR08-33]. http://www.fra.dot.gov/eLib/Details/L04249

• Ranney, J., Wu, S., Austin, C., and Coplen, M. (2008, June). Positive Safety Outcomes of Clear Signal for Action Program at Union Pacific Yard Operations, U.S. DOT Federal Railroad Administration [RR08-09]. http://www.fra.dot.gov/eLib/Details/L04250

• Zuschlag, M., Ranney, J. and Coplen, M. (2008, June). Promising Evidence of Impact on Road Safety by Changing At-risk Behavior Process at Union Pacific, U.S. DOT Federal Railroad Administration [RR08-08]. http://www.fra.dot.gov/eLib/details/L03483

• Ranney, J. and Nelson, C. (Coplen, M. COTR ). (2007).The Impact of Participatory Safety Rules Revision on Incident Rates, Liability Claims, and Safety Culture in the U.S. Railroad Industry. U.S. DOT Federal Railroad Administration, Washington, DC [Report No. DOT/FRA/ORD-07/14]. http://www.fra.dot.gov/eLib/details/L01613

• Zuschlag, M., Ranney, J. and Coplen, M. (2007, February). Clear Signal for Action Program Addresses Locomotive Cab Safety Related to Constraining Signals, U.S. DOT Federal Railroad Administration [RR07-08]. http://www.fra.dot.gov/eLib/details/L03505

• Coplen, M., Ranney, J. and Zuschlag, M. (2007, February). Behavior-Based Safety at Amtrak-Chicago Associated with Reduced Injuries and Cost, U.S. DOT Federal Railroad Administration [RR07-07]. http://www.fra.dot.gov/eLib/details/L03506

• Coplen, M. (2007, January). Proactive Risk Management Safety Approaches for Managing Human-Factors-Caused Accidents in the Railroad Industry: Alternatives to Compliance. Pre-conference Human Factors Workshop #141. Transportation Research Board Annual Conference, Washington, D.C. http://www.fra.dot.gov/eLib/details/L03506

• Lee, M. and Ranney, J. (2006). Example of Investigation Best Practices: Interim Findings from an Evaluation of Canadian Pacific Railways' Investigation of Safety-Related Occurrences Protocol (ISROP), U.S. DOT Volpe National Transportation Systems Center, Cambridge, MA. http://www.fra.dot.gov/eLib/Details/L04251

• Ranney, J. and Lee, M. T. (2006, September). Canadian Pacific Railway Services’ 5-Alive Safety Program Shows Promise in Reducing Injuries, U.S. DOT Federal Railroad Administration [RR06-14]. http://www.fra.dot.gov/eLib/details/L03517

• Ranney, J. and Lee, M. T. (2006, September). Canadian Pacific Railway Investigation of Safety-Related Occurrences Protocol Considered Helpful by both Labor and Management, U.S. DOT Federal Railroad Administration [RR06-13]. http://www.fra.dot.gov/eLib/details/L03518

FRA Safety Culture Selected Bibliography (cont.)

• Ranney, J. and Zuschlag, M. (2006, September). Behavior-Based Safety at Amtrak-Chicago Associated with Reduced Injuries and Costs, U.S. DOT Federal Railroad Administration [RR06-12].

• Coplen, M. (2006, January). Danger, people working: multi-modal lessons on improving safety through work process observations and process improvement methods. Pre-Conference Human Factors Workshop. Transportation Research Board Annual Conference, Washington, D.C.

• Ranney, J., Nelson, C. and Coplen, M. (2005). The Efficacy of Behavior-based Safety in the U.S. Railroad Industry: Evidence from Amtrak-Chicago. Transportation Research Board 85th Annual Meeting, Pp. P06-0633. http://www.fra.dot.gov/eLib/details/L04252

• Ranney, J. and Nelson, C. (2004). Impacts of Participatory Safety Rules Revision in U.S. Railroad Industry: An Exploratory Assessment. Transportation Research Record: Journal of the Transportation Research Board of the National Academies, Washington, D.C., pp 156-163. http://www.fra.dot.gov/eLib/Details/L04253

• Ranney, J. (2004, January). Safety Rules Revisions Impact on Safety Culture, Incident Rates, and Liability Claims in the U.S. Railroad Industry: A Summary of Lessons-Learned, U.S. DOT Volpe National Transportation Systems Center, Cambridge, MA. http://www.fra.dot.gov/eLib/Details/L04258

• Coplen, M. (2003, August). FRA R&D Pilot Safety Initiative: Proactive Preventative Safety. Presentation on behavior-based safety at BNSF labor/management meeting. Alliance, NE.

• Coplen, M. and Ranney, J. (2003, January). The Impact of Safety Rules Revisions on Safety Culture, Incident Rates, and Liability Claims in the U.S. Railroad Industry, U.S. DOT Federal Railroad Administration [RR03-03]. http://www.fra.dot.gov/eLib/details/L03554

• Ranney, J. (2003). Safety Culture: TRB Human Factors in Transportation Workshop 105. Pre-conference workshop presented at the 82nd Transportation Research Board Meeting, January 12, 2003, Washington, DC. http://www.fra.dot.gov/eLib/details/L04259

• Ranney, J. and Coplen, M. (2000). Assessing At-Risk Behavior in Railroad Operations. Presentation at Transportation Research Board Annual Conference. January, 2000.

• Coplen, M. (1999).Compliance with Railroad Operating Rules and Corporate Culture Influences: Results of a Focus Group and Structured Interviews. U.S. DOT Federal Railroad Administration, Washington, DC [Report No. DOT/FRA/ORD-99/09]. http://www.fra.dot.gov/eLib/details/L04185

FRA Safety Culture Selected Bibliography (cont.)

Conclusion: Research and development programs in federal agencies can play a major role influencing industry safety culture, but it is much more than simply publishing results. Incorporate Evaluation as a Key Strategy Tool.

• Ask, then answer, questions that matter.

About processes, products, programs, policies, and impacts Helped identify, develop, and design pilot safety culture implementation projects

• Monitor the extent to which, and the ways, in which projects and programs are being

implemented. What’s working, and why, or why not? Monitored pilot implementations for ongoing improvement

• Measure the outcomes and impacts.

Inform others about lessons learned, progress, and program impacts Documented safety and safety culture outcomes from pilot implementations

• Refine program strategy, design, and implementation.

Where successful programs are confirmed, supports broad-scale adoption across the industry Helped identify industry partners and inform strategy for company and industry-wide scale-up

• Systematically engage key stakeholders to improve program success.

Identify and actively involve intended users Clarify intended uses and potential misuses Increased the utilization, impact, and effectiveness of pilot safety culture project outcomes for broader scale

adoption and sustainability

Contact Information

MICHAEL COPLEN • Senior Evaluator • Office of Research and Development • Office of Railroad Policy and Development • Federal Railroad Administration Michael.coplen@dot.gov JOYCE RANNEY • Safety Culture, Senior Evaluator • Surface Transportation Human Factors • Volpe, The National Transportation Systems Center Joyce.ranney@dot.gov

23

EXTRA SLIDES

24

Evaluation Standards *Guiding principles for conducting evaluations

Utility (useful)

Feasibility (practical)

Propriety (ethical)

Accuracy (valid)

Evaluation Accountability (professional)

• Evaluator Credibility

• Attention to Stakeholders

• Negotiated Purposes

• Explicit Values • Relevant

Information • Meaningful

Processes & Products

• Timely & Appropriate Reporting

• Concern for Consequences & Influence

• Project Management

• Practical Procedures

• Contextual Validity

• Resource Use

• Responsive & Inclusive Orientation

• Formal Agreements

• Human Rights & Respect

• Clarity & Fairness • Transparency &

Disclosure • Conflicts of Interest • Fiscal

Responsibility

• Justified conclusions & decisions

• Valid Information • Reliable

Information • Explicit Program &

Context Description

• Information Management

• Sound Design & Analyses

• Explicit Evaluation Reasoning

• Communication & Reporting

• Evaluation Documentation

• Internal Metaevaluation

• External Metaevaluation

Note: The Program Evaluation Standards were developed by the Joint Committee on Educational Evaluation and have been accredited by the American National Standards Institute (ANSI). 25

American Evaluation Association (http://www.eval.org) •3000 members in 2001 •over 7700 members today • all 50 states •over 60 countries •$95/year membership, includes

– American Journal of Evaluation – New Directions in Evaluation – online access to full journal articles

Evaluation Resources

26

• Affiliate Evaluation Associations – Washington Research and Evaluation Network (WREN) – Federal Evaluator’s Network

• Evaluation Journals

– American Journal of Evaluation (AJE) – New Directions for Evaluation (NDE) – Evaluation Review – Evaluation and the Health Professions

• The Evaluator’s Institute (http://tei.gwu.edu/courses_dc.htm)

– George Washington University

• The Evaluation Center (http://www.wmich.edu/evalctr/) – Western Michigan University

Evaluation Resources

27

• Commitment from all key stakeholders

• Voluntary, confidential/anonymous, non-punitive participation

• Systematic and objective data gathering, analysis, and reporting

• Problem solving, barrier identification and removal, corrective action process

• Long-term sustaining mechanisms

Common Elements of Successful Safety Culture Change

28

Corrective actions were not just focused on the individuals involved in the event

ISROP results led to system-wide improvements

Safety Alert issued across company in 2004

Updated jacking guidelines prepared in 2006

Safety Culture Rail Industry Policy Influences: ISROP Case Study – System Wide Impact

29

‘‘§ 20156. Railroad safety risk reduction program ‘‘(1) PROGRAM REQUIREMENT.— ... the Secretary of Transportation . . . shall require each railroad carrier ... ‘‘(A) to develop a railroad safety risk reduction program under subsection (d) that systematically evaluates railroad safety risks on its system and manages those risks . . . ‘‘(2) RELIANCE ON PILOT PROGRAM.—The Secretary may conduct behavior-based safety and other research, including pilot programs, before promulgating regulations under this subsection and thereafter. The Secretary shall use any information and experience gathered through such research and pilot programs under this subsection in developing regulations under this section.”

110TH CONGRESS of the United States of America

H. R. 2095 Rail Safety Improvement Act of 2008

Policy Influence at U.S. Congress

Policy Influence at U.S. DOT Safety Council

Safety Culture Action Team

–Safety Culture Research Paper

–DOT Safety Policy Statement

31

DOT Safety

Culture

DOT Actions

Safety Culture:

Industry Organizations

States Public

Actions: Industry

Organizations States Public

Safety Outcomes

The traditional view of Federal intervention

The prevailing view of safety culture

U.S. DOT and Safety Culture

CSA Overview What is CSA?

Safer Culture

Peer-to-peer Feedback

Safer Practices

Continuous Improvement

Safer Workplace

Confidential Data

Safety Leadership

33

Program Evaluations: Company and Industry-wide Safety Culture Change Initiatives*

Organization/Program Company or Industry-wide Changes Confidential Close Call Reporting System (C3RS)

Transitioning from pilots to systemwide program

CSA PAX Operations No-cost training and educational materials

Amtrak “Safe-to-Safer” Program

BNSF

Safe Align® “Approaching Others” (EST/AO) Behavioral Accident Prevention Process (BAPP)® Safety System Organizational Restructuring

ASLRRA Short Line Safety Institute * Ongoing FRA R&D Evaluation Projects

34

• Vision: for the short line and regional railroad industry to be recognized as one of the safest in the world

• Mission: to enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships

• Strategic Goals: • to enhance and improve safety practices • to increase the short line and regional railroad

industry’s culture of commitment to safety

Short Line Safety Institute Vision, Mission, and Strategic Goals Mission: Enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships.

35

• FRA R&D Team (FRA, Volpe, UCONN) Activities: – Tool development: Pilot test, design, and implement protocols, tools, and

procedures for assessing safety culture and safety compliance – Program development: Conduct organizational structure needs assessment,

and recommend appropriate and effective organizational structure for Safety Institute

– Evaluate project development, implementation processes, and ongoing outcomes to inform efforts for larger-scale Safety Institute

• ASLRRA/Safety Institute Team Activities: – Conduct safety culture and safety compliance assessments – Provide safety education, training, and development to managers and

employees – Implement the Safety Institute’s organizational development plan

• Transition: Following Pilot Project evaluation, transition to a permanent, expanded Short Line Safety Institute with embedded continuous improvement

Short Line Safety Institute Pilot Project Overview Mission: Enhance safety culture and safety compliance of short line and regional railroads through voluntary, nonpunitive partnerships.

36