Final Report Development of an Operation and Maintenance Plan and Training and Certification for Direct Potable Reuse (DPR) Systems

Development of an Operation and Maintenance Plan and Training and Certification for Direct Potable Reuse (DPR) Systems

Troy Walker, MIE (Aust) Benjamin D. Stanford, Ph.D. Aaron Duke, P.E., BCEE Meric Selbes, Ph.D. Doug Kobrick, P.E. Ryan Nagel, P.E., PNV EP Sean Pour, P.E., Ph.D. Robert Boysen, P.E. Hazen and Sawyer

Debra Burris, P.E., BCEE DDB Engineering Inc.

John Caughlin OperatorStar

Jim Vickers, P.E. Separation Processes Inc.

Co-sponsors Metropolitan Water District of Southern California City of Los Angeles, CA City of San Diego, CA Gwinnett County, GA West Basin Municipal Water District, CA Orange County Water District, CA Suez Environnement Trojan UV Separation Processes Inc. Additional Anonymous Utilities in the U.S. and Australia

Water Environment & Reuse Foundation Alexandria, VA

ii Water Environment & Reuse Foundation

The Water Environment & Reuse Foundation (WE&RF) is a 501c3 charitable corporation seeking to identify, support, and disseminate research that enhances the quality and reliability of water for natural systems and communities with an integrated approach to resource recovery and reuse; while facilitating interaction among practitioners, educators, researchers, decision makers, and the public. WE&RF subscribers include municipal and regional water and water resource recovery facilities, industrial corporations, environmental engineering firms, and others that share a commitment to cost-effective water quality solutions. WE&RF is dedicated to advancing science and technology addressing water quality issues as they impact water resources, the atmosphere, the lands, and quality of life. For more information, contact: Water Environment & Reuse Foundation 1199 North Fairfax Street, 9th Floor Alexandria, VA 22314 Tel: (571) 384-2100 www.werf.org werf@werf.org © Copyright 2016 by the Water Environment & Reuse Foundation. All rights reserved. Permission to copy must be obtained from the Water Environment & Reuse Foundation. Library of Congress Catalog Card Number: 2016953442 WE&RF ISBN: 978-1-94124-255-1 This report was prepared by the organization(s) named below as an account of work sponsored by the Water Environment & Reuse Foundation (WE&RF). Neither WE&RF, members of WE&RF, the organization(s) named below, nor any person acting on their behalf: (a) makes any warranty, express or implied, with respect to the use of any information, apparatus, method, or process disclosed in this report or that such use may not infringe on privately owned rights; or (b) assumes any liabilities with respect to the use of, or for damages resulting from the use of, any information, apparatus, method, or process disclosed in this report. Hazen and Sawyer This document was reviewed by a panel of independent experts selected by WE&RF. Mention of trade names or commercial products or services does not constitute endorsement or recommendations for use. Similarly, omission of products or trade names indicates nothing concerning WE&RF's positions regarding product effectiveness or applicability.

Water Environment & Reuse Foundation iii

About WE&RF The Water Environment & Reuse Foundation (WE&RF) is a 501c3 charitable corporation seeking to identify, support, and disseminate research that enhances the quality and reliability of water for natural systems and communities with an integrated approach to resource recovery and reuse; while facilitating interaction among practitioners, educators, researchers, decision makers, and the public. Our research represents a portfolio of more than $200 million in water quality research. WE&RF operates with funding from subscribers, donors, state agencies, and the federal government. Our supporters include wastewater treatment facilities, stormwater utilities, and regulatory agencies. Equipment companies, engineers, and environmental consultants also lend their support and expertise. WE&RF takes a progressive approach to research, stressing collaboration among teams of supporters, environmental professionals, scientists, and staff. All research is peer reviewed by leading experts. For the most current updates on WE&RF research, sign up to receive Laterals, our bi-weekly electronic newsletter. Learn more about the benefits of becoming a WE&RF supporter by visiting www.werf.org.

iv Water Environment & Reuse Foundation

Abstract & Benefits

Abstract: This project was developed in response to the need to provide meaningful, actionable advice to the direct potable reuse (DPR) initiative at the Water Environment & Reuse Foundation and to provide information that can assist the Expert Panel, CDPH, and SWRCB in making DPR a reality for California. Specifically, there was an opportunity to provide input on permitting structure, operations and maintenance protocols, and training and certification programs, as well as to develop some of the content/curriculum that can be used in the training and response setting. In order to address the gaps in training, certification, and permitting programs and to support the California DPR initiative, the objective of this project was to develop a standard operations and maintenance plan framework for various DPR treatment processes and to develop a DPR Training and Certification Framework for DPR system operators. As the perceived “human element” in the process, operation teams must have robust and reliable operational plans, systems, and processes to ensure safety and reliability – essential elements for the advancement of public acceptance of DPR. Relative to existing water and wastewater treatment systems, operations teams are under much greater scrutiny for performance, and must therefore have adequate training and certification processes in place to provide a framework for developing and evaluating the necessary skills. The objective was to identify the key requirements for operation of DPR systems, and articulate this in an operational framework. Specific attention was also given to the requirements of a training and certification framework. To deliver this project, a team of experts were assembled who have direct experience in the establishment of operations and maintenance protocols for recycled water schemes around the world, have direct experience in the permitting of IPR schemes in California and have direct experience with training and certification of IPR operators in California. To meet these objectives, the project followed a two-phase approach to effectively answer two questions: What are the vital operational and maintenance requirements that must be integrated into a permitting

scheme to ensure the success of DPR? Based on these requirements, how can we ensure operators can meet these requirements?

The project was developed in two phases: Phase 1: Develop a Standard Operations and Maintenance Plan for DPR Schemes. Phase 2: Develop a DPR Training and Certification Curriculum Framework for DPR System Operators.

The overall output from this research was the outline of an operational framework for DPR. The framework incorporates risk management principles and in particular the HACCP (Hazard Analysis and Critical Control Point) methodology considering a holistic review of operational requirements. In addition, a detailed gap analysis of California’s Code of Regulations governing both recycled water and drinking water were reviewed to determine what changes are required to accommodate DPR. Key recommendations are included in the report.

Key operational procedures and requirements were further developed in the report, based on both an RO-based treatment train for DPR (MF/RO/UV-H2O2-Cl2) and a non-RO treatment train (Ozone/Floc-sed/BAC/GAC/UV/Cl2). For each process key operating parameters, operating procedures, key maintenance requirements and critical control point response procedures were outlined in the context of DPR which can be adopted for site-specific operational plans.

A staffing analysis for DPR systems was also considered which included a review of roles that will be required in the context of a DPR system. A staffing assessment was made of a theoretical DPR system to demonstrate a method of staffing assessment. This included both operations and maintenance roles.

Water Environment & Reuse Foundation v

A last, but very significant part of the report is a development of an operator certification framework for DPR. With a review of existing operator certification and training systems across the United States and internationally, this part of the report provides recommendations for a future DPR operational and training certification system. Benefits: Provides a comprehensive review of operational requirements in the consideration of a DPR facility. Provides an operational framework that can be used as a whole, or in part to inform and support

operational planning for future DPR, as well as IPR facilities. Supports designers, operators, and regulators in ensuring the appropriate level of design and

operational planning requirements. Operations and training certification element included in this report provides additional material for

consideration for the development of certification and training programs – adding and support other efforts currently underway in California and elsewhere.

Keywords: Direct potable reuse, HACCP, critical control points, risk reduction, operations, operational plan, recycled water plan, operator training, operator certification, regulations.

vi Water Environment & Reuse Foundation

Table of Contents Abstract and Benefits ................................................................................................................................... iv List of Figures ............................................................................................................................................... x List of Tables .............................................................................................................................................. xii List of Acronyms and Abbreviations ......................................................................................................... xiv Acknowledgments ...................................................................................................................................... xvi Chapter 1 Introduction and Background ............................................................................................. 1 1.1 Understanding the Importance of Operations in DPR ........................................................ 1 1.2 Integration of HACCP into Operations ............................................................................... 2 1.3 Permitting in California Title 17 and Title 22 Recycled Water .......................................... 5 1.4 Current Operator Training and Certification Requirements ............................................... 6 1.5 Project Objective and Technical Approach ........................................................................ 6 1.5.1 Organization of this Report .................................................................................... 7 1.6 References ........................................................................................................................... 8 Chapter 2 California Code of Regulations and Permit Structure ........................................................ 9 2.1 Overview of the Task .......................................................................................................... 9 2.2 Approach ............................................................................................................................. 9 2.3 Summary of Operations Regulatory Changes ................................................................... 12 Chapter 3 Operation and Maintenance Framework ........................................................................... 21 3.1 Operational Management .................................................................................................. 21 3.2 Risk Management Process ................................................................................................ 22 3.2.1 Risk Assessment Framework ............................................................................... 23 3.2.2 Source Water Quality Risk Assessment .............................................................. 27 3.2.3 Health and Safety Risk Assessment ..................................................................... 32 3.2.4 Environmental Risk Assessment.......................................................................... 33 3.2.5 Asset Condition and Risk Assessment ................................................................. 34 3.3 Critical Control Points – the HACCP Process .................................................................. 38 3.3.1 Hazard Analysis and Critical Control Points – HACCP ...................................... 38 3.3.3 HACCP Team ...................................................................................................... 42 3.3.4 RO-Based Treatment Train Critical Control Point Selection .............................. 43 3.3.5 Non-RO Based Treatment Train Critical Control Point Selection ...................... 45 3.3.6 Log reduction of Microorganisms ....................................................................... 48 3.3.7 Critical Control Point Selection – Wastewater Treatment ................................... 51 3.3.8 Critical Control Points – Water Treatment Plant ................................................. 53 3.3.9 Critical Control Points and Critical Operating Points .......................................... 53 3.3.10 Establishing Critical Limits ................................................................................. 54 3.4 System Validation and Verification .................................................................................. 56 3.4.1 Validation ............................................................................................................ 56 3.4.2 Pre-Commissioning Validation ............................................................................ 56 3.4.3 Commissioning Validation .................................................................................. 56 3.4.4 Commissioning Verification ................................................................................ 57 3.4.5 Revalidation ........................................................................................................ 57 3.5 Operational Control and Monitoring ................................................................................ 57 3.5.1 DPR Control Systems .......................................................................................... 57 3.5.2 Performance Trending and Data Management .................................................... 59 3.5.3 Alarms .................................................................................................................. 60 3.5.4 Response Procedures and Implementation of Alarm Strategies .......................... 64 3.5.5 Specific Alert and Alarm Procedures for Identified CCPs .................................. 68

Water Environment & Reuse Foundation vii

3.5.6 Control System Testing ....................................................................................... 69 3.5.7 Analyzer Management ......................................................................................... 69 3.5.8 Dashboard Reporting ........................................................................................... 71 3.6 Operating Interfaces .......................................................................................................... 75 3.6.1 Operating Across Different Jurisdictions ............................................................. 75 3.6.2 Source Water Management .................................................................................. 75 3.6.3 Operational Interface Protocols ........................................................................... 76 3.7 Asset Management ................................................................................................................... 77 3.7.1 Development of an Asset Management Framework for DPR ............................. 77 3.7.2 Asset Management Business Planning ................................................................ 80 3.7.3 Infrastructure Evaluation / Planning .................................................................... 82 3.7.4 Current state of assets .......................................................................................... 83 3.7.5 Asset Risk Assessment ........................................................................................ 85 3.7.6 Financial Management / Planning ....................................................................... 87 3.7.7 Performance Reporting ........................................................................................ 90 3.7.8 Maintenance Management Strategy ..................................................................... 91 3.7.9 Work Order Management System ....................................................................... 93 3.8 Water Quality Monitoring................................................................................................. 94 3.8.1 Advanced Treatment Plant Source Monitoring ................................................... 94 3.8.2 Operational Monitoring ....................................................................................... 97 3.8.3 Verification Sampling Programs (Advanced Treatment Plant Product Sampling) ... 98 3.8.4 Sampling and Analysis Management................................................................. 101 3.9 Incident and Emergency Management ............................................................................ 102 3.9.1 Principles of Incident and Emergency Management Plan ................................. 102 3.9.2 Incident Types and Categories ........................................................................... 103 3.9.3 Incident Response .............................................................................................. 106 3.9.4 Incident Investigation ........................................................................................ 109 3.9.5 Incident Prevention and Preparedness ............................................................... 109 3.9.6 Scenario Training ............................................................................................... 110 3.10 Change Management ...................................................................................................... 110 3.10.1 The Change Management Process ..................................................................... 110 3.10.2 Change Management Procedure ........................................................................ 111 3.11 Operating Manual and Procedures .................................................................................. 114 3.11.1 Operation and Maintenance Plans ..................................................................... 114 3.11.2 Standard Operating Procedures ............................................................................ 114 3.11.3 Unit Process Guidelines ....................................................................................... 115 3.12 Case Study – Operation and Maintenance of Groundwater Replenishment System ...... 116 3.12.1 Project Description ............................................................................................ 116 3.12.2 Permits ............................................................................................................... 117 3.12.3 Interagency Agreements and Policies ................................................................ 118 3.13 Operator Training ........................................................................................................... 120 3.14 Transparency and Auditing ............................................................................................. 120 3.14.1 Internal Auditing ................................................................................................ 120 3.14.2 External Auditing ............................................................................................... 121 3.15 References ....................................................................................................................... 121 Chapter 4 Key Operational Procedures and Requirements ............................................................. 123 4.1 Introduction ..................................................................................................................... 123 4.2 RO-Based Process – Chloramination .............................................................................. 124 4.2.1 Process Overview .............................................................................................. 124 4.2.2 Key Operating Parameters ................................................................................. 125 4.2.3 General Operating Procedures ........................................................................... 126

viii Water Environment & Reuse Foundation

4.2.4 Critical Control Point Response Procedure ....................................................... 126 4.2.5 Regular Maintenance Activities ......................................................................... 129 4.3 RO-Based Process – Microfiltration ............................................................................... 129 4.3.1 Process Overview .............................................................................................. 129 4.3.2 Key Operating Parameters ................................................................................. 130 4.3.3 General Operating Procedures ........................................................................... 131 4.3.4 Critical Control Point Response Procedures ...................................................... 131 4.4 RO-Based Process – Reverse Osmosis ........................................................................... 135 4.4.1 Process Overview .............................................................................................. 135 4.4.2 Key Operating Parameters ................................................................................. 135 4.4.3 General Operating Procedures ........................................................................... 136 4.4.4 Critical Control Point Response Procedures ...................................................... 136 4.4.5 Regular Maintenance Activities ......................................................................... 140 4.5 RO-Based Process – UV/H202 ........................................................................................ 140 4.5.1 Process Overview .............................................................................................. 140 4.5.2 Key Operating Parameters ................................................................................. 141 4.5.3 General Operating Procedures ........................................................................... 142 4.5.4 Critical Control Point Resources ....................................................................... 143 4.5.5 Maintenance Activities ...................................................................................... 147 4.5.6 Operations-Focused Design Considerations ...................................................... 147 4.6 RO-Based Process – Stabilization .................................................................................. 148 4.6.1 Process Overview .............................................................................................. 148 4.6.2 Key Operating Parameters ................................................................................. 149 4.6.3 General Operating Procedures ........................................................................... 149 4.6.4 Critical Control Point Response Procedures ...................................................... 149 4.6.5 Regular Maintenance Activities ......................................................................... 152 4.7 RO-Based Process and Non-RO-Based Process – Chlorine CT ..................................... 152 4.7.1 Process Overview .............................................................................................. 152 4.7.2 Key Operating Parameters ................................................................................. 152 4.7.3 General Operating Procedures ........................................................................... 154 4.7.4 Critical Control Point Response Procedures ...................................................... 155 4.7.5 Regular Maintenance Activities ......................................................................... 159 4.8 Non-RO-Based Process – Ozone .................................................................................... 159 4.8.1 Process Overview .............................................................................................. 159 4.8.2 Key Operating Parameters ................................................................................. 160 4.8.3 General Operating Procedures ........................................................................... 163 4.8.4 Critical Control Point Response Procedures ...................................................... 165 4.8.5 Maintenance Activities ...................................................................................... 172 4.8.6 Operations-Focused Design Considerations ...................................................... 172 4.9 Biologically Active Carbon (BAC) ................................................................................ 173 4.9.1 Process Overview .............................................................................................. 173 4.9.2 Key Operating Parameters ................................................................................. 174 4.9.3 General Operating Procedures ........................................................................... 175 4.9.4 Critical Control Point Response Procedures ...................................................... 177 4.9.5 Maintenance Activities ...................................................................................... 184 4.9.6 Operations-Focused Design Considerations ...................................................... 184 4.10 GAC…. ........................................................................................................................... 185 4.10.1 Process Overview .............................................................................................. 185 4.10.2 Key Operating Parameters ................................................................................. 186 4.10.3 General Operating Procedures ........................................................................... 187 4.10.4 Critical Control Point Resources ....................................................................... 188

Water Environment & Reuse Foundation ix

4.10.5 Maintenance Activities ...................................................................................... 192 4.10.6 Operations-Focused Design Considerations ...................................................... 192 4.11 UV ................................................................................................................................... 193 4.11.1 Process Overview .............................................................................................. 193 4.11.2 Key Operating Parameters ................................................................................. 194 4.11.3 General Operating Procedures ........................................................................... 195 4.11.4 Critical Control Point Responses ....................................................................... 195 4.11.5 Maintenance Activities ...................................................................................... 199 4.11.6 Operations-Focused Design Considerations ...................................................... 199 4.12 References ....................................................................................................................... 200 Chapter 5 Recommended Staffing ................................................................................................... 201 5.1 Approach to Determining O&M Staffing for Water Reuse Facilities ............................ 201 5.1.1 Overview ............................................................................................................ 201 5.1.2 Approaches ........................................................................................................ 202 5.1.3 Planning Stage – Benchmarking Assessment .................................................... 202 5.1.4 Design Stage – Zero-Based Assessment ............................................................ 208 5.2 Zero-Based Assessment Example ................................................................................... 211 5.2.1 Example – RO Membrane-Based DPR .............................................................. 211 5.2.2 Estimation of Hours Available per Full Time Equivalent. ................................ 213 5.2.3 Task-Based Analysis – Operations .................................................................... 214 5.2.4 Task-Based Analysis – Maintenance ................................................................. 216 5.3 Minimum Operator Certification Requirements ............................................................. 219 5.4 Important Roles and Responsibilities ............................................................................. 220 Chapter 6 Operator Certification and Training ................................................................................ 221 6.1 Introduction ..................................................................................................................... 221 6.2 California – Existing Training and Certification Programs ............................................ 221 6.2.1 Drinking Water Training and Certification ........................................................ 222 6.2.2 Wastewater Training and Certification .............................................................. 226 6.2.3 Example Operator Certification at Advanced Reuse Facilities in California .... 230 6.3 Certification in Other States/Countries ........................................................................... 231 6.3.1 Washington ........................................................................................................ 231 6.3.2 Virginia .............................................................................................................. 231 6.3 3 Texas .................................................................................................................. 231 6.3.4 Arizona .............................................................................................................. 232 6.3.5 North Carolina ................................................................................................... 235 6.3.6 Wisconsin .......................................................................................................... 236 6.3.7 Australia ............................................................................................................. 237 6.4 Summary of Existing Systems ........................................................................................ 246 6.5 Proposed Operator Certification IPR/DPR Curriculum .................................................. 247 6.5.1 Treatment Processes for Consideration ............................................................. 247 6.5.2 Existing Water and Wastewater Curricula in California ................................... 248 6.5.3 Existing Sources for Additional Curriculum ..................................................... 258 6.6 Proposed Operator Certification Framework for DPR ................................................... 259 6.6.1 Option 1 – Specific DPR Certification Curriculum ........................................... 259 6.6.2 Option 2. DPR “Add On” to Existing Certification Frameworks ...................... 262 6.6.3 Option 3 – DPR “Add On” to Existing Certification Frameworks Covering Water/Wastewater Certification Gaps ............................................................... 263 6.6.4 Consideration of Competency Based Curriculum ............................................. 263 6.6.5 Educational Requirements and Continuing Education Credits .......................... 264 6.6.6 Overall Recommendation .................................................................................. 264 6.7 References ....................................................................................................................... 265

x Water Environment & Reuse Foundation

List of Figures 1.1 The Western Corridor Recycled Water Management Plan Integrates HACCP Through the

Entire Operations and Maintenance Framework. ............................................................................ 4 2.1 Drinking Water Operations Regulatory Changes to Accommodate DPR ..................................... 12 2.2 Recycled Water Operations Regulatory Changes to Accommodate DPR ..................................... 13 3.1 Proposed DPR Operational Management Framework ................................................................... 22 3.2 General Risk Assessment Approach .............................................................................................. 24 3.3 Risk Ranking Matrix Reuse-13-03 ................................................................................................ 26 3.4 Possible Symbols and Removal Rates for Risk Register ............................................................... 29 3.5 Extract of the Inherent and Residual Risk Assessment .................................................................. 30 3.6 Diagram of a CCP and Associated Monitor for an RO System ..................................................... 39 3.7 Diagram of Critical Operating Points (COPs) Supporting the RO CCP and Plant Production ..... 40 3.8 HACCP System Decision Tree for Defining Critical Control Points in DPR Facilities ................ 41 3.9 Critical Control Points (Outlined) – RO Membrane-Based Treatment Train ................................ 45 3.10 Critical Control Points (Outlined) – Ozone-BAC-Based Treatment Train as Shown with

Optional Pre-Ozonation Step. ........................................................................................................ 47 3.11 General Schematic of Urban Infrastructure with DPR .................................................................. 51 3.12 Control System Elements ............................................................................................................... 58 3-13 Graphical Depiction of a Time Delay Alarm Trigger (True) and a False Event Resulting in

No Alarm ....................................................................................................................................... 62 3-14 Graphical Depiction of a Moving Average Alarm (True) and a False Event Resulting in

No Alarm ....................................................................................................................................... 62 3-15 Graphical Depiction of a Block Average Alarm Trigger (True) and a False Event Resulting in

No Alarm ....................................................................................................................................... 63 3-16 Graphical Depiction of a Point to Point Alarm Trigger (True) and a False Event Resulting in

No Alarm ....................................................................................................................................... 63 3-17 Generic Alert Level Response Procedure ...................................................................................... 66 3-18 Generic Critical Alarm Response Procedure ................................................................................. 67 3.19 RO-Based Process – CCP Dashboard ............................................................................................ 71 3.20 Example Overall Dashboard Report - RO Based Treatment Process ............................................ 72 3.21 Non-RO-Based Process – CCP Dashboard .................................................................................... 73 3.22 Overall Dashboard Report Non-RO-Based Process ...................................................................... 74 3.23 Key Asset Management Program Elements ................................................................................... 77 3.24 Levels of Service and Asset Management Program Cascade ........................................................ 78 3.25 ISO 55001 Asset Management Framework ................................................................................... 79 3.26 Example ISO 55001 Gap Analysis ................................................................................................ 80 3.27 Data and System Management for DPR Facilities ......................................................................... 82 3.28 Core Questions of Asset Management ........................................................................................... 83 3.29 Probability of Failure (PoF) Methodology..................................................................................... 85 3.30 Consequence of Failure (CoF) Categories ..................................................................................... 86 3.31 Example 100-year Asset Renewal Projection ................................................................................ 88 3.32 Example Prioritized CIP ................................................................................................................ 90 3.33. Test to Determine Proactive Maintenance ..................................................................................... 92 3.34. Preventative versus Predictive Maintenance .................................................................................. 93 3.35 Advanced Treatment Plant Feed Sampling Parameter Ongoing Selection Process

(Adapted from Western Corridor Recycled Water Project, Queensland Australia). ..................... 95 3.36 Annual Review of Verification Monitoring –

Adapted from Western Corridor Recycled Water Project – SEQWater. ....................................... 99

Water Environment & Reuse Foundation xi

3.37 Responsive Verification Monitoring Program – Approach Taken from Western Corridor Recycled Water Project – SEQWater ......................... 100

3.38 Example Change Management Process for Major Change.......................................................... 113 4.1 RO-Based Treatment Train .......................................................................................................... 123 4.2 Non-RO-Based Treatment Train .................................................................................................. 124 4.3 Chloramine Dosing Critical Alert (Warning) Response .............................................................. 127 4.4 Chloramine Dosing Critical Alarm (Failure) Response ............................................................... 128 4.5 Membrane Filtration Guidance Manual Equation 4.1 (U.S. EPA) .............................................. 130 4.6 MF/UF Alerts (Warning) Response ............................................................................................. 133 4.7 MF/UF Critical Alarm (Failure) Response .................................................................................. 134 4.8 RO Alert (Warning) Response ..................................................................................................... 138 4.9 RO Critical Alarm (Failure) Response......................................................................................... 139 4.10 UV/AOP Alert (Warning) Response............................................................................................ 145 4.11 UV/AOP Critical Alarm (Failure) Response ............................................................................... 146 4.12 Chemical Stabilization Alert (Warning) Response ...................................................................... 150 4.13 Chemical Stabilization Alarm (Failure) Response ....................................................................... 151 4.14 Disinfection System (Chlorine) Alert (Warning) Response ........................................................ 157 4.15 Disinfection System (Chlorine) Alarm (Failure) Response ......................................................... 158 4.16 Reactor Configuration and Monitoring Equipment < 3 consecutive reaction chambers ............. 162 4.17 Reactor Configuration and Monitoring Equipment >= 3 consecutive reaction chambers ........... 162 4.18 Ozone Low CT and Low Flow Alert Procedures ........................................................................ 168 4.19 Ozone Low CT and Low Ozone Residual Alert Procedures ....................................................... 169 4.20 Ozone Low CT and Low UVT Alert Procedures ........................................................................ 170 4.21 Ozone Low CT Critical Failure Response ................................................................................... 171 4.23 Ozone/BAC Alert Response Procedures...................................................................................... 180 4.24 Ozone/BAC Critical Failure Response Procedures ..................................................................... 181 4.25 Coagulant/BAC Alert Response Procedures ................................................................................ 182 4.26 Coagulant/BAC Critical Failure Response Procedures ............................................................... 183 4.27 GAC Alert Response Procedures ................................................................................................. 190 4.28 GAC Critical Failure Response Procedures ................................................................................. 191 4.29 UV Disinfection Alert Response Procedures ............................................................................... 197 4.30 UV Disinfection Critical Failure Response Procedures ............................................................... 198 5.1 Process Flow Diagram OCWD GWRS – Courtesy OCWD ........................................................ 203 5.2 Processes – West Basin Recycled Water Treatment System (Courtesy WBMWD) ................... 205 5.3 Existing and Proposed Process Trains, Gwinnett County F. Wayne Hill Facility, GA. .............. 205 5.4 RO-Based Advanced Treatment Process – For Staffing Evaluation ................................................. 5.5 Maintenance Estimate FTE by Plant Process – 10 MGD RO-Based Process ............................. 218 5.6 Maintenance Estimate FTE by Trade/Skill Discipline – 10 MGD RO-Based Process ............... 219 6.1 Existing Water and Wastewater Operator Certification Framework – California ....................... 230 6.2 Process of Water and Wastewater Treatment Plant Operator Certification ................................. 246 6.3 RO Membrane-Based Treatment Train........................................................................................ 247 6.4 Ozone-BAC-Based Treatment Train ........................................................................................... 248 6.4 DPR Operator Certification Framework Option 1 ....................................................................... 260 6.5 DPR Operator Certification Framework 2 ................................................................................... 262 6.6 DPR Operator Certification Framework 3 ................................................................................... 263

xii Water Environment & Reuse Foundation

List of Tables 2.1 Summary of Recommended Operations Regulatory Changes to Accommodate DPR .................. 10 2.2 Detailed Gap Recycled Water and Drinking Water Regulation Gap Analysis .............................. 14 3.1 Risk Likelihood Descriptor – Source Water Risk Assessment Example ....................................... 25 3.2 Risk Consequence Descriptor – Source Water Risk Assessment Example Reuse-13-03 .............. 25 3.3 Examples of Categories of Typical Hazards and Sources ............................................................. 27 3.4 Risk Consequence Descriptors – Health and Safety Risk Assessment .......................................... 32 3.5 Risk Descriptors – Likelihood Health and Safety Risk Assessment .............................................. 33 3.6 Asset Condition and Likelihood of Failure .................................................................................... 34 3.7 Physical and Performance Condition Failure Modes ..................................................................... 35 3.8 Criticality Indices ........................................................................................................................... 36 3.9 Asset Risk Assessment Matrix Example ....................................................................................... 37 3.10 Example Asset Assessment Scoring .............................................................................................. 37 3.11 CCP Selection Process and Indicators – RO Membrane-Based Treatment ................................... 43 3.12 Monitoring Parameters for CCPs in RO-Based Treatment ............................................................ 45 3.13 CCP Selection Process – Non-Membrane Treatment Process ....................................................... 46 3.14 Monitoring Parameters for CCPs in Ozone-BAC-Based Treatment ............................................. 47 3.15 Log-Removal Targets Based on CA IPR Regulations, WRRF-11-02, and TCEQ Requirements .... 48 3.16 Virus Creditable Log Reduction by Treatment Process ................................................................. 49 3.17 Cryptosporidium Creditable Log reduction by Treatment Process ................................................ 49 3.18 Giardia Creditable Log Reduction by Treatment Process ............................................................. 50 3.19 Examples of Critical Operating Points .......................................................................................... 54 3.20 Example Consequence of Failure Summary for a DPR Facility.................................................... 87 3.21 Potential Basis of Advanced Water Treatment Feed Analysis ...................................................... 97 3.22 Example External Laboratory Sampling Frequency for Operational Monitoring. ........................ 98 3.23 Example Monitoring Frequency from Western Corridor Project ................................................ 101 3.24 Examples of Incident Categories for Some Operational Areas ................................................... 105 3.25 Incident Category and Reporting Time Frame ............................................................................ 106 3.26 Incident Response for Individuals and Management Teams ....................................................... 108 3.27 Training Types and Frequency .................................................................................................... 110 3.28 Examples and Definitions of Change Categories ........................................................................ 111 3.29 Operator Staffing for OCWD’s GWRS ....................................................................................... 118 4.1 Chloramine Alert and Alarm Example Setpoints ........................................................................ 126 4.2 Microfiltration/Ultrafiltration Alert and Alarm (Summary) ........................................................ 132 4.3 Reverse Osmosis Membrane Alert and Alarm Summary ............................................................ 137 4.4 CCP Monitoring Parameters for UV AOP ................................................................................... 141 4.5 Chemical Stabilization Alert and Alarm Summary ..................................................................... 149 4.6 Chlorine Disinfection Alert and Alarm Level Summary ............................................................. 156 4.7 Ozone Low CT Alert and Alarm Summary ................................................................................. 167 4.8 Ozone-BAC Alert and Alarm Summary ...................................................................................... 178 4.9 Coagulation-BAC Alert and Alarm Summary ............................................................................. 179 4.10 GAC Alert and Alarm Summary ................................................................................................. 189 4.11 UV Dose (mJ/cm2) Required to Achieve Pathogen Inactivation (Ref: U.S. EPA UVDGM) ...... 194 5.1 Available Information during Stages of DPR Facility Development .......................................... 202 5.2 Advanced Reuse Staffing Benchmark Survey – RO-Based Treatment Facilities ....................... 206 5.3 Survey Results – Non-RO-Based Process ................................................................................... 207 5.4 Average Available Days Per Year – Municipal Staff .................................................................. 208 5.5 Example – Instrument and Control Maintenance Staff Time Required ....................................... 210 5.6 Specific Task Assessment – RO-Based DPR Process ................................................................. 214

Water Environment & Reuse Foundation xiii

5.7 General Task Assessment– RO-Based DPR Process ................................................................... 215 5.8 Example Estimate of Equipment Maintenance Requirements – Major Pumps ........................... 217 6.1 Water Treatment Plant (WTP) Operator Certification Requirements California (1) ................... 223 6.2 Water Treatment Plant Classification – California (3) ................................................................ 225 6.3 WWTP Operator Certification Requirements California (4) ....................................................... 228 6.4 Wastewater Treatment Plant Classification Table (5) ................................................................. 229 6.5 ABC Water and Wastewater Core Competencies ........................................................................ 234 6.6 Wisconsin Wastewater Operator Subclass Categories (15) ......................................................... 237 6.7 Scoring Table for Water Supply System Public Health (microbial) Risk Classification (16) ..... 238 6.8 Microbial Risk Plant Classifications (16) .................................................................................... 239 6.9 Minimum Qualifications, Experience and Refresher Training Requirements for the

Responsible Person at Water Treatment Systems per Classification (16) ................................... 240 6.10 Certificate III Water Operations Competency Requirements

(Water Industry Training Center NWP-07 Training Handbook) ................................................. 241 6.11 Example of a Competency Module – NWP355B Monitor Operate and Control Membrane

Filtration Processes – Australian Government Industry Skills Panel NWP07 Water Training Package (17) ................................................................................................................................ 243

6.12 Wastewater Treatment Plant Operator Examination Content – California .................................. 249 6.13 Drinking Water Treatment Exams Expected Range of Knowledge ............................................ 250 6.14 Water and Wastewater Operations Certification Curriculum – Gap Analysis............................. 251 6.15 Availability of Training Content for DPR Processes ................................................................... 256 6.16 Required Additional Material for DPR Processes ....................................................................... 257 6.17 General Curriculum for DPR Certification Grade Levels............................................................ 261

xiv Water Environment & Reuse Foundation

Acronyms and Abbreviations

AOC Assimilable Organic Carbon

AOP Advanced Oxidation Process

AWPF Advanced Water Purification Facility

BAC Biological Activated Carbon

BDOC Biodegradable Organic Carbon

CDPH California Department of Public Health

CCP Critical Control Point

CCPP Calcium Carbonate Precipitation Potential

CCR California Code of Regulations

COP Critical Operating Point

CT Concentration × Time as mg-min/L

DBPs Disinfection Byproducts

DDW Division of Drinking Water

DOC Dissolved Organic Carbon

DPR Direct Potable Reuse

EED Electrical Energy Dose

EfOM Effluent Organic Matter

EPA Environmental Protection Agency

GAC Granular Activated Carbon

GWRS Groundwater Replenishment System

HACCP Hazard Analysis and Critical Control Point

IPR Indirect Potable Reuse

ISO International Standards Organization

LPHO Low Pressure High Output

LRV Log Reduction Value

LSI Langlier Saturation Index

LT2ESWTR Long Term 2 Enhanced Surface Water Treatment Rule

MF Microfiltration

MGD Million Gallons per Day

MOC Membrane Operator Certification

MP Medium Pressure

NDMA n-Nitrosodimethylamine

Water Environment & Reuse Foundation xv

NTU Nephelometric Turbidity Units

O&M Operations and Maintenance

OMMP Operation, Maintenance, and Management Plan

PDT Pressure Decay Test

RO Reverse Osmosis

RWQCB Regional Water Quality Control Board

SB Senate Bill

SCADA Supervisory Control and Data Acquisition

SEDA Southeast Desalting Association

SOC Synthetic Organic Compound

SWRCB State Water Resources Control Board

UF Ultrafiltration

UFRV Unit Filter Run Volume

U.S. EPA United States Environmental Protection Agency

TOC Total Organic Carbon

UV Ultraviolet (Light)

UVT Ultraviolet Light Transmittance

VOC Volatile Organic Compound

WHO World Health Organization

WRF Water Reclamation Facility

WE&RF Water Environment & Reuse Foundation

xvi Water Environment & Reuse Foundation

Acknowledgments

This project was funded by the Water Environment & Reuse Foundation in cooperation with Metropolitan Water District of Southern California.

The project team would like to thank all of the utilities, consultants, and operational staff provided data and access to facilities throughout this project. They especially appreciate the workshops supported by Mehul Patel and the team at Orange County Water District, as well as Mark Starr at the City of Los Angeles’ Terminal Island Reclamation Facility. The team also thanks Justin Mattingly his role as Project Manager from Water Environment & Reuse Foundation and the Project Advisory Committee for their review provided throughout the project.

This report is dedicated to the memory of Cecile Bèle, who developed the CCP response procedure format and approach used on the Western Corridor Recycled Water Project, and recommended in this report.

Principal Investigators Troy Walker, MIE (Aust), Hazen and Sawyer Ben Stanford, Ph.D., Hazen and Sawyer Project Team Debra Burris, P.E., BCEE, DDB Engineering John Caughlin., OperatorStar James C. Vickers, P.E., Separation Processes, Inc. Aaron Duke, P.E., BCEE, Hazen and Sawyer Meric Selbes, Ph.D., Hazen and Sawyer Ryan Nagel, P.E., PNV EP, Hazen and Sawyer Sean Pour, P.E., Ph.D., Hazen and Sawyer Robert Boysen, P.E., Hazen and Sawyer Participating Agencies West Basin Municipal Water District, CA Orange County Water District, CA City of Los Angeles, CA Gwinnett County, GA Trojan UV Project Advisory Committee Jim Crook, Ph.D., P.E., Environmental Engineering Consultant Phil Friess, Sanitation Districts of Los Angeles County Mike Gritzuk, Pima County Wastewater Reclamation Department Tom Richardson, P.E., RMC Water and Environment Mike Wehner, Orange County Water District

Water Environment & Reuse Foundation 1

Chapter 1

Introduction and Background

1.1 Understanding the Importance of Operations in DPR

As continued population growth, increasing urban density and varying climate place heavy burdens on our nation’s water supplies, water agencies and policy makers are examining innovative ways to stretch water supplies, to sustain projected population growth rates and to provide reliability and redundancy in their supply portfolio. As such, many water agencies around the United States and around the world have been turning to the potable reuse of municipal wastewater, either directly or indirectly, to help meet growing demands.

It is important for any water treatment facility to have a high level of reliability to ensure water quality is delivered to an acceptable standard and the health risks and aesthetic impacts to public are minimized. This importance is underlined in the case of potable reuse, where the real risks of higher contaminant levels in plant feed water (e.g., during epidemics or after industrial accidents), along with perceived risks associated with public perception of reuse, require a high level of operational surety. Consistent and assured levels of reliability can be met only with a holistic asset management framework including a robust design, effective and transparent operational management, a carefully managed maintenance strategy, and proven response procedures. The plant must be designed correctly, it must be operated well with realistic and practical demands on operations staff, and the assets and infrastructure must be maintained in a highly reliable condition.

A key element of success of any water treatment system relies on its operators and the ability of those operators to evaluate and respond to any issues that may arise. As the perceived “human element” in the process, operations must have robust and reliable operational plans, systems, and processes to ensure safety and reliability – essential elements for the advancement of public acceptance of direct potable reuse (DPR). Relative to existing water and wastewater treatment systems, operations teams are under much greater scrutiny for performance, and must therefore have adequate training and certification processes in place to provide a framework for developing and evaluating the necessary skills for successful operation and management of water recycling systems.

Recent work funded by WateReuse Research Foundation has outlined the importance of the “Four R’s” of potable reuse as they relate to the treatment processes: Redundancy, Reliability, Robustness, and Resilience (Pecson, Trussell, Pisarenko, et al., 2015). However, this concept could be extended beyond the treatment processes to the operations team who is in charge of managing those processes, from validating that each process and process monitor is working as intended to responding to events and reporting information to the appropriate groups (e.g., regulators, supervisors, management teams). Thus, when considered holistically, the four R’s expand to Six R’s: Redundancy, Reliability, Robustness, Resilience, Response, and Reporting as a means of incorporating not only the technology needs for potable reuse, but also the operations team that is integral to the proper functioning and maintenance of the processes. Fortunately, the Hazard Analysis and Critical Control Point (HACCP) methodology contains each of these elements and can be used to incorporate each of the Six R’s into potable reuse system, including the training and management of operations teams (Halliwell, Burris, Deere, et al., 2015; Walker, Stanford, Khan, et al., 2016, in press).

Water Environment & Reuse Foundation 2

1.2 Integration of HACCP into Operations

Initially developed for the food industry, the HACCP framework has been adopted internationally by a number of utilities to manage microbiological and chemical contaminants in water treatment systems, including recycled water systems (Halliwell, Burris, Deere, et al., 2015). HACCP is a logical, scientific process control system designed to identify, evaluate, and control hazards. Its purpose is to put in place process controls that will detect and correct deviations in quality processes at the earliest possible opportunity. It focuses on performance and quality monitoring and maintaining the barriers of treatment, rather than on end of pipe sampling and treatment.

Recently, the Reuse-13-03 project entitled Critical Control Point Assessment to Quantify Robustness and Reliability of Multiple Treatment Barriers of a DPR Scheme (Walker, Stanford, Khan, et al., 2016, in press), was completed as a direct demonstration of the applicability of HACCP to DPR systems and included full-scale data evaluation to assess the ability of critical control points (CCPs) to manage human health risks. The project focused on working through the HACCP methodology to identify health risks, identify water quality objectives, and develop CCPs for both reverse osmosis (RO) membrane-based treatment (MF/UF-RO-UV/H2O2-Cl2) and a non-membrane-based treatment process (O3-BAC-GAC-UV-Cl2). A key outcome of that project was the development of a sound technical foundation upon which to provide standard design guidelines and operational response procedures. Having the right procedures in place to control the treatment process is essential, but equally important is the ability to transparently deal with incidents of failure in a timely and effective manner in order to safeguard public health and maintain public and regulatory confidence. As one of the key deliverables of Reuse-13-03, standard operating responses were developed in an incident-response framework that incorporated immediate responses to correct process or equipment failures; management of out-of-specification water; monitoring and investigation of the incident; and, equally important, pathways of communication.

However, these standard operating responses are only a piece of the overall operations and maintenance system that is required to support the successful operation of a DPR system. In the context of DPR scheme operation, these standard operating responses would be a part of a broader, over-arching operational framework (i.e., operation, maintenance, and management plan or “OMMP”) that would contain the objectives, strategies, and management actions required for the supply of recycled water for DPR. With the principles of HACCP as its foundation, the OMMP would contain within it all of the important elements that are required to ensure safe, reliable operations of the DPR treatment infrastructure.

An example of this overarching operational plan is the Recycled Water Management Plan that was developed for the Western Corridor Recycled Water Scheme, a large scale indirect water recycling scheme in Brisbane, Australia. With a foundation in HACCP, this plan provided a framework for the operational management of the scheme, ensuring that the management of risk to public health from the use of recycled water is fully integrated into the processes as well as the operations, maintenance, and management systems at all levels.

A simplified overview of the Recycled Water Management Plan framework is shown in Figure 1.1. This is instructive in identifying the important elements of operations and maintenance management that must be considered for a recycled water scheme and while this plan was developed for an indirect potable reuse scheme (IPR), the same elements are easily adapted to a DPR treatment system.

This report builds upon the work reported in Reuse-13-03 to focus on the development of the key elements required for the operations and maintenance management. In the case of an actual DPR treatment plant or scheme, these elements would be fully developed, with a great deal of information and detail specific to the plant or scheme itself including specific treatment processes, regulatory environment

Water Environment & Reuse Foundation 3

and operating entity. For this report, the framework will be useful in the identification of common requirements for either of the two DPR process trains under consideration (RO membrane-based treatment (MF/UF-RO-UV/H2O2-Cl2) and the non-RO membrane process of floc/sed-O3-BAC-GAC-UV-Cl2), with a set of recommendations to ensure those common requirements are met.

This framework is by no means the only approach that can be taken in developing an OMMP. The framework shown here is simply used as a means of clearly identifying the important elements that must be considered in developing OMMPs. Utilities and private operators have their own systems and processes that have been developed over time and are a best fit for that organization, though the core principles remain consistent between the OMMPs. The World Health Organization’s Drinking Water guidelines use the concept of a “water safety plan” which mirrors the HACCP process but is re-branded for the water industry to avoid confusion with HACCP as it applies to the food industry (WHO, 2011). It is our vision that the elements of HACCP can be applied in a bottom-up structure such that shapes the way utilities manage public health protection, operations, and maintenance for compliance with state regulations and permits rather than in a top-down structure that forces HACCP and ISO 22000 type compliance within the regulation and permit.

ISO 22000 is a standard developed by the International Organization for Standardization the deals with food safety. It incorporates HACCP principles along with the important elements of interactive communication and system management to ensure that safe food products are delivered to the final consumer. ISO 22000 has been adopted at some water treatment and advanced recycled water facilities as a framework for ensuring safe drinking water to consumers. Examples include the Western Corridor Recycled Water Project indirect potable reuse system, along with the Sydney Desalination Plant.

Water Environment & Reuse Foundation 4

Figure 1.1 The Western Corridor Recycled Water Management Plan Integrates HACCP through the Entire Operations and Maintenance Framework.

Operational Monitoring

Water Quality Sampling and Analysis

Non Conformances Corrective/Preventative

Actions

Managing Incidents and Emergencies

Operating Interface Protocols

Operating Procedures

Asset Management and Maintenance

Op

erat

ion

s M

anag

em

ent

On Line Quality Monitoring

Process Performance

Upstream Wastewater Interface Protocol

Critical Control Point Response Procedures

Process operating procedures

Instrument Calibration and Verification

Maintenance Management

Asset Condition and Risk Assessment

Specific process equipment maintenance (e.g. membrane management)

Critical Control Point Response Procedures

HA

CC

P

Operator Skills and Training

Training Requirements Certification

Validation and Auditing

Emergency Response Procedures

Emergency Response Communication

Downstream Water Interface Protocol

Ris

k M

anag

emen

t

HACCP Team Selection

Critical Control Point Selection

Risk Management Process

Operational Risk Assessment

Recycled Water Management Plan

Roles and Responsibilities

Water Quality Risk Assessment

(HACCP)

Water Environment & Reuse Foundation 5

1.3 Permitting in California Title 17 and Title 22 Recycled Water

Equally important in the development of operation and maintenance requirements is the permitting context in which the recycled water plant is to be operated. In California, the California Code of Regulations (CCR) sets forth requirements for water recycling in Title 17 and Title 22 (SWRCB, 2015).

CCR Title 17, Division 1, Chapter 5, Group 4, Articles 1 and 2, Sections 7583 through 7605 contain regulations pertaining to cross-connection control and backflow prevention. The purpose of these Title 17 requirements is to protect the potable water system from contamination from wastewater and/or recycled water systems. Various backflow prevention devices, construction methods, and testing/maintenance procedures are set forth in these regulations.

Changes in the Title 17 regulations will be needed to accommodate DPR systems because of the obvious linkage of the recycled water system with the drinking water treatment plant and/or distribution system. A broader sense of the “one water concept” will require changes in the regulations, while at the same time being protective of public health. For DPR, strict adherence to safeguards offered by a HACCP system and incorporated into operations and maintenance plans will help minimize health risks that Title 17 is designed to address.

CCR Title 22, Division 4, Chapter 3 “Water Recycling Criteria” regulates water recycling. The majority of Title 22 provisions specify requirements for production and reuse of non-potable recycled water, including irrigation, impoundments, industrial, cooling water, and dual-plumbing uses; Articles 5.1 and 5.2 allow recycled water for groundwater replenishment as reviewed and approved on an individual case-by-case basis by the California Department of Public Health (CDPH) (now the State Water Resources Control Board Division of Drinking Water (DDW), particularly where the use of recycled water may involve a potential risk to public health. Indirect potable reuse (IPR) projects, such as the Orange County Water District (OCWD) Groundwater Replenishment System (GWRS), have been approved and permitted under this element of Title 22. To date, six IPR projects are operational in California under the auspices of this permit system.

In 2010, the California Water Code was revised when Senate Bill (SB) 918 was approved, requiring that CDPH adopt uniform water recycling criteria for: 1) groundwater recharge by December 31, 2013, and 2) surface water augmentation by December 31, 2016. While CDPH has prepared earlier versions of the draft groundwater recharge requirements, this legislation provided momentum for the regulatory process. In response, CDPH issued Draft Groundwater Replenishment Reuse Regulations on June 26, 2013 and adopted regulations for groundwater recharge which became effective on June 18, 2014. At the present time, CDPH is developing regulations for surface water augmentation projects.

The next step in structuring permitting criteria for potable reuse will likely involve developing regulations for DPR projects, to manage current and future severe drought conditions that will likely continue to impact California and other western states (Barnett and Pierce, 2008; Beuhler, 2003; Castle, Thomas, Reager, et al., 2014). Consolidation of the CDPH drinking water program, which also oversees recycled water, into the State Water Resources Control Board (SWRCB) umbrella, which includes the Regional Water Quality Control Boards (RWQCBs), should help address permitting, operator certification, and operational compliance for future IPR and DPR systems. It is anticipated that DPR regulations may be based on IPR draft regulations, incorporating many of the same requirements for enhanced source control, full advanced treatment with multiple barriers, as well as online monitoring of critical control points for compliance with critical limits to control public health risks in manner similar to that of HACCP.

Water Environment & Reuse Foundation 6

1.4 Current Operator Training and Certification Requirements

As described in the previous sections, existing indirect potable reuse facilities in California are regulated and permitted by the Division of Drinking Water (DDW) at the SWRCB which provides input into the permit process regarding drinking water quality and treatment requirements. This permit provides a number of specific and prescriptive operational and maintenance requirements based on individual plant treatment trains, as well as operator certification requirements.

Currently, a certification framework for recycled water operations exists as a subset to SWRCB wastewater requirements and so utilities instead must rely on existing wastewater and drinking water certification systems. These are administered by SWRCB for wastewater and DDW (also part of SWRCB) for drinking water respectively. The focus of both of these certification models is more on the conventional wastewater and drinking water treatment trains respectively, and as such there is a gap in the specific process and operating requirements for recycled water facilities. Additionally, key technologies, such as membrane treatment, ozone, and BAC are not well covered.

In the current context, utilities and operators managing IPR schemes use a combination of water and wastewater treatment plant certifications, with supplemental training provided for specific process equipment and regulatory requirements. For example, the Orange County Water District Groundwater Replenishment System requires all operators to have wastewater treatment plant certifications. In addition, OCWD operators have many years of experience with membrane and advanced oxidation processes (AOP). Training and capability for these processes in managed in house by the district, and not currently well supported by external training and certification agencies.

The gap in training has nonetheless been identified not only by plant utilities and operators, but also a number of associations, engineering consultants as well as technical equipment vendors. As an example, the South East Desalination Association (SEDA) has developed a Membrane Operator Certification (MOC), which provides a multi-day training program, supplemental learning materials, and a testing/certification process that is currently accepted in Florida as a membrane operators licensing program.

1.5 Project Objective and Technical Approach

This project was developed in response to the need to provide meaningful, actionable advice to the DPR initiative at WateReuse Research Foundation and to provide information that can assist the Expert Panel, DDW of SWRCB, and SWRCB in making DPR a reality for California. Specifically, there was an opportunity to provide input on permitting structure, operations and maintenance protocols, and training and certification programs as well as to develop some of the content/curriculum that can be used in the training and response setting.

In order to address the gaps in training, certification, and permitting programs and to support the California DPR initiative, the objective of this project was to develop a standard operations and maintenance plan framework for various DPR treatment processes and to develop a DPR Training and Certification framework for DPR system operators.

To deliver this project, we assembled a team of experts who have direct experience in the establishment of operations and maintenance protocols for recycled water schemes, have direct experience in the permitting of IPR schemes in California and have direct experience with training and certification of IPR operators in California. To meet these objectives, the project followed a two-phase approach to effectively answer two questions:

Water Environment & Reuse Foundation 7

What are the vital operational and maintenance requirements that must be integrated into a permitting scheme to ensure the success of DPR? (Addressed in Chapters 3 and 5)

Based on these requirements, how can we ensure operators can meet these requirements? (Addressed in Chapters 4 and 6)

To accomplish the objective of developing recommendations around operation, maintenance, and management plans for DPR and developing recommendations on permitting and operator training, the following tasks were completed:

Phase 1: Develop a Standard Operations and Maintenance Plan for DPR Schemes o Task 1.1: Evaluate the California Code of Regulations and its Adequacy for DPR Systems and

Recommend any Changes

o Task 1.2: Identify and Recommend a DPR Permit Structure, Operator Certification Program, and Permitting Authority

o Task 1.3: Develop O&M Protocols/Framework for DPR Systems from Source to Tap

Phase 2: Develop a DPR Training and Certification Curriculum Framework for DPR System Operators o Task 2.1: Recommend DPR System Staffing o Task 2.2: Develop Recommended Operator Staff Training and Certification Framework

1.5.1 Organization of this Report

The remainder of this report outlines the findings and recommendations of the Project Team, including a review of regulatory and training requirements among multiple states with IPR or DPR frameworks or legislation in place. Chapter 2 provides a thorough review of the California Code of Regulations and identifies gaps that

will need to be addressed in developing DPR regulations Chapter 3 provides the key elements that are required in an operational management plan. It provides

detail and examples for each of the elements described. Chapter 4 outlines key operational procedures and requirements for the specific process barriers

(critical control points) for both of the DPR systems being considered. For completeness, this includes CCP response procedures developed as a part of Reuse-13-13.

Chapter 5 provides a review of staffing approaches including staff benchmarking at existing IPR facilities and provides a recommendation for assessing staff requirements at future DPR facilities

Chapter 6 provides an assessment of operator certification and training needs in a DPR context, building upon existing water and wastewater frameworks in the US and Australia. This chapter also provides a list of curriculum needs and content that will need to be developed.

Water Environment & Reuse Foundation 8

1.6 References Barnett, T.P. and D.W. Pierce (2008). "When Will Lake Mead Go Dry?" Water Resources Research 44(3): 10.

Beuhler, M. (2003). "Potential Impacts of Global Warming on Water Resources in Southern California." Water Science and Technology 47(7-8): 165-168.

Castle, S.L., B.F. Thomas, J. T. Reager, M. Rodell, S. C. Swenson and J. S. Famiglietti (2014). "Groundwater Depletion During Drought Threatens Future Water Security of the Colorado River Basin." Geophysical Research Letters 41(16): 2014GL061055.

Halliwell, D., D. Burris, D. Deere, G. Leslie, J. Rose and J. Blackbeard (2015). Utilization of HACCP Approach for Evaluating Integrity of Treatment Barriers for Reuse. WateReuse Research Foundation, Alexandria, VA.

Pecson, B.M., R.S. Trussell, A.N. Pisarenko and R.R. Trussell (2015). "Achieving Reliability in Potable Reuse: The Four Rs." Journal American Water Works Association 107(3): 48-58.

SWRCB (2015). Regulations Related to Recyled Water (Title 17 and Title 22 California Code of Regulations), Updated 2015.

Walker, T., B.D. Stanford, S. Khan, R. Valerdi, S. A. Snyder and J. Vickers (2016, in press). Critical Control Point Assessment to Quantify Robustness and Reliability of Multiple Treatment Barriers of a DPR Scheme (Reuse-13-03). WateReuse Research Foundation, Alexandria, VA: 310.

WHO (2011). Guidelines for Drinking Water Quality, 4th Edition. World Health Organization (WHO), Geneva. 1: 493. http://apps.who.int/iris/bitstream/10665/44584/1/9789241548151_eng.pdf

Water Environment & Reuse Foundation 9

Chapter 2

California Code of Regulations and Permit Structure

2.1 Overview of the Task

In this chapter, the California Code of Regulations (CCR) are reviewed to identify sections governing recycled water and drinking water systems that may be applicable to developing operations plans for DPR systems. The criteria for recycled water and drinking water facility operations and operator certifications are compared and a gap analysis is developed to identify potential contradictions where regulatory changes may be needed to accommodate DPR.

2.2 Approach

A review of the CCR and related State Water Resources Control Board (SWRCB) Division of Drinking Water (DDW) policy documentation was conducted to compare and contrast regulatory criteria related to the operation of recycled water and drinking water facilities in California. These regulations were examined for requirements pertaining to operations, operator training and certification, and operating records maintenance. An initial listing of notes and references was compiled and subsequently developed into a matrix summarizing the requirements and documenting the specific references.

As background information, the CCR is the official compilation and publication of regulations, or laws, which have been adopted, amended or repealed by state agencies in California. The CCR is organized by Titles that are comprised of various Divisions, grouping regulations by subject.

Specific water recycling regulations used in this analysis included: Title 17 CCR “Public Health,” Division 1 “State Department of Health Services” [now SWRCB

DDW], Chapter 5, “Sanitation (Environmental)” Title 22 CCR “Social Security,” Division 4 “Environmental Health,” Chapter 3, “Water Recycling

Criteria” California Water Code, Sections 13625-13633 “Wastewater Treatment Plant Classification and

Operator Certification” Title 23 CCR “Waters,, Division 3 “State Water Resources Control Board and Regional Water

Quality Control Boards,, Chapter 26 “Classification of Wastewater Treatment Plants and Operator Certification”

Specific drinking water regulations used in this analysis included: Title 22 CCR “Social Security,, Division 4, “Environmental Health,, Chapter 12 “Safe Drinking

Water Project Funding,, Chapter 13 “Operator Certification,, Chapter 14 “Water Permits,” and Chapter 15 “Domestic Water Quality and Monitoring Regulations,” Chapter 16 “California Waterworks Standards,” Chapter 17 “Surface Water Treatment” (chapters not listed regulate other aspects of drinking water unrelated to operation, which was the focus of this review)

California Department of Health Services [now SWRCB DDW], Policy Memo 97-005 “Policy Guidance for Direct Domestic Use of Extremely Impaired Sources”

The compilation was then reorganized to group common requirements that more easily facilitated their comparison. Many commonalities were observed between the operations requirements for recycled water and drinking water systems. For some requirements, however, differences were found and subsequently

Water Environment & Reuse Foundation 10

evaluated for potential conflicts with DPR systems. The end product of this evaluation was development of a gap analysis matrix. By contrasting the differences, recommended changes in operational regulations were developed to bridge the gaps.

2.2.1 Gap Analysis and Recommended Operations Regulatory Changes The gap analysis found that, for the most part, only minimal changes in the existing California recycled water and drinking water regulations will be needed to accommodate DPR implementation. The regulatory provisions involving more significant changes are related to operator certification and staffing requirements.

Table 2.1 summarizes key regulatory requirements related to the operations, operator training and certifications, and operating records maintenance for recycled water and drinking water systems that will need to be modified for DPR. The text on the following pages describes the context for each of these concluding regulatory strategies. For the analysis, the operations requirements were grouped in five topics: 1) permits and reporting, 2) source water, 3) monitoring and water quality, 4) staffing and operator qualifications/certifications, and 5) backflow prevention and dual plumbing. Two categories of regulatory changes, minimal and significant, were determined, and recommended changes to facilitate DPR were developed. Appendix A presents the more detailed gap analysis matrix upon which the recommendations are based. Table 2.1 Summary of Recommended Operations Regulatory Changes to Accommodate DPR

Category Reuse-13-13 Recommended Regulatory Changes

Drinking Water Recycled Water

Permits and reports Develop a single combined permit for DPR systems (in lieu of two separate permits for drinking water and recycled water)

Source control Expand source control requirements for extremely impaired sources to include monitoring and minimization of contaminant discharges, inventory chemical use and outreach/education

Monitoring and water quality Comply with notification levels for extremely impaired sources

Staffing and operator certifications

Maintain detailed operating records of water quality analyses, operational problems, equipment breakdowns, diversions, corrective or preventive actions, and alarms

Establish more detailed operator classifications, certification levels and training plus number of staff requirements

Backflow prevention and dual plumbing

Revise backflow and cross-connection control requirements to exempt DPR from being considered a cross-connection

Revise backflow and cross-connection control requirements to differentiate between recycled water systems and DPR systems and exempt DPR from being considered a cross-connection

Water Environment & Reuse Foundation 11

2.2.2 Permitting With regard to permitting, minimal revisions in existing regulations would be necessary to permit DPR systems. In lieu of two separate permits (one for the water recycling facility and another for the drinking water facility) it is envisioned that a combined system permit for DPR systems will need to be developed under new DPR regulations. For example, a single DPR permit could specify enhanced communications, alarms, and critical control points to link the operations of both recycled water and water treatment plants. Separate permits for each plant could miss that degree of required coordination, whereas a unified permit would improve operation and enhance overall public trust in DPR facilities’ performance. While recycled water could simply be recognized as a new raw water supply source at the water treatment plant, separate permits for the facilities would be cumbersome to administer and would inadvertently separate the operational, monitoring, and reporting responsibilities. Joint responsibility under a single permit would resolve operational divisions by tying the entire DPR to common requirements. This is probably most evident for a DPR facility discharging directly to the drinking water distribution system where the operations regulatory requirements imposed upon the DPR facility would likely incorporate requirements for both recycled water and drinking water production, effectively producing a hybrid DPR permit. A new separate permit for all DPR facilities would provide clarity in the operating requirements.

2.2.3 Source Control Water recycling facilities, particularly those designated as indirect potable reuse (IPR) systems, are subject to on-going, enhanced source control requirements. An enhanced source control, or expanded industrial pre-treatment program, for the raw wastewater stream requires a continuously active effort to minimize the risk of contamination of the source, thereby safeguarding the wastewater quality that becomes feed water for production of recycled water for IPR. For drinking water facilities, such as those handling extremely impaired source waters, the requirements appear to focus on cleanup of contaminated sources with expected gradual improvement of the source water quality. For those types of drinking water projects, the contaminants in the source water are already known, and the drinking water treatment plant is designed to treat and recover the impaired source water. Source control for a DPR system may be more similar to that for an IPR recycled water system, requiring only minimal, if any, revisions in the recycled water regulations. Instead of focusing on the mindset of gradual cleanup of impaired sources over time (drinking water), DPR, which starts with raw wastewater as the source, source control will depend upon a constant vigilance to reduce the risk of contamination.

2.2.4 Backflow Prevention Title 17 of the California Code of Regulations (Div1, Chapter5, Group 4, Article 1) covers the requirements for backflow prevention to minimize the risk of contamination from unwanted cross connections. Recycled water is considered in this regulation as a specific risk that must include back flow prevention. For DPR, minimal changes could simply exempt DPR systems from being considered as cross-connections and add a reference to new DPR regulations, thereby avoiding any misinterpretations that might hinder DPR.

Operator Certification and Staffing Requirements More significant differences were noted in the operator certification and staffing requirements for recycled water and drinking water systems. Per the California Water Code, water recycling facility operators are required to have wastewater operator certifications; however, with SWRCB approval, a water treatment plant operator certification may be substituted at a water recycling facility. With the advanced treatment processes employed at many water recycling plants, wastewater certifications may not be as applicable as water certifications. For example, operators with wastewater certifications may have little experience with membrane treatment. In comparison, water systems have more detailed certification classifications, separating treatment plants and distribution systems; yet membrane or specific process experience could still be limited. More discussion of the differences between operator

Water Environment & Reuse Foundation 12

certification requirements for recycled water and drinking water systems can be found in Chapter 6. In general, it seems likely that DPR operator certifications may be based on general qualifications and also require separate testing for specific treatment processes utilized by the DPR facility. For instance, DPR regulations may require certifications for membrane treatment. Staffing requirements at DPR facilities may be based on size of the facility, similar to water systems, as well as the types of treatment processes. It is envisioned that these issues could be effectively addressed by enacting separate DPR operator certification requirements.

2.3 Summary of Operations Regulatory Changes

Figures 2.1 and 2.2 summarize the operations regulatory changes that may be required to accommodate DPR. Drinking water operations regulations may need to incorporate expanded source control requirements and more detailed operating records. Recycled water operations regulations may need to comply with notification levels set forth for extremely impaired water sources. Separate permitting, operator certification, and staffing requirements would need to be developed for DPR applications. New separate DPR regulations would help address operational requirements for DPR systems.

Table 2.2 includes the detailed gap analysis for current recycled water requirements and drinking water requirements in California that pertain to operations, certification and training. Specific requirements are reviewed, and recommended changes for DPR to be accommodated as a drinking water supply option are included

Figure 2.1 Drinking Water Operations Regulatory Changes to Accommodate DPR

13 Water Environment & Reuse Foundation

Figure 2.2 Recycled Water Operations Regulatory Changes to Accommodate DPR

14 Water Environment & Reuse Foundation

Table 2.2 Detailed Gap Recycled Water and Drinking Water Regulation Gap Analysis

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

Permits and Reports

Obtain a permit for operation of a Groundwater Replenishment Reuse Project (GRRP)

Obtain a permit. A permit issued by DDW for use of an extremely impaired source shall include all necessary treatment, compliance monitoring, and operational and reporting requirements.

DPR systems will likely need to obtain a separate permit specifically for DPR. This single, combined DPR permit would be in lieu of securing two separate permits, one for water recycling and another for drinking water.

Engineering report shall indicate the method for compliance with the regulations and contain a contingency plan.

Public water system shall submit a technical report to DDW as part of the permit application. The report shall include detailed plans, specifications, water quality information, physical descriptions of the system, and financial assurance information.

The engineering report for recycled water systems is similar to the technical report for water systems, except that financial assurance information is currently not required for a recycled water system. A requirement may include the financial assurance information for DPR.

Source Water

15 Water Environment & Reuse Foundation

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

Recycled water used for a GRRP shall be from a wastewater agency that administers an industrial pretreatment program that assesses the fate of and monitors for Division of Drinking Water/ Regional Water Quality Control Board -specified contaminants through the treatment systems, includes an outreach program to minimize discharges of contaminants at the source, and maintains an inventory of chemicals and contaminants at the sources.

Monitoring at the extremely impaired source shall be provided to control the level of contamination and assure that levels of contaminants will not increase beyond the proposed treatment system's capability. Prepare and submit an extremely impaired source water quality surveillance plan that includes monitoring between the origin of contamination and the extremely impair source that is proposed for drinking water. Sanitary surveys shall be conducted initially and repeated every 5 years.

Source control requirements for DPR systems would likely more closely resemble those for recycled water in lieu of the extremely impaired sources for drinking water because of the potential to receive contaminants into the wastewater system from multiple sources.

Monitoring and Water Quality

16 Water Environment & Reuse Foundation

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

Lab analyses shall be performed by laboratories approved by DDW using DDW-approved drinking water methods. Chemical analyses other than for those having primary or secondary MCLs shall be described in the Operation Optimization Plan (OOP)

Lab analyses shall be performed by laboratories certified by DDW using DDW-approved drinking water methods. For extremely impaired sources, describe the proposed monitoring and treatment including performance standards to achieve compliance with Maximum Contaminant Levels (MCLs) and Notification Levels (NLs).

Drinking water systems using extremely impaired sources are required to comply with NLs. Recycled water systems are not required to comply with NLs. If it is considered similar to extremely impaired sources, DPR systems would need to comply with NLs unless the regulations are revised.

OOP shall include a monitoring plan for regulated contaminants and physical characteristics of applied recycled water

Describe the proposed monitoring and treatment including performance standards to achieve compliance with MCLs and NLs. Prepare and submit a compliance monitoring and reporting program.

DPR systems may be required to monitor for and comply with NLs similar to water treatment system requirements.

Staffing and Operator Qualifications/Certifications

Prior to start-up, prepare, submit for approval by DDW and RWQCB an OOP that identifies and describes the operations, maintenance, analytical methods and monitoring for the GRRP. Update the OOP to represent current operations, maintenance and monitoring of the GRRP.

Operations plan identifying all operational procedures, failure response triggers, and loading rates including: 1) process monitoring plan; 2) process optimization procedures; 3) established water quality objectives or goals; and 4) level of operator qualification.

Operations plan requirements for DPR systems would likely include level of operator certification in a similar fashion to those for drinking water systems. DPR systems will likely have operator certifications based on: a) general qualifications and b) specific qualifications based on treatment processes utilized by that DPR facility.

17 Water Environment & Reuse Foundation

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

Wastewater treatment plant supervisors and operators shall possess a certificate of the appropriate grade. State Board shall develop and specify in its regulations the training necessary to qualify a supervisor or operator for certification of each type and class of plant.

DPR system operator classifications, certifications, and training may be more prescriptive and similar to those for drinking water plants. Recycled water system operators are required to have wastewater certifications unless water certifications are approved. DPR requirements may need to consolidate the operator certification and training requirements. DPR systems will likely have operator certifications based on: (a) general qualifications and (b) specific qualifications based on treatment processes utilized by that DPR facility.

For supervisors and operators of water recycling plants, the SWRCB may approve use of a water treatment plant operator of appropriate grade certified by CDPH (DDW) in lieu of a certified wastewater treatment plant operator, provided that the SWRCB may suspend or revoke its approval if the operator commits any of the prohibited acts described in Article 7, of Chapter 26 of Division 3 of Title 23 of the CCR.

The operations plan for an extremely impaired source shall include the level of operator qualification. Drinking water operator certifications consist of two classifications: treatment facilities and distribution systems. Certification requirements are detailed and dependent upon the size of the facility and other factors. Water treatment plants are

DPR system operator classifications, certifications, and training may be more prescriptive and similar to those for drinking water plants. Recycled water system operators are required to have wastewater certifications unless water certifications are approved. DPR requirements may need to consolidate the operator certification and training

18 Water Environment & Reuse Foundation

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

classified using a point system that tallies points for the following and requiring higher grade operator certifications for water treatment plants with higher point tallies: 1) source water type, 2) raw water microbiological quality, 3) raw water turbidity, 4) raw water perchlorate, nitrate and nitrite concentrations, 5) raw water chemical and radiological contaminant levels, 6) type of surface water filtration, 7) type of disinfection, 8) type of disinfection/oxidation treatment without inactivation credit, and 9) any other treatment process that alters the physical or chemical characteristics of the drinking water. Distribution system certification is based on the population served, with higher classifications required for larger systems.

requirements. DPR systems will likely have operator certifications based on: (a) general qualifications and (b) specific qualifications based on treatment processes utilized by that DPR facility.

19 Water Environment & Reuse Foundation

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

Each water recycling plant shall be staffed with a sufficient number of qualified personnel to operate the facility efficiently so as to achieve the required level of treatment at all times.

Staffing levels are dependent upon the size of the facility. Public water systems have specific requirements for numbers of certified water treatment operators and water distribution system operators at designated grades for various sizes of facilities.

Drinking water system requirements are more detailed and prescriptive than those for recycled water system requirements. . It is anticipated that DPR systems will have specific requirements for staffing based on both facility size and technical complexity. Chapters 5 and 6 of this report discuss options for staffing and operator certification for DPR in more detail.

Operating records shall be maintained at the water recycling plant or central depository within the operating agency of: water quality analyses, operational problems, equipment breakdowns, diversion to emergency storage or disposal, corrective or preventive actions taken, and alarms.

Maintain records of the public water system operation and make those available for inspection by DDW.

Recycled water requirements for maintaining operating records are more detailed than those for drinking water systems. DPR requirements may incorporate the more detailed approach.

Operate the GRRP to provide optimal reduction of all chemicals and contaminants

Operate the public water system to comply with primary and secondary drinking water standards, not be subject to backflow, provide a reliable and adequate supply of pure, wholesome, healthful, and potable water, utilize certified water treatment operators of the appropriate grade, and comply with

Drinking water requirements for backflow prevention would need to be revised to accommodate DPR. Title 17 backflow prevention regulations could simply state that DPR is not classified as a cross-connection and reference separate DPR regulations. DPR system

20 Water Environment & Reuse Foundation

RECYCLED WATER REQUIREMENTS (EXISTING)

DRINKING WATER REQUIREMENTS (EXISTING)

Relative Degree of Regulatory Changes Required to Accommodate DPR

Recommended Changes and Notes to Accommodate DPR Water Recycling Criteria/Requirements

Pertaining to Operation, Certifications, and Training

Drinking Water Criteria/Requirements Pertaining to Operation, Certifications, and Training Minimal Significant

the operator certification program. requirements would likely include operator certification provisions similar to those for drinking water systems. DPR systems will likely have operator certifications based on: (a) general qualifications and (b) specific qualifications based on treatment processes utilized by that DPR facility.

Backflow Prevention and Dual Plumbing

The water supplier's cross-connection control program shall include: (1) operating rules or ordinances to implement the cross-connection program; (2) Conducting surveys to identify where cross-connections are likely to occur; (3) at least one person trained in cross-connection control to carry out the program; (4) establishment of procedures for testing backflow preventers; and (5) maintenance of records of locations, tests, and repairs of backflow preventers.

The public water system shall not be subject to backflow under normal operating conditions.

Title 17 backflow prevention requirements will need to be revised to exempt DPR from being considered a cross-connection and reference separate DPR regulations.

21 Water Environment & Reuse Foundation

Chapter 3

Operation and Maintenance Framework

3.1 Operational Management

Successful operation of any water or wastewater treatment facility requires an effective strategy to ensure that the necessary resources in terms of personnel, training, operational procedures, information, funds, infrastructure and work environment are employed to consistently and reliably meet the requirements of the facility. An operational framework is required that will articulate this strategy and integrate all of the required operational management elements to ensure that the strategy is delivered. In the case of DPR, it must provide a framework that contains the necessary elements to deliver a safe and reliable supply of drinking water.

Fundamentally, effective operational management must ensure the delivery of the facility’s goals, and in doing so strike a sustainable balance between managing the various risks to achieving those goals at an acceptable level of effort and cost. In the case of DPR, the framework should provide a cost effective, long term sustainable supply of drinking water that ensures the protection of public health.

This protection of public health is paramount in the success of DPR, and an operational management framework must integrate this as a central requirement. The Water Environment & Reuse Foundation project Reuse-13-03 “Critical Control Point Assessment to Quantify the Robustness and Reliability of Multiple Barriers of a DPR Scheme” has outlined the use of elements of the HACCP process as a means of ensuring that treated water quality reliably meets public health requirements. HACCP is a logical, scientific process control system designed to identify, evaluate and control hazards, which are significant for food safety. The purpose of a HACCP system is to put in place process controls that will detect and correct deviations in quality processes at the earliest possible opportunity. HACCP focuses on monitoring and maintaining the barriers of treatment, rather than on end of pipe sampling and testing. This provides the dual advantage of ensuring poor quality is prevented in the first place, and allows for a reduction in end of pipe monitoring and associated costs.

HACCP, or at a minimum the use of critical control points (CCPs), has been applied at a number of water recycling projects in order to demonstrate the management of microbiological and chemical risk via multiple barrier processes. Specifically, CCPs are points in the treatment process that are specifically designed to reduce, prevent, or eliminate a human health hazard and for which controls exist to ensure the proper performance of that process. The HACCP or CCP process is not a stand-alone feature of the operational management framework, but is integral to the operational framework while specific requirements that support the framework flow down through many of the operational elements. The full implementation of the HACCP process is possible, such as at the Western Corridor Recycled Water Project in Queensland, Australia where operations were certified to ISO 22000 (ISO 2005, Food Safety Management Systems). However this report does not explicitly recommend that full HACCP implementation is necessary, but rather the important elements from within permeate the operational framework. These key elements include: The development and maintenance of a water quality risk assessment and register. The selection of CCPs. The selection of CCP monitoring parameters and instrumentation and development of a plan to

integrate monitoring into SCADA.

22 Water Environment & Reuse Foundation

Development and application of CCP critical limits (alert and alarm levels) and response procedures. An effective asset management and maintenance system to provide cost effective and reliable

custodianship of CCPs. An effective auditing process. In addition, an operational framework must also incorporate other elements necessary for operational management including: Occupational health and safety. Recruitment, training, and certification of operations staff. Comprehensive procedures for the operation of plant and equipment. Incident and emergency management. Effective communication for internal and external stakeholders. Regulatory requirements including specific DPR and other general requirements.

Figure 3.1 outlines the key elements for a DPR operational framework. The remainder of this chapter provides details of each of the elements.

Figure 3.1. Proposed DPR operational management framework

3.2 Risk Management Process Risks are defined as “the effect of uncertainty on objectives” (ISO 2009, Standard 31000, Risk Management--Principles and Guidelines). In general, they are the factors that can prevent the objectives

DPR Operations Management Plan

Risk Management CCP Operations Management

Risk Management

Process

Operational Risk

Assessment

Water Quality Risk

Assessment

Critical Control Point

Selection

Operating Interfaces

Validation and

Auditing

Operational Monitoring

Non Conformances

Corrective/Preventative

Actions

Managing

Incidents and

Emergencies

Operating

Procedures

Asset Management

and Maintenance

Operator Skills and

Training

Roles and

Responsibilities

Critical Control Point

Management

Communication

23 Water Environment & Reuse Foundation

of a process operation from being met. To provide robust, reliable, and cost-effective operation of any water or wastewater treatment facility, these risks must be either eliminated or mitigated to an acceptable level. Very often, these risks are addressed based on a combination of technical knowledge, effective plant design and operational experience, however increasingly operating utilities are employing a more coordinated approach to risk management. Risk management is the identification, assessment and prioritization of risks, followed by coordinated and economical application of resources to minimize, monitor and control the probability and/or impact of adverse events or to maximize the realization of uncertainties (ISO 2009). Effective risk management provides utilities with strategies for treating risks that might impede it in pursuit of its objectives, and also provides the flexibility and resilience to respond to unexpected risks and take advantage of unexpected opportunities. Risk management through the process of identification, analysis, assessment, treatment, monitoring and review is a tool operational teams can use for developing and maintaining cost effective controls.

In the case of the operation of a DPR facility, there are clear risks that must be managed in the protection of public health. As a key element of the Critical Control Point approach, a hazard analysis and water quality risk assessment is the first step in the process. Other potential risks areas for operations also include: Regulatory compliance. Occupational Health and Safety. Environmental. Financial Reputational. Asset, infrastructure and equipment. Natural disaster. Operational (including equipment technical performance, equipment risk and production risk). In all cases there are several approaches to dealing with risk as outlined in ISO 31000 which include: Avoiding the risk by deciding not to start or continue with the activity that gives rise to the risk. Accepting or increasing the risk in order to pursue an opportunity. Removing the risk source. Changing the likelihood. Changing the consequences. Sharing the risk with another party or parties (including contracts and risk financing). Retaining the risk by informed decision.

In order to make the assessment of what risks are present, and how they should be managed, risk assessment frameworks should be employed.

3.2.1 Risk Assessment Framework

A risk assessment framework provides a methodology by which risks can be identified, articulated, ranked (in terms of likelihood and consequence) and any further actions that are required to reduce or eliminate the risk to an acceptable level.

Figure 3.2 provides a general risk assessment approach that can be used to manage risk. Typically, this process is captured in a risk register table.

24 Water Environment & Reuse Foundation

Figure 3.2. General Risk Assessment Approach

Establish the Context

Risk Identification

Likelihood and consequence of the risk

Determine the Inherent risk level

Describe existing controls.

Is the risk accepted ?

Is further action required?

Describe further actions.

Evaluate the residual risk level.

Monitor and ReviewCommunicate and Consult

Yes

No

Yes

Yes

NoNo

Risk Register

25 Water Environment & Reuse Foundation

Establish the Context of the Risk – Is this risk related to operations, public health, regulatory, financial, or other area?

Identification of the Risk – A brief description of what the risk is, and what can go wrong. (For example, oxidation damage to RO membrane from exposure to chlorine).

Likelihood of Risk Occurring – How likely is this risk to occur? This is often assessed as a ranking from very low to very high, or numbered. Very often, the ranking is defined clearly in the context of the type of risk being assessed. For example, in the case of source water hazard risk assessment conducted as a part of Reuse-13-03 (Walker, Stanford, Khan, et al., 2016), the following likelihood descriptors in Table 3.1 were applied. Table 3.1. Risk Likelihood Descriptor - Source Water Risk Assessment Example

Likelihood Description Almost Certain Is expected to occur with a probability of multiple occurrences within a year. Likely Will probably occur within a 1 to 5 year period. Possible Might occur or should be expected to occur within a 5 to 10 year period. Unlikely Could occur within 20 years or in unusual circumstances. Rare May occur only in exceptional circumstances. May occur once in 100 years.

Alternative likelihood rankings may be adjusted relevant to the context of the risk. For example, an asset risk assessment may consider the condition of an asset in terms of its likelihood of asset equipment failure over some period of time (a year for example).

Consequence if the Risk Occurs – What happens if the risk does occur? This is similarly ranked over a range from low to extreme impact. Again, in the source water hazard risk assessment of Reuse-13-03, this was ranked from insignificant to catastrophic. The descriptors are listed in Table 3.2. Table 3.2. Risk Consequence Descriptor - Source Water Risk Assessment Example Reuse-13-03

Consequence Description Detailed Example

Catastrophic Major impact for a large population

Widespread acute health impact expected, resulting in hospitalization and/or decreased life expectancy

Major Major impact for a small population

Potential acute health impact affecting a limited number of the community

Moderate Minor impact for a large population

Repeated breach of a chronic health parameter, long-term/lifetime exposure required or potential widespread aesthetic impact.

Minor Minor impact for small population

Elevated levels of a chronic health parameter no health impact expected or potential local aesthetic impact.

Insignificant Insignificant impact or not detectable

No expected health impacts or an isolated exceedance of an aesthetic parameter

Another approach, again using the asset risk assessment, a consequence may be considered a criticality – in other words what impact would the failure of that piece of equipment have on the performance of the system.

26 Water Environment & Reuse Foundation

Inherent Risk Level (Likelihood x Consequence) – The inherent risk level is the risk that would be carried by operations if no risk controls were implemented. For example, in our case of membrane oxidation by chlorine, this would be the risk if there was no prevention, monitoring or control action taken to prevent free chlorine oxidizing the membranes. The inherent risk ranking is determined on a matrix based on the previously assessed likelihood and consequence. An example of this ranking is shown in the Figure 3.3.

Likelihood Consequence

Insignificant Minor Moderate Major Catastrophic Almost Certain Low (E1) Moderate (E2) High(E3) Very High

(E4) Very High

(E5)

Likely Low (D1) Moderate (D2) High (D3) Very High (D4)

Very High (D5)

Possible Low (C1) Moderate (C2) High (C3) Very High (C4)

Very High (C5)

Unlikely Low (B1) Low (B2) Moderate (B3) High (B4) Very High

(B5) Rare Low (A1) Low (A2) Low (A3) High (A4) High (A5)

Figure 3.3. Risk Ranking Matrix Reuse-13-03

The level of acceptable risk is generally low to moderate. If the risk assessment concludes that the risk ranks as high or very high, then additional controls will be essential to return to an acceptable level of risk.

Describe Further Actions – If the level of risk is unacceptable, additional actions must be taken to reduce the risk. This can include addition of automated controls, operational procedures, design improvements, additional maintenance or other actions required to reduce the risk to an acceptable level.

Evaluate the Residual Risk – The residual risk is the risk that is left following the implementation of further action. It is assessed as described above and is the final risk score that will be evaluated.

This risk assessment process is articulated on a risk register which maintains a record of the current state of assessed risk. These risk registers should be used on a regular basis to review any changes to the profile of risk for each of the areas covered. Also, should there be any changes to risk noted during the course of operations, these should also be captured and updated.

The development, measurement and articulation of the risk profile provide a guide to operational management and staff on where to expend effort to ensure reliable performance. They can often provide a useful guide for budgeting planning, and can assist in the justification of investment. The risk register is a very useful tool for operational decision making. It also provides an effective communication to stakeholders as to how risks are managed.

Ultimately, the success of the risk assessment process depends on the implementation of recommended risk control actions throughout the operational framework. The operational requirements that are required to mitigate the identified risks must inform the requirements of the operational framework and flow through to elements such as operational procedures, operational training, performance monitoring, asset management and other important elements.

27 Water Environment & Reuse Foundation

3.2.2 Source Water Quality Risk Assessment

Key to the success of DPR operations is the management of water quality risks. As the first step in the HACCP process, the development of a source water quality risk assessment is critical in understanding what operational strategies must be employed to reduce those risks to a level that will protect public health. A source water hazard review allows for identified water quality risks to be managed, establishes the level of removal required over the entire treatment train to meet water quality objectives, and identifies what barriers (processes) will be required for the removal of specific contaminants or categories of contaminants.

A source water hazard review typically consists of a significant site-specific sampling campaign over an extended period of time to understand the likely concentration of contaminants and any seasonal or temporal (even diurnal) variations in source water quality. Monitoring would be tailored to understand the occurrence of acute risk chemical or biological contaminants (i.e., monitoring would need to capture extreme events and diurnal variability) or chronic risks (i.e., less frequent monitoring over an extended period to benchmark background concentrations). Guidance on developing monitoring strategies for contaminants has been reported elsewhere (Park, Reckhow, Lavine, et al., 2014; Petrovic, Eljarrat, Alda, et al., 2004; Stanford, Reinert, Rosenfeldt, et al., 2014). Additionally during this review, literature and industry data should be collected on the effectiveness of treatment processes to remove the identified hazards, focusing on the processes that are present in the process train. Through the site-specific sampling campaigns and the literature and industry information, typical categories of hazards as well as ranges were developed and are outlined in Table 3.3. Table 3.3. Examples of Categories of Typical Hazards and Sources

Category Typical Hazards (Examples) Typical Sources

Biologicals

Include protozoa (e.g., Cryptosporidium and Giardia), bacteria (e.g., E.coli), and viruses (enteric)

Fecal matter found in domestic waste as well as animal contamination of storage reservoirs

Inorganics and Metals

Nitrate, nitrite, perchlorate, arsenic, lead, manganese, nickel, and antimony

Trade waste, domestic waste and illegal discharge to sewer

Radionuclides Uranium, radium, strontium, tritium, and iodine

Environmental input and domestic waste (hospitals and hospital discharge patients, laboratory research facilities, defense facilities)

Volatile Organic Compounds (VOCs)

Industrial compounds such as carbon tetrachloride, dichlorobenzene, dichloromethane, MTBE, tetrachloroethylene, vinyl chloride

Trade waste, domestic waste and illegal discharge to sewer, catchment specific as linked to specific industries

Synthetic Organic Compounds (SOCs)

Pesticides1 such as bentazon, carbofuran1, diquat, heptachlor1, molinate and simazine

Run-off/infiltration and illegal discharge

1The reader should check the current regulatory status of any pesticides as some (including carbofuran and heptachlor) may be currently banned or in the process of being phased out of use.

28 Water Environment & Reuse Foundation

With the potential hazards identified, the next steps are to identify the hazards and hazardous events that may affect the quality of the final treated water; assess the risks posed by the relevant hazards and hazardous events; describe how the risks posed by the relevant hazards and hazardous events are to be managed; and note which control measures need to be implemented.

The risk assessment process relies on two main aspects: The characterization of the inherent risk in the source water to be treated and the expected performance of the treatment train. The source water component is very specific to the project being assessed and is considered on a case by case basis as each collection system is subject to a different range of chemical and microbial inputs. The actual water needs to be thoroughly characterized to establish the range of contaminant concentrations which might be expected. In the absence of site specific information, models can be used to simulate source water composition based on existing water quality data from secondary effluents in similar facilities. The second component, the expected performance of the treatment train, usually can be predicted at a design stage by combining existing performance data and specific design parameters. A treatment train can, therefore, be assessed for various risks without the actual infrastructure being built.

In identifying source water hazards, the suitability of the treatment train to reduce any inherent risk to an acceptable level must be evaluated. Listing and categorizing all potential health hazards and assessing the inherent public health impact associated with each of these hazards without any form of treatment and reviewing the treatment barriers and the predicted residual risk after treatment must be assessed. The design of the treatment train should be sufficient to reasonably ensure that no hazards are present and prose unacceptable risk after treatment when all barriers are operating correctly. However, not all barriers operate per design at all times and as such, evaluating the impacts of the failure of a specific treatment step and how risk would be mitigated by implementing critical and other quality control measures must be completed. Through systematically listing hazardous events that could occur, establishing the public health impacts associated with each of the possible hazardous events, and establishing additional control and mitigation measures for hazardous events form the basis of the critical control point identification for source water risk assessment. Possible source water hazards cover a wide range of biological contaminants, inorganics and metals, radionuclides, volatile and soluble organic compounds, disinfection byproducts, and disinfectants. As such, with a wide range of contamination possible a wide range of inherent risks are also possible based on treatment and source water quality; however, a few overarching observations have been found: All biological contaminants have a very high risk based on their prevalence, likely concentration, and

public health impact. Depending on the nature of the collection system, and in particular the impact of heavy industries on

the total flow, heavy metals and some inorganics may present a high risk in source waters. Similarly, volatile organic compounds from industrial activities in a collection system could also

present some significant levels of contaminants such as carbon tetrachloride, dichloromethane, MTBE, and trichloroethylene and could incur high risk classification.

The risk associated with synthetic organic compounds will be very dependent on the impact of run-off on the quality of the collection system and therefore the source water to be treated. Most synthetic organic compound risks are related to the presence of pesticides in the source water and could pose very high risks, and therefore should be evaluated for potential presence during seasonally-appropriate monitoring periods.

Disinfection by products (DBPs) are expected to present a significant risk due to the presence of precursors in secondary effluent, through actual risk will be strongly related to the specific process parameters. A conservative risk based approach is proposed to be taken when assessing the risk associated with DBPs.

29 Water Environment & Reuse Foundation

Through the overall evaluation of all aspects of the source water – industrial pre-treatment, collection, wastewater treatment, advanced treatment drinking water treatment, and distribution, the risks must be fully categorized and recorded in a risk register. This risk register will allow the assessment team to further evaluate any additional treatment steps that may be necessary to mitigate risks or any treatments upstream that might negatively impact the quality of the water. Finally, it should be noted that hazards should be reevaluated periodically as new information or new hazards identified. This living document will provide the basis for transparent decision making for utility managers, engineers, and operations staff.

While there are different ways of developing a risk register, an example is shown below whereby likelihood and consequence risk descriptors used are noted above in the previous Section 3.2.1 and a simple color coded graphic system (Figure 3.4) is used as a part of project, adapted from Reuse-13-03), is useful to clearly express the presumed or expected ability of each treatment barrier to remove a particular hazard or class of hazards on a semi quantitative/qualitative basis. An even more qualitative version of this register notation would be at three levels: (a) the process step is intended to control that particular hazard, (b) the process step is not intended to control that hazard but provides some additional ancillary removal, or (c) the process step is not intended to control that hazard and provides little-to-no removal of that hazard.

○ Poor = no significant expected removal (<20%)

◑ Fair = up to 60% removal

◕ Good = up to 90% removal

● Excellent = greater than 90% removal

No data available

Figure 3.4 Possible Symbols and Removal Rates for Risk Register

30 Water Environment & Reuse Foundation

The overall ability of the train to remove a particular hazard is expressed using the same system and this can be applied to all hazards for which the inherent risk is assessed conservatively as High or Very High. Figure 3.5. Extract of the Inherent and Residual Risk AssessmentFigure 3.5 provides an extract of the risk register showing how this was applied in the case of the RO membrane-based train. (More details on all aspects of the risk register can be found in Reuse-13-03).

Figure 3.5. Extract of the Inherent and Residual Risk Assessment

Ha

za

rd

Ta

rge

t

(lo

we

r o

f E

PA

& C

DP

H)

Ma

x c

on

ce

ntr

ati

on

in

so

urc

e

Un

it

Ra

tio

Ma

x/T

arg

et

Imp

ac

t

So

urc

e

Co

ns

eq

ue

nc

e

Lik

elih

oo

d

Ris

k

Cryptosporidium 0Acute

Health

Domestic waste - human and animal faecal matter

Contamination of storage reservoirsCatastrophic

Almost

Certain

Very High

(E5)

Giardia lamblia 0Acute

Health

Domestic waste - human and animal faecal matter

Contamination of storage reservoirsCatastrophic

Almost

Certain

Very High

(E5)

Barium 2 0.006 mg/L 0.003Chronic

HealthTrade waste, Domestic waste, Illegal discharge Minor Possible

Moderate

(C2)

Beryllium 0.004 0.005 mg/L 1.25Chronic

HealthTrade waste, Domestic waste, Illegal discharge Moderate Possible High (C3)

Nitrate (as N) 10 32 mg/L 3.2Acute

HealthTrade waste, Domestic waste, Illegal discharge Major Possible

Very High

(C4)

Inherent Risk(drinking feedwater directly at 2L

per day)

Ha

za

rd

MF

/UF

Re

ve

rse

Os

mo

sis

UV

-AO

P

Ch

lori

ne

/ D

isin

fec

tio

n

Ov

era

ll

Re

qu

ire

d t

rea

tme

nt

eff

icic

ien

cy

Tre

atm

en

t B

arr

iers

Co

ns

eq

ue

nc

e

Lik

elih

oo

d

Ris

k

Cryptosporidium ◕ ● ● ○ ● 10 log UF, RO, UV, Chlorine Insignificant Rare Low (A1)

Giardia lamblia ◕ ● ● ● ● 10 log UF, RO, UV, Chlorine Insignificant Rare Low (A1)

Barium ○ ● ○ ○ ● N/A RO Minor Rare Low (A2)

Beryllium ○ ● ○ ○ ● N/A RO Minor Rare Low (A2)

Nitrate (as N) ○ ◕ ○ ○ ◕ N/A RO Minor Unlikely Low (B2)

Barrier Assessment(drinking product water assuming all barriers worked as designed)

Treatment Effectiveness

31 Water Environment & Reuse Foundation

In addition to water quality risks, hazardous events which could affect the treatment train should also be assessed in terms of their likelihood and the hazards such events would introduce. It should be noted that the hazardous events risk assessment would be quite specific to the particular design of the treatment plant, the catchment as well as the downstream distribution system, thus the analysis provided here is generic and would need to be modified for a specific facility.

The example below is of a number of events that could have an impact on the quality of the product water, and these can be separated in the categories listed below (Reuse-13-03). However, it is important to keep in mind that “unit processes” are in often in reality a set of multiple parallel operating and/or standby units. Thus, the likelihood of failure of an entire process would imply that all filters (for example) across the entire plant fail at the same time. Thus, the more likely scenario is that a portion of a given process step may fail. However, both eventualities must be considered in the risk assessment.

Upstream of the advanced treatment: o Accidental contamination of the catchment (such as the one-off discharge of large quantities

of industrial chemicals); o Outbreaks of infectious diseases in the community leading to unusually high levels of

pathogens in the source water; o Failure of biological processes that form part of the sewage treatment process; o High rainfall events leading to bypass of the sewage treatment process;

Within the advanced treatment: o Catastrophic integrity breaches of MF/UF or RO membrane filtration system components or

racks; o Catastrophic failures of filtration processes; o Overloading of filters; o Failure of dosing/control systems for UV, Ozone or chlorine based disinfection; o Formation of DBPs; o Failure of dosing systems in relation to product water stabilization (increased corrosivity) o Overdosing, underdosing, or contamination of chemicals added as part of the treatment

process; Downstream of the advanced treatment:

o Formation of DBPs after the treatment process within the distribution systems; o Downstream impact of water quality on distribution system, including corrosion and release

of contaminants (metals) into the system.

This list is non-exhaustive and more details specific to a given facility should be considered as part of the actual risk assessment. For each of the hazardous events, control measures were identified which became critical control points (CCPs) when it was found that these are critical to guarantee an acceptable risk to public health.

32 Water Environment & Reuse Foundation

3.2.3 Health and Safety Risk Assessment

Any water, wastewater or recycled water facility will require activities that have a potential to pose health and safety risks to plant operators, site contractors and visitors alike. DPR does not present substantially different risks to those encountered at other treatment plants, however operational staff transitioning to a DPR plant may encounter several treatment technologies which contain safety risks that they are as yet unfamiliar with. These may include items such as: Specific chemicals (e.g., hydrogen peroxide, ozone). High pressure (reverse osmosis, microfiltration air systems). High temperature (chemical cleaning systems). Microbiological risks (source water handling/sampling). Handling risks (membrane loading, UV lamp handling).

A health and safety risk assessment is a valuable tool to capture safety risks and provides a tool to help adopt safety controls and continually re-assess the health and safety risk profile.

Risks are identified through a number of avenues by operations staff. It is useful to begin the process at the plant design phase, as this provides an opportunity for operations staff in particular to review the operability of the plant. HAZOP (Hazard and Operability studies) is one useful approach that will highlight safety risks and provide an opportunity for these to be resolved at the design phase. Other opportunities to identify risk include: Analysis of industry and project safety history. Consultation and workplace risk assessments with operations staff. Review of vendor and equipment literature. Lessons learned from prior operation.

Risk consequence should be classified pending the significance of potential injury or harm to personnel. Table 3.4 provides an example taken from an operating IPR plant in Australia (confidential plant).

Table 3.4. Risk Consequence Descriptors – Health and Safety Risk Assessment

Insignificant Minor Moderate Major Extreme

Small First Aid Injury

Medical treatment injury. Near miss with the potential for minor or moderate incident consequences.

Lost time injury (without hospitalization). Near miss with the potential for major incident consequences.

Lost time injury with hospitalization and an ability to return to work in any capacity after treatment. Near miss with the potential for extreme incident.

Fatality or multiple serious injuries to staff, contractors or the general public. Lost time injury resulting in hospitalization and permanent disability.

33 Water Environment & Reuse Foundation

Likelihood is determined by examining the frequency and exposure of a hazard occurring. Frequency of exposure is the extent to which a source of risk exists, and probability is the chance that when that source of risk exists, consequences will follow. Table 3.5 describes the various levels of likelihood. Table 3.5. Risk Descriptors – Likelihood Health and Safety Risk Assessment

Likelihood Level Descriptor Description

5 Almost certain. The event is likely to occur daily or weekly over a period of one calendar year.

4 Likely The event is likely to occur on a monthly or quarterly basis over one calendar year.

3 Possible The event is likely to occur on a half yearly or yearly basis over a one calendar year.

2 Unlikely The event is likely to occur once in 2 or 3 years.

1 Rare The event is likely to occur in exceptional circumstances.

As for other risk registers described, a final risk matrix is developed using the combination of likelihood and consequence. It should be a goal for the operations team to ensure all health and safety risk be controlled to a level as low as reasonably possible.

Typically the management of health and safety risk is covered in operational procedures, equipment and process safety guidelines, chemical material safety data sheets and specific emergency responses for incidents included in an overall emergency response plan.

3.2.4 Environmental Risk Assessment

The operation of a DPR facility will require environmental considerations. As for health and safety, these environmental risks are not necessarily unique to DPR, but in combination may present a range of risks that operational teams have not yet encountered. This may include risks such as: Reverse osmosis concentrate disposal. Ozone release to atmosphere. Chemical spills, release to the environment. Solids waste management risks. Non-compliance with regulatory requirements. Pest, weed control. Greenhouse gas/emission targets.

As for previous risk assessments, likelihood and consequence must be considered to ensure risks are minimized at an acceptable level. A useful guide for the environmental risk register is to include a reference to the applicable regulation (if any) for each hazard listed.

Typically, the management of environmental risk is covered in operational procedures, with specific emergency responses for incidents included in an overall emergency response plan. An example has not been included here for brevity.

34 Water Environment & Reuse Foundation

3.2.5 Asset Condition and Risk Assessment

The success of DPR processes relies on good maintenance of assets. From a public health standpoint, critical control processes must be maintained to standards to ensure effective treatment, and critical control monitors must be highly reliable. Assets must also provide reliable production, and be cost effective as a water supply option. To ensure that assets are maintained as reliably as possible while maintaining a cost effective supply it is important to understand the both the condition of assets, but also the criticality of those assets to ensure that resources are applied strategically to minimize risks of failures and adverse impacts to budgets from unanticipated or early asset replacements.

The purpose of the asset condition and risk assessment is to provide a structured and consistent approach to review the condition and expected life of the asset, maintenance and renewal requirements, and the management of asset risks. This ultimately feeds into renewal and replacement programs. A risk-based approach represents leading practice in determining the method of assessment, and the frequency of monitoring activities. Asset condition rating and criticality factor are used to determine the asset risk profile.

An asset’s condition is an indication of its likelihood of failure. The condition grading takes into account the physical condition of the asset from visual inspections, maintenance records, condition monitoring, asset age, operating environment and the expected useful life of the asset (Table 3.6). The useful life of an asset is the period where the asset provides the designated level of service at an economical cost. The asset life is dependent on a number of factors, which include material, construction methods, operating environment and levels of maintenance. Table 3.6. Asset Condition and Likelihood of Failure

Condition Index

Condition Grade

Chance of loss of performance (12-month period)

Likelihood of Failure in Next 12 Months

1 Excellent Condition is as new. Performance is optimal.

Almost None

2 Good Only minor signs of deterioration. Performance is sufficient.

Very Unlikely

3 Average Noticeable signs of deterioration. Performance is efficient.

Unlikely

4 Fair Concerning levels of deterioration. Performance is borderline.

Possible

5 Poor Unacceptable levels of deterioration. Performance is adequate.

Expected

Condition assessment usually involves two types of condition ratings: Physical Condition – The current state of repair and operation for the equipment item. The physical

condition is determined by an inspection in the field. Performance Condition – The ability of the equipment item to meet operational requirements now

and in the future. The performance condition is determined in discussions with operations and maintenance staff after the physical condition is assigned. Typical physical and performance condition criteria are shown in Table 3.7.

35 Water Environment & Reuse Foundation

Table 3.7. Physical and Performance Condition Failure Modes

Condition Failure Mode Evaluation Criteria Probable Approach

Physical Mortality Information Capital OR Maintenance Mechanical and Electrical Testing

Performance Mortality Reliability (breakdowns) Capital OR Maintenance

Capacity Current capacity testing Capital OR Maintenance

Future capacity needs Capital

Level of Service

Current and future regulatory needs Capital

Other LOS measures Capital OR Maintenance

Efficiency Obsolescence Capital O&M issues (not breakdowns) Capital OR

Maintenance

The criticality, or consequence of failure, for an asset is the measurement of the impact of failure of that asset on the operation of the treatment plant. Consideration is given to the individual process and the plant as a whole.

In assigning the asset criticality factor, it is important to consider the following: Impact on the safety of personnel. Impact to public health/water quality. Impact to the natural environment. Impact on contractual requirements. Cost of replacement of the asset. Current maintenance requirements. Critical spares availability.

36 Water Environment & Reuse Foundation

A list of criticalities is shown in Table 3.8. Often asset criticality is listed as simply non-critical, low criticality and critical. In addition, for DPR it is recommended that a fourth category, “CCP critical,” be added to clearly define those items that are critical to a critical control point (that is the process barrier itself or the critical monitor). Table 3.8. Criticality Indices

Criticality Index Criticality Description Example

1 Non critical Failure does not have an adverse impact on safety, performance or the environment.

Sample valve.

2 Low criticality Failure would have an adverse impact, but back-up protection, such as redundancy, protects against it.

Failure of a duty pump in a duty/standby arrangement.

3 Critical Failure does have adverse impact on safety, performance or the environment.

Clarifier drive failure.

4 CCP Critical Failure has an adverse impact on critical control points or critical control monitoring.

Failure of chlorine monitoring for final chlorine CT.

The asset risk assessment is conducted using the asset condition and criticality ratings. The ‘likelihood’ of an event occurring is a qualitative or quantitative description of probability or frequency of it occurring. In this context the condition rating is a measure of the likelihood of that asset failing. The poorer the condition of an asset the more likely it is to fail. The ‘consequence’ of this failure is the impact it has on an event expressed qualitatively or quantitatively, being a disadvantage or a gain. In this context the criticality factor determines the impact the failure of the asset has on the plant operation and hence loss of service. The assessment score is a measure of the condition rating multiplied by the criticality factor.

37 Water Environment & Reuse Foundation

Table 3.9. Asset Risk Assessment Matrix Example

Criticality Factor “consequence”

Non-Critical

1

Low Criticality

2

Critical

3

CCP Critical

4

Con

ditio

n R

atin

g “l

ikel

ihoo

d” 1 Excellent 1 2 3 4

2 Good 2 4 6 8 3 Average 3 6 9 12 4 Fair 4 8 12 16 5 Poor 5 10 15 20

The asset assessment score can be grouped into five levels for analysis and review and determination of future strategies for asset maintenance. The descriptions for these scores are shown in Table 3.10

Table 3.10. Example Asset Assessment Scoring

Risk Index Risk Level Requirement for Action 1-4 Low Asset replacement/renewal is not a priority. 5-6 Moderate Condition to be monitored, asset renewal timing

to be reviewed. 7-10 High Further investigation to be carried out and failure

contingency strategy to be in place. Closer monitoring to be brought forward. Asset renewal may be brought forward.

11-15 Very High Failure contingency strategy to be in place. Immediate action/intervention is required.

16-20 Extreme CCP As for Very High, however communicated as critical for protection of public health.

This scoring gives an indication of the plant asset risk profile. This condition assessment/risk analysis is typically included in an overall asset management plan.

38 Water Environment & Reuse Foundation

3.3 Critical Control Points – the HACCP Process

3.3.1 Hazard Analysis and Critical Control Points – HACCP

The HACCP methodology has been adopted internationally by a number of countries to manage microbiological and chemical contaminants in water treatment systems, including recycled water systems (Halliwell, Burris, Deere, et al., 2015). An excellent review of the history of HACCP and its applications worldwide are provided in the WateReuse-09-03 report by Halliwell et al. (2015).

The HACCP system was originally developed as an engineering means of controlling microbial hazards in consumed food. HACCP is a logical, scientific process control system designed to identify, evaluate and control hazards, which are significant for food safety. The purpose of a HACCP system is to put in place process controls that will detect and correct deviations in quality processes at the earliest possible opportunity. It focuses on monitoring and maintaining the barriers of treatment, rather than on end of pipe sampling and testing. This provides the dual advantage of ensuring poor quality is prevented in the first place, and allows for a reduction in end of pipe monitoring and associated costs.

In its essence, the HACCP process is categorized into seven principles that are used to assess risk and determine a well-defined path forward for managing those risks and operation of the facility. The principles, whether part of a true HACCP/ISO 22000 accredited system or one that is using the principles to guide them through DPR assessment and operation, can be used to guide the process of developing critical control points for potable reuse: Principle 1: Conduct a hazard analysis (as noted in Section 3.2.2 of this report). Principle 2: Determine the Critical Control Points. Principle 3: Establish Critical Limits. Principle 4: Establish a system to monitor the control of a CCP. Principle 5: Establish the corrective action to be taken when monitoring a CCP is not under control. Principle 6: Establish procedures for verification to confirm that the HACCP system is working effectively. Principle 7: Establish documentation concerning all procedures and records appropriate to these principles and their application.

It is important to note that the HACCP system identifies critical control points (CCPs) as points in the treatment process that are specifically to reduce, prevent, or eliminate a human health hazard and for which controls exist to ensure the proper performance of that process. Highlighting these points assists operators, regulators and other stakeholders to place a primary focus on public health, as distinct from other important operational elements.

From an operations and asset management standpoint, there are other points in the treatment train that are important to ensure the production capacity of the facility and the effective maintenance of equipment. These points are often named critical operating points (COPs) and are important in the overall operation and maintenance of the facility but do not by themselves directly influence water quality and public health. As such, COPs are points in the treatment process that are specifically designed to maintain the production capacity of the facility and protect working assets.

As an example to demonstrate the difference between CCPs and COPs, the following is provided:

The RO process step is defined as a CCP for microorganism removal as well as for chemicals of concern (Figure 3.6). It operates as a CCP, with a critical monitor of electrical conductivity (EC). If the EC increases above an alert limit (set at some margin below what would be considered a breach), an alarm

39 Water Environment & Reuse Foundation

will be raised on the plant SCADA system and corrective action is taken by the plant operations staff. If the EC increases above a critical limit (set close to, or at what would be considered a breach), the control system will automatically shut the unit down, an alarm condition is raised at the SCADA, automated control actions will be enacted and operators will take corrective action.

Figure 3.6 Diagram of a CCP and Associated Monitor for an RO System For the RO unit, there are a number of other very important monitoring parameters, control actions and operating responses that are critical to the success of that process, but are not of direct relation to public health protection. For example, antiscalant dosing and pH correction are important to manage scaling, RO recovery is important also in the management of scaling and other flow settings for the RO must be maintained for correct operation. In these cases, control response and operational responses can be articulated as COPs (Figure 3.7), using the same format for operations for consistency, and yet differentiating the COPs from CCPs.

40 Water Environment & Reuse Foundation

Figure 3.7. Diagram of Critical Operating Points (COPs) Supporting the RO CCP and Plant Production Both CCPs and COPs are important for the successful operation of a DPR plant. However, DPR facilities contain a number of different process treatment units with high levels of plant automation and importantly SCADA alarms. By differentiating CCPs, the operational team can clearly demonstrate and communicate the consideration of public health. This assists in both ensuring that public health is considered of paramount importance to operations while at the same time maintaining a clear focus on health related performance for regulators and other stakeholders, without becoming distracted by the multitude of other operational considerations. As an example, the Western Corridor Recycled Water Project in Australia adopted the HACCP methodology for the three advanced water treatment plants operating to produce IPR. At each of these plants (utilizing microfiltration, reverse osmosis, advanced oxidation and final chlorination), a separate SCADA screen was dedicated to CCP only, indicating current condition for each process barrier and providing information on any current alarm condition.

3.3.2 Selecting Critical Control Points – Overall The HACCP process provides a strict definition of a critical control point. This is useful, as it ensures that only process barriers with a significant impact to protection of public health are selected. This is important, as there may often be a tendency to select a large number of critical control points throughout a treatment process, resulting in a system that may be cumbersome for operational teams to maintain.

The following critical control assessments, based on the work of Halliwell (2015), and further developed by WE&RF project Reuse-13-03 (Walker, Stanford, Khan, et al., 2016) contain a flow diagram of five questions to determine whether a process or point in a process train qualifies as a CCP. The five questions are as follows: 1. Is there a hazard at this process step? (And what is it?) 2. Do control measure(s) exist for the identified hazard? 3. Is the step required to achieve a log reduction of microorganisms and/or to meet water quality targets? 4. Could contamination occur at or increase to unacceptable level(s)? 5. Will a subsequent step or action eliminate or reduce the hazard to an acceptable level?

41 Water Environment & Reuse Foundation

A classic example of applying this methodology to a complete process configuration from water resource recovery to drinking water production is the bar screens at the head of the wastewater treatment plant:

Is there a hazard at this step? Yes, microbial and chemical hazards exist with near certainty in raw sewage.

Do control measures exist for the identified hazard? No, this step in the process is not designed to disinfect or remove dissolved constituents, therefore it is not a CCP.

However, from a facility operations perspective, having the bar screens operating properly is critical to the overall facility operation and production, therefore it would fall into the COP category. By questioning each step in the complete process train in the manner shown in Figure 3.8, it allows operations to focus on aspects of the facility that may need immediate attention for public health protection vs. aspects of the facility that need attention from a plant production and asset maintenance perspective.

Figure 3.8. HACCP System Decision Tree for Defining Critical Control Points in DPR Facilities

42 Water Environment & Reuse Foundation

The selection of critical control points (CCPs) for two different treatment trains have been developed as a key outcome of project Reuse-13-03. This included the assessment of an RO based treatment train (consisting of chloramination/MF/RO/UV-H202/Stabilization/Chlorine and a Non-RO based treatment train which consisted of Coagulation-Settling/Ozone/BAC/GAC/UV/Chlorine). The selection of these two process trains does not imply that these are the most appropriate technological choices for all DPR applications. They do, however, provide useful examples of systems operating with the “conventional” MF/RO approach employing the RO as a desalination process, as well as a non-RO based approach which may be more applicable where TDS reduction is not required and/or RO concentrate disposal is prohibitive.

3.3.3 HACCP Team

The HACCP process has been developed from the food industry, and as such recommends that a “Food Safety Team” be convened to determine CCPs for a food production facility. For water treatment applications this may be translated into a “Water Safety” group or “HACCP team”. The success of implementation and maintenance of the HACCP system depends on a team that is committed to the process, and is representative of a range of disciplines covering operations, asset management, engineering, planning, environmental management and compliance. Overall, the team should be designed to cover elements of water quality risk and control of risk, and ensure that the HACCP process is integrated into all aspects of plant operation.

A typical HACCP team should consist of:

A HACCP team leader. This person acts as the coordinator and champion of the HACCP process and will have overall responsibility of developing HACCP plans. This person is often from an operations or water quality management background (Halliwell, Burris, Deere, et al., 2015).

A Water Quality Manger or Water Quality Specialist. Operations manager and senior plant operators. Asset management/maintenance representative. Planning and development representative. Trade Waste/Environmental Manager. Risk Management/Compliance.

The HACCP team begins by developing a source water quality risk assessment, as described in Section 3.2.2. These risk assessments can be time consuming, and it is often more practical that it is developed by a sub-group of the HACCP team (usually the water quality manager) prior to a larger group review. The source water quality risk assessment as developed in Reuse-13-03 was intended to cover a range of possible source water qualities, and was not specific to a particular plant, but rather considered water quality health risks in source water from a number of operating indirect potable reuse plants as well as an extensive review of literature.

Very often, the source water quality data (i.e., treated effluent) that is available for a new reuse facility (either IPR or DPR) is limited and does not contain water quality parameters of interest for a substantial review of public health. Instead, the HACCP team may be presented with historical water quality data that is focused on wastewater treatment discharge permits. Additionally, even the data that is available may not be sampled at sufficient frequency to provide knowledge of seasonal or temporal plant performance – important in developing an effective source water quality risk assessment. In the absence of detailed water quality data, the assessment conducted in Reuse-13-03 is recommended as a basis and starting point for individual plant risk assessments (included in Reuse-13-03). This can provide a baseline risk review, while a sampling and analysis campaign is developed to provide specific knowledge of water quality.

43 Water Environment & Reuse Foundation

Once an agreed source water quality risk assessment has been developed, the team then considers their specific treatment process, working through the questions in Figure 3.8 in a workshop setting. In addition to the HACCP team listed above, there may be value in including expertise external to the water utility including water quality/laboratory analysts, engineering consultants and health regulators for this workshop. This is especially important for smaller utilities which may not have the same depth of specialist staff internal to their organization as some larger utilities.

The methodology provides a logical sequence of questions to ensure rigor in the selection of the CCPs. For DPR applications, a clear focus should be placed on whether the identified barrier would be counted on to provide either a specific, measurable microorganism log reduction, or provide a measureable reduction in another contaminant or contaminants that will assist in achieving final water quality targets. This assists in filtering out marginal removal capabilities, or those that may not be accepted by regulators.

The following two sections provide two examples of critical control point selection as developed for the RO and non-RO based treatment trains as a part of Reuse-13-03.

3.3.4 RO-Based Treatment Train Critical Control Point Selection

Table 3.11 outlines the critical control point selection that was determined as a part of Reuse-13-03. The CCP team’s response to each particular question (those listed in Figure 3.8) is included to demonstrate clearly the rationale for decisions taken, and can act as an example for future CCP selection. CCP monitoring parameters are also described in Table 3.11. The CCPs are also indicated by solid outlines in Figure 3.9.

Table 3.11. CCP Selection Process and Indicators – RO Membrane-Based Treatment

Process step CCP Decision Monitoring Parameters Chloramine Dosing

Q1 Yes Q2 Yes Q2a N/A

Q3 No Q4 Yes Q5 No → CCP

In relation to Question 5, some levels of DBP may persist through remaining process

Total (combined) chlorine

Inlet Strainer Q1 Yes Q2 Yes Q2a N/A

Q3 No Q4 No Q5 Yes → NOT A CCP

In relation to Q1, the strainer can potentially remove some contaminants, either biological or chemical

N/A

Ultrafiltration Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q3, this filtration process removes microbiological hazards

Pressure Decay Integrity Test and Individual (or combined) filter effluent turbidity Pressure decay integrity testing provides superior resolution, however it is a discrete test. Turbidity can provide an effective continuous back up measure.

Reverse Osmosis

Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A

In relation to Q3, this process removes microbiological and chemical hazards

Electrical conductivity On line TOC

44 Water Environment & Reuse Foundation

→ CCP

Electrical conductivity and/or on line TOC are currently the most sensitive analyzers for this task. More sensitive on line analytical techniques are emerging which will improve the resolution of this monitoring point.

UV-H2O2 Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q3, this process removes microbiological and chemical hazards (and specifically NDMA)

UV Present Power Ratio (Ratio of EED/EER) Confirmed dose of Hydrogen peroxide; UVT of feed water. The UV dose provided for advanced oxidation is significantly higher than required for disinfection. Operationally the energy utilized to meet NDMA removal will provide sufficient UV dose for disinfection targets.

Stabilization Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q1, the hazardous event is the downstream mobilization of the hazards lead and copper if the stabilization process is not well controlled. Note that this may not be as significant if the final product is delivered to the head of a water treatment plant, or is effectively blended prior to introduction to the distribution system.

pH, applied chemical dose, TDS, periodic alkalinity checks, CCPP (calculation) LSI (calculation) (Breaks down as hardness, alkalinity, pH and TDS

Chlorination Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q1, the hazards are both microorganisms and the potential addition of perchlorate. Note that this may not be as significant if additional disinfection credit can be achieved by introduction to the inlet of a drinking water treatment plant

Free chlorine residual Chlorine dose CT (calculated)

45 Water Environment & Reuse Foundation

Figure 3.9. Critical Control Points (Outlined) – RO Membrane-Based Treatment Train

For each of the selected critical control points, the Project Team also identified monitoring parameters to ensure that the CCP is meeting its required removal performance. These, in turn, will inform the development of operational response procedures, as described Table 3.12. Table 3.12. Monitoring Parameters for CCPs in RO-Based Treatment Process Step Health Risk Management Monitoring Parameters

Pre-Chloramination NDMA Control Mechanism Total (combined) chlorine

MF/UF Microorganism Control Pressure Decay Integrity Test Individual Filter Effluent Turbidity

RO Microorganisms and Chemicals of Concern

Electrical Conductivity Online TOC

UV/H2O2 Microorganisms and Chemicals of Concern (e.g., NDMA and 1,4-dioxane)

UV Present Power Ratio Hydrogen Peroxide UVT of Feed Water Turbidity of Feed Water

Stabilization Lead and Copper pH TDS Alkalinity Applied Chemical Dose CCPP and LSI Calculations

Chlorine Microorganisms, Chlorate, DBP Management

Free Chlorine Residual and Dose CT (calculated)

3.3.5 Non-RO-Based Treatment Train Critical Control Point Selection

The non-RO treatment train selection decisions are shown in Table 3.12. For this process train of ozone-BAC-GAC-UV-chlorine, our team encountered initial difficulty in meeting the required log reduction of

46 Water Environment & Reuse Foundation

micro-organsims via the process identified at the outset of the project. Specifically, the biological activated carbon (BAC) process was unable to be considered as a CCP as a stand-alone process because there was no control mechanism to adjust its ability to achieve pathogen reduction or contaminant removal. Instead, by modifying the process to incorporate a coagulation step ahead of filtration, the BAC process could now be incorporated as it would be effective at turbidity removal (and hence a level of microorganism) if operated like a biological filter with turbidity performance goals. Additionally, by considerating ozone-BAC as a single process barrier (similar to including UV-H2O2 as a single barrier) it could also be incorporated as a CCP. Therefore the revised non-membrane treatment process became ozone-flocculation/sedimentation-BAC-GAC-UV-chlorine. This process is more consistent with the actual process train at the Goreangab plant in Windhoek, Namibia, though could include various combinations of ozonation before and/or after the flocculation/sedimentation step. This decision making, while in a sense straight forward from a process design and selection viewpoint, was nonetheless facilitated by CCP selection process. The workings of this process are outlined in Table 3.13 and a summary figure depicting the CCPs is shown in Figure 3.20. It should be noted that in Figure 3.20, the pre-ozonation step is shown as an optional configuration and is not listed as a CCP though in many DPR settings this may be selected as a CCP.

Table 3.13. CCP Selection Process –Non-Membrane Treatment Process Process Step CCP Decision Monitoring Parameters Ozone Q1 Yes

Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q1, this process step is concerned with the treatment of microorganisms and chemicals but also DBP formation and control

Ozone dose Ozone residual CT (calculated) Change in UVT (Delta UVT)

Biologically Activated Carbon (BAC)

Q1 Yes Q2 No Q2a Yes

Q3 N/A Q4 N/A Q5 N/A → NOT A CCP

In relation to Q2a, BAC alone cannot be considered a CCP, the process must be considered in combination with ozone for chemical removal, and have modifications to provide removal for microorganisms

Ozone-BAC (combined step)

Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q3, this process step is concerned with the treatment of chemical hazards. Note that the process has been combined to provide a CCP for chemicals

Ozone dose EBCT

Coagulation-BAC (combined step)

Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q3, this process removes microbiological hazards. Note that the process has been combined to provide a CCP for microorganisms

Filtered water turbidity Coagulant Dose Ratio TOC post BAC to feed TOC

Granular Activated Carbon (GAC)

Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q1, a range of chemical hazards are managed by this process step, including TOC, DBPs, DBP precursors and other chemicals

Carbon life TOC and/or UVT

UV Disinfection Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q1, the hazards are microorganisms.

UV Dose UV Transmissivity

Chlorine CT Q1 Yes Q2 Yes Q2a N/A

Q3 Yes Q4 N/A Q5 N/A → CCP

In relation to Q1, the hazards are both microorganisms and the potential addition of perchlorate. Note that this may not be as significant if additional disinfection credit can be achieved by introduction to the inlet of a drinking water treatment plant.

Free chlorine residual Chlorine dose CT (calculated)

47 Water Environment & Reuse Foundation

Figure 3.10. Critical Control Points (Outlined) – Ozone-BAC-Based Treatment Train as Shown with Optional Pre-Ozonation Step

Table 3.14. Monitoring Parameters for CCPs in Ozone-BAC-Based Treatment

Process Step Health Risk Management Monitoring Parameters

Ozone Microorganism Control DBP Control Mechanism Dissolved Organic Constituents

Ozone dose Ozone residual CT (calculated) Change in UVT (Delta UVT)

Ozone/BAC Combined Step Dissolved Organic Constituents DBPs and DBP Precursors

Ozone dose EBCT

Coagulation/BAC (Combined step)

Microorganisms and Turbidity Filtered water turbidity Coagulant Dose Ratio TOC post BAC to feed TOC

Granular Activated Carbon (GAC)

Dissolved Organic Constituents, DBP Control

Carbon life TOC and/or UVT

UV Disinfection Microorganism Control UV Dose UV Transmissivity

Chlorine Microorganism Control, DBP Management, Chlorate

Free Chlorine Residual and Dose CT (calculated)

48 Water Environment & Reuse Foundation

3.3.6 Log Reduction of Microorganisms

As noted in Section 3.3.2, one of the key questions to be asked in the determination of a process unit as a CCP is any log reduction (that is percentage removal expressed as a logarithm) credit for microorganisms that can be achieved by each identified CCP. While the CCP process requires the review of all water quality health related risks, microorganisms are considered an acute health risk, and as a result specific attention is given to identifying log reduction capability.

The log reduction required for microbial pathogens is described for virus, Cryptosporidium, Giardia and bacteria. Giardia is sometimes mentioned separately, or in some cases is implied with the Cryptosporidium removal requirement.

WRRF-11-02 established microorganism log reduction requirements from raw wastewater to final reclaimed water (Trussell, Salveson, Snyder, et al., 2013). The requirements were based on maximum concentrations of microorganisms from literature in raw wastewater, and a requirement to meet a 1/10,000 (one infection per ten thousand people per year) as an acceptable level of risk in final drinking water. The proposed removal targets are noted in Table 3.15.

Table 3.15. Log-Reduction Targets Based on CA IPR Regulations, WRRF-11-02, and Texas Commission for Environmental Quality Requirements

Microbial Pathogen Virus Giardia Cryptosporidium Total Coliform

Log reduction-CA 12 10 10 --

WRRF-11-02 12 10 10 9

Log reduction-TX 8 6 5.5 --

California does not currently have specific requirements for DPR, however it has adopted regulations for groundwater recharge IPR and at the time of writing was developing regulations for surface water augmentation. The California Groundwater Replenishment Rules Title 22, CCR, Division 4, Chapter 3, Article 1 in document “DPH-14-003E, GW Replenishment Using RW, May 30, 2014” requires that “recycled municipal wastewater used as recharge water for a GRRP receives treatment that achieves at least 12-log enteric virus reduction, 10-log Giardia cyst reduction, and 10-log Cryptosporidium oocyst reduction.

Texas also does not currently have specific log reduction requirements for DPR. The Texas Water Development Board (TWDB) Direct Potable Reuse Research Document Final Report (McDonald and Nellor, 2015) notes that in contrast to California, the Texas Commission on Environmental Quality (TCEQ) has established minimum log reduction requirements aimed at the same 1/10,000 acceptable risk as noted for California, however using treated wastewater effluent data as its starting point. Table 3.15 is a starting point for removal requirements, and is assessed on a case by case basis depending on microbial pathogen concentrations in the specific wastewater effluent to be treated with a requirement of 8-log virus, 6 log Giardia and 5.5 log Cryptosporidium oocyst reduction.

California further requires that there is a multiple barrier approach to ensure that removal is not reliant upon too few barriers. Specifically, it requires that for each pathogen “as separate treatment process may be credited with no more than 6-log reduction, with at least three processes each being credited with no less than 1.0 log reduction” (CDPH, 2014).

49 Water Environment & Reuse Foundation

Requirements such as this need to be taken into consideration in the determination of CCPs, and importantly the determination if sufficient log reduction has been achieved. If sufficient log reduction cannot be established for a given treatment train, an additional process barrier may need to be incorporated.

Along with these constraints, a particular challenge for treatment process selections is the determination of what log reduction of microorganisms can be reliably validated for particular technologies. Again in California, a report prepared by a certified engineer may be used to propose a given log reduction credit for a given process if the report provides “evidence of the treatment process’ ability to reliably and consistently achieve the log reduction” and provides a “microbial, chemical or physical surrogate parameter(s) that verifies the performance of each treatment processes’ ability to achieve its credit log reduction” (CDPH, 2014).

The Reuse-13-03 report provided a compendium of the current creditable log reduction by treatment process for virus, Cryptosporidium and Giardia, as shown in Tables 3.16 through 3.18. Table 3.16. Virus Creditable Log Reduction by Treatment Process (Footnotes Follow Tables)

Treatment California Maximum

Creditable U.S. EPA

Maximum Creditable* Free chlorine 4.0-log (1)

Ozone 4.0-log (1)

UV Disinfection 4.0-log (2) 3.0-log (3)

UV AOP 6-log(7) MF (membrane filtration) [UF]

0.5-log (4) [4-log](4)

As determined by the state (4)

RO 2.0-log (5, 7) As determined by the state (4) Coagulation/Filtration

Conv. Sed/Filt Direct Filtration

2.0-log (1)

1.0-log (1)

*CT calculations depend on temperature and pH Table 3.17. Cryptosporidium Creditable Log Reduction by Treatment Process (Footnotes Follow Tables)

Treatment California Maximum

Creditable U.S. EPA

Maximum Creditable* Free chlorine 0-log (6)

Ozone 3-log (2)

Log credit variable (6) UV Disinfection 4-log

Log credit variable (6) UV AOP 6-log(7) MF/UF (membrane filtration)

4.0-log (4) As determined by the state (4)

RO 2.0-log (5, 7) As determined by the state (4) Coagulation/Filtration

Conv. Sed/Filt Direct Filtration

2.5-log (up to 3.5-log)(4)

3-log (up to 4-log) (4)

Lime Softening 0.5-log (2)

* CT calculations depend on temperature and pH

50 Water Environment & Reuse Foundation

Table 3.18. Giardia Creditable Log Reduction by Treatment Process (Footnotes Follow Tables)

Treatment California Maximum

Creditable U.S. EPA

Maximum Creditable* Free chlorine 3-log (1)

Ozone 3-log (1)

UV Disinfection 4-log(6)

UV AOP 6-log(7) MF/UF (membrane filtration)

4.0-log (4) As determined by the state (4)

RO 2.0-log (5) As determined by the state (4) Coagulation/Filtration

Conv. Sed/Filt Direct Filtration

2.5-log (1)

2.0-log (1)

* CT calculations depend on temperature and pH

Footnotes: Legal Document Supporting Information/References in Tables 2-2 through 2-4: 1. U.S. EPA (1999). Alternative Disinfectants and Oxidants Guidance Manual, U.S. EPA Office of

Water. EPA/815/R-99/014. 2. U.S. EPA (2007). Simultaneous Compliance Guidance Manual for the Long Term 2 and Stage 2DBP

Rules, U.S. EPA Office of Water. EPA 815-R-07-017 3. U.S. EPA (2004). Comprehensive Surface Water Treatment Rules Quick Reference Guide: Systems

Using Conventional or Direct Filtration, U.S. EPA Office of Water. EPA 816-F-04-003 4. U.S. EPA (2005). Membrane Filtration Guidance Manual, U.S. EPA Office of Water. EPA 815-R-06-

009 5. P.36 “Desalination of Seawater: Manual of Water Supply Practices”. AWWA. Copyright 2011 6. U.S. EPA (2007). Determining Virus and Giardia Inactivation with Chlorine, U.S. EPA Region 8.

EPA Region 8 PowerPoint (https://www.epa.gov/dwreginfo/surface-water-treatment-rules), Accessed 6/27/16

7. CA Title 22, CCR, Division 4, Chapter 3, Article 1 in document “DPH-14-003E, GW Replenishment Using RW, May 30, 2014”

The assessment of process log reduction presents a particular challenge as a result of the aforementioned capped maximum log reduction at any barrier, and imposed multiple barrier approach. An additional challenge comes from the fact that in many cases the actual log reduction achieved by the process cannot be reliably measured at some process barriers by currently available monitoring techniques. A good example is reverse osmosis, which is currently limited to 2.0 log reduction credits (or less in some cases) as a result of the resolution of electrical conductivity or total organic carbon (TOC) measurement used to determine system integrity to sufficient resolution to better demonstration log reduction capacity. In the case of RO, significant research work (WRRF 09-06 (Frenkel and Cohen, 2014)) and (Reuse-12-07 (Jacangelo Ongoing Research)) among others are continuing to advance integrity monitoring techniques with a goal of improving the creditable log reduction at that step.

An ongoing review of technical developments for critical monitoring should be considered both at the outset of a new project, and for implementation during the life of a project if additional log reduction can be assessed at each CCP, or if new CCPs can in fact be assigned as the result of improved monitoring techniques.

51 Water Environment & Reuse Foundation

3.3.7 Critical Control Point Selection – Wastewater Treatment

It should be noted that the two examples noted in Sections 3.3.3 and 3.3.4 provide a CCP assessment for processes and treatment trains within only the advanced treatment component of the DPR system, and have not considered either wastewater collection, wastewater treatment or downstream drinking water treatment processes.

Because of the enormous variability in types of processes used in wastewater treatment and the variable quality of water produced from those facilities, an assessment of CCPs was not considered as a part of Reuse-13-03 project, but rather all risks were considered within the advanced treatment system alone.

This is a very conservative approach, and places a heavy emphasis on managing water quality risks within the advanced water treatment facility, particularly if the Californian IPR requirement of 12 virus, 10 Cryptosporidium and 10 Giardia are targeted. In a Texas context however, where the risk is assessed downstream of wastewater treatment, this may be more applicable.

Ideally, for a HACCP analysis of an existing or planned facility, the entire system from sewer collection to drinking water distribution should be considered during the risk analysis and when identifying the CCPs. This would include a review of particular source water contaminants (particularly those of industrial, medical or agricultural origin) known to be present in the sewer catchment, particularly where source control strategies are in place.

Figure 3.11. General Schematic of Urban Infrastructure with DPR

52 Water Environment & Reuse Foundation

For this project the water reuse facility was considered “in a box” but in reality all points from collection through each stage of treatment and on through distribution must be considered for adequate risk assessment and designation of critical control points

A particular challenge for CCP selection, however, concerns biological treatment processes. While biological treatment processes are known to reduce the levels of many contaminants of concern, as well as provide significant reduction of microorganisms, an effective critical monitoring control that can provide good correlation for removal has proven difficult to determine. For this reason, and due to the normal variability in plant operation, the development of critical control points within the wastewater treatment process may be difficult.

Additionally, the upstream wastewater treatment plant may be operated and managed by a different entity than the DPR water recycling plant and would require an integration of operational systems and practices in order to effectively operate a CCP approach. As noted in the previous review of regulatory requirements, this level of co-operation is desirable but may be practically difficult to achieve.

Research work conducted by the Australian Recycled Water Centre of Excellence (AWRCoE) National Validation Study SP3 – “Development of Validation Protocols for Activated Sludge Process in Water Recycling,” (2016 In Print) has conducted research into the assessment of suitable surrogate monitoring for activated sludge processes to determine effective surrogate monitoring. Initial research suggests that human adenovirus is a promising candidate for log reduction validation due to its high presence in raw wastewater and lower log reduction than polyomavirus and somatic coliphage. However, this validation relies on water quality sampling and analysis with a commensurate lag time in obtaining results.

In terms of real time analysis with online analyzers, a poor correlation was established between on line measurable parameters (DO, temp, pH, electrical conductivity and turbidity) which suggests that the development of an online critical monitor may be difficult to establish that can correlate to a specific microorganism log reduction.

In essence, in terms of selecting a CCP within wastewater biological processes, it is difficult to correlate any specific water quality analyzer or surrogate measurement that can validate that the processes are providing sufficient removal of microorganisms and chemicals of concern in real time. Therefore the monitoring and “control” aspect is missing from the process performance metric and thereby eliminates its candidacy as a true CCP.

However, an example where biological treatment systems have been used is at the Western Corridor Recycled Water System for which the advanced water treatment plants are fed with secondary effluent from fully nitrifying MLE (Modified Lutzack-Ettinger) wastewater treatment processes. In this case, the activated sludge process (specifically the nitrification process) was identified as a CCP, with process monitoring achieved by means of an on line ammonia analyser at the inlet to each plant. As full nitrification was anticipated to produce continuously low ammonia levels, this monitor could provide evidence in upsets in the nitrification process and control actions could be taken (in this case by not accepting feed water to the advanced plants under conditions of higher ammonia). Additionally, under wet weather flow conditions, when a portion of the wastewater did not receive full biological treatment due to hydraulic limitations and process bypass, the water was not sourced for further treatment at the advanced plant.

While neither the log reduction of microorganisms nor a removal of other chemicals of concern could be easily validated by this approach, it did at least work to minimize the risk of treating water outside the normal expected range for key wastewater monitoring parameters.

53 Water Environment & Reuse Foundation

With the wide array of different wastewater treatment process trains and resulting water quality, while correlation with specific removal of microorganisms and other constituents is difficult to correlate, the use of key monitoring (such as turbidity and nitrogen if targeted for removal) are useful indicators for any potential adverse or upset operational conditions in the wastewater treatment process. In the context of reuse, they can also provide helpful information to wastewater treatment operations in terms of real time monitoring, which may not be common on existing wastewater facilities.

3.3.8 Critical Control Points – Water Treatment Plant

As opposed to biological wastewater treatment, conventional water treatment systems are amenable to CCP selection. In fact, many of the same processes that are utilized in advanced treatment systems for reuse such as membrane filtration, UV disinfection, chlorination, coagulation and filtration are commonly used at water treatment plants. In fact, the CCP methodology is applicable for water treatment facilities and has been adopted at a number of worldwide locations unrelated to reuse, including the Gold Coast seawater desalination plant in Australia. This framework is described elsewhere as a water safety plan (Bartram, Corrales, Davison, et al. 2009).

The adoption of the CCP process at the water treatment facility can assist in providing additional validated microorganism log reduction and relieve pressure on providing all of this at the advanced treatment plant upstream. With a fully integrated system, this may provide significant cost savings if an entire process step is no longer required and/or operational flexibility if a lower level of removal is required at any one step. For example, the direct integrity test critical limits on a membrane filtration system could be relaxed, or a lower chlorine CT employed at the advanced treatment plant if sufficient log reduction can be demonstrated in the drinking water facility. Processes used in drinking water facilities that could be considered for CCP selection include: Coagulation, flocculation and filtration. UV disinfection. Ozone disinfection Ozone-biological activated filtration. Granular carbon filtration. Microfiltration/ultrafiltration. Chlorine disinfection.

3.3.9 Critical Control Points and Critical Operating Points

An important benefit of HACCP process is to highlight the protection of public health, enabling a clear focus for operations and reporting to stakeholders. In applying HACCP, it is easy for a HACCP team to become side-tracked away from the decision making required to evaluate CCPs from a public health standpoint. Plant designers and operators are concerned with a range of issues surrounding the performance of the plant including maintaining plant production, maintaining good condition of assets, optimizing energy consumption and providing safety to operational staff among others.

It is very important that at least one member of the HACCP team be tasked with maintaining focus for the group and ensuring that processes that are not directly related to protecting public health are not inadvertently identified as CCPs and likewise that CCPs are not missed if the team becomes overly focused on certain types of risks (contaminants) or perceived risks.

It is vitally important, however, that these other items that are not directly related to health are captured and managed to ensure that important elements are integrated into design and operational management. As noted in Section 3.3.1, this can be managed using the concept of critical operating points (COPs). It is recommended that during the development of CCPs, a list of COPs also be included for development.

54 Water Environment & Reuse Foundation

Specifically, COPs are points in the treatment process that are specifically designed to maintain the production capacity of the facility and protect working assets.

Some examples of COPs are included in Table 3.19. These examples are not exhaustive, and additional COPs may be included to manage risk of equipment and process. Table 3.19. Examples of Critical Operating Points

Reverse Osmosis Based Treatment Non-RO Based Treatment

Process Area Critical Operating Point

Process Area Critical Operating Point

Microfiltration Backwash water Flow Rate

Ozone Dry gas monitor, off-gas analyzer

Backwash air scour flow rates

BAC Filter differential pressure

Fouling indication (trans membrane pressure or permeability).

GAC Filter Differential pressure

Reverse Osmosis Dose of antiscalant BAC Filter differential pressure

pH correction dosing GAC Filter Differential pressure

Recovery

Fouling indication (trans membrane pressure, permeability, normalized parameters).

Cleaning Waste Neutralization pH

3.3.10 Establishing Critical Limits

For each of the critical control points selected, a critical limit must be established for each of the monitoring parameters that are indicated in the preceding sections. These limits are set to ensure that the critical control point is operating at a minimum standard or above, to maintain sufficient removal of the noted health risks at that step. In the context of DPR, this is very often a level of microorganism removal, but may also be other parameters such as a UV power input limit to ensure removal of NDMA or a limit on chemical dosing for chloramine or final water stabilization control as noted above in Sections 3.3.4 and 3.3.5.

Advanced treatment processes operate with highly automated systems, and so for the vast majority of critical limits there is a high reliance on online analyzers. These analyzers provide operational monitoring data to plant SCADA (Supervisory Control and Data Acquisition) systems, as a part of the overall automated control of the plant. Critical limits are managed as control system alarm set points.

It is important that there is warning where possible of a critical control point process operating outside of its required range, and hence it is recommended to use an “Alert” limit as a warning, when the performance has deteriorated close to the minimum required. This equates to a “Warning” alarm in the normal management of control systems. In this case, a warning alarm should be generated by the SCADA

55 Water Environment & Reuse Foundation

system alerting operational staff to review system performance and take required corrective actions. A “Critical” limit is that at which the system has fallen below the minimum requirement. In this case typically that critical process system will automatically shut down, or automatically switch to a safe state.

Often, the critical limits are applied to individual process units operating in parallel trains. For example, a microfiltration system will usually consist of multiple units operating in parallel. Each one of those units will conduct a pressure decay integrity test on a regular basis (daily or more regular) that monitors the membranes within that unit. If any of those units has a loss of membrane integrity below the minimum required, that unit alone will be automatically shut down. The remaining units may continue to operate, and hence this will result in a reduction in plant output. In some cases (for example final chlorination or ozone dosing) a failure will impact the entire treatment train and will result in a full plant shut down.

Greater discussion of critical limits, alarms and response procedures is discussed in Chapter 4.

In establishing the critical limits, i.e., defining what values of the monitoring parameter the alarm set points should be, there are a number of important considerations: Resolution of Monitoring. A key challenge for critical limit establishment is sufficient resolution of

the monitoring parameter to gauge actual performance. A good example of this is the reverse osmosis process, where electrical conductivity is unable to provide resolution better than 1.5 to 2.0 log reduction (of electrical conductivity). Studies have shown greater removal of micro-organisms based on challenge test studies, and so the use of this monitoring parameter for critical limits in effect limits the removal of microorganisms and other contaminants that can be validated across this process. For this particular example, extensive research is ongoing to improve monitoring resolution for this process.

Practical Operability. The critical limit must not only have sufficient monitoring resolution, but it must also be a limit to which the critical process can be reasonably operated. For example, the pressure decay integrity test for some microfiltration systems can provide a resolution up to 5.0 log of Giardia and Cryptosporidium particles. However, this is a very high level of integrity and for some membrane systems may require a very high level of maintenance (pinning fibers, replacing o-rings). While it is possible, it is operationally less practical and hence the critical limit is best matched to a 4.0 log reduction.

Regulatory Limitations. In some jurisdictions, there may be a limitation applied to the removal capability granted to a specific process critical control point. For example, in California a maximum 6.0 log of microorganism removal (for virus) is accepted for UV/H2O2. Consideration of these regulatory limits must also be taken into account.

Time for response. Consideration should also be given to the response time for the monitor to detect a change in the process, and consequently provide an alarm and automated control response. In most cases, the response time for a specific analyzer is in seconds. However, for some processes this can take longer. The microfiltration pressure decay test requires that an automated sequence of operation takes place on the membrane unit in order to take the measure for the critical monitor. This test requires between five to10 minutes of operational down time and as a result is performed often no more than once per day on a specific MF unit. This means that it is possible that a unit could operate outside the required range of performance for up to 24 hours (i.e., the time interval between tests) prior to an alarm and automated response condition. Reuse-12-06 discusses this scenario with respect to engineered storage (Salveson, Steinle-Darling, Trussell, et al., 2015).

System validation. System validation is the process of developing supporting evidence for the selection of each critical limit for critical control point processes. This is discussed in Section 3.4 below.

56 Water Environment & Reuse Foundation

3.4 System Validation and Verification

3.4.1 Validation

Validation is the process of proving the effectiveness of control measures in reducing the risk posed by hazards or hazardous events. It is the process which demonstrates that the critical control point selection, control, automation and operational response procedures have been correctly implemented. A useful approach, developed by the State of Queensland’s Department of Energy and Water Supply (2008) notes three key stages of the validation process: Pre-commissioning Validation. Commissioning Validation. Commissioning Verification.

3.4.2 Pre-Commissioning Validation

Pre-commissioning validation is a study that is conducted to support the critical control point selection process, and in particular assist in the determination of the extent of removal possible at each critical control point. In other words, how much removal can be achieved at each critical control point? This is both for microorganism removal (log reduction) and also removal of other specific constituents of concern such as NDMA, 1,4-dioxane and other risks identified in the water quality risk assessment. The critical control point selection process should be a part of the validation framework. Validation involves the use of expert technical knowledge, literature reviews, full scale plant operational data review and, if required (or desired), pilot testing. This work can assist in providing suitable evidence that the critical limits established for each critical control point are suitable to provide not only sufficient removal, but also sufficient safety margin for operations. That is, it helps to define both the alert and critical limit selection. This study is usually conducted during the design phase of the project and includes the source water hazard assessment, critical control point selection and establishment of critical limit steps. Fortunately, significant full-scale data that can be used for validation (from a literature review perspective) can be found with detailed analysis in Reuse-13-03 (Walker, Stanford, Khan, et al., 2016).

3.4.3 Commissioning Validation

Commissioning validation is the confirmation that the critical control point processes adequately operate to remove microbial and chemical hazards on the constructed facility. It can be considered as the equivalent of a plant performance test. It must test the selection of processes, critical limits and also the operational systems and processes that support it. A commissioning validation must: Confirm that the treatment processes (CCPs) in combination will adequately remove the identified

microbiological and chemical hazards when operated within critical limits. Demonstrate that the critical control point processes consistently operate within the critical limits. Demonstrate that corrective actions and response procedures are implemented, and if breaches occur

that these actions control the hazards. To achieve this, a commissioning validation plan will require: Monitoring of CCPs using critical monitoring parameters to prove that performance within critical

limits can be achieved.

57 Water Environment & Reuse Foundation

Where possible, the monitoring of indicator and surrogate parameters around each CCP and the treatment system as a whole to demonstrate the efficacy of hazard removal within critical limits.

Demonstrate that the required water quality can be achieved while the CCPs are operated within their critical limits.

Commissioning validation can be conducted in conjunction with an overall plant performance test. This test is a useful exercise in ensuring that operational monitoring and data reporting systems are sufficiently developed for the long term operation of the treatment system. In addition, it provides an effective road-test of control and operational response procedures.

3.4.4 Commissioning Verification

Commissioning verification is the end of pipe treated water quality testing, which provides a verification that the CCP process and critical limits are consistently achieving the desired water quality. It is a relatively short term period of extensive water quality testing to verify that the identified source water hazards have been consistently and extensively removed.

The commissioning verification is in effect a short term, intensive water quality sampling regime. The Queensland Government in their Recycled Water Management Plan and Validation Guidelines (2008) require 13 weeks of twice weekly sampling for this step. This regime can then be modified and reduced in both number of samples, and frequency of sampling, for the ongoing water quality sampling and testing. It should be noted that this timeframe was based on the time available to the Western Corridor Recycled Water Project at the frequency that sampling was already in place due to the emergency nature of this project. Projects may require a larger number of sample events to establish a basis for the reduction of sampling. This is further discussed in Section 3.8.

3.4.5 Revalidation

If there are significant changes to the DPR treatment system, either from a source water, wastewater treatment plant operational or infrastructure changes, advanced treatment or water treatment changes, or identification of new hazards, then revalidation should be conducted for the system.

Significant changes may include: Introduction of new processes or equipment Changes to source water quality Increases in water quality hazard concentration or identification of a new or emerging hazard Repeated systematic failures Addition of a new influent source – such as a major new sewershed connection to the system

A restart after any extended period of plant shutdown (of more than six months duration)

3.5 Operational Control and Monitoring

3.5.1 DPR Control Systems

Advanced water treatment facilities, including those used for recycled water, are designed and operated as highly automated systems. Effective system control, and in particular rapid and appropriate operational response is as much a function of correct controls programming, supervisory control and data acquisition (SCADA) and human machine interface (HMI) configuration. The success of an effective and reliable operating response relies on the interface of the automation to the operator. The operations team must be diligent in ensuring that the controls and automation works correctly, and they must interface effectively to: Proactively review performance to anticipate problems before they occur. Respond effectively to alerts, alarms, and shut down conditions.

58 Water Environment & Reuse Foundation

Provide a thorough investigation of why the problem occurred, and transfer lessons learned to improve future operations.

Return systems safely and effectively to service in a timely manner. Ensure transparency to provide trust with stakeholders.

Treatment systems and their process components are configured with Programmable Logic Controllers (PLCs) connected to field devices (valve, pumps, analyzers and switches) necessary for operation. The PLC executes a series of programmed instructions in a continuous loop manner which are repeated (scanned) multiple times per second. Data is stored in PLC registers which may contain simple binary (discrete) status (e.g., 1/0, on/off, run/stop, open/closed) or more complex analog process values (e.g., flow, pressure, level, turbidity or conductivity). The PLC is able to communicate with a SCADA which has the ability to access the PLC registers in a bi-directional (read/write) manner. A HMI is used to display the status and analog values in a visual format (typically a table or schematic) and allows the operator to enter binary or analog values into the PLC for subsequent execution. The HMI is usually configured to limit the value of an operator input to within a specified range, to prevent an inadvertent or unanticipated data entry.

Figure 3.12. Control System Elements

59 Water Environment & Reuse Foundation

Operational data from the PLC registers is retained in a database by time and/or event and subsequently used to create operating trends or reports using SCADA itself or other supplementary software programs. Operator interaction at the HMI is monitored as well and may be recorded as event or change in operation. Programs that log events, alarms, and response (known as “historians”) provide a record of activities that can be used to troubleshoot events or other changes and are also used to provide a visual display of numeric values in a graphical format over a period of time.

Most control systems are accessible through an internet connection behind a firewall, thus the HMI may be not be located at the facility. In some cases, security concerns may prevent an internet connection. Likewise, most systems will have multiple HMIs with partial or complete view of the system. From the operations perspective, the link between process components, PLC, SCADA, and HMI must be maintained at all times to call relevant information to the attention of staff so that they can respond in a timely manner to events that range from minor changes for process optimization and production to catastrophic events which require system shutdown and repair. Likewise, management staff can access stored history to review operator response to various events that occur throughout the day.

When selecting control systems for water treatment and specifically for DPR scenarios, the engineer should consider the amount of data being collected and analyzed, the frequency at which the data are produced and sent to the operations team, and the amount of time required to identify and respond to an alarm and true system failure (i.e., “failure response time,” as described Salveson, Steinle-Darling, et al., 2015). The following sections of this chapter provide context on water quality alarms, strategies for identifying false alarms, response procedures for “alerts” and “critical alarms,” and then guidance on how these principles can be applied in designing and operating DPR facilities. The Reuse-13-03 report also provides additional detailed descriptions of water quality alarms, alerts, and response procedures which, much of which has been included in Chapters 3 and 4 of this report for ease of reference while providing expanded content and explanation in the context of operator training and certification frameworks.

3.5.2 Performance Trending and Data Management

One of the benefits of HMI systems is the ability to gather extensive amounts of data from an operating plant. However, if not well organized and managed, the vast amount of data that is collected by the HMI can be overwhelming, with a significant challenge for staff to regularly collect, sort and organize into meaningful operating information. It is important that data reporting and management is set up effectively to ensure that data can be turned into operational knowledge in time to make informed, beneficial operational decisions.

HMI systems can provide trending of any analog data from the PLC. This includes results from on line water quality analyzers, level transmitters, pressure transmitters and flow transmitters; as well as a number of calculated variables that may rely on a number of individual analyzer readings (for example chlorine or ozone CT, or log reduction if calculated via the MF pressure decay test). While it is possible to set up any trended parameter, it is highly recommended to develop a set of standard trends for important key operating parameters. A list of some of these important parameters has been listed in Chapter 4.

In developing standard trends, important management of data must be taken into consideration to ensure that the data is representative and can be meaningfully interpreted. This includes: Sampling of trend data at an appropriate frequency. For many parameters, it is important to have

a high data sampling frequency (for example on line conductivity from an RO can be in the range of seconds) in order to determine performance over shorter periods of time. However for other data points such as normalized reverse osmosis data, this can be less frequent – with one or two data points per day sufficient to provide meaningful operating performance.

60 Water Environment & Reuse Foundation

Reviewing data over a meaningful span of time. In the case of some data, it may be helpful to review data over a period of a single day. This may include trends of advanced water treatment feed water quality parameters to determine diurnal quality variation for example. For other data however, again using the example of reverse osmosis normalized performance data, a period of many months may be appropriate to determine a meaningful trend in performance.

Data filtering is important to ensure that there is no erroneous data reported that may be misinterpreted as actual performance. Examples include not trending data from analyzers when process units are off line, note data as in error if an analog signal is lost or analyzer is faulted and noting if any analyzer is in calibration mode. It is often useful to set a number of rules on trending data such that if the result is significantly outside of an expected range then this data is discounted from the trend. This is not to be confused with possible outliers from failed process or instrument, but rather for situations that cannot occur in all practicality.

Comparison with range of expected performance, and reference to alert and critical levels. Ensuring that the trend screen has reference to an expected range of performance, and also reference to alarm set points is important to put context to the operating data. Often operators are engaged in the numerous processes of advanced treatment systems for the first time when coming to the plant, and so understanding what “normal” operation looks like is helpful with guidance on the trends themselves. This can be further supported with the development of dashboard type reports (noted in Section 3.5.8).

Important parameters for trending have been identified for specific process elements in Chapter 4 of this report. A key recommendation is that each of the CCP monitoring parameters has a trend screen that is developed and is readily accessible and highlighted within the HMI menu of trend screens. These can be used for routine assessment of performance and reporting.

3.5.3 Alarms

Alarms are used to inform operations staff about recorded events within the system that may indicate an “alert” or “critical” notification about a given process or process monitor. Because of the highly configurable and programmable nature of control systems, the structure and management of alarms can be widely variable from facility to facility and are based upon the regulatory requirements, preferences of the utility, configuration and limitations of the PLC or programming software, and understanding of the control system programmer. Specific terminology and practices may vary, however, several common themes and practices exist.

In general, alarms are categorized in terms of the severity and structure at the SCADA level. The assignment of severity to an alarm condition can be somewhat subjective in nature, with categorization established using project guidelines or prior experience of the programmer. A solid understanding of process function (e.g., determining whether it is a critical control point or not), acceptable ranges of performance, and outcomes when the process or process monitor fails will help guide the development of alarm set points and severity levels.

Events are categorized as routine start and stop of a process based upon a field parameter. (Hi or Lo). Events do not require acknowledgement or action by the operator. These are really not alarms, but will result in the issuance of a notification.

Alerts (Warning) are categorized as non-routine events that require acknowledgement by the operator. This alarm condition may require a field investigation and corrective action by the operator prior to placing equipment back into service. The typical convention for an alert alarm is Hi or Lo status indication.

61 Water Environment & Reuse Foundation

Critical Alarms (Failure) are triggered by conditions that result in the disabling of a system or a loss of its functionality until the condition in the field is resolved. This is categorized as an abnormal condition, beyond the typical or anticipated operating parameter of the system and is normally associated with a failure of the process. The typical convention for an alarm of this type is Hi-Hi or Lo-Lo condition.

In a well-designed system alarms are minimized to limit any “nuisance” or “transitional” alarms that may be encountered. There are two common practices associated with the management of nuisance and/or transitional alarms: delay and disable. Delay of an alarm generally involves the establishment of an HMI adjustable time delay in the PLC. Disabling alarms may be possible by PLC programming or adjustable through the HMI, thus restriction on access may be necessary.

Alarm set point values may also be adjusted through the HMI, and may require similar restrictions on the access, especially for critical alarms. With some control systems, it may be possible for the operator to overwrite or simulate a process value into a PLC register from the HMI on a temporary basis in order to disable an alarm.

Examples of questions that can be asked to determine if the programming of the PLC or HMI is well designed are as follows: Can a process or sub-process be started and/or stopped without nuisance alarms or operator

intervention to disable or modify set-points in the control system? Are process control sequences able to be completed without failure? What are the number of critical alarms that occur in a day? Is there a process in place to identify and manage false alarms?

A key consideration in the proper commissioning of any treatment system is whether the number of alarms is manageable for normal operation. Alarm flooding is an all too common, and significant issue for operators, a condition with more alarms than can be reasonably managed by an operations team, where there is a high risk of missing very important alarms.

Alarms may also be process or water quality related. The remaining discussion will focus upon water quality based alarms that are focused on CCPs in potable reuse and that would result in a non-compliance regulatory permit violation.

3.5.3.1 Alarm Types and Triggers Within the PLC, the process value from the field device can be scanned multiple times per second. The PLC may be programmed with specialized instructions to assure that a true alarm condition exists before subsequent action is taken. An example of an alarm configuration strategy is called “Time Delay On” (Figure 3-13). The process value (indicated by the red line) has to be continuously above the threshold value (high set point) in order to initiate the timer. If the monitor/field device continues to produce a signal above the high set point beyond a pre-determined period of time, the alarm notification is issued (true). In the event that the value drops below the threshold value, for any period of time, the timer is reset (false) and will not time until the value is above its threshold value again. Additional programming may be used to ensure the alarm condition has cleared before the timer is reset.

62 Water Environment & Reuse Foundation

Figure 3-13. Graphical Depiction of a Time Delay Alarm Trigger (True) and a False Event Resulting in No Alarm

A second alarm configuration is called the “Moving Average” (Figure 3-14). In this case the sample is averaged on a periodic basis (1/sec, 1/min) and the values reported are in reality an average of the past number of events over a pre-defined interval. At the next interval, the oldest value is discarded, other values indexed and the most recent value added to calculate the moving average.

Figure 3-14. Graphical Depiction of a Moving Average Alarm (True) and a False Event Resulting in No Alarm

A third alarm configuration is the “Block Average” (Figure 3-15). In this case, an input is averaged over a period of time (e.g., 30 seconds) and then averaged. All samples are discarded and the process repeated. In the example below the alarm notification (Block Average > High Set Point) would occur for block 3 of the true example.

63 Water Environment & Reuse Foundation

Figure 3-15. Graphical Depiction of a Block Average Alarm Trigger (True) and a False Event Resulting in No Alarm

The fourth alarm configuration is called “Point to Point” (Figure 3-16). In this case, a single value from the PLC is used as the basis for alarm determination. Process values between the points of measurement are disregarded. An alarm is triggered when P1 and P2 is greater than the high set point for two consecutive measurements. Analyzer measurements between P1 and P2 are ignored.

Figure 3-16. Graphical Depiction of a Point to Point Alarm Trigger (True) and a False Event Resulting in No Alarm

The development of alarm configurations for a water treatment has historically used the scenario of an operator recording data (by hand) and subsequently calculating compliance as a strategy for rule making. As a result, there are numerous potential water quality scenarios that are likely to occur but which may not be captured from a compliance monitoring perspective. Such scenarios would result in excessively long failure response time intervals and would be nearly impossible to manage from a production and CCP perspective.

Therefore, from a practical perspective, continuous online monitoring for process parameters for regulatory compliance with data averaging via PLC and SCADA offers the potential to improve the overall process reliability as increased monitoring is possible and vastly reduces the failure response time. Information about sampling frequency should be considered when developing alerts and alarms for

64 Water Environment & Reuse Foundation

continuous processes as intermittent sampling (e.g., 15 minute point to point sampling or peak hour flow measurement) can be more challenging to monitor and respond to in a timely manner than an alarm configured in the manner typically associated with an online monitor and PLC system.

3.5.3.2 Alarm Management Considerations Alarms are often not managed effectively in any of the process industries, including the water/wastewater industry. Often, little thought has generally been given to what constitutes an alarm, how they should be displayed, and how they should be managed. All modern HMI applications have the ability to select alarm levels within their analog point configuration database. In general, the “event”, “alert”, and “alarm” levels for these analog signals are selected during the initial configuration of the control system and are rarely modified thereafter. The result is that the SCADA system will generate many alarms, only some of which are really important enough for operators to act upon. Excessive alarms can quickly overwhelm operators or hide critical process alarms within a long list of lower priority alarms, thereby extending response time. Excessive alarms can also lead to a condition of alarm flooding, and the natural human tendency in such cases is to simply ignore them. Following some incidents in the process industries where accidents occurred when operators missed critical alarms, a more concerted effort was made to develop standards for alarm management. The relevant standard for alarm management is “ANSI/ISA-18.2-2009 Management of Alarm Systems for the Process Industries”. One of the key concepts that came out of this standard development effort was that, if a signal does not require an operator action, it is not an alarm. This concept helps to eliminate the bulk of the alarm “noise” at a facility so that those alarms that are integral to CCPs and COPs will be immediately apparent and quickly acted upon. Suppression of other alarms such as alarms associated with equipment that is intentionally out of service (or needing scheduled routine maintenance) further reduces the number of alarms generated by a SCADA system. The key to successful alarm management is the application of such principles in a systematic manner at each facility. The first step is to create an alarm philosophy to define what actually constitutes an alarm. For existing facilities, the amount and type of alarms must be analyzed, and those alarms that are truly critical enough to require operator action retained while others are placed in the “event” category, meaning they are simply information. Another key concept of the standard is that the alarm management process continues throughout the life of the system. Alarms are constantly monitored and those deemed no longer relevant are deleted from the system. It is recommended that DPR utilities implement an alarm management strategy based on the applicable portions of the ANSI/ISA-18.2 standard for all facilities (considering that it was written for a variety of process industries, not just the water industry). Additional information on this subject will be available in the future as part of WRRF-14-01 Integrated Management of Sensor Data for Real Time Decision Making in DPR Systems.

3.5.4 Response Procedures and Implementation of Alarm Strategies

An important operational consideration is the operational response procedure for when a critical control point may fail. Critical control point response procedures provide a clearly articulated set of responses for plant operators to take should a critical monitoring point determine that a barrier is no longer fully intact. As a starting point for operational safety, each critical monitoring point has an “alert” limit that acts in the manner of a warning alarm, providing an indication that the monitoring limit is approaching the critical limit yet providing time for both automated control responses and operational responses to occur in a proactive fashion – to resolve the issue before a critical limit is breached. For more severe issues that may occur with or without a preceding alert notification, “critical alarm” limits indicate failure of a process or failure of a monitor for a CCP that requires immediate attention to ensure water quality goals are met.

65 Water Environment & Reuse Foundation

In most cases for recycled water plants, processes and plants are highly automated. As a result, the operating procedures focus on ensuring that automated processes have operated correctly. There is a heavy reliance on instrumentation and as a result the procedures tend to focus heavily on checking instrumentation and verifying that the monitoring limit is real and that analyzers and instruments are operating correctly.

Response procedures must also clearly articulate which stakeholders are to be notified, how to correct the issue and how to return the equipment safely to service once the issue has been resolved. Importantly, the procedures require that an investigation be conducted to ensure the cause of the breach is understood, so that actions can be taken to prevent it in the future.

As a general starting place when developing site-specific and process-specific response procedures, the following sequence of steps are suggested for the configuration of Alerts (Warning) and Critical Alarms (Failure) and responses. The information provided here is based upon work that has been previously conducted as a part of the Western Corridor Recycled Water Project Recycled Water Management Plan, developed by Veolia Water Australia in conjunction with Seqwater.

Alert (Warning) Perform a water quality test, if alert condition is obtained. Validate the instrumentation is properly functioning. Repeat the test, if the alert condition is confirmed. Diagnose and repair the condition. Record the event and remedial action. Repeat the test with acceptable results.

Critical Alarm (Failure) Process unit automatically shuts down and is taken out of service. Perform a water quality test, if alarm condition is obtained. Validate the instrumentation is properly functioning. Repeat the test, if the alarm condition is confirmed. Notify Supervision that a Critical Alarm Condition exists. Diagnose and repair the condition. Repeat the test; when acceptable results are obtained the unit may be returned to service. Record the Event and remedial action. Notify regulatory authorities if water quality was compromised.

A generic flow diagram of response procedures for alerts and alarms is provided on the following pages (Figures 3-17 and 3-18).

66 Water Environment & Reuse Foundation

Figure 3-17. Generic Alert Level Response Procedure

67 Water Environment & Reuse Foundation

Figure 3-18. Generic Critical Alarm Response Procedure

68 Water Environment & Reuse Foundation

3.5.5 Specific Alert and Alarm Procedures for Identified CCPs

In Chapter 4, alert and critical alarm strategies are presented to illustrate how alarms are addressed for the critical control points identified in the two process trains. While examples here are provided for the reader to see how response procedures can be developed for specific unit processes, it is important to realize that each facility will need to develop and/or modify its own procedures that are specific to their equipment, monitors, and water quality goals. However, there are a few key considerations that can be carried between systems and they are included here based on professional judgment and experience with operating full-scale potable reuse facilities: Control System Events should not be identified or included in the alarm historian, though they should

be documented in the overall system historian. Alerts (Warning) should be used to address the normal issues that are associated with maintenance or

repair of equipment. Alerts (Warning) should not shut down or disable the operation of the equipment. Critical Alarms (Failure) should be used to trigger immediate and automatic shut down and disabling

of equipment until corrective action is successful. Critical Alarms should include parameters for method of measurement (time delay, moving average)

including sampling frequency (min, sec) and time basis (min, hr) if deemed appropriate by the regulatory authority.

Critical Alarms (Failure) should be structured to reflect an unusual/catastrophic occurrence, such as an (equipment failure) resulting in equipment operation. Ideally the occurrence can be captured as an alert (warning) before the critical (condition is obtained)

Critical Alarms (Failure) may or may not result in the loss of water quality. Notification of regulatory authorities shall only occur when the water quality from the system is compromised.

Critical Alarms (Failure) should have restricted (supervisor) access at the HMI level or be programmed at the PLC level to limit inadvertent changes to the alarm setpoints.

Alarms should be redundant at the unit and system (common) level if possible. Consideration should be given to redundant instruments where those instruments are monitoring a

CCP. This can be achieved with a single duty instrument responsible for critical alarm trigger, but with reference to a redundant unit where a difference between the two readings is monitored to account for instrument failure or loss of calibration.

69 Water Environment & Reuse Foundation

3.5.6 Control System Testing With such a heavy reliance on controls and automation, it is important that thorough testing of control systems takes place at the end of the construction phase of a project, as a part of commissioning and performance testing, and then on a regular basis to ensure that control loops, control responses, analyzers and instruments are working effectively. Many treatment systems with complex automation systems may often encounter programming “bugs” and other problems which could undermine confidence in the overall treatment system, despite the robust treatment technologies that have been employed.

There are a number of points within the life of a project where the controls and automation system can be thoroughly tested to minimize the risk of a control system failure: Control Philosophy/Functional Specification review – The control philosophy and functional

specification provides a written narrative of how the overall control system must operate. This document is produced during the design phase and includes a description of instrument control loops, alarm setpoints, control setpoints, automated control sequences and safe shutdown conditions. This document should clearly include CCP and COP requirements and importantly must include any automation aspects of CCP response procedures. This is the first point at which it is ensured that CCP responses are automated.

PLC/HMI Functional Acceptance testing – This is a simulation based testing process that occurs at the bench top level, prior to final plant completion. This test ensures that the PLC program and HMI operate as designed. It should include as a key component a check on all CCP and COP automated response procedures – to ensure that these have been programmed and work as intended prior to implementation on the full scale plant. CCP alarms can be simulated during this test, and functionality of the PLC programming can be checked to ensure that the control response is correctly programmed.

PLC/HMI Site Acceptance testing – This is the next step of control testing that occurs on the full scale plant. This is a key part of the pre-commissioning process prior to start up for the plant in which the PLC program and HMI is operated at site. Some simulation still occurs, as water treatment is not occurring during this test. The site acceptance test provides a check for actual analyzers and other instruments to ensure that physical control loops are in place. Alarms can be simulated during this test as for the functional acceptance test, with the added advantage of checking some full scale equipment.

Performance Testing/Validation testing – This is final step in the startup of the plant in which the treatment plant is operated under normal conditions. This performance test will ensure that the process is operating as designed, and will have a range of parameters to test including overall production, production rates at each process unit, and testing of operating sequences. It is recommended that CCP alarm conditions be simulated during this test to provide a full scale test of the control response procedure. The performance test may run concurrently with commissioning validation and verification as noted in previous sections.

Re-check of plant and equipment – Over the life of the plant, process equipment, analyzers and other equipment may need to be replaced or upgraded. Whenever this occurs, a change management system should be in place to ensure that all changes are adequately checked. A functional acceptance test that checks on particular CCP automated responses should be carefully checked in these cases. For example, if an online chlorine analyzer is replaced with an upgraded unit, or if a microfiltration membrane system is upgraded, the CCP response and other control elements should be carefully checked prior to service to ensure that CCP automated controls in particular, but also other important operations occur correctly.

3.5.7 Analyzer Management

In most cases for CCPs, as well as many other aspects of process operation, are heavily reliant on monitoring from analyzers. In fact, the more significant risk to the reliability of a CCP is less the CCP process barrier itself, but more the reliability of the analyzer that is detecting a failure.

70 Water Environment & Reuse Foundation

CCP monitoring analyzers specifically should be noted as critical items in terms of asset management. As noted in Section 3.7 (Asset Management) below, a specific level of criticality is recommended (CCP critical) which identifies those particular assets upon which the CCP system relies. As a part of project Reuse-13-03, a review of the reliability assessment of typical on line analyzers for CCPs for both treatment trains under consideration was conducted. Details of analyzer reliability have not been included here for the sake of brevity. However, a few key summary considerations in the management of analyzers are listed below. Track record of performance – As for selection of any process equipment, a proven and reliable

track record of performance is important. Performance of the analyzer in a similar environment, with the same or similar water type being analyzed. This can either be gathered from application in other existing utilities or if necessary be incorporated in a pilot study. For existing plant where an analyzer is looking to be upgraded or newly installed, a temporary installation for a period of three to six months can provide better knowledge of long term performance.

Local support – Some analyzers (for example low level on line TOC) have a high level of complexity and can be challenging to calibrate and maintain. In these cases a review of local support options from analyzer vendors, and/or a review of trained instrument technicians that have experience with that particular instrument/analyzer is important. Consideration of plant location and access and capability of local support is an important consideration of analyzer selection. Even some large utilities may outsource instrument maintenance for complex instruments. In the case of smaller utilities, outsourcing of a larger number of analyzers may be considered.

Spares management – Many analyzers contain parts that may require frequent replacement (for example pH probes) or may have a long lead time for replacement. It is important to ensure that there are sufficient analyzer spares available such that important equipment can be returned to service quickly following an instrument or analyzer failure. Some analyzers require the use of chemical reagents, and these should be stored appropriately in sufficient quantities to ensure no down time in the case of exhaustion of supplies. At the same time, care must be taken to ensure that inventories stored at site remain in date.

Regular and checked record of verification and calibration – Verification of the analyzer refers to a cross check of the analyzer against either a known standard or a known calibrated instrument. It is a cross check that can be performed more frequently than a full instrument calibration. A routine program of verification and calibration is essential to minimize the risk of incorrect analyzer reading. Analyzer drift, offset or other poor calibration can result in a failure to detect a CCP operating outside its acceptable range of performance. A well-documented schedule that outlines time of verification and calibration and results is essential.

Calibration events carefully managed on SCADA – When analyzers are calibrated, the result must be masked from the PLC system so that the operating plant does not respond to an errant analyzer reading (for example, when a pH meter undergoes calibration the probe is immersed in standard buffer solutions of pH 4 and/or 10 and pH 7. This is sometimes handled by placing the output from that analyzer on the HMI in a “calibration” or “force” mode in which the control system no longer reads the actual reading, but defaults to a manual input or the last reading prior to calibration. Care must be taken following calibration to ensure that the system is returned to normal operation immediately after calibration and the input does not remain in forced mode for long periods. This can be easily achieved with a time-out function that will return the input to the actual analyzer after a maximum time allowed for calibration to occur. Careful review of other parameters during the calibration period should also be assessed.

Analyzer redundancy – Consideration can be given to analyzer redundancy for some CCPs, particularly where there is a single process train with a single monitoring point (for example multiple chlorine analyzers as part of the chlorine CT calculation). In these cases, important consideration of control systems should include the use of a primary/secondary reference for monitoring and control

71 Water Environment & Reuse Foundation

action, the use of result difference as an indicator for instrument calibration and control methodology to alternate to secondary instrumentation in the case of primary failure. Other CCPs, such as membrane filtration, may have multiple process trains in parallel where multiple analyzers are used in parallel. An effective instrument validation, calibration and maintenance schedule along with a well-managed spares inventory may eliminate the use of double validation in some cases. The decision for design in this application should be based on a risk assessment for each unit process (CCP), the reliability of the analyzer in question and asset management strategies in place.

3.5.8 Dashboard Reporting

Routine reporting of system performance across all elements of the process, including wastewater treatment processes, advanced treatment processes and water treatment processes are critical to ensure operational success. A useful operational reporting tool is the dashboard report, in which a number of important operational parameters can be reported at a glance. It is useful to include in these reports key performance indicators – that is the key parameters that indicate process, equipment or system performance.

The following figures show examples of dashboard reports developed for both the RO based process and non RO based processes that have been considered for this report. Site- and process-specific dashboards should be considered for advanced treatment processes because they provide a clear, logical way to review process performance and prepare operating reports.

Figure 3.19. RO-Based Process – CCP Dashboard

72 Water Environment & Reuse Foundation

Figure 3.20. Example Overall Dashboard Report – RO-Based Treatment Process

73 Water Environment & Reuse Foundation

Figure 3.21. Non-RO-Based Process – CCP Dashboard

74 Water Environment & Reuse Foundation

Figure 3.22. Overall Dashboard Report Non-RO-Based Process

75 Water Environment & Reuse Foundation

3.6 Operating Interfaces

3.6.1 Operating Across Different Jurisdictions

From source to tap, DPR treatment systems are likely to include multiple operational entities. Municipal wastewater is often managed by a different agency to an advanced water treatment facility, with an operational focus traditionally on the removal of pollution. Environmental discharge permits are often focused on achieving water quality results for a limited number of water quality parameters (carbonaceous material, nutrients and suspended solids) on a percentile basis that allows for fluctuations in water quality and flow on a diurnal basis, and during high flow/wet weather type conditions.

Advanced water treatment facilities by contrast require knowledge and monitoring of a multitude of additional water quality parameters for both protection of public health, but also to ensure reliable operation of process systems (for example, to minimize scaling in reverse osmosis). In addition to the requirement for knowledge of more water quality parameters, performance at the advanced treatment facility also requires more real-time analysis and cannot rely on percentile performance (e.g., monthly averages).

Water treatment plant or distribution systems also must be aware of conditions at the upstream advanced treatment plant. Impacts from an advanced treatment plant turning on and off can impact water supply and quality. For example, the presence or absence of advanced treated water fed to a water treatment plant may impact chemical coagulation conditions, changes of water temperature may impact flocculation processes and changing water stability may impact downstream pipeline assets and pipeline biofilm populations.

Effective operating protocols across these different elements of the overall DPR systems are therefore critical to assure operational success.

3.6.2 Source Water Management

Sewer-shed source control programs provide the first line of defense in the control of source water hazards, specifically working to minimize pollutants from commercial, retail and industrial customers as well as engagement with the general public.

At a minimum, utilities should inventory the type of industries and associated range of contaminants that may be discharged and use this information in the risk assessment and development of a risk register described in Section 3.2. More information regarding source control is available in Chapter 5 of “Framework for Direct Potable Reuse” project 14-20. Importantly, information on newly identified source control risks, or source events, must be effectively communicated to all entities operating facilities in the DPR treatment system.

For example, the California Code of Regulations (CCR) specifies source control requirements for groundwater recharge projects. Specifically, CCR requires that:

“… the recycled municipal wastewater used for a GRRP shall be from a wastewater management agency that:

a) Administers an industrial pretreatment and pollutant source control program; and

b) Implements and maintains a source control program that includes, at a minimum;

76 Water Environment & Reuse Foundation

1) An assessment of the fate of Department-specified and Regional Board-specified chemicals and contaminants through the wastewater and recycled municipal wastewater treatment systems,

2) Chemical and contaminant source investigations and monitoring that focuses on Department-specified and Regional Board-specified chemicals and contaminants,

3) An outreach program to industrial, commercial, and residential communities within the portions of the sewage collection agency's service area that flows into the water reclamation plant subsequently supplying the GRRP, for the purpose of managing and minimizing the discharge of chemicals and contaminants at the source, and

4) A current inventory of chemicals and contaminants identified pursuant to this section, including new chemicals and contaminants resulting from new sources or changes to existing sources, that may be discharged into the wastewater collection system.

3.6.3 Operational Interface Protocols

An important operational element in the management across interfaces is the development of an operating protocol between the wastewater treatment facility and downstream advanced treatment facility as well as from the advanced facility to the downstream water treatment plant (or distribution system). Even in cases where these facilities are managed at the same site or by the same operational entity, there are sufficient risks at this interface to warrant the articulation of a clear protocol. The purpose of the protocol is to set out the allocation of responsibility between the parties for the supply of recycled water and ensure that risks are effectively managed. Key elements of the protocol should include:

Clear definition of interface points. That is, a clear articulation of the physical location at which the water transitions from the responsibility of one party to another. This should be described in terms of process and instrumentation diagram (P&ID) as an identified termination point, including a piping layout drawing with GIS (geographic information system) coordinates. For example, this may be at a fence line, or a specific flange in a pipeline. A schematic diagram included within the protocol is helpful.

A communication protocol. Communication between parties must be undertaken regularly and in a clear and efficient manner. A communication protocol may include regular meetings of operations staff and the discussion of water quality, planned changes to infrastructure, planned maintenance activities and a review of that period’s performance. Valuable communication may also include the sharing of water quality data, including any on line data that may be available at the operational interface. For example, the Western Corridor Recycled Water Project in Australia reported online ammonia, nitrate, TOC, pH and turbidity at the inlet to each of their three advanced water treatment facilities directly on line via a SCADA link to the wastewater treatment plants. This provided valuable real-time performance data that had hitherto been unavailable.

Regular reporting of water quality and production. This is the “business as usual” content of communication, and provides an ongoing understanding between entities of the key aspects of operation.

Management of maintenance schedule. As noted above, maintenance activities at either side of the operational interface can impact operations. The sharing and regular updating of planned maintenance activities – over a period of six to 12 months can provide certainty for planning and minimize unforeseen impacts. Benefits of aligning maintenance (for example coordinating downtime) can improve overall system production.

77 Water Environment & Reuse Foundation

Co-ordination of Incident and Emergency Management. Effective coordination between operational entities in the event of an emergency event is critical. The co-ordination plan should include responses to supply interruption, circumstances under which water cannot be accepted (either not acceptable to the advanced treatment system or not acceptable to drinking water/distribution, management of water quality incidents and critical control point failures.

While operational interfaces between entities is important, regular meetings and communication between all elements of the DPR system (from source control to tap) is important to ensure the overall success of the system. As noted in Chapter 2, it has been recommended to consider all entities involved in the treatment of water through to distribution be covered under a single permit for DPR production. Whether this is legislatively feasible or not, organizationally the entities must operate together in lock step co-operation to ensure success of DPR.

3.7 Asset Management

3.7.1 Development of an Asset Management Framework for DPR

A key to success with any infrastructure relies on good stewardship of plant and equipment. The success of the treatment process can only be as reliable as the equipment itself. As a part of the overall DPR O&M framework, asset management is vital to help inform operations and maintenance decision making to underpin this reliability. An asset management framework not only helps to identify the entire portfolio of assets to be maintained, but also provides a better understanding of their entire life cycle. This in turn helps the O&M team to plan appropriately for repairs and replacement and helps to fine tune maintenance strategies to maintain the high level of service required for a DPR system as assets age.

There are varying approaches to asset management in the water industry, and a wide spectrum of definition as to what asset management means. In some organizations, asset management is considered maintenance and capital investment planning, while in others it is considered a holistic management approach for the entire utility operation. The ISO Standard for Asset Management (55001 series) defines asset management as “The coordinated activities of an organization to realize value from assets”. According to the U.K.’s Institute for Asset Management, asset management is not so much about doing things to assets, but about using assets to deliver value and achieve an organization’s explicit purposes (An Anatomy of Asset Management, https://theiam.org/ama). This section of the report does not aim to provide an overarching discussion of asset management and its various incarnations within organizations, but rather aims to describe practical approaches to asset management that can be adapted to DPR. An asset management framework is designed to both articulate an organization’s approach and commitment to asset management, and also to guide the asset management program’s implementation. Figure 3.23 illustrates five key elements of any successful asset management framework or program.

Figure 3.23. Key Asset Management Program Elements

78 Water Environment & Reuse Foundation

The identification of Service Levels, Asset Management Gap Analysis, and Strategy Development provide high-level direction for asset management at the DPR facility and help to define overall purpose and intent. It also establishes a link between the asset management processes and practices (work performed) and the levels of service (work measured).

These two major considerations in turn drive the DPR facility’s level of service.

3.7.1.1 Levels of Service Levels of service represent a utility’s operational commitment to provide its customers and stakeholders a specified level of quality and reliability for a given level of cost. The service levels are targets of performance of the system, and of various system elements, that are required to meet overall system objectives. Levels of service are usually externally focused, strategic in nature, and relate to things that customers are concerned with such as quality, quantity, reliability, responsiveness, and cost of service. They are driven and informed by an organization’s overall strategy, which is influenced by elected and appointed governmental officials, customers and outside stakeholders. This strategy also influences the environmental, regulatory, and public policies that utility organizations implement, all of which drive the levels of service that an organization develops to inform its asset management program.

Figure 3.24. Levels of Service and Asset Management Program Cascade

In addition, they provide a means to communicate performance, both internally and externally, and prioritize infrastructure investment. Finally, these levels of service represent the linkage between the strategic goals an organization has and the operational tactics that it must employ to deliver on those goals. This ensures that operational and maintenance efforts and resources are focused on the right areas.

Examples of levels of service for a DPR treatment plant might be treatment capacity availability (i.e., service level target > 95%) and critical control point exceedances allowed in a given year (i.e., service level target of no more than five). To meet these service level targets, individual assets (process equipment, mechanical equipment, electrical equipment, etc.) must be maintained to a condition where those established level of service targets can be achieved. Furthermore, they must be continuously monitored to ensure that the levels of service are consistently achieved.

3.7.1.2 Gap Analysis In order for organizations to understand what asset management functions and activities they are currently performing that could be leveraged for their DPR facility compared to those recommended by industry standard frameworks, such as the ISO 55001 Standard for Asset Management, a gap analysis on those

79 Water Environment & Reuse Foundation

activities can be performed. An asset management framework, like ISO 55001, is intended to provide a more comprehensive perspective for the development of asset management practices and programs. It defines good asset management practice and provides guidance for the assessment of an organization’s current asset management maturity.

ISO 55001 is built around the Plan-Do-Check-Act cycle of continuous improvement. It specifies 27 elements of good practice that a competent organization should have in place as part of its asset management system. These elements can be categorized into seven fundamental areas of asset management, as shown in Figure 3.25 below.

Figure 3.25. ISO 55001 Asset Management Framework

The asset management framework consists of an organization’s asset management policy, strategy, objectives, asset management plan(s), and the activities, processes and organizational structures necessary for their development, implementation and continual improvement. This provides alignment for an organization’s strategic vision and goals to be achieved through asset-centric strategies and plans and represents a single ‘line of sight’ from an organization’s leadership team down to the asset operations and maintenance staff, so everyone has a clear understanding of what they are required to do to achieve the organization’s strategic asset management goals. The framework includes organizational structure, roles and responsibilities, standards, information management systems, processes, and resources.

Using the ISO 55001 framework as a guide, organizations can grade their current asset management activity performance against recommended levels of achievement in each of the 27 areas of asset management program proficiency, as shown on the example gap analysis spider chart in Figure 3.26. Any gaps that result help formulate an organization’s asset management roadmap and implementation plan. This should be performed at an organizational level, but can be applied at a facility level as well for organizations that are looking to leverage existing asset management practices for their DPR facility.

80 Water Environment & Reuse Foundation

Figure 3.26. Example ISO 55001 Gap Analysis

3.7.1.3 Asset Management Strategy In the case of a DPR facility/treatment system, the overarching strategy consists of providing a cost effective, reliable and safe additional supply of drinking water for the utility’s customers. In order to achieve that strategy, two primary goals also need to be achieved: 1. Water Quality: To ensure that the water reliably meets the quality requirements, and specifically the

protection of public health. This is a function of full compliance with CCP, regulatory, aesthetic and reliability requirements.

2. Production: To ensure that the process train meets the production goals to provide a reliable supply of water. This is a function of demand and ensures water availability is consistent with current and future customer needs through long-term supply and demand analysis.

The best way to ensure that each of these goals, and in turn, the overarching DPR facility strategy is achieved, a utility would need to identify specific actions necessary to close any asset management gaps that it has at its facility, along with the allocation of appropriate resources.

3.7.2 Asset Management Business Planning

The business planning side of asset management consists of ensuring that there is organizational support for successful asset management to be achieved. – namely that people/roles/responsibilities are identified and allocated to perform and manage the DPR facility’s assets in an appropriate manner, and that the supporting information management infrastructure is in place to allow asset management practices to be performed.

3.7.2.1 Asset Management Roles and Responsibilities In line with an overall approach that does not require major organizational structural changes to an organization, but does support a more formalized asset management approach to maintaining DPR facilities, it is important to identify at least one individual at the facility as the lead for asset management. This role would have enhanced responsibility for overall program development and maintenance

81 Water Environment & Reuse Foundation

management. The “Asset Manager” would also take the lead role in coordinating and facilitating the various work efforts established to ensure achievement of the DPR facility’s high level asset management strategies and goals.

It is often helpful to establish several work groups or committees to set overall DPR asset management guidelines and lead initial high priority aspects of an asset management program. These committees should be facilitated by the Asset Manager. Many of these committees will also have to interface and/or collaborate with existing utility departments, to ensure Input and perspective from across the organization. It is most helpful if these committees include representation from various utility departments including O&M and Engineering to ensure collaboration as well as include active participation from field staff. It is also critical for each committee to have a strong lead and champion that has the ability to drive organizational change. Several suggested committees, and their primary functional responsibilities are listed below.

Asset Condition and Criticality Committee o Develops guidelines and methodology for asset inventory, attributes, hierarchy, condition,

criticality, and risk assessment. o Provides guidance on establishing and tracking standard inspection and reporting intervals

and data maintenance procedures. Service Level, Reliability, and Reporting Committee

o Develops and tracks DPR facility levels of service, reliability, and regulatory metrics in alignment asset management program objectives.

o Designs and issues monthly/quarterly/annual performance reports on DPR facility Work Planning and Scheduling Committee

o Develops business processes and procedures for work planning and scheduling, primarily interfacing with existing Computerized Maintenance Management System (CMMS) / Enterprise Asset Management (EAM) systems.

o Provides the Planning/Scheduling function that is primary interface with the CMMS/EAM and the coordination point between maintenance crews/supervisors and the maintenance system.

Preventive Maintenance (PM) and Reliability Centered Maintenance (RCM) Committee o Develops the processes and procedures for DPR maintenance strategies including standard

preventive and predictive techniques for critical assets o Ensures that a consistent process is applied across all DPR maintenance and that formal

practices are documented through SOPs.

3.7.2.2 Supporting Information Management Infrastructure In any DPR facility asset management effort, knowing the needs of a utility is of value only when compared with what is currently available and/or achievable. It is important to ensure that DPR maintenance management systems are used as effectively and efficiently as possible in supporting asset management. It involves a thorough evaluation of the existing environment from the perspectives of People/Systems/Processes: People – Understanding of how your people – specifically your IT support staff and users of your

systems - serve and support asset management at the DPR facility. Systems – Understanding how core systems – CMMS and SCADA in particular – are configured and

used. This also includes an understanding of the performance of underlying IT infrastructure. Processes – Understanding how business and operational processes work (or don’t) in support of DPR

facility asset management.

82 Water Environment & Reuse Foundation

Figure 3.27. Data and System Management for DPR Facilities

3.7.3 Infrastructure Evaluation/Planning

Asset management processes and practices not only support planning for equipment failure by identifying the failure modes and predicting failures but also support optimized rehabilitation and replacement efforts through maintenance management systems. Establishing a comprehensive condition assessment strategy for DPR facility assets is a major step in developing an efficient capital plan for the facility and a requirement of sound financial planning.

The following section outlines a framework for asset management which is centered on three core questions (Figure 3.28). 1. What is the current state of the assets? 2. Which assets are critical to achieving required performance? 3. How can O&M costs and Capital Investment Planning (CIP) be optimized?

83 Water Environment & Reuse Foundation

Figure 3.28. Core Questions of Asset Management

The physical assets in treatment plants and associated infrastructure undergo wear and tear over time. Therefore, knowing the real rate of wear and tear and costs associated with asset maintenance is critical information for optimal operation of a treatment plant. Without knowing how fast the plant and equipment is wearing, proactive operation is almost impossible. The wear rate (and its associated cost) helps utilities to estimate the amount of reinvestment required in the plants. There is a significant correlation between the amount of reinvestment in assets and the performance of the system. A utility with a reactive operation and maintenance culture that reinvests minimally in its assets usually ends up spending more on emergency and breakdown maintenance, which jeopardizes the reliability of the system while having less money to spend earlier and pro-actively to prolong the life of assets.

This is especially true of DPR facilities and infrastructure, as the reliability of processes is only as good as the condition of equipment. From a CCP perspective, an acute area of concern is for online analyzers and automation systems as these are heavily relied upon to ensure adequate performance.

3.7.4 Current State of Assets

In order to understand the true cost of maintaining a DPR facility and its associated equipment, a utility needs to know what assets they have and where they are located. Then they need a reasonable estimate of the remaining useful life of each asset, which is dependent on the asset’s condition.

The first step in DPR facility asset management is understanding the current state of the assets. This step helps the O&M team to identify specific attribute information about each asset, where the assets are located, and what current state of condition they exhibit. A best practice is to record details of asset condition in line with the asset inventory hierarchy that corresponds to how work will be ordered and managed for that

WHAT do we have? Asset inventory, GIS, field verification

WHAT is its condition? Modeling, vibration analysis, oil sampling, visual inspection

WHAT is it worth? Asset valuation, depreciation

WHAT do we need to do with it? Rehab, replacement, maintain, risk analysis

WHERE do we start? Desktop condition, criticality, and asset risk assessment

WHEN do we need to do it? Prioritization, CIP, master plans

HOW MUCH will we finance it? Construction cost estimates, O&M, life cycle cost

HOW will we finance it? Rate studies, bond issues, grants, and loans

HOW do we do it? Planned field condition assessment

HOW do we measure value? Service levels and key performance indicators

HOW do we define and measure Program success?

Continuous improvement through active tracking and reporting frameworks for KPIs

HOW do we continuously manage the enterprise Asset Management Program?

Program management through effective strategy alignment, program governance, benefits measurement, and stakeholder engagement

84 Water Environment & Reuse Foundation

equipment (for example, a utility’s asset register may drill down to the pump, valve or other mechanical equipment level). This level contains what is referred to as Maintenance Managed Items (MMI). This is the level at which an asset is maintained or decisions are made to repair, refurbish, or replace.

Understanding the current state of a DPR facility’s assets includes the following components:

Asset Inventory The asset inventory includes the recording of all assets located in the DPR facility, their asset identification, location, installation year, size, asset class, asset type, material, and condition scores with associated pictures for each asset. The final product of an asset inventory process is an asset register that is integrated into the utility’s maintenance management system.

Condition Assessment The condition assessment is the technical assessment of the operational performance and physical conditions of an asset, using a systematic method designed to produce consistent, relevant and useful baseline information. Condition assessment provides insight into the nature and timing of possible failure. Condition assessment results, together with asset functionality, utilization and cost considerations, are used to support a wide range of asset planning decisions, particularly in relation to asset repair and replacement needs. Condition assessment can be conducted at different levels of detail, ranging from high-level, low-detail to low-level, high-detail.

Level 1: Basic / high-level Level 2: Intermediate / high-criticality Level 3: Advanced / high-detail

A risk-based approach is recommended to maximize the efficiency of the condition assessment. Those assets that present a high risk to facility performance should be assessed frequently and in detail. Others that present a lower risk can be reviewed less often, and perhaps at a lower level of detail. This process is often referred to as “asset filtering” that is, focusing attention where it will provide the best value.

With an increased level of detail comes a higher level of sophistication and expense. As a result, the assessment process is applied to fewer assets at each progressive level. Filtering is highly efficient because additional resources are applied only to those assets requiring attention after an initial, higher level, assessment of all assets has been made.

For example, critical monitors including online analyzers that provide the assurance of CCP performance will require advanced, higher level of detail assessments due to their high criticality for CCP performance, whereas an asset such as a chemical feed pump may only require basic or intermediate condition assessment

Residual Life Calculation This step includes assessing remaining useful life by identifying projected asset useful life for all DPR facility assets. If historical condition assessment data is available, deterioration curves can be developed for each asset class. With time, many facilities build knowledge from the experience of asset performance over time, coupled with manufacturer’s recommended useful life information, which can then be catalogued to better refine residual life and associated costs.

Replacement Cost Determination Replacement costs estimate the actual cost to replace an asset with equal capacity, material, size, or

85 Water Environment & Reuse Foundation

type. Historical cost data is a good source of information to determine replacement costs of assets. These estimated replacement costs should be included in the CMMS asset register.

3.7.5 Asset Risk Assessment

Asset condition and risk assessment, as described in Section 3.2.5, is used to set the priority of cost and resources in a transparent and consistent manner. The objectives of the asset risk assessment are to:

1. Identify assets representing the greatest risk to the DPR facility and to the organization. 2. Promote efficient use of resources by focusing on high-risk assets (i.e., capital and operational

expenditures, staff hours). 3. Highlight assets requiring detailed condition assessment or renewal. 4. Prioritize inspection, cleaning, and preventative maintenance schedules.

3.7.5.1 Probability of Failure/Likelihood of Failure Figure 3.29 shows a more detailed example of a probability of failure (PoF) methodology for facility assets. The assessment starts with reviewing existing asset information available from various sources, including the CMMS database, master plans, and as built drawings, CCTV data, or site visits. When condition data is available, the PoF is calculated utilizing a condition score (as described in Section 3.2.5). Where condition data is not available, an age-based methodology utilizing expected useful life, deterioration curve information, and asset age can used to estimate the PoF (noting however that age is not necessarily a surrogate for asset condition).

Review existing data:

GIS database Rehab history Master plans As-built drawings

Calculate

Condition-Based

PoF

Calculate

Age-Based PoF

Does asset have

high CoF?

Apply Normal

Management

Strategy

No

Apply Aggressive

Management

Strategy

Yes

Site Visit

(Visual Condition Assessment)

Is asset visually

accessible?

Determine

Condition ScoreYesNo

Figure 3.29. Probability of Failure (PoF) Methodology

86 Water Environment & Reuse Foundation

3.7.5.2 Consequence of Failure As noted in Section 3.2.5, not every asset presents the same failure risk, or is equally critical to the system. This step includes ranking assets according to how critical they are from triple bottom line perspective (i.e., economic, social, and environmental). Redundancy or the presence of backup equipment, helps to decrease the consequence of failure, when available. The criticality rating reflects the potential impact of an asset failure to the performance of a group of assets, process, and the entire system. A two-tiered approach is recommended for the criticality assessment. 1) Process level. 2) Asset level.

At the process-level, the impact of failure is measured system-wide and the focus is quantifying the impact of process failure on the performance of the entire system. At the asset-level, the focus is to measure the impact of asset failure on the performance of the process.

A triple bottom line approach is recommended to measure the process-level criticalities. The triple bottom line approach considers economic/financial, social, and environmental impacts of failures. Each of the triple bottom line aspects of Consequence of Failure (CoF) are defined in Figure 3.30.

TBL Category CoF Considerations

Economic O&M Impacts Direct cost to repair, additional cost to operation (labor, power, chemicals)

Impact on Other Processes Additional operating costs on any related plan procedures as a result of failure

Social Level of Service Failures Service disruption, taste, odor, hardness, etc.

Safety Consider utility staff and public

Environmental Regulatory compliance Any permit requirements at the facility

Figure 3.30. Consequence of Failure (CoF) Categories

As previously discussed, the two primary levels of service for DPR facilities are often water quality and water production. The corresponding CoF may be different for each of these levels of service, and it may be useful and important to examine CoF impacts for both, as shown in Table 3-20. As shown, process-level and asset level CoF scores are assigned to each asset. The asset-level consequences are only scored based on their impact to the functionality of the process.

In this example, quality and production are weighted as being equally important at the overall process-level; thus, if the failure does not have any impact on the water quality and has a major impact on production, then the process-level impact is considered major. Asset-level CoF provides the impact of asset failure on the overall functionality of the process of which it is a part.

87 Water Environment & Reuse Foundation

Table 3.20. Example Consequence of Failure Summary for a DPR Facility

Process Asset

Process-Level CoF Asset-Level CoF

Quality Production Impact of asset failure on the functionality of

the process Pre-chloramination

Chlorine-Chlorine Storage Tank

2-Minor Impact

3-Moderate Impact 2-Minor Impact

Pre-chloramination

Chlorine-Instrumentation & Control

4-Major Impact

3-Moderate Impact 4-Major Impact

Pre-chloramination

Chlorine-Chemical Dosing Pumps

3-Moderate Impact

3-Moderate Impact 4-Major Impact

Pre-chloramination

Chlorine-Chlorine/Chloramine Analyzer

4-Major Impact 3-Moderate

Impact 4-Major Impact

Pre-chloramination Ammonia-Storage Tank 2-Minor

Impact 3-Moderate Impact 2-Minor Impact

Pre-chloramination Ammonia-Dosing Pumps 3-Moderate

Impact 3-Moderate Impact 4-Major Impact

Pre-chloramination

Ammonia-Chlorine Analyzer

3-Moderate Impact

3-Moderate Impact 4-Major Impact

Inlet Strainer Strainer 1-No Impact

3-Moderate Impact 3-Moderate Impact

Inlet Strainer Instrumentation & Control 1-No Impact

1-No Impact 2-Minor Impact

Inlet Strainer Motor 1-No Impact

3-Moderate Impact 3-Moderate Impact

MF/UF Membrane Unit 4-Major Impact

4-Major Impact 5-Catastrophic

MF/UF Skid Structure 1-No Impact

1-No Impact 1-No Impact

MF/UF O-Ring 3-Moderate Impact

3-Moderate Impact 3-Moderate Impact

MF/UF Air Compressor 3-Moderate Impact

4-Major Impact 2-Minor Impact

3.7.6 Financial Management/Planning

The development of a financial optimization analysis based on asset condition (PoF) and criticality (CoF) is often the key ingredient of a utility’s overall financial management plan. It enables an organization to identify and prioritize its capital and O&M projects so that the most critical projects are funded and executed first. This helps to ensure that any available funding is allocated to the repair or replacement of assets with the highest risk of failure based on their overall condition and criticality.

Optimizing capital and operation activities requires a detailed knowledge of the asset renewal and operations costs. In order to evaluate which decisions will reduce risks and meet desired levels of service, it is necessary to develop “whole life” cost models for assets. Replacement cost of each asset can be calculated using historical bid tabulations, purchase orders, financial inventory, and cost tables. Where available, operations and maintenance costs can be broken down and assigned to individual assets. With the listing of assets, their costs, and risks all recorded in the asset register within a utility’s CMMS, it is possible to determine replacement, rehabilitation, and major maintenance trends for DPR facility assets.

88 Water Environment & Reuse Foundation

Maintenance management strategies can be used as the logic to model the activities and costs of all assets into the future, in order to predict and project future maintenance, replacement, and rehabilitation needs. Budgetary requirements and average investment requirements can be modeled for the next 20 years or extend beyond 100 years to ensure the planning horizon is long enough to capture a full life-cycle of all DPR assets (see Figure 3.34).

Figure 3.31. Example 100-year Asset Renewal Projection

The long-term asset renewal plan can identify the following for each year: Assets requiring action. Type of action required (replacement, refurbishment, maintenance). Estimated cost of each action. Estimated total budget required for the year. List of capital projects prioritized by risk.

The long-range budgetary needs can be available at any level within the asset hierarchy (i.e., by plant, by process, by system, by asset). In addition, future renewal needs can be compared with current budget levels to determine necessary annual rate increases. It can also estimate the current work backlog and develop visualization of budget versus work backlog.

Realizing the necessary funding for the next 5, 20, and 100 years of capital and major maintenance activities allows utilities to analyze various funding scenarios and options. Utilities can determine the appropriate balance between borrowing money or increasing rates and increasing maintenance budgets. Understanding upcoming asset renewals and costs allows utilities to establish required reserve levels and allows staff to optimize the use of budget and resources by validating the need for identified CIP projects.

The CIP project validation process incorporates the full project life cycle costs (from design to decommission), alternative project solutions, associated risks, and confidence level ratings to develop a business case that justifies the need for a project. The validation process includes a comparison between the CIP projects that are driven by asset management study and the CIP projects already funded by looking at their associated risks and confidence level ratings. This comparison indicates how the budget is spread among CIP projects which will enable the utility to prioritize the projects, rearrange the allocation of funding, and invest more on CIP projects with higher associated risks and confidence level ratings.

89 Water Environment & Reuse Foundation

3.7.6.1 CIP Prioritization One of the key aspects of asset management is asset renewal and project justification. Utilities need to ask themselves, are we making the right decision, at the right time, at the right cost, for the right reason? With large capital investment required for DPR facilities and watchful eyes of stakeholders, it is reassuring to have confidence in the investment decisions through a business case that documents, validates, and justifies the investment decision.

A CIP project validation process is used to quickly filter and prioritize capital projects. It utilizes core asset management elements to help prioritize projects in a consistent and justified way. Major components of a CIP project validation process include:

1. Project Identification – Identifies and documents the project details. 2. Confidence Level Rating (CLR) – Measures the quality of the data used to generate the project. 3. Business Risk Exposure (BRE) – Measures the risk associated with the project. 4. Life Cycle Cost Analysis (LCC) – Evaluates the life cycle cost of ownership. 5. Business Case Evaluation (BCE) – Evaluates other alternatives and documents the recommended

action.

The CIP project validation process is represented as the first five steps of the overall CIP budgeting process. Each component acts as a filter to justify the progress of the project. For example, both Confidence Level Rating (CLR) and Business Risk Exposure (BRE) components require an achievement of a minimum score before the project can be considered for business case evaluation. Only when the project meets or surpasses the minimum CLR score will the project qualify for BRE assessment. A Business Case requires both CLR and BRE requirements to be met before consideration.

This step-by-step process ensures a quality check and consistent and justified decision-making. Business Case Evaluation is an international asset management best practice methodology for optimizing, justifying, and documenting the investment decisions in a consistent and transparent manner. It provides a well-structured document that highlights the following key decision-making elements: Project information and requirements. Business drivers. Alternative options. Capital cost. Life-cycle cost. Risks. Benefits. Benefit cost analysis. Net present value analysis. Payback period / return on investment.

A formal business case should be designed to evaluate feasible solutions and to provide an optimized recommendation based on minimizing the risk, life-cycle cost, and maximizing benefit. Incorporating the concept of Benefit/Cost Ratio (B/C) and/or Net Present Value (NPV), each feasible solution can be carefully analyzed with respect to initial capital cost, on-going operation and maintenance costs, risk costs, and potential benefits. The solution providing the greatest benefit will then formally be recommended for implementation (Figure 3.32).

90 Water Environment & Reuse Foundation

Figure 3.32. Example Prioritized CIP

3.7.7 Performance Reporting

Integrating a performance tracking system with an organization’s Asset Management information management systems (i.e., CMMS, Human Resources, Financial, Laboratory Information Management System, SCADA, GIS, etc.) allows for the utility to properly track performance and define program progress and implementation success. These tracking systems can range from spreadsheets and simple database applications to full-fledged, commercial off-the-shelf performance management dashboard systems.

Simple data analytics reveal basic insights; more sophisticated analytics, applied to data that has been pooled into a “data lake” with data from external and enterprise sources allow utilities to unearth deeper insights that will help to optimize DPR facility performance.

Because of the growing volume, complexity and strategic importance of asset management data, it is no longer desirable or even feasible for each department/unit/division/function within a utility to manage this data by itself, or to build its own data analytics capabilities.

To get the most out of the new data resources, utilities today are creating dedicated data groups that are potentially embedded within the core asset management program team to consolidate data collection, aggregation and analytics.

Recently, advances in technology have revolutionized data and performance reporting so that users (with limited IT development expertise) can perform data mining and develop high impact visuals for performance reporting. Benefits of this “Business Intelligence” approach include: Eliminating the reliance on core IT developers to develop and manage reporting frameworks.

Business Intelligence is now integrated with common applications such as MSExcel, putting the non-IT user in a position to perform complex data analysis and develop aesthetically pleasing visualizations.

Significantly reducing development cost and level of effort. Through the concept of data “lakes,” data models can be constructed using data from various sources

(CMMS, GIS, SCADA, project management, financial and customer information systems) with ease.

91 Water Environment & Reuse Foundation

Eliminating the extensive costs and need for complex and disparate system integration that is typically required to connect data for effective performance reporting.

Reducing the time to develop high impact visualizations to hours or days, rather than weeks, months, and years.

Complete transferability to mobile devices for use at meetings and workshops.

3.7.8 Maintenance Management Strategy

One of the goals of an asset management strategy is to support the shift from reactive maintenance to proactive maintenance. Proactive maintenance focuses on managing asset failure rather than waiting for the assets to fail and then replacing them with new assets. Running the asset to failure can in some cases be an appropriate approach and the most cost effective, however applying this strategy to all the assets reduces the reliability of the system and can result in significant O&M costs.

An optimized maintenance strategy ensures that the condition of assets is assessed at appropriate intervals based on costs, safety, and reliability, risk limits and/or indications of degraded performance or failure. Critical equipment is examined more often and more thoroughly. The examination process identifies needed enhancements or corrective actions. A proper review will emphasize the critical equipment; formalize current practices and schedules; and adjust the scope and/or frequency of inspections, surveillance, and testing activities to ensure they will detect and mitigate mechanisms contributing to failure. Sometimes, the enhancement may involve changes in operation, production, or utilization resulting in the need for a cultural change in the organization to move from reactive to proactive.

Depending on whether the progression of asset failure can be detected (is the failure predictable?), whether there is enough time to respond to the failure (is the failure preventable?), and whether the consequence of failure exceeds cost of cure (is the asset critical?) an appropriate strategy needs to be developed. In the case of DPR, a proactive maintenance strategy is required in particular for core process equipment (in particular critical control points). However, even for a DPR system, for some items it may be appropriate to run to fail for low risk profile items, or those for which failure is difficult to predict.

The flowchart in Figure 3.33 can help decide which assets need to be maintained and which assets are appropriate to be managed to failure. To decide whether or not to adopt a proactive maintenance strategy for a particular asset, the first question is whether the asset failure is predictable or preventable. If the failure is not predictable or preventable then a Run to Failure strategy needs to be considered and the O&M team needs to plan for corrective failure response, such as appropriate spares inventory, or rapid repair response. As a default, maintenance activities recommended by the manufacturers should be followed.

If the failure is predictable and/or preventable, then the criticality of the asset needs to be assessed. If the asset is critical and its failure results in significant consequences, then a condition-based or usage-based maintenance strategy needs to be considered depending on how the failure is predicted. If the asset is not critical then the proactive maintenance can still be an option if the preventive maintenance is cost effective, otherwise a run to failure approach can be considered.

As an example, an asset for which an agency may choose to have a run to failure strategy is O-rings. Typically, O-rings are changed when the membranes are changed in order to optimize maintenance costs, in particular the labor can be shared for both O-ring and membrane replacement. However, the O-rings can reach the end of their useful lives in some cases before the membrane does. The failure of O-rings cannot be predicted and prevented. The consequence of failure for an O-ring is high because of its impact on the quality of the product. Therefore, a corrective failure response can be put in place to mitigate the risk of failure.

92 Water Environment & Reuse Foundation

The corrective failure response has two components: 1) Monitoring strategy. 2) Response strategy.

In this example the failure of an O-ring will be recognized for the microfiltration via the pressure decay integrity test, and for an RO system via an increase in measured electrical conductivity in the RO permeate. The response strategy could be having the spare parts and the right O&M crew to replace the asset in case of failure.

Figure 3.33. Test to Determine Proactive Maintenance

Maintenance activities should be adjusted corresponding to asset life, in particular based on the remaining time to predicted failure. At the beginning of the asset life cycle, preventive maintenance is recommended such as time – or usage-based maintenance strategies. Most manufacturers typically have specific recommendations for the scope and frequency of preventive maintenance. As the asset ages, the maintenance strategy gradually shifts to predictive maintenance or condition-based maintenance as operations and maintenance staff gather more knowledge and experience with the equipment. Examples of this predictive maintenance are the maintenance activities that are triggered by a diagnosis of a stress in the asset through the condition assessment process (e.g., vibration, noise, heat).

As the asset becomes closer to the end of its useful life, monitoring intervals are increased to make sure that the need for replacement is identified before the failure of the asset (Figure 3.34). This enables tracking the asset to failure which gives enough time for the operators to intervene and perform the corrective action.

93 Water Environment & Reuse Foundation

Figure 3.34. Preventative versus Predictive Maintenance

Major Maintenance Management Strategies:

There are three major maintenance management strategies that support proactive maintenance.

1) Total productive maintenance – The goal of this maintenance strategy is to maximize overall equipment effectiveness. It focuses on developing a comprehensive asset management plan for each asset for the life of the asset.

2) Reliability centered maintenance – This maintenance strategy lowers asset deterioration rate and reduces variation in failure intervals. It also helps lengthen equipment life by periodically predicting remaining useful equipment life from its condition and restoring the condition of the asset.

3) Zero breakdown maintenance – This maintenance strategy is built around criticality that is ensuring that highly critical assets do not break down. This is the recommended approach for CCPs : What is this asset supposed to do? What causes each failure? Why does it matter? What should be done if a suitable proactive task cannot be found?

3.7.9 Work Order Management System

The output of a maintenance management system is the work order which is the link between asset management and maintenance management systems. Traditionally, work orders contain information such as actual labor and material used, description of the maintenance activity, and the primary of cause of failure. However, the true benefit of asset management is not realized until the work orders contain information such as asset type, size, condition, performance history, and failure modes. By including asset level information in the work orders (asset-linked costs rather than only crew costs) failure patterns and their associated costs start to be discovered. This requires a shift of focus to performance of assets over their life cycle which not only track the work done by the crew but also tracks the work performed on a specific asset. Tracking the work orders at the asset level would support agencies to start changing their maintenance techniques from unplanned (reactive) to proactive maintenance strategies.

94 Water Environment & Reuse Foundation

3.8 Water Quality Monitoring Comprehensive water quality monitoring is one of the most important operational tasks for a DPR system. It must take place for a number of operational considerations including: Source control monitoring. Wastewater treatment process monitoring. Wastewater effluent/advanced treatment source monitoring Advanced treatment plant operational monitoring (within and between process units). Analyzer calibration/verification monitoring. Advanced treatment plant final product (verification) monitoring. Source control monitoring will not be discussed in this report, but is covered in Chapter 5 of Framework for Direct Potable Reuse, WRRF project 14-20. Wastewater treatment process monitoring is also not further discussed in this report.

3.8.1 Advanced Treatment Plant Source Monitoring

Due to the difficulties of thorough water analysis in the matrix of raw sewage, wastewater treated effluent (or the advanced treatment plant source water) provides the ability of a more thorough and detailed assessment of water quality risks.

The characteristics of water quality presented to the advanced water treatment plant may vary over time. In Section 3.3.7 of this chapter, the selection of CCPs is determined by a review of water quality hazards at the inlet of the advanced water treatment plant. Water quality analysis from wastewater treatment facilities is often limited to a small range of parameters consistent with environmental discharge permits. Therefore prior to and during the design phase of the advanced plant, extensive water quality analysis must be conducted to help characterize the water quality to assess the broader range of public health water quality risks to develop a comprehensive assessment of risks. However, once the risk assessment has been conducted and CCPs identified, the water quality assessment must be ongoing to identify a change in risk profile, and in particular identify new and emerging water quality risks.

Selecting parameters for advanced treatment feed characterization should consider variation with time and season, new hazards that may be presented in the sewershed such as industrial inputs as well as advances in research and analytical techniques which may allow measurement of previously unknown or unmeasurable contaminants. Table 3.21, adapted from the Western Corridor Recycled Water Project (Seqwater), demonstrates a process that can be used to decide which parameters to include in an advanced treatment source water monitoring program and provides guidance on sampling frequency. This particular example is for the RO based treatment process, however can be adapted for the non-RO process.

95 Water Environment & Reuse Foundation

Figure 3.35. Advanced Treatment Plant Feed Sampling Parameter Ongoing Selection Process (Adapted from Western Corridor Recycled Water Project, Queensland Australia)

Chemical parameter identified in drinking

water regulations, research or other

literature.

Could it form or be added in the process above a health risk

standard?

Is there evidence it is not present in the

sewershed?

Is retrospective analysis possible?

Is the detection limit < health standard ?

Was it detected?

Is it already monitored in

advanced treated product water?

Formed or added before RO ?

Quantify the parameter in final advancedtreatment plant product water or after formation

within the process.

Do not include in advanced treatment plant source monitoring.

Is it likely to be rejected by RO ?

Do not include in advanced treatment plant source monitoring.

Quantify the parameter in advanced treatment plant source or after

formation in the process.

Do not include in advanced treatment plant source monitoring.

YES NO

NO

NO

YES

YES

NO

YES NO

Include in annual review of parameters and determine sampling frequency

Include in annual review of parameters and determine sampling frequency

YES

NO

YES

YES

NO

NO

YES

NO

96 Water Environment & Reuse Foundation

Prior to the initial water quality risk assessment, and at regular intervals throughout the life of the system (typically yearly), a formal review of water quality parameters being assessed should be conducted.

Information sources that can inform the list of parameters to monitor for include:

EPA primary drinking water regulations. EPA secondary drinking water regulations. EPA identified future contaminants identified in Unregulated Contaminant Monitoring Rules

(UCMR) and/or contaminant candidate list (CCL). o Ideally, any contaminant monitored will have an associated health reference level or other

benchmark against which measured concentrations can be compared in order to develop an action plan. Without a benchmark action level, monitoring provides information on occurrence but no set path for action.

Relevant state drinking water regulations if more stringent than EPA. Chemicals that have health standards in other recycled water guidelines or drinking water guidelines

internationally (for example Australian Guidelines for Water Recycling). Results of research ongoing with Water Research Foundation (WRF), Water Reuse Research.

Foundation (WRRF), Water Environment Research Foundation (WERF), or other research organizations.

Information from industrial chemical registers. Information from city source control chemical registers. Surveys and contacts with other advanced recycled water facilities for IPR or DPR.

Additional parameters should be included in the monitoring of the wastewater effluent following any abnormal events including, for example, high TOC in reverse osmosis permeate, or other CCP processes operating outside of their critical limits. This may be useful to identify potential contamination events from the sewershed.

Initial sampling frequency for parameters should be relatively high in order to gain a baseline of source water quality information and provide a statistically relevant estimate of variability of concentration in the source. This may include a six-month or longer intensive sampling campaign to review all identified parameters from the lists above. This frequency may be weekly or even higher, in order to obtain sufficient data. Guidance on developing monitoring campaigns and benchmarking chemical constituents is provided elsewhere (Khan, 2010a; Khan, 2010b; Stanford, Reinert, Rosenfeldt, et al., 2014).

Following the establishment of this baseline, the sampling regime can be refined to a lower frequency of sampling and quantity of parameters, based on whether: There is a likelihood of encountering this contaminant in the sewershed. There is published evidence of this contaminant in other wastewater sources. The concentration has been observed to approach known health reference levels. It has been validated as well removed by the advanced treatment processes operating within CCP

limits.

Concentration of contaminants in wastewater effluent are assumed to follow a log normal distribution (Khan, 2010b). The estimated log normal percentile concentration of a parameter in the wastewater treatment plant effluent can be used as a guide to the determination of monitoring frequency. Table 3.21, adapted from the Western Corridor Recycled Water source water monitoring program, can act as a guide to a typical sampling frequency.

97 Water Environment & Reuse Foundation

Table 3.21. Potential Basis of Advanced Water Treatment Feed Analysis (Adapted from Western Corridor Recycled Water Project, Queensland Australia)

Condition Sampling Frequency

All identified water quality health risks, whether or not previously detected.

Yearly

99th Log normal percentile in source at 0.001 to 0.1 of MCL or relevant health target.

Twice per year

99th Log normal percentile in source at > 0.1 of MCL or relevant health target.

Monthly

Source parameter of interests from parameter selection process (Figure 3.35)

Monthly

Microorganism Indicators

Weekly

Event Screening GCMS screening of advanced treatment plant feed and other locations within the advanced treatment plant. This would be included as part of a CCP response procedure.

Source control monitoring may also include on line water analysis at the operational interface between the advanced treatment process and upstream wastewater treatment plant (as noted in Section 3.6). As an example, all three advanced treatment plants at the Western Corridor Recycled Water Plants in Australia included on line turbidity, pH, TOC, ammonia, nitrate and phosphorus at the advanced treatment plant inlet. This data was useful to determine diurnal water quality variation in the wastewater treatment plant, provided early warning of wastewater treatment plant upsets and provided useful operational feedback to the wastewater treatment plant operators.

3.8.2 Operational Monitoring

Operational monitoring includes both on line analysis as well as water quality sampling and analysis within the advanced treatment plant. On line water quality analysis for each CCP has already been discussed in this report. Other analyzers within the plant provide important operational information for other aspects of plant and process, and may be incorporated as COPs or just used for other monitoring.

Regular sampling for the verification of on line analyzer results is also important to ensure that analyzers are reading accurate results. These are typically conducted on site using an on-site laboratory, bench top test or hand held test equipment. Regular calibration and testing of on-site sampling and analysis equipment, as well as management of testing reagents is of vital importance.

Other water quality sampling and analysis is conducted within the advanced treatment plant for other purposes including: Providing further verification of CCP performance. For example, regular sampling of RO feed and

permeate for some divalent ions and other key indicators can provide greater resolution of removal performance. For GAC, sampling of TOC can provide a better indication of removal capability over time.

98 Water Environment & Reuse Foundation

Ensuring processes are not subject to a changed operational risk profile. For example, a routine analysis of inorganic cations and anions for RO feed can ensure that feed pH setting, antiscalant selection and dose, and RO recovery are set appropriately for the feed water quality to prevent scaling.

The following table outlines an example sampling frequency for external laboratory analyses, taken from the Western Corridor Recycled Water Project. This is representative of a RO-based advanced treatment process, however it can be adapted to a non-RO-based process. Table 3.22. Example External Laboratory Sampling Frequency for Operational Monitoring

Parameter Frequency

Microorganism indicators at each CCP Quarterly

Divalent ion (sulfate) at RO feed and permeate Weekly

Indicators of chemical parameters > 10 x limit of reporting detected in advanced treatment plant feed. RO feed and RO permeate Note that 10 x LOR was selected as 90% removal was claimed across the reverse osmosis system.

Quarterly

GCMS scan on RO permeate and UV H202 treated water

Quarterly

3.8.3 Verification Sampling Programs (Advanced Treatment Plant Product Sampling)

Verification monitoring provides a final demonstration that water quality is meeting the requirement for public health and other targeted water quality criteria in the advanced treatment system product water. While the CCP approach manages the risk at a process unit basis, the verification sampling provides a final demonstration of compliance. The water quality analysis for verification monitoring will typically be conducted at an accredited external laboratory to provide transparency in results.

Not all treated water parameters need to be routinely tested, nor all tested at the same frequency, especially if they are not detected in the sewershed and/or are known to be well removed by treatment processes within CCP limits. As for advanced treatment feed water sampling noted in Section 3.8.1, methodologies can be applied to rationalize both the frequency and range of parameters measured. Performance of the advanced treatment plant may be adequately demonstrated by routinely monitoring the parameters that present a significant risk, monitoring of surrogates that demonstrate process performance (e.g., TOC breakthrough for GAC vessels), as well as indicators that are representative of groups of parameters.

The verification monitoring (or advanced treatment plant product water monitoring) can be reviewed on an annual basis via an annual review process.

99 Water Environment & Reuse Foundation

Figure 3.36. Annual Review of Verification Monitoring (Adapted from Western Corridor Recycled Water Project – Seqwater)

In addition, a responsive monitoring program provides a rapid review of changes to water quality health risks encountered at the advanced treatment plant. This can be triggered if: Results of research or investigation at the advanced treatment plant determine the parameter is formed

in the process itself. The concentration of a water quality parameter in the advanced treatment feed water is detected at

levels significantly above the median (10 times). There is a regulatory concern or identification of a contaminant of concern becomes known from

other similar systems or from general production.

The responsive process also provides additional sampling and analysis in the event that a parameter is found to be above the MCL, health reference level, or other treatment target in the advanced treatment product water.

From Response Monitoring Results

Previous 24 months advanced treatment plant feed and product results

Annual Review of previous 24 months advanced treatment

plant feed and product results

Was the parameter detected in > 5% of samles of advanced

treatment plant feed ?

Was the parameter detected in advanced

treatment product water?

Has the parameter beendetected in any previous feed or product sampling

at the advanced treatment plant ?

Remove from advanced treatment plant product

monitoring.

Parameter of interest identified in advanced

treatment feed.

Amend monitoring to include advanced treament feed and

product monitoring.

Is concentration in advanced plant feed > 10 x limit of reporting?

Include as an indicator parameter in advanced

treatment feed and product water.

Include in annual scan of advanced treatment plant

product water.

Periodic review using parameter selection decision

tree.

YES

YES

NO

YES

NO

NO

End

YES

NO

100 Water Environment & Reuse Foundation

Figure 3.37. Responsive Verification Monitoring Program – Approach Taken from Western Corridor Recycled Water Project – Seqwater

Sampling frequency, again based on the sampling frequency at the Western Corridor project, is noted in Table 3.23 and may be used for guidance but is not meant to imply a recommendation of this frequency for monitoring.

Research work or other investigation identifies

parameter formed in advanced treatment process.

Parameter median result in advanced treatment plant feed

> 10 x MCL.

Other identified water quality concern (press, public

pressure, related issue at other facility)

Minimum of 5 sampling events in advanced treatment plant

product water.

Was parameter detected in advanced treatement product water > 0.5 x MCL or

target level?

Include in monitoring of advanced product water

weekly.

Is the parameter above the MCL or

target?

Use the result in the annual review of parameters.

NO

YES

NO

Monitor twice weekly until investigation has identified the cause or 5 consecutive samples

are < MCL or target.

YES

101 Water Environment & Reuse Foundation

Table 3.23. Example Monitoring Frequency from Western Corridor Project Parameter Frequency

Microorganism indicators Weekly

Indicators detected in advanced treatment plant feed water > 10 x detection limit

Every two weeks

Hazards detected in advanced treatment plant product water > 0.5 x MCL or target limit.

Weekly

Hazards detected in advanced treatment plant product water < 0.5 x MCL or target limit.

Every two weeks.

Any previous detection in advanced treatment plant feed water

Yearly

99th Log normal percentile in advanced treatment plant feed at 0.001 to 0.1 of MCL or treated water target.

Quarterly

99th Log normal percentile in advanced treatment plant feed at > 0.1 of MCL or treated water target.

Every two weeks

It should be noted that these sampling frequencies were predicated by the fast construction and startup time required for the Western Corridor facility, and that the high frequency was set to gather sufficient statistical data within a 13-week period that was required to start plant operation. 3.8.4 Sampling and Analysis Management

Water quality sampling and analysis in a DPR context may provide a significantly greater challenge for operations than wastewater treatment systems and drinking water systems alike. There can be a much greater volume of sampling and testing that is required, with in many cases a greater level of sophistication required in sample planning and management. The following considerations are needed in the management of sampling and analysis: Sampling and analysis management plan. Training for correct sampling, sample storage and transport. Accreditation of the external laboratory. Laboratory and field QA/QC (e.g., replicate samples, blind duplicates, analysis technique, matrix

spikes, accounting for matrix effects)

A detailed sampling and analysis plan must be developed across the advanced treatment plant. This should include: Detailed schedules for sampling and analysis for water quality analysis performed both internally and

externally to the operating facility. Effective sample planning, ensuring that resources such as sample containers, reagents, storage

availability and integration into staff operating routines takes place. Sampling and analysis procedures, taking into account best practice sample collection techniques,

preservation, storage and transportation requirements for external analyses. This should include reference to Standard Methods for the Examination of Water and Wastewater (22nd edition AWWA)

102 Water Environment & Reuse Foundation

in addition to the latest information for contract laboratory companies and the literature on sample handling and analysis best practices (e.g., Vanderford, Mawhinney, Trenholm, et al., 2011).

An effective chain of custody system for collection, recording and transferring of samples to external laboratories.

With a wealth of data, careful attention must be paid to water quality data management. While some organizations are able to manage data using simple spreadsheets or simple databases, a water quality database approach is recommended. These databases may be linked to external laboratory LIMS (Laboratory Information Management Systems) for rapid response of results. There is a high reliance on external laboratories for water quality results, and as such a laboratory with a wide range of analytical capability is important. Use of laboratories that are suitably accredited (for example ELAP, NELAC, etc.) is also an important requirement. With a high volume of samples and wide range of parameters to be analyzed, quality assurance and quality control must be managed. It is recommended that regular use of duplicate samples and blind duplicate samples be delivered to laboratories as a part of quality assurance – on top of the quality assurance that should already be provided by the external laboratories. For all sampling to be sent to external laboratories, it is highly recommended to maintain a duplicate sample, where storage of this sample for up to a month is practical but understanding the limitations of potential analyte degradation during storage. This can enables a second cross check of any result, especially in the case of potential analysis error or errant result. Selection of samples for maintaining duplicates should depend on those analytes considered a higher risk (noted in the water quality risk assessment) and the allowable holding time. A good communication with the external laboratories is also very important to ensure that any result that is outside the expected range is quickly reported to allow as fast an operational response as possible. By providing the laboratory with an expected range of result, a communication can be delivered to the treatment facility if a result is found outside this range. It is best that this communication occurs as soon as the sample is measured, with an interim result communicated to allow fast response. Work can begin on determining the cause for an errant result prior to the finalization of those results.

3.9 Incident and Emergency Management

3.9.1 Principles of Incident and Emergency Management Plan

Incidents and emergencies can and do occur in operating facilities of all kinds over the life of operation. It is the obligation of any operating utility or agency to implement comprehensive and effective risk management systems to lower the risk of incidents occurring to as low as reasonably practicable. Implementation of an effective, and rigorously tested crisis, incident and emergency management system to manage and minimize the impact of any incident is of vital importance. This is especially true in the case of DPR, where there is a high level of both regulatory and public scrutiny.

An effective incident and emergency management plan must be in place in order to: Reduce the risk of incidents occurring. Reduce the impacts of incidents on personnel, the community, the environment, sites, assets and

systems. Promote and support the maintenance of effective incident management processes. Provide guidance to operations on the correct response to an incident.

103 Water Environment & Reuse Foundation

Key principles that must be addressed by an effective Incident and Emergency Management Plan include:

Principle Description

Risk Analysis The identification of hazards and risks which could impact on the utility, agency, community, the environment or key stakeholders. A risk assessment process has been described in Section 3.2.3 of this chapter.

Prevention The planning and documentation of prevention and mitigation activities for all major hazards and allocation of responsibility for their implementation.

Preparedness The development, implementation and review of specific incident management plans and processes to manage identified risks, training staff and establishment of facilities to ensure the utility can respond effectively to the incident.

Response The issue of warnings and establishment of processes for effective notification of incidents and the mobilization of resources to combat the incident or threat.

Recovery The return to normal operations, implementing lessons learned from the response process.

It is assumed that in writing this report that utilities and agencies that will become engaged in DPR (from source through to tap) will have an incident and emergency management system or systems already in place which will form the framework to incorporate important elements that are important and/or unique to DPR. The following information is designed as guidance for elements that should be included in planning.

3.9.2 Incident Types and Categories

Incidents may be related to safety, environment, community relations, asset failure, and security of assets, operational non-conformance or water quality. In the case of DPR, CCP failures and water quality incidents are two significant incident types that must be included to assist in developing effective responses.

The severity of incidents is categorized based on the level of management and the resources required for controlling the incident, mitigating impacts and returning the system to normal operations. Organizations often have a number of incident levels or categories. This may be similar to the example below which includes: No Impact – Incidents that do not cause any injury, illness and/or damage but are reportable within

the operations team to ensure lessons are learned and systems/processes are improved to lower the risk a greater incident occurring in the future.

Minor Incident – Incidents that can be managed as part of routine operations under the control of an operator or supervisor within an operating facility.

Moderate Incident – Incidents that require a coordinated response by operators, led by a responsible manager and which may require specialist assistance. If more than one entity in a DPR system (i.e., wastewater plant and advanced treatment plant) are impacted.

Major Incident – Incidents that require a coordinated response by operators led by a responsible manager that does require specialist assistance.

Crisis – A crisis more significant than a major incident and will cause or have the potential to cause:

104 Water Environment & Reuse Foundation

o Death or serious risk of injury. o Contamination of drinking water supplies by microorganisms as the result of failure of the DPR

system. o Serious impact to the environment caused by malfunctioning of a facility or service. o Serious act of willful misconduct. o Threat to public health and continuity of supply. o Adverse attention by media.

Table 3.24 provides an example of incident categorization for a selected range of incident categories including Health and Safety, Environmental, Water Quality and CCP. Other areas that would be considered for a DPR operational system will include asset failure, commercial, criminal or security activity or property damage.

105 Water Environment & Reuse Foundation

Table 3.24. Examples of Incident Categories for Some Operational Areas

Incident Category No Impact Minor Incident Moderate Incident Major Incident Crisis Health and Safety Report only (no

treatment given). Near miss that would have resulted in a first aid injury.

First aid injury. Near miss that would have resulted in a medical treatment injury.

Medical treatment injury. Near miss that would have resulted in a lost time injury.

Lost time injury. Near miss that would have been life threatening. Injury to the general public.

Fatality or multiple serious injuries. Lost time injury resulting in permanent disability.

Environmental Near miss. Environmental incident has minimal impact to land, water or air. Minor or minimal clean up. Toxic/flammable release, spill or leak that is contained or controlled.

Environmental incident requiring clean up but no long term impact. Toxic/flammable release, spill or leak that is contained or controlled with minimal impact to land, water or air. Impact on sensitive flora and fauna.

Environmental incident requiring substantial clean up. Toxic/flammable release, spill or leak impacting land, water or air. Reportable to environmental authority.

Environmental incident requiring ongoing clean up and/or monitoring. Fire or toxic/flammable release, spill or leak requiring major emergency services and major multiple external agency response Evacuation of site, neighbors. Possible threat to life

Water Quality Advanced treatment product water chemical parameter > MCL/treated water target and not supplied to distribution.

Advanced treatment product water chemical parameter > MCL/treated water target and supplied to distribution. One microorganism count confirmed in advanced treatment product water and not supplied to distribution.

Advanced treatment product water chemical parameter 95th percentile > MCL/treated water target and supplied to distribution. One microorganism count confirmed in advanced treatment product water and supplied to distribution.

Advanced treatment product water microorganism 98th percentile > 1 and supplied to distribution.

CCP CCP alert condition raised.

CCP critical limit exceeded on one process unit in parallel train of multiple units. Automated CCP response works correctly.

CCP critical limit exceeded on single unit or entire train of parallel units. Automated CCP response works correctly. Product water not supplied to distribution.

CCP critical limit exceeded on single unit or entire train of parallel units. Automated CCP response does not stop operation or prevent water from proceeding to supply. Product water supplied to distribution/process unit does not shut down automatically.

Multiple CCP critical limits exceeded. Product water supplied to distribution/process units do not shut down.

106 Water Environment & Reuse Foundation

The risk assessment process, identified earlier in this chapter, provides a methodology to identify potential hazards and risks that could impact the successful operations of the DPR system. Each of the Incident Categories noted above should be linked to a specific risk assessment. For example, the Health and Safety category is linked to the Health and Safety Risk assessment, Water Quality and CCP are linked to the source water quality risk assessment. As noted in Section 3.2 the risk assessment process is an ongoing process that requires regular reviews and incorporates new information and conditions as they evolve. Any additional incidents that occur should be re-evaluated via the risk process to identify if the mitigations in place are suitable.

3.9.3 Incident Response

3.9.3.1 Timely Communication Effective incident response requires timely communication to ensure that all parties are able to support in that response. An incident and emergency management contact list is an important requirement to be clear on names, telephone numbers and email addresses that must be used in the case of an appropriate level of incident. These must be updated on a regular basis to ensure they are current.

The time frame required for notification is significant, especially as the severity of the incident category increases. It must be clear who needs to be contacted, and by when. For minor incidents this may be a follow up with supervision and management within a day or a few days, but for more significant incidents this must be done in a shorter timeframe to ensure an effective response. For example, an incident reporting timeframe may be as follows in Table 3.25. Table 3.25. Incident Category and Reporting Time Frame

Incident Category Reporting Timeframe

No Impact Operations manager – within one week. Shift supervisor – within 24 hours.

Minor Incident Operations manager – within one week. Shift supervisor – within 24 hours.

Moderate Incident Operations manager – within 24 hours. Shift supervisor – immediately. Stakeholder representatives across operational interfaces (WWTP – advanced plant or advanced plant to drinking water/distribution – within 24 hours if impacted.

Major Incident Operations manager – immediately. Shift supervisor – immediately. Stakeholder representatives across operational interfaces (WWTP – advanced plant or advanced plant to Drinking water/distribution) – immediately.

Crisis Operations manager – immediately. Shift supervisor – immediately. Stakeholder representatives across operational interfaces (WWTP – advanced plant or advanced plant to Drinking water/distribution) – immediately.

This table provides an example only. Incident notifications will be developed for specific organizations and reflect their organizational management structure.

107 Water Environment & Reuse Foundation

3.9.3.2 Incident Management Teams Responding to an incident requires a range of skills, expertize and management capability, dependent upon the nature and severity of the incident.

Minor incidents can be managed by one or a small number of staff locally to the incident. Moderate incidents, major incidents and crises will require the establishment of a site based incident management team, with a more formal process in place including:

Identify an individual as the incident manager to provide overall coordination of the response.

Identify other operational and specialist members of the incident management team. This team should identify those responsible for the technical response, as well as important functions of overall site management and communications.

Distribute incident management team checklists to key incident management team members.

Deal with the incident immediately to limit any impact on staff, operations, the community, the environment or public health.

Ensure that the best qualified external resources are called to respond to the incident (for example fire or toxic waste spill teams).

Develop an overall response strategy to the incident. For each site related to the DPR system, a set of instructions should be developed that cover the majority of potential incident events that can occur. This can then provide guidance on the steps to deal with the technical aspects of the incident. For example, CCP events have specific operational response procedures which act as incident response instructions (Section 3.5). Specific response procedures for each CCP identified have been included in Chapter 4.

Notify relevant management.

Coordinate the response to all communications.

After the incident is brought under control and clean up/recovery can begin it is essential that an incident debrief is carried out. Feedback should be used to improve incident and emergency management plans and the individual incident response instructions.

The following pages (Table 3.26) provide a summary of how individuals, incident management teams (IMT), and crisis management teams (CMT) may respond to various events at an operating facility.

108 Water Environment & Reuse Foundation

Table 3.26. Incident Response for Individuals and Management Teams

No Impact/Minor Moderate/Major Crisis

Individual response required when:

An incident can be controlled quickly by normal staff.

No specialist resources or equipment is required

No emergency services assistance is required

There is no effect on outside parties including neighbors, the community, contractors, other stakeholders

Site Incident Management Team (IMT) activated when:

Site staff cannot cope and specialist advice and support are (perceived to be) required.

Major interest is shown by the authorities and media, or potential for public outrage

There is a threat to the Project with potential implications for relations with key stakeholders.

An incident cannot be controlled quickly by staff Specialist resources or equipment are required. The emergency services are called to assist. Outside parties are, or could be, affected, including

neighbors, community or the public.

A Crisis Management Team (CMT) is activated when:

There is a real potential for an Incident to have a broader impact on the Project or Utility/Agency.

There are multiple injuries/fatalities, threats to life, significant damage to property, environment, or major potential impact on financial results.

Site Resources cannot cope and specialist advice and support are (perceived to be) required.

Major interest is shown by the authorities and media, or potential for public outrage.

There is a threat to the project and/or utility reputation, with potential implications for relations with key stakeholders.

Individual response is:

Handled by most qualified individual, e.g., first aid attendant, operator, fitter, etc.

Based on site No specialist training or equipment

required in response

The Incident Management Team is:

Led by Site Incident Manager, with designated support officers handling operations, communications, planning, support and administration issues.

Based at the designated incident room in the main office or other location as nominated by the Incident Manager or delegate.

Supplemented by designated functional specialists and other advisors as appropriate.

Specially trained and equipped for spill containment, first aid, firefighting, etc.

Crisis Management Team is:

Led by the Crisis Controller, with designated support officers handling operations, communications, planning, support and administration issues.

Based at the designated incident room in the main office or other location as nominated by the Incident Manager or delegate.

Supplemented by designated functional specialists and other advisors as appropriate

Individual shall:

Handle the incident onsite, coordinating internal resources for response.

Notify the Supervising Operator and ensure normal incident reporting procedures are followed.

Site IMT shall:

Assess the actual / potential impact on the operations and develop an overall response strategy

Coordinate functional inputs such as legal, insurance, safety and environment

Treat direct incident, with emphasis on human life Assist emergency services Restore control and effect recovery. Deal with local issues such as neighbors and authorities. Keep staff, Management and other stakeholders informed. Assess the impact on the business and develop an overall

response strategy. Coordinate corporate functional inputs such as legal,

insurance, safety and environment.

CMT shall: Determine if a Crisis Alert has been triggered and act

accordingly Assess the impact on the business and develop an overall

response strategy Provide support to the site on the form of advice, resources

and assess need for further input Handle major external communications e.g., with client or

customer, Govt Departments, authorities, councils, media, “relatives”, contractors, pressure groups, etc.

Coordinate corporate functional inputs such as legal, insurance, safety and environment

Notify key stakeholders and staff

109 Water Environment & Reuse Foundation

3.9.4 Incident Investigation

In order to continually improve operations and minimize the risk of recurring incidents, a thorough investigation of incidents should be conducted. Incident investigation should proceed in a timely fashion. The following incidents should commence investigations within a short timeframe of occurrence: Any near miss incidents or incidents resulting in injury. Environmental incidents. Major process breakdowns. Recurring incidents. Any water quality results exceeding MCL or other defined water quality target. Any CCP critical limit exceeded.

Root cause analysis should be applied to all incident investigations. This involves examining the cause and effect chain of events leading to an incident. The main steps are: Working backwards in the system, starting from the incident, identify all the possible causes relating

to environment, training, procedures, communication, hazards, work behavior and equipment. From a list of all possible causes determine the major cause. Analyze the major cause. Keep asking “why did this happen” until the root cause or causes are

determined. Identify the corrective steps for the major cause and for all contributory causes.

3.9.5 Incident Prevention and Preparedness

A number of initiatives can be used at the operating facilities to prevent incidents. These include: Planned inspections of the operating facilities. Appropriate and regularly refreshed training. Comprehensive induction training for new staff, contractors visiting site, and all visitors. Effective work permit systems. Preventative measures and control actions.

Incident management training in particular is of great importance to underpin success in the management of incidents. It will enable staff to: Effectively manage incidents within the operating facility. Effectively coordinate incident response with stakeholders. Recognize the diversity of functions within incident management. Describe the concept of comprehensive incident management. Obtain a perspective of roles and functions. Accept the roles and functions of other organizations during major incidents.

Training should be provided early in the engagement of new staff, with regular refresher and scenario training provided at regular intervals. Incident management training should be an integral part of operational staff training.

110 Water Environment & Reuse Foundation

3.9.6 Scenario Training

Scenario training is particularly valuable training for incident management. By running exercises, staff can be well prepared for the management of incidents. These scenarios provide the benefits and outcomes of: Reviewing existing processes, plans, and procedures. Increasing awareness of processes, plans, and procedures. Identifying resource shortfalls and limitations. Improving coordination and clarifying roles and responsibilities.

A typical frequency for scenario training is included in Table 3.27. Table 3.27. Training Types and Frequency

3.10 Change Management

3.10.1 The Change Management Process

Change management is the process of handling material (equipment/process/staff) change systematically to minimize and mitigate any potential risks that result to a change to equipment, operational procedures, documentation or operational planning. The objective is to adequately manage change through four major change management steps of initiation, assessment, authorization and implementation. This is done through: Establishing a clear mechanism for dealing with change issues as they arise through the performance

of DPR facilities. Enforcing an evaluation of a proposed change in terms of its benefit, its cost and its risk. Managing and tracking changes, and enabling all stakeholders to assess the impact and risk of the

change to operations. Ensuring that all changes undertaken have been authorized.

Exercise/training type To be undertaken by: Frequency:

Minor Incidents Drills and testing of alarms and evacuation processes

Operations/Project Team lead by Supervising Operator

2 per year

Major Incidents Desktop exercise to test interfaces and understanding

Operations/Project Team led by a Potential Incident Manager

1 per annum

Major Incidents On site major exercise involving external emergency services

Operational Staff. 1 per annum

Critical Control Point Response Procedures/Events.

Operational Staff 6 per year

111 Water Environment & Reuse Foundation

3.10.2 Change Management Procedure

A change management procedure should be developed and used by any party (operations team, contractors, utilities) requesting changes to the facilities, systems and processes including: Changes to plant and equipment. Civil/building work changes Control system hardware changes. Control system software changes Laboratory and sampling and analysis changes. Standard operating procedures. Documentation changes. Software version upgrades (e.g., HMI software upgrade) or enhancements. Planned/scheduled outages.

The procedure should include the following elements: That the initiator of the change outline the details of the proposed change, reviewing the likely impact

of the change, and any risks associated. A classification of the change in terms of impact. An example of possible change categories are listed

in Table 3.28. Table 3.28. Examples and Definitions of Change Categories

Change Category Definition Examples Minor Change A change that does not significantly

impact on operations. Often with a relatively low cost.

Change to operating setpoints with minor process and operating impacts. Minor corrections/enhancements to chemical dosing systems or analyzer racks. Minor software upgrades.

Significant Change A change that has a significant impact and presents some risk to operations.

Change to CCP setpoints or other major operational setpoints. Introduction of a new analyzer type. Any change that impacts an operational interface. Requires re-training of some operational staff.

Major Change Major change that requires considerable planning and risk management.

Major change to plant or CCP process (for example new process added). Training of a large number of employees is required. Major cost of implementation.

Emergency Change In some cases changes must be done very quickly. This category enables a fast response when this is required. For these changes, sufficient approval can be obtained in the form of verbal or email instruction from appropriate authorized person.

Response to equipment failure that may result in extended loss of production. Urgent change to ensure plant water quality can return to compliance.

112 Water Environment & Reuse Foundation

A list of authorizations required for each change. This will be dependent on the nature of the change, and the change category as listed above. Development of an authorization matrix is useful to outline which level of operational management is required to authorize each change. The technical nature of the change should highlight the appropriate level of technical capability (for example a SCADA change should require a controls engineer review, changes to sampling and analysis will require a water quality manager). In addition, the authorization will require an appropriate level of operational/financial responsibility to provide approval.

Implementation of the change. Communication with stakeholders. The stakeholders impacted must be identified and outlined in the

details of the proposed change, and communication prior to, during and after the change has been completed is required.

Update to all relevant documents. This involves updates to technical drawings, operating procedures, plans, schedules, PLC and SCADA files. The change process is only completed when systems are fully updated.

An example change management process, for an identified major change, is shown in Figure 3-38.

113 Water Environment & Reuse Foundation

Figure 3.38. Example Change Management Process for Major Change

114 Water Environment & Reuse Foundation

3.11 Operating Manual and Procedures

3.11.1 Operation and Maintenance Plans

Each of the treatment systems comprising the DPR system will include an operations and maintenance manual that has either been developed as part of the plant construction, or has been developed by the facility.

In the case of Orange County Water District’s Groundwater Replenishment System (GWRS), an operations, maintenance, and monitoring plan (OMMP) has been developed and submitted to the California Department of Drinking Water (DDW). The OMMP requires annual review and, when updated, is submitted to the DDW and Regional Water Quality Control Board for review and approval.

Details of GWRS and OMMP are included in the case study in the following section.

3.11.2 Standard Operating Procedures Standard operating procedures (SOPs) provide clear instructions for operational staff to cover the full range of operational tasks that are performed at each DPR facility. These contain step by step instructions on how to operate the plant effectively and safely to achieve the required outcomes. SOPs for plant operation generally fall in to one of five main groups: Equipment SOPs – These are used for general equipment operation and routine maintenance items. Monitoring SOPs – These outline the monitoring requirement for process, plant and equipment. Sampling SOPs – These are incorporated into the sampling and analysis management planning

process, and detail sampling methodology, sample storage, transport and chain of custody requirements.

Analysis SOPs – These are also incorporated into the sampling and analysis management planning process and cover analysis techniques for analysis that is conducted at the site.

CCP response procedures – These take the form of a flow chart. A full set of CCP response procedures was developed as a part of Reuse-13-03, and have been included as a part of Chapter 4.

SOPs are often developed initially as a part of the overall plant O&M manual at the time of plant construction by engineers, construction personnel, and equipment vendors. Nonetheless, it is highly recommended that standard operating procedures be further developed and managed by the operations team – to ensure that there is a consistent format, and to make sure that changes to plant and equipment and lessons learned from operation can be incorporated. This may include reference to, or the incorporation of other documents (for example vendor equipment manuals) however care should be taken to ensure that information from external sources remains up to date. Reliance on the external material alone, without reference to a site specific SOP, is not recommended.

SOPs should be managed by the change management process (see Section 3.10) with high importance on document control to ensure that all SOPs operated in the facility are up to date, and superseded SOPs are not inadvertently used. Maintaining online, electronic SOPs is a useful strategy for managing document version control, although operations must remain vigilant as paper copies invariably are used and may not be effectively updated.

A standard template is useful to assist in easier development of plant SOPs as well as familiarity with operational staff. The following sections, taken from SOPs used at the Beverly Hills Water Treatment Plant in California, provide guidance on potential document layout. The sections include:

Purpose – This is a clear, brief description of the intent of the SOP.

115 Water Environment & Reuse Foundation

Scope – Identifies the scope. What does this SOP cover (and what are the limits of what it covers) Definitions – This is important, as with any field of endeavor or technology there is jargon and

terminology which may become a barrier to clear understanding. Clear definition of technical and other terms used in the SOP is important.

Responsibility and authority – this clearly identifies which staff will be responsible for the task itself and scheduling of the task.

Task Description – This can provide an overview and context of the task providing context and background.

Safety – This provides important safety requirements for the task to protect staff and other personnel. It may include personal protective equipment to be used, information on important isolations/lock-out tag-outs and other considerations.

Procedures – The clear step by step instructions. For both this section and the Task Description, images, diagrams and photographs are very helpful to ensure clear descriptions and minimization of ambiguities.

Normal Operating Setpoints – May be included for process units, to provide guidance. Records – Covers the information that should be recorded during or at the completion of the

procedure and where these records are to be stored. Document Control – Provides the current version of the document, along with relevant responsible

signatories.

3.11.3 Unit Process Guidelines Unit process guidelines (UPGs) provide a higher level overview of process plant and equipment. These complement SOPs, but provide more overarching detail of operation. The purpose of a UPG is to: Describe the theoretical basis for the operation of the process unit. Define the validation criteria that are used to check that the process is operating correctly and

achieving the objectives. Describe the process limiting factors that affect the process. Describe the monitoring requirements for the process. Outline strategies to control and optimize the process to achieve the desired objectives. Provide a trouble-shooting guide for the process unit.

These procedures also contain a description of all equipment and each system and subsystem, and include a summary of the function and design specification of the equipment.

116 Water Environment & Reuse Foundation

3.12 Case Study – Operation and Maintenance of Groundwater Replenishment System

3.12.1 Project Description

The Groundwater Replenishment System (GWRS) is a water supply project jointly sponsored by Orange County Water District (OCWD) and Orange County Sanitation District (OCSD) that supplements existing water supplies by providing a new, reliable, high-quality source of water to recharge the Orange County Groundwater Basin (the Basin), to protect it from degradation due to seawater intrusion..

The GWRS is located in central Orange County, California, and operated by OCWD. The GWRS is an indirect potable reuse project and consists of four major components:

The Advanced Water Purification Facility (AWPF), which is located in Fountain Valley and features treatment processes and pumping stations;

The Talbert Seawater Intrusion Barrier (Talbert Barrier), which is located in Fountain Valley and Huntington Beach, and is comprised of a series of injection wells and pipelines that are supported by an extensive network of groundwater monitoring wells;

The Kraemer-Miller-Miraloma Basins along with other nearby spreading basins, which are located in Anaheim and supplied by the GWR Pipeline, and are supported by numerous groundwater monitoring wells; and

The Demonstration Mid-Basin Injection (DMBI) Project, which is located in Fountain Valley and Santa Ana, and is comprised of one test injection well supplied by the GWR Pipeline and two groundwater monitoring wells.

The AWPF receives secondary-treated wastewater from OCSD Reclamation Plant No. 1 and treats it to better than drinking water standards using microfiltration (MF), reverse osmosis (RO), advanced oxidation/disinfection consisting of hydrogen peroxide addition and ultraviolet light exposure (UV/AOP), followed by decarbonation and lime stabilization. Pumping stations and pipelines convey the purified recycled water to the Talbert Barrier, Kraemer-Miller-Miraloma Basins, and/or industrial users.

The original AWPF began operation in January 2008 and was designed to produce 70 million gallons per day (MGD), or approximately 72,000 acre-feet per year (AFY) of highly treated recycled water based on a greater than 90% online factor. In 2015, the initial expansion was completed, increasing the AWPF design production capacity up to 100 MGD, or approximately 103,000 AFY based on a 90% online factor. Historically, about half of the purified recycled water was injected at the Talbert Barrier and the other half was percolated at Kraemer-Miller-Miraloma Basins. The flow split varies depending on the needs of the Basin with roughly one third of the flow currently injected into the Talbert Barrier and the balance percolated through the spreading basins in Anaheim. OCWD have advanced internal laboratory capabilities to analyze a large range of microbial and chemical constituents.

The Talbert Barrier consists of a series of 36 injection well sites that are supplied by pipelines from the AWPF Barrier Pump Station. OCWD constructed the injection barrier to form an underground hydraulic mound, or pressure ridge, that helps prevent seawater intrusion near the coast in the Talbert Gap area. Without the Talbert Barrier, seawater would migrate inland and contaminate the fresh groundwater supply of the Basin. In addition to providing seawater intrusion control, the Talbert Barrier also injects purified recycled water into the deeper Main aquifer with the primary purpose of replenishing the Basin. Potable water may also be injected at the barrier, although blending is not required.

GWRS purified recycled water and other waters are percolated at Kraemer-Miller-Miraloma Basins. Other waters may include Santa Ana River (SAR) captured storm flow, SAR base flow, and purchased

117 Water Environment & Reuse Foundation

imported water. GWRS recharge at Kraemer and Miller Basins began in 2008 along with start-up of the original GWRS components. Miraloma Basin began spreading purified recycled water in 2012. Purified recycled water is conveyed from the AWPF to these three spreading basins by the 13-mile GWR Pipeline along the west levee of the SAR. While purified recycled water recharge is restricted to Kraemer-Miller-Miraloma Basins, other waters may be recharged at those three basins as well as nearby spreading basins, Anaheim Lake, Mini-Anaheim Lake, and La Jolla Basin. Blending of purified recycled water with other waters is no longer required.

The DMBI test injection well began recharging the Main aquifer with GWRS purified recycled water in 2015. The purpose of the DMBI Project is to determine the feasibility of replenishing the heavily pumped Main aquifer with purified recycled water in the center of the groundwater basin. Depending on the outcome of the DMBI Project, OCWD plans to integrate the DMBI facilities into a future multiple mid-basin injection well system that would increase replenishment in an area of high groundwater production.

Besides these major components, GWRS also serves industrial customers purified recycled water for cooling water and industrial uses.

3.12.2 Permits

OCWD operates GWRS, including the AWPF, Talbert Barrier, Kraemer-Miller-Miraloma Basins, and industrial uses under the requirements of the “Producer/User Water Recycling Requirements and Monitoring and Reporting Program for the Orange County Water District Interim Water Factory 21 and Groundwater Replenishment System Groundwater Recharge and Reuse at Talbert Gap Seawater Intrusion Barrier and Kraemer/Miller Basins” adopted by the RWQCB as Order No. R8-2004-0002 in 2004, and two subsequent amendments, RWQCB Order No. R8-2008-0058 in 2008, and RWQCB Order No. R8-2014-0054 in 2014. The latter amendment increased the allowable purified recycled water capacity up to 100 MGD and incorporated Miraloma Basin. Collectively, these RWQCB Orders comprise the permit for the GWRS. The permit incorporates groundwater recharge criteria, findings and conditions, and recommendations from the DDW.

OCWD also maintains a separate permit for the GWRS emergency discharge to the SAR, “Waste Discharge Requirements for the Orange County Water District Groundwater Replenishment System Advanced Water Purification Facility to Reach 1 of the Santa Ana River”, adopted by the RWQCB as Order No. R8-2014-0069, NPDES No. CA8000408, in 2014. During peak wastewater flow events, the AWPF can provide hydraulic relief for the OCSD ocean outfall by discharging up to 100 MGD of microfiltered, UV-disinfected recycled water (bypassing RO) to the SAR. With completion of the initial expansion, the AWPF can provide hydraulic relief for the outfall by continuing normal operation and production of up to 100 MGD of purified recycled water.

Source water for GWRS is secondary effluent from OCSD Reclamation Plant No. 1 located in Fountain Valley adjacent to the AWPF. OCSD operates the wastewater facilities under the requirements of “Waste Discharge Requirements and National Pollutant Discharge Elimination System Permit for Orange County Sanitation District Reclamation Plant No. 1 and Treatment Plant No. 2”, adopted by the RWQCB as Order No. R8-2012-0035, NPDES No. CA0110604, in 2012.

Each of these permits reference the other aforementioned permits, effectively creating regulatory requirements for each element as well as the collective GWRS program.

118 Water Environment & Reuse Foundation

3.12.3 Interagency Agreements and Policies OCWD and OCSD are committed to working together on GWRS as documented in an Agreement entitled “Amended Joint Exercise of Powers Agreement for the Development, Operation and Maintenance of the Groundwater Replenishment System and the Green Acres Project” that was executed on May 5, 2010. This Agreement updates and amends the original November 12, 2002, Agreement between the two agencies. (“Joint Exercise of Powers Agreement for the Development, Operation and Maintenance of the Groundwater Replenishment System and the Green Acres Project”)

The original 2002 Agreement identified each agency’s responsibilities pertaining to the GWRS design, cost sharing, operational and maintenance obligations. It established property leases whereby OCWD agreed to lease OCSD property for GWRS facilities for a nominal sum ($1.00). The 2002 Agreement established the GWRS Steering Committee, which is comprised by designated members of Boards of Directors of both OCWD and OCSD, for the purpose of facilitating development of GWRS. At that time, ultimate authority for the planning, design and construction of GWRS rested with the Boards of Directors of OCWD and OCSD. The Agreement established cost sharing whereby OCSD reimbursed OCWD for 50%, less grant receipts, for planning, design and construction of the initial (70 MGD) GWRS facilities. With regard to operation and maintenance, the 2002 Agreement stated that OCWD shall at its sole expense operate, maintain and be responsible for regulatory compliance for GWRS. Furthermore, the original Agreement defines acceptable source water quality and flow rates (treated secondary wastewater) delivered by OCSD from Plant No. 1 to GWRS. OCSD also agreed to accept waste streams from GWRS.

The updated 2010 Agreement amended the prior agreement and maintained the continuity of the working relationship between the governing bodies of OCWD and OCSD. The 2010 Agreement added provisions for the GWRS initial expansion, operation of wastewater facilities, including effluent flow and quality, and operation of recycled water facilities, including peak flow relief. The Agreement includes provisions for a comprehensive industrial pretreatment and source control program to support indirect potable reuse. Financial responsibilities are specified, as well as permitting and regulatory reporting duties are assigned. In summary, the 2010 Agreement memorializes the understanding between OCWD and OCSD under which the GWRS is operated and maintained.

3.12.4 GWRS Operation and Maintenance: Staff Benchmarking As of December 2015, OCWD’s GWRS operations staff includes a total of 22 operators with wastewater and water treatment certifications of various grades: Table 3.29. Operator Staffing for OCWD’s GWRS Grade Level Number of Operations Staff (as of December 2015)

5 7

4 3

3 7

2 2

1 -

T-4* 1

OIT-II 1

OIT-I 1

Total 22 *Water treatment certification. All others are wastewater certifications.

119 Water Environment & Reuse Foundation

The GWRS Operation Optimization Plan (OOP) describes the operating parameters, critical control points, maintenance schedules and troubleshooting guides for the AWPF, injection barrier, and spreading basins. The OOP is an update of the Operation, Maintenance and Monitoring Plan (OMMP) required by the GWRS permit, initially approved in 2008 and updated in 2010.

The purpose of the OOP is to provide guidance for operating, maintaining, and monitoring the GWRS and for training personnel. The OOP is divided into sections focusing on specific process areas of the facilities:

OOP Section Description 1 Project Overview 2 Air Gap Structure and Pump Station 3 Influent Screening Facilities 4 Secondary Effluent Flow Equalization 5 Microfiltration 6 Green Acres Project (GAP) 7 Chemical Storage and Feed Systems and Cartridge Filters 8 Reverse Osmosis System 9 Advanced Oxidation / UV Disinfection Process

10 Decarbonation / Post Treatment (Lime Stabilization) 11 Product Water (Finished Water) Pumping Facilities 12 Substation / Switchgear Building 13 Kraemer-Miller-Miraloma-La Palma Spreading Basins 14 Talbert Barrier Injection Wells 15 Plant Utilities 16 Process Control System 17 Water Quality Monitoring 18 Staffing, Quality Assurance and Contingency Plans

Actual day-to-day operation, maintenance recording, and performance monitoring of GWRS facilities are accomplished by OCWD staff using the Process Control System (PCS). The PCS is a distributed control system that provides for operational control, status monitoring, and data collection for the AWPF processes. The PCS allows for the transmission of equipment status, alarms, process control information to the Main Control Room at the AWPF. It monitors online water quality and process performance, triggering alarms as necessary and allowing the operator to view the status or make operational adjustments.

OCWD staff uses the Online Operations and Maintenance Manual (OLM) for operational information about treatment processes, systems within processes. Routine testing procedures, maintenance and calibration schedules, and other detailed information can generally be found in the OLM or in Standard Operating Procedures (SOPs) developed by OCSD operations staff for specific processes or systems. New SOPs are developed and existing SOPs are updated as the need arises. OCWD operations management staff maintains the “library” of SOPs as a source of information for operation of GWRS. Maintenance and calibration work is tracked by the Computerized Maintenance Management System (CMMS).

The OOP and OLM are used together. In some areas, the OOP and OLM overlap; in other areas, the OLM provides more detailed step-by-step instructions for the operator. The OOP is reviewed, updated as necessary and submitted to DDW and the RWQCB in accordance with the GWRS permit. The OLM can be revised by authorized personnel at any time based on operating experience. OCWD’s goal is to use the

120 Water Environment & Reuse Foundation

SOPs and CMMS as references to update the OLM on a frequent basis so that it contains up-to-date information. Collectively, all these documents are valuable tools, but they should not replace operator experience and judgment, and safety should always take precedence over the guidelines in the OOP, OLM, SOPs, and CMMS.

The Joint Operations Committee, which consists of OCWD and OCSD staff, meets on a regular basis to communicate operations and maintenance plans, implement joint policies and procedures, cross-train operations staff, and address issues that arise to optimize both the GWRS and Plant No. 1 facilities. Reports on the Joint Operations Committee activities are presented to the OCWD and OCSD Boards of Directors.

Examples of the cooperation fostered by the Joint Operations Committee include development of the following Joint Standard Operations Procedures (SOPs), which address issues shared by both OCWD and OCSD:

Screening Facilities – Describes operation of the AWPF influent screening facilities that receive secondary effluent from Plant No. 1 as feedwater to the AWPF.

Reject Streams – Describes operating procedures for many AWPF reject streams that are returned to OCSD, including MF backwash and cleaning waste, RO concentrate, cleaning and flush waste, lime sludge, and other miscellaneous drains.

Miscellaneous Wastes – Describes procedures for miscellaneous waste streams from other OCWD facilities (e.g., Green Acres Project and injection well redevelopment waste) returned to Plant No. 1.

Chemical Deliveries – Provides instructions for chemical delivery truck access and routing through the Plant No. 1 site to the AWPF.

Bypass to 66-inch Interplant Line – Describes procedures for the AWPF to discharge to the OCSD 66-inch diameter Interplant Line, which conveys wastewater from Plant No. 1 to Plant No. 2.

Santa Ana River Discharge – Describes operation of the AWPF in the emergency peak flow mode to provide hydraulic relief for the OCSD ocean outfall.

3.13 Operator Training Operator training and certification is the subject of Chapter 6 of this report.

3.14 Transparency and Auditing

3.14.1 Internal Auditing

Internal and external auditing provides a valuable check on performance by ensuring that operations management is regularly assessed in a consistent and systematic manner.

Internal audits are carried out by personnel within the operating facility or utility organization. These are self-checks that assist to verify: The effectiveness of operational systems to meet all of the DPR system objectives. The adequacy of implemented controls to minimize high risk activities. Compliance with permits, codes, legislation and industry standards. Whether the stated objectives in operating plans and documents are being met Identify areas for ongoing improvement.

Internal audits are best conducted by personnel that are not directly or regularly involved in the subject matter. This can provide a level of objectivity in comparing plans and documentation with actual operational management performance.

121 Water Environment & Reuse Foundation

3.14.2 External Auditing

External audits provide a level of transparency to an organization and its operational management. Some organizations adhere to a range of standards which provide an organizational framework which can be independently audited. ISO 9000 quality assurance provides a framework for quality within an organization overall. Both the Western Corridor Recycled Water Project and Santa Clara Valley Water District are certified to this standard for the operation of their advanced reuse treatment facilities. ISO 22000, the Food Safety Standard incorporates HACCP principles, and as such is also a useful standard for accreditation. The Western Corridor Project is also accredited to this standard, and Singapore’s Newater is accredited to a similar Singapore-based standard.

3.15 References (2008). Recylced Water Management Plan and Validation Guidelines. Australia, State of Qld, Dept of Energy and Water Supply. https://www.dews.qld.gov.au/__data/assets/pdf_file/0020/45164/rwmp-and-validation.pdf

Bartram, J., L. Corrales, A. Davison, D. Deere, D. Drury, B. Gordon, G. Howard, A. Rinehold, and M. Stevens (2009). Water Safety Plan Manual: Step-by-Step Risk Management for Drinking Water Suppliers. World Health Organization (WHO), Geneva.

California Code of Regulations (CCR 60020.106 Wastewater Source Control).

CDPH (2014). DPH-14-003e Groundwater Replenishment Using Recylced Water, Tilte 22, California Code of Regulations. SWRCB. http://www.cdph.ca.gov/services/DPOPP/regs/Pages/DPH14-003EGroundwaterReplenishmentUsingRecycledWater.aspx

Frenkel, V. and Y. Cohen (2014). New Techniques for Real-Time Monitoring of Membrane Integrity for Virus Removal: Pulsed-Marker Membrane Integrity Monitoring System (WRRF-09-06b). WateReuse Reserach Foundation, Alexandria, VA.

Halliwell, D., D. Burris, D. Deere, G. Leslie, J. Rose, and J. Blackbeard (2015). Utilization of HACCP Approach for Evaluating Integrity of Treatment Barriers for Reuse. WateReuse Research Foundation, Alexandria, VA.

ISO (2005). ISO 22000: Food Safety Management. International Standards Organization, Geneva, Switzerland.

ISO (2009). ISO 31000:2009, Risk Management--Principles and Guidelines. International Standards Organization, Geneva, Switzerland.

Jacangelo, J. (Ongoing Research). Standard Methods for Integrity Testing and Online Monitoring of NF and RO Membranes (WRRF-12-07). WateReuse Research Foundation, Alexandria, VA.

Khan, S.J. (2010a). Chapter 6: Safe Management of Chemical Contaminants for Planned Potable Water Recycling. Issues in Environmental Science & Technology. Hester, R.E. and R.M. Harrison. Cambridge, UK, Royal Society of Chemistry RSC Publishing. 31: 114-137.

122 Water Environment & Reuse Foundation

Khan, S.J. (2010b). Quantitative Chemical Exposure Assessment for Water Recycling Schemes. National Water Commission, Canberra, ACT, Australia. Waterlines Report Series Number 27, March 2010: 207 pages.

McDonald, E. and M. Nellor (2015). TWDB Final Report: Direct Potable Reuse Resource Document. Texas Water Development Board, Austin, TX. Volume 1 of 2: 178 pages.

Park, M., D. Reckhow, M. Lavine, E. Rosenfeldt, B. Stanford and M.-H. Park (2014). "Multivariate Analyses for Monitoring EDCs and PPCPs in a Lake Water." Water Environment Research 86(11): 2233-2241.

Petrovic, M., E. Eljarrat, M.J.L. Alda, and D. Barceló (2004). "Endocrine Disrupting Compounds and Other Emerging Contaminants in the Environment: A Survey on New Monitoring Strategies and Occurrence Data." Analytical and Bioanalytical Chemistry 378(3): 549-562.

Salveson, A., E. Steinle-Darling, S. Trussell, B. Trussell, and L. McPherson (2015). Guidelines for Engineered Storage for Direct Potable Reuse: Final Report of Project Reuse-12-06. Water Environemtn & Reuse Foundation, Alexandria, VA.

Stanford, B.D., A.M. Reinert, E.J. Rosenfeldt, D. Dryer, M.-H. Park, and D. Reckhow (2014). "Sampling Frequency, Location, and Reporting Limit Influence on Benchmarking EDC/PPCPs (Pdf)." Journal American Water Works Association 106(9): E362-E371.

Trussell, R.R., A. Salveson, S.A. Snyder, R.S. Trussell, D. Gerrity, and B.M. Pecson (2013). Potable Reuse: State of the Science Report and Equivalency Criteria for Treatment Trains. WateReuse Research Foundation, Alexandria, VA: 276.

Vanderford, B., D. Mawhinney, R. Trenholm, J. Zeigler-Holady, and S. Snyder (2011). "Assessment of Sample Preservation Techniques for Pharmaceuticals, Personal Care Products, and Steroids in Surface and Drinking Water." Analytical and Bioanalytical Chemistry 399(6): 2227-2234.

Walker, T., B.D. Stanford, S. Khan, R. Valerdi, S.A. Snyder, and J. Vickers (2016). Critical Control Point Assessment to Quantify Robustness and Reliability of Multiple Treatment Barriers of a DPR Scheme (Reuse-13-03). Water Environment & Reuse Foundation, Alexandria, VA: 310.

123 Water Environment & Reuse Foundation

Chapter 4

Key Operational Procedures and Requirements

4.1 Introduction This project has focused on two proposed treatment approaches to DPR, namely:

RO based treatment train (Chloramine-MF-RO-UV/H2O2-Cl2). Non RO based treatment train (Ozone-Flocculation-Ozone-BAC-GAC-UV-Cl2).

As previously discussed in Chapter 3, and based on the work of project Reuse-13-03 (Walker, Stanford, Khan, et al., 2016), critical control points (CCPs) were identified for both process trains. As a part of the CCP review and in order to effectively manage water quality risks and meet targets for primary and secondary EPA and Californian MCLs, the non RO based treatment train was modified to include a coagulation/flocculation step ahead of biological activated carbon.

The two graphics below illustrate both processes including identified CCPs.

Figure 4.1. RO-Based treatment Train

124 Water Environment & Reuse Foundation

Figure 4.2. Non-RO-Based Treatment Train

This chapter provides a review of each of the identified CCPs, providing an overview of important elements for a DPR operational framework. Each section of the chapter is identified by the process train (e.g., “RO-Based Process”) and the critical control point that is discussed in that section (e.g., “Chloramination”). This includes:

A brief process overview. Key operating parameters for the process. General operating procedures that are required. Critical control point resources. Critical control point response procedures (developed as a part of Reuse-13-03). Key maintenance activities required. Operations focused design considerations.

4.2 RO-Based Process – Chloramination

4.2.1 Process Overview

Chloramination is used to provide a disinfection residual through the membrane treatment processes, particularly the reverse osmosis process, to minimize biological fouling. Reverse osmosis membranes will rapidly oxidize in the presence of free chlorine, however have a substantially higher tolerance to chloramine.

The biological fouling of membranes is not a health concern in itself, but rather there is a risk of chloramine dosing forming unwanted byproducts such as nitrosamines and other disinfection byproducts.

The approach to chloramine dosing varies from plant to plant, and depends in particular upon how much ammonia is in the secondary or tertiary effluent stream that feeds the advanced treatment process. In cases where there is no or incomplete nitrification, chlorine may be dosed directly into the effluent stream to form chloramine. This is the approach taken at the West Basin Municipal Water District’s Edward C. Little Recycled Water facility, Orange County’s Groundwater Replenishment Scheme, and the Scottsdale Water Campus. In other facilities where the effluent has been fully nitrified, ammonia (in the form of ammonium sulfate or ammonium hydroxide) is dosed to provide sufficient ammonia to form chloramine. This is the approach taken on the Western Corridor Recycled Water Project in Brisbane, Australia.

125 Water Environment & Reuse Foundation

In the case of the Bundamba Advanced Water Treatment Plant (one of the three Western Corridor Recycled Water Plants), the original dosing configuration of ammonium sulfate followed immediately by sodium hypochlorite was shown to result in an increase of NDMA (Walker and Roux 2009). The dosing approach was modified, whereby the ammonium sulfate and sodium hypochlorite were both dosed to form chloramine in a single carrier water stream of RO permeate, which was consequently dosed directly as chloramine. While the exact reaction kinetics and pathway were not studied in this case, empirical evidence showed that NDMA formation effectively ceased. In cases where ammonia and chlorine must be added to the effluent stream, this could be a recommended approach, though site specific reaction characteristics/chemistry should be investigated prior to implementing a potential solution at full scale.

Dosing is typically targeted to provide a dose of between 2 mg/L and 5 mg/L chloramine, measured as Cl2 at a chlorine-to-ammonia ratio of 4:1.. The target dose may vary slightly from plant to plant, as dictated by different warranty requirements determined by individual reverse osmosis membrane suppliers. The dose does not need to be fine-tuned, but rather control is aimed at maintaining an average dose (on a daily average basis) without exceeding a required maximum dose. Short term losses of chloramine dosing (a few hours) provide no additional health risk, however may increase the rate of membrane fouling and result in additional operating costs and/or a loss of plant production.

A higher than desired chlorine dose may increase the risk of unwanted byproduct formation. The impact of chlorine dose on byproduct formation will vary from plant to plant and may vary from day to day and the best approach in terms of maximum chlorine dose may be determined empirically at each plant with a program of sampling and analysis. This may assist in the appropriate chlorine target setpoint.

The protection of membranes from free chlorine oxidation is also required and may utilize free chlorine, ammonia, and oxidation reduction potential (ORP) analyzers to detect the presence of free chlorine. This is however related to protecting the membrane asset and is not directly related to mitigating a health risk and so is not considered an aspect of the CCP. Operationally this will be considered as a critical operating point or COP.

4.2.2 Key Operating Parameters

Chloramine is not intended to provide any significant log reduction of microorganisms and is not utilized as a CCP in this instance to achieve this. Rather, the intent is to minimize fouling but requires consideration towards minimizing the risk of unwanted byproducts. Set points are typically set to achieve a specified level of chloramine at the feed to the RO system while avoiding DBP formation. Total Chlorine – Total chlorine residual at the inlet to the reverse osmosis system is often the governing setpoint for operation. This is used to infer the chloramine dose. The chemical dose required to achieve this may vary depending on the chlorine demand of the inlet wastewater effluent. Chloramine – In some cases a chloramine (or total chlorine) analyzer can be used to measure chloramine directly, rather than inferring a chloramine dose. Some of these instruments can also provide an ammonia reading. These analyzers have been improving for use in this application in recent years and have been seeing increased use at advanced treatment plants. If no analyzer is used, spot checks of chloramine can be conducted with small on site test kits. Ammonia – At some plants ammonia is present in the wastewater effluent at sufficient levels for chloramination. At other plants which fully nitrify, ammonia may need to be added. Particularly in this latter case, or where ammonia levels may become low, on line monitoring of ammonia is useful.

126 Water Environment & Reuse Foundation

4.2.3 General Operating Procedures

The operation of chloramine dosing is performed by automated dosing of chlorine to the plant feed water, as well as ammonia if there is insufficient ammonia present.

General operating procedures should include: Regular check of chloramine by test kit (if no chloramine analyzer). SCADA review of ammonia trends (if available) to determine variation in influent ammonia. Review of chlorine dose to meet chlorine setpoint requirements. An increase in chlorine dose may be

indicative of increased feed water chlorine demand or stock chlorine solution degradation.

4.2.4 Critical Control Point Response Procedure

4.2.4.1 Chloramine Dosing Alert (Warning) Figure 4.3 illustrates the proposed Chloramine Dosing Alert (Warning) for the Chloramination step. In contrast for most of the selected CCPs, the health risk that is mitigated is a chronic chemical risk rather than an acute microbiological risk. As a result, actions that are required to be taken need not be as immediate which is reflected in the response procedures. In this case, a daily 24 hour or similar average period can be taken to assess this parameter.

4.2.4.2 Chloramine Dosing Alarm (Failure) Figure 4.4 illustrates the Alarm (failure) procedures for the Chloramine Dosing step. Critical failure alarms for chloramine processes are configured in a manner similar to the Alert (warning) status alarm. The primary difference is that a failure will result in a stoppage of the chloramine dosing, whereas an alert will allow for operator intervention before this controlled action takes place. As for the warning alarm condition, as this is not an acute health risk, a longer time period prior to failure alarm will be provided (Table 4.1). Table 4.1. Chloramine Alert and Alarm Example Setpoints

Monitoring Parameter Alert Level

Critical (Failure) Level Notes

Total Chlorine

Cl2 > 1 mg/L from setpoint 1hr moving average

Cl2 > 1 mg/L from setpoint 24 hr moving average

A high level of chlorine is an indication of poor dosing control. The ideal chlorine setpoint is a balance of both membrane warranty requirements and empirical data from sampling and analysis at the individual plant. As the health risks from chloramine are not acute, but rather chronic, the control action is to shut down the chloramine system rather than the entire plant.

127 Water Environment & Reuse Foundation

Figure 4.3. Chloramine Dosing Critical Alert (Warning) Response

Cl2 Alert triggered?

Review SCADA trends (Flow, Cl2 dose,

ammonia)

Check analyser(flow, chamber

cleanliness , signal, calibration)

Validate actual Cl2using grab sample.

Alert is real?

Conduct risk assessment with supervisor and review corrective

actions

Normal Operations

Y

Notify as per Incident Response

Plan

Staff to carry out investigation and implement maintenance/corrective actions.

Record Event

Parameter Alert level

Total Chlorine1 hr moving average

Cl2 > 1 mg/L from setpoint

Plan analyzermaintenance/calibration

Remove analyzer from service

N

Level back to

normal ?

Y

N

Y

Measure Cl2, chloramine and ammonia.

Take sample of final treated water for analysis of DBPs, NDMA.

N

128 Water Environment & Reuse Foundation

Figure 4.4. Chloramine Dosing Critical Alarm (Failure) Response

Critical limit

triggered?

Notification to plant manager and supervisors

Automated shutdown of chloramine dosing

system.

Review Alert response steps

Record EventRestart process under Supervisor direction

Normal Operations

Y

Incident Response Process:• Verbal notification to authorities and customers• Initial investigation and risk assessment (potential

exposure, process failure analysis)• Incident report• Engagement with authorities to agree subsequent steps

Process engineer to carry out investigation and implement maintenance and

corrective actions.

Continue monitoring of trends (including

sampling if required)

Complete Incident Report (<24hrs)

-> Preliminary risk assessment

Complete Investigation Report (<1wk)

-> Corrective Actions

Parameter Critical limit

Total Chlorine24 hr moving average Cl2 > 1 mg/L from setpoint

Repairs Successful

?

Water Quality Failure?

Y

N

N

Y

N

129 Water Environment & Reuse Foundation

4.2.5 Regular Maintenance Activities

The following lists the maintenance activities that should be followed to keep the instrumentation and equipment associated with chloramine dosing in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual, rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.). Weekly Validation Checks Total chlorine analyzers. Ammonia analyzers (if used). Chloramine analyzers (if used). SDI (or daily, depending on warranty requirements). Monthly Chemical metering pump calibration. Check on chlorine stock solution strength. Sampling and analysis of NDMA in MF filtrate or RO feed.

4.3 RO-Based Process – Microfiltration

4.3.1 Process Overview

Microfiltration (MF) or ultrafiltration (UF) membranes serve the function of providing a very high level of filtration for the downstream RO system. This allows for a minimization of downstream particulate fouling, less cleaning required, and more economical operation of the RO system. In reuse applications they are typically configured as hollow fiber polymeric membranes, although ceramic membranes are making forays into this area. An additional benefit of microfiltration and ultrafiltration is that a log reduction of some microorganisms can be validated using the pressure decay integrity test.

Microfiltration and ultrafiltration are essentially comparable technologies in this application. Both technologies straddle the divide of 0.1 micron which differentiates microfiltration (above) and ultrafiltration (below).

MF and UF systems are configured in units with multiple modules of hollow fiber polymeric (or ceramic) membranes. The units are either configured as pressurized systems in which water is pumped through the membranes, or submerged systems in which water is drawn through by suction. Each unit undergoes regular backwashing to remove build-up of suspended material, using a combination of water and low pressure air scour. Regular chemical washing is also conducted to remove material that is not removed effectively by backwashing.

The pressure decay test (PDT) (sometimes known as the membrane integrity test or pressure hold test) is configured on most systems. It is by far a higher resolution test than turbidity with log reduction values of greater than 4 for particles of > 3 microns capable of being confirmed. This provides validation for giardia, cryptosporidium and bacteria. The shortcoming of this test is that it requires a unit to pause from filtration mode and is not continuous, occurring at most plants around once per day.

Online turbidity can provide a continuous reading of treated water quality, however it is capable of only verifying a log reduction of at best 2 to 3 log due to the limitations of feed turbidity and measurement accuracy of instrumentation on the filtrate. (Typical inlet turbidity of < 10 NTU and outlet of 0.01 NTU turbidity = 3 log reduction). Turbidity will provide an indication only if there is a gross failure of the

130 Water Environment & Reuse Foundation

membrane system. As noted in Reuse-13-03, significant damage to the MF or UF membrane fibers is required to result in significant turbidity increase.

Challenge testing is a valuable means of measuring membrane integrity. The use of MS-2 phage has been used to provide validation of removal capability, however microbiological testing is relatively slow and expensive and not practical for very large systems. PDT provides sufficient resolution and is more practical for operations.

The PDT is a method of direct integrity testing used to identify and isolate breaches in the membrane barrier. The test involves pressurizing one side of the membrane fibers with air and measuring the rate of pressure decay with the other side exposed to atmospheric pressure.

A methodology for converting the PDT test readings to a log reduction value is detailed in the U.S. EPA Membrane Filtration Guidance Manual (Chapter 4). The test is based on the phenomenon that when a certain air pressure is applied to one side of an integral, fully wetted membrane (all membrane pores are filled with liquid) almost no air will pass through the membrane. This is because a minimum air pressure (the bubble point) is required to break the surface tension forces and displace the liquid.

Provided the air test pressure is below the bubble point, no air will escape from the pressurized side of the membrane except for a small flow that diffuses through liquid in the pores (diffusive air flow). If a leak or defect is present, air will displace the liquid and flow freely at this point, provided the defect size is not so small that the applied test pressure cannot overcome the surface tension forces.

A membrane’s bubble point and hence the test pressure required to demonstrate a 3 micron compliance is calculated from the Bubble Point Equation (U.S. EPA Membrane Filtration Guidance Manual Equation 4.1).

Figure 4.5 .Membrane Filtration Guidance Manual Equation 4.1 (U.S. EPA)

4.3.2 Key Operating Parameters

The MF/UF unit and overall system has a number of key operating parameters that are important to maintain sustainable operations and meet the requirements of the CCP: Operating flow rates including feed flow, filtrated flow and any cross flow rates the system is

operated in cross flow. The flow rates of specific stages are also important to allow for effective stage by stage monitoring and diagnostics.

Temperature has an impact on the permeability of the membrane as well as the membrane’s salt rejection characteristics.

Membrane differential pressure (trans-membrane pressure (TMP)).

131 Water Environment & Reuse Foundation

Membrane permeability and/or normalized trans-membrane pressure. Membrane TMP is impacted by temperature and unit flow rate. By normalizing the TMP the actual performance of the membrane can be reviewed with changes due to temperature and flow taken into account. The use of normalized membrane permeability (membrane flux per unit driving pressure) provides similar information. By trending this information an understanding of fouling and indication of when membrane cleaning is required are provided.

Membrane pressure decay integrity test performance. This is usually provided as a pressure decay rate in psi/min or kPa/min. The MF/UF unit will perform a PDT once daily and the result captured in the SCADA. This result should be trended over time to identify any overall loss in membrane integrity with time.

MF/UF feed and filtrate turbidity.

4.3.3 General Operating Procedures

General required operating procedures for the MF/UF system include: Regular verification of turbidity analyzers (cross checks with grab samples) and regular calibration

are an important requirement for management of this system. Review of permeability and normalized TMP data trends. Review of PDT test results. Regular cleaning of MF/UF membranes with chlorine, and acid based cleaning solutions as

required/recommended by the system supplier.

4.3.4 Critical Control Point Response Procedures

4.3.4.1 Microfiltration/Ultrafiltration Alert (Warning) Figure 4.6 illustrates the proposed Alert (warning) scenario for hollow fiber (microfiltration /ultrafiltration) membrane integrity verification. For these systems, turbidity is a continuously available water quality parameter that has been demonstrated to be less sensitive than the Pressure Decay Rate (PDR) (U.S. EPA, 2001) but can provide continuous information to the SCADA system. However, PDR testing is performed on a periodic basis (typically 1/day) and thus does not provide the confidence of continuous monitoring of a CCP. Thus under normal operation, it is recommended that turbidity and/or a particle counter should be included as an online monitor such that if turbidity is observed above a trigger setpoint, then actions would be to investigate the turbidimeter and conduct an off-cycle PDR test. During normal turbidity meter operation and reading a pressure decay test would be performed on a scheduled basis.

With the exception of some small plants, most plants have multiple microfiltration or ultrafiltration units operating in parallel. It should be noted, that the alert and critical alarms are monitored at the individual unit level. This is an inherently conservative approach, as the alert and critical alarms are set to achieve the full removal requirements, ignoring that any issue will be diluted by other units that are operating correctly.

In the event that high turbidity is confirmed, the unit would be removed from service and the PDR established. If the PDR indicates an unacceptable test result the unit would remain out of service and diagnostic and repair activities would be performed to obtain acceptable an acceptable PDR Test result prior to returning the unit to service.

132 Water Environment & Reuse Foundation

4.3.4.2 Microfiltration/Ultrafiltration Alarm (Failure) Figure 4.7 illustrates the Critical (Failure) response procedures for a hollow fiber (microfiltration / ultrafiltration) membrane system. The first action for a critical (failure) limit exceedance to inhibit the operation of the unit/system. The next action is to confirm that the critical (failure) limit has been exceeded, and that once confirmed managers and supervisors are notified of the failure condition. The second difference with the issue is that once the corrective action step has been performed, an assessment is made to determine if water quality was actually compromised during the failure event. If water quality was compromised, additional actions and interaction with external entities may be required.

It should be noted that for Figure 4.7, a filtrate turbidity critical condition is highly unlikely and will most likely only exist with failure of the turbidimeter. Membrane units often have redundant turbidimeters which would also have to fail in order for the common filtrate to fail critically. Recall that from Chapter 5 the Risk Priority Number (RPN) of 72 was assigned to the turbidimeter while an RPN of only 36 was applied to the PDR test. In both cases the occurrence (O) number was “2” indicating a low likelihood of failure while severity (S) was “9” indicating that the monitor provides a critical function and is key to public health protection. Because of the automation of the turbidimeter, the detection (D) number was “4” while it was only “2” for the PDR test. The take-away message is that the two tests (turbidity and PDR) are highly reliable in terms of being able to detect and trigger alarm events with a low likelihood of “failure to notice failure”. Table 4.2. Microfiltration/Ultrafiltration Alert and Alarm (Summary)

Monitoring Parameter Alert Level Critical (Failure) Level Notes

Unit Pressure Decay Rate (Daily Integrity Test)

PDR (in psi/min) target calculated to achieve desired slightly above (LRV + 0.2) target LRV of 3 micron particles, based on calculation in U.S. EPA Membrane Filtration Guidance Manual. The PDR target calculated will account for membrane type, unit configuration, flow rate, water temperature and degree of fouling.

PDR (in psi/min) target calculated to achieve desired LRV of 3 micron particles, based on calculation in U.S. EPA Membrane Filtration Guidance Manual.

LRV targeted is typically 4.0 for 3 micron particles which caters for Giardia, Cryptosporidium and bacteria. Lower targets are sometimes used if LRV requirements can be achieved with other processes. As LRV is logarithmic, a PDR target slightly above (0.1 to 0.2 log higher than target) is sufficient to provide sufficient buffer between the alert and critical levels.

Unit Filtrate Turbidity (15 min moving average)

Typically > 0.2 NTU or similar depending on the accuracy of the analyzer. If the analyzer is a laser turbidimeter, consideration of a lower figure may be suitable.

Typically > 0.5 NTU or similar depending on the accuracy of the analyzer. If the analyzer is a laser turbidimeter, consideration of a lower figure may be suitable.

Turbidity is far less sensitive relative to PDR. This back up provides a continuous monitor in case of gross failure and does not provide a guarantee of log reduction. Specific targets may be adjusted from site to site.

133 Water Environment & Reuse Foundation

Figure 4.6. MF/UF Alerts (Warning) Response

Turbidity Alert

triggered?

Review combined and individual SCADA trends (Feed, filtrate turbidity, cleaning process, TMP,

flow)

Check analyser(flow, chamber

cleanliness , signal,

calibration)

Validate actual turbidity using

handheld analyzer

Alert is real?

Conduct risk assessment with supervisor and review

corrective actions

Normal Operations

Y

Y

N

Notify as per Incident

Response Plan

Remove Unit from Service

Investigation (Diagnostic membrane bubble testing, equipment fault

identification) and implement maintenance/corrective actions

Record Event

Parameter Alert level

Unit Pressure Decay Rate(Daily Integrity Test)

PDR > 0.2 psi/min(eq 4 LRVs)

Unit Filtrate Turbidity(15 min moving average)

> 0.2 NTU (unit/combined)

Plan analyzermaintenance/calibration

Remove individual analyzer/unit from

service

PDR Alert triggered?

N

N

Y Is PDR Alert

confirmed?

Y

Repeat PDR Test

N

PDT Level back to normal?

N

Perform PDR Test

Y

134 Water Environment & Reuse Foundation

Figure 4.7. MF/UF Critical Alarm (Failure) Response

135 Water Environment & Reuse Foundation

4.4 RO-Based Process – Reverse Osmosis

4.4.1 Process Overview

Reverse osmosis utilizes semi-permeable membranes to provide a separation of soluble materials such as dissolved ions and organic materials from water. In the RO based process, this technology provides the removal of the bulk of dissolved contaminants and chemicals of concern. As noted in Chapter 3, it is currently limited in the microorganism log reduction that can be validated due to limitations in the suitability of an on line analyzer for a suitable surrogate parameter at reasonable resolution and time frames.

The reverse osmosis system operates typically at a fairly constant flow rate, and hence flow of the overall system must be governed by starting up or turning off individual trains. In the case of smaller plants, this may limit the flow set points at which the plant can operate.

Each individual RO unit operates with a feed pump that delivers water at high pressure to RO membranes. Treated water with a significant reduction in dissolved constituents, known as permeate, is delivered to downstream processes. A concentrated waste stream is delivered to waste. The ratio of permeate to feed water is known as the recovery. The recovery rate is set typically using a flow control valve.

The RO membranes are spiral wound elements which are housed in fiber glass pressure vessels. They are arranged in staged arrays with typically about 50% or so production of water produced from the first stage. In order to respect hydraulic requirements at each membrane element, a second and sometimes third stage of membrane pressure vessels takes the concentrate flow from the preceding stage and produces further permeate water. The total permeate is combined into a single stream per unit.

The use of pH correction via acid dosing and antiscalant chemicals are utilized to maximize the RO unit recovery while minimizing the risk of scaling with sparingly soluble salts at the membrane surface.

There are a number of different membrane manufacturers, each of whom produces a range of membrane models. These range with different rejection capabilities and operating permeability. Typically, reuse applications utilize low pressure brackish water membranes or in some cases nanofiltration membranes. An exhaustive discussion of RO design is not included here, however a list of operating parameters and general operating procedures are discussed below.

4.4.2 Key Operating Parameters

The RO unit has a number of key operating parameters that are important to maintain sustainable operations and meet the requirements of the CCP: Operating flow rates including feed flow, concentrate flow and permeate flow rates. The flow rates of

specific stages are also important to allow for effective stage by stage monitoring and diagnostics. Temperature has an impact on the permeability of the membrane as well as the membrane’s salt

rejection characteristics. Electrical conductivity is a well-accepted surrogate for total dissolved solids, and is used to gauge the

rejection of the RO unit. Pressure is an important parameter in understanding the amount of fouling or scaling on the

membrane surface. Operating data normalization. RO membranes are impacted in their performance characteristics by

temperature, total dissolved solids and flow rate. Normalization is a technique that allows comparison of operation at a specific set of conditions to a reference set of conditions. This allows the user to determine whether changes in flow or rejection are caused by fouling, damage to the membrane, or

136 Water Environment & Reuse Foundation

are just due to different operating conditions. The parameters of flow, temperature, electrical conductivity and pressure above are input into these formulae. These formulae will not be included here, however are available from membrane suppliers and engineering consultants. Typically, longer term trends are of more value and hence one or two data points per day is often sufficient. Key normalized data trends include normalized permeate flow (or specific flux), normalized salt rejection and normalized differential pressure. For each of these parameters, a set point will provide indication of when RO membrane cleaning is required.

pH is required to ensure that the feed water to the membranes is not prone to scaling once concentrated inside the RO unit.

Silt Density Index – This is a test that provides a plugging rate of solids through a 0.45 micron filter. It can be performed manually or in some cases with automated instrumentation. It provides surety of the potential of particulate fouling to the RO unit. It is often a requirement of membrane warranty.

4.4.3 General Operating Procedures

General required operating procedures for the RO system include: Regular verification of pH analyzers (cross checks with grab samples) and regular calibration are an

important requirement for management of this system. Regular verification of conductivity analyzers. Regular SDI test. Typically weekly as a minimum with timing often dictated by warranty conditions. Review of normalized data trends including normalized flow (or specific flux, normalized differential

pressure and normalized salt rejection. Regular cleaning of RO membranes with detergent cleaners, caustic based cleaning solutions and acid

based cleaning solutions as required.

4.4.4 Critical Control Point Response Procedures

4.4.4.1 RO Membrane Alert (Warning) Figure 4.8 illustrates the proposed Alert (warning) scenario for an RO membrane system using Electrical Conductivity (EC) Percent Removal as the method to trigger an alert response. Electrical conductivity is a continuously available water quality parameter that feeds data to the PLC. Under normal operation, if low percent removal is observed, actions would include investigating the feed and permeate conductivity. RO membranes have conductivity-based water quality monitors located on the influent as well as the permeate (effluent) which are used to determine the percent removal. The percent removal is calculated simply as the relationship between feed and permeate (i.e., a more complex calculation involving temperature compensation or normalization of water quality is not being used). Percent removal is used to represent the actual (non-normalized) removal across the membrane system.

TOC can be used to provide an additional monitoring technique to indicate the level organic material removal across the RO system. Reverse osmosis permeate has a relatively low level of TOC, and as such a low detection analyzer such as UV/persulfate oxidation must be used to achieve consistent results in the < 100 ppb range. These instruments are relatively costly, and it is generally uneconomical to include one analyzer per RO train, but rather one is installed on the common RO permeate line. This approach has been taken at Orange County Water District’s Groundwater Replenishment Scheme, The Western Corridor Recycled Water Project, and on Singapore’s Newater facility. While this instrument can be used as a CCP monitor, it has most often been used as an indicator of upstream process upset or indicator of an unanticipated pollution spill to the sewer shed.

137 Water Environment & Reuse Foundation

Current research is working towards improvements in RO system integrity monitoring (Frenkel and Cohen, 2014; Jacangelo Ongoing Research). This work may result in the potential for alternative monitoring approaches with higher sensitivity. Some approaches under current investigation include rhodamine dye injection, uranine injection, and sulfate monitoring, among others. Once validated, these may be substituted as monitors for this CCP.

4.4.4.2 RO Membrane Alarm (Failure) Figure 4.9 illustrates the Critical (Failure) response procedures for an RO membrane system. The first action for the critical (failure) limit exceedance is to inhibit the operation of the unit/system. The next action is to confirm that the critical (failure) limit has been exceeded, and that once confirmed managers and supervisors are notified of the failure condition. The second difference with the issue is that once the corrective action step has been performed, an assessment is made to determine if water quality was actually compromised during the failure event. If water quality was compromised, additional actions and notification may be required. Table 4.3. Reverse Osmosis Membrane Alert and Alarm Summary

Monitoring Parameter Alert Level Critical (Failure) Level Notes

RO EC Percent Removal

(15 min moving average)

(common feed/unit permeate)

Less than 94% removal

(For a target LRV of 1.2)

Less than 90% removal

For a target LRV of 1.0

EC removal is calculated from a common RO feed EC and an individual unit EC.

Targets can be adjusted for higher or lower LRV and may depend on the specific RO membrane rejection.

RO TOC Percent Removal

(15 min moving average)

(common feed/common permeate)

Less than 98 %

(For a target salt rejection LRV of 1.7)

Less than 95%

(For a target LRV of 1.2)

Analyzers for permeate measurement are expensive and are thus used on the common RO permeate line. These analyzers have typically been used for monitoring of upstream process upset or sewershed contamination.

138 Water Environment & Reuse Foundation

Figure 4.8. RO Alert (Warning) Response

139 Water Environment & Reuse Foundation

Figure 4.9. RO Critical Alarm (Failure) Response

140 Water Environment & Reuse Foundation

4.4.5 Regular Maintenance Activities

The following lists the maintenance activities that should be followed to keep the instrumentation and equipment associated with the RO in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual, rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks pH monitors. Conductivity monitors. SDI (or daily, depending on warranty requirements).

Monthly Chemical metering pump calibration. pH monitors. Temperature monitors. RO feed water quality analysis. RO membrane vessel permeate conductivity grab sample check.

4.5 RO-Based Process – UV/H202

4.5.1 Process Overview

UV AOP is an effective process for the removal of trace contaminants. The high energy UV light convert the hydrogen peroxide to hydroxyl radicals. Hydroxyl radicals are non-selective, have very high oxidation potential, and can readily react with organic contaminants in the water such as, synthetic organic compounds (SOCs). Two UV lamp technologies, low-pressure high-output (LPHO) and medium-pressure (MP), are commonly considered during the design of UV systems. LPHO lamps are each lower power (wattage) than MP lamps, and thus, the LPHO systems require more lamps to achieve a desired level of energy delivery. The delivery of target electrical energy dose (EED) is essential for the formation of hydroxyl radicals in the UV AOP system.

From a process operation standpoint, the key to the UV AOP process is to operate the UV reactor within its validated range. The performance of UV AOP in contaminant destruction is a function of both the UV dose (mJ/cm2) delivered by the reactors and the hydrogen peroxide concentration (mg/L) injected upstream of the reactors. As design UV dose increases, the concentration of H2O2 required for a given log-removal target decreases (and vice versa). UV doses for AOP systems are significantly higher than that required for UV disinfection and typically range from approximately 200 mJ/cm2 to over 1,200 mJ/cm2. Typical hydrogen peroxide doses can range from approximately 3 mg/L to 15 mg/L. The values for EED and hydrogen peroxide dose parameters are set during the full-scale validation (or testing) of UV reactors, where operating window is established for proper reactor performance. These parameters are then used in the calculation of UV dose delivered through the reactor. If the UV reactor is operated within this operating window of tested parameters, then the UV reactor can be assumed to have achieved the targeted pathogen inactivation via dosing delivery.

The use UV AOP in treatment does not have any impacts on downstream treatment processes; however, it is highly dependent on the feed water quality. For UV AOP systems, it is important to consider the impact of hydroxyl radical scavengers. Hydroxyl radicals are non-specific oxidizing agents and will react with numerous constituents within the water in competition with the target compounds, which in this case can be trace organic contaminants including NDMA, pharmaceuticals, and personal care products. As such,

141 Water Environment & Reuse Foundation

the presence, concentration, and rate of reaction of these radical scavengers substantially impact the efficiency of the advanced oxidation process. The most common scavengers are dissolved organic compounds (i.e., DOC), nitrate, bicarbonate and carbonate (i.e., alkalinity) (Metz et al., 2011; NWRI, 2000). Chemicals added within the treatment process also will scavenge radicals, including hypochlorous acid, the hypochlorite ion, and combined chlorine (chloramines). Due to these reasons, UV AOP is typically included at the end of the treatment process train prior to chlorination.

To achieve proper oxidation, target UV and hydrogen peroxide doses should be satisfied at all times. Water quality changes resulting in an inefficient pathogen inactivation typically involves a decrease in UV transmissivity due to increase in turbidity, and increase in scavenging demand. In the context of the direct potable reuse treatment, the water quality is expected to be of excellent quality as impact to UVT or scavenging demand changes not likely to occur. This expected UVT and scavenger stability makes any observed change in UV AOP system a good indicator of upstream process upset(s).

Based on these UV doses in AOP mode, the doses needed to obtain disinfection of pathogens such as Cryptosporidium, Giardia, and viruses would be greater than what is obtained for disinfection alone. However, the UV reactors have not been validated for pathogen and virus inactivation in UV AOP mode. Therefore, the 4-log reduction of pathogens may not uniformly be granted by state regulatory agencies for UV AOP systems. In most cases, it is likely to be evaluated on a case-by-case evaluation conducted by the states and California, for example, grants up to 6-log reduction for UV/AOP systems. In order to be credited for pathogen inactivation, the UV AOP system would have to have the same monitors as a disinfection system, in terms of monitoring UVT, intensity, calibration, etc.

4.5.2 Key Operating Parameters

As noted above, UV AOP is important for the removal of SOCs that may pass through the RO system. Although, the concentration of these contaminants is expected to be low after RO treatment, UV AOP provides an additional barrier for the removal of these contaminants. Furthermore, as discussed in the previous section, on a case-by-case evaluation some states may provide disinfection credits for the UV AOP systems. 4-log inactivation credit for Cryptosporidium, Giardia, and viruses in the RO treatment train would also contribute to achieve the California DDW goal of 12-10-10 for reuse systems.

UV AOPs performance depends on the formation of hydroxyl radicals. This would require proper propagation of the UV light in the reactor and thus, hydrogen peroxide, electrical energy dose, and UV lamps are identified to be the key monitoring parameters for UV AOP systems as they are directly related oxidation efficacy. Alert level and critical level values identified in Reuse-13-03 for these monitoring parameters are summarized in Table 4.4. Table 4.4. CCP Monitoring Parameters for UV AOP

Parameter Alert Level Critical Level

H2O2 25% below target level 50% below target level

EED < 105% target < 100% target

UV Lamp >10% >15%

In addition to parameters above, flow rate, UV intensity, and UVT, would also have an influence on the operation of the reactor which are discussed in further detail in the UV disinfection section.

142 Water Environment & Reuse Foundation

In operating UV AOP systems, there are a number of parameters that must be monitored to ensure proper treatment is being provided (e.g., formation of hydroxyl radicals is achieved). These monitoring parameters are described below.

Hydrogen Peroxide Dose – This parameter is recommended for daily monitoring because it factors directly into the delivery of hydrogen peroxide that will form the hydroxyl radicals that are essential for the UV AOP system. A decrease in peroxide could be attributed to issues with, metering pumps, or issues with hydrogen peroxide monitoring equipment (if applicable).

Electrical Energy Dose – The Electrical Energy per Order (EEO), is used to determine the removal efficiency of organic contaminants in a UV AOP system. EEO is defined as the electrical energy required to reduce the concentration of a pollutant to below target levels, typically measured in log order reduction, per volume of water treated. It enables a direct assessment of the effectiveness of removing different organic compounds in UV AOP systems, though the concept can also be applied to other advanced oxidation processes. The EED is the Electrical Energy Delivered by the AOP system. If the EED drops below the range required for EEO, it would indicate that some photolyzable SOCs (e.g., NDMA) may not be removed in the UV AOP system. If the EED approached the set boundary, then warnings/alarms should be triggered. The EEO is dependent on water quality and is measured at the optimum hydrogen peroxide dose. The water quality is a factor for the UV disinfection systems and those factors are discussed further in the UV disinfection section.

In some instances, the EED is incorporated into what is termed the Present Power Ratio or PPR, which simply compares the EED to the required EEO in a ratio. Thus, the PPR must be > 1 in order to achieve expected reduction in contaminants.

UV Lamp – this parameter was selected because the UV lamps are the sole source of the UV light beams. Any failure in the UV lamps can lead to insufficient formation of hydroxyl radicals which in turn would adversely affect the UV AOP systems performance.

4.5.3 General Operating Procedures

UV AOP reactors should be operated to achieve the target removal of SOCs. This is most often tracked by ensuring that the design UV dose is met, which in turn involves ensuring proper dosing of the hydrogen peroxide and delivery of electrical energy dose are within the validated range for a given reactor and UV lamps are functional. Hydrogen peroxide feed system should be maintained regularly (e.g., feed pump calibration, flowmeter calibration). The dosing of the hydrogen peroxide should be tracked and recorded by the plant SCADA system.

The delivery of EED is most commonly determined by a UV sensor incorporated in the UV reactor design. The sensor(s) tracking the UV intensity should be maintained properly and calibrated periodically. Low UV intensity can be caused by failure of a UV lamp. UV lamps have a set lifespan, which is typically within the range of 8,000 to 14,000 hours. The plant SCADA system should keep track and record the use of each lamp. Any lamp failures should be replaced immediately and the SCADA system should be updated. During the operation of the UV reactors, quartz sleeves around the UV lamps should be periodically cleaned to prevent accumulation of any foulants. This can be done manually or through pneumatically or electrically actuated systems. The operation of the wiper system should also be tracked and recorded in the plant SCADA system.

Conversion of the hydrogen peroxide is dependent on the delivery of light waves. Flowrate determines how long the water to be treated stays within a UV reactor and dictates the energy delivered to the water. Flow rate can be easily tracked by the flowmeters of each UV AOP reactor. The plant automation system should track and record the per reactor flow rate. Operators should routinely monitor this parameter and

143 Water Environment & Reuse Foundation

compare them to the design values for the reactor. If the flowrate is higher than the design capacity, then insufficient contact time is being provided in the UV reactor. In this situation, the flow to the individual reactor should be reduced until the flowrate is within the design range. UV transmittance and UV sensors are also important for the proper delivery of EED. These parameters are discussed further in the UV disinfection section. Operators should also routinely monitor these parameters in addition to the ones listed above and compare them to the design values for the reactor.

Operators should also routinely monitor the reactor temperature to prevent overheating of the UV lamps. If the UV AOP reactors are not within the validated range, then corrective actions must be taken as described in the next section.

4.5.4 Critical Control Point Resources

4.5.4.1 UV AOP Alert (Warning) Response Figure 4.10 illustrates the proposed Alert (warning) scenario for UV/H2O2 Advanced Oxidation Process. In a UV/AOP system, three modes of action are occurring simultaneously for chemical and microbial control (Kruithof, Kamp, and Martijn, 2007). First, UV irradiation is being used to disinfect the water and, in the case of UV/AOP, the energy is typically at least 10 times greater than that used for UV disinfection. Secondly, the high energy UV light will photolyze some compounds such as NDMA that may be less amenable to removal by other treatment technologies (Plumlee, López-Mesas, Heidlberger, et al., 2008; Steinle-Darling, Zedda, Plumlee, et al., 2007). Third, hydrogen peroxide (H2O2) is added to the water and reacted with the high energy UV light to form into hydroxyl radicals. The hydroxyl radical acts as a strong oxidant that reduces the amount of other chemical constituents such as 1,4-dioxane in the water (Stefan and Bolton 1998). Hydrogen peroxide is metered as a liquid into the process stream, and can lose flow through off-gassing.

The amount of light exposure achieved in a UV/AOP reactor can be measured as a dose (mJ/cm2) or as a reaction Energy Equivalent per log-Order of reduction (EEO) for a target contaminant. Additional parameters such as UV Transmittance, UV Lamp Intensity, and flow are used to calculate the dose/EEO. Under certain circumstances, including end of useful life, UV lamps may fail resulting in decreased dosage/EEO. However a UV-oxidation control system that is operating based on a target EEO will be able to adjust the delivered UV energy to maintain the target EEO. The Contaminant Log Reduction (comparison of target and calculated actual log reduction) is referred to as the Electrical Energy Dose (EED).

Likewise, the amount of peroxide added can impact the degree of production of hydroxyl radicals formed and UV photolysis and disinfection that can occur. Because only about 10% of the peroxide actually is reacted to create hydroxyl radicals, too little peroxide addition will result in a low hydroxyl radical yield and potentially insufficient advanced oxidation (Rosenfeldt and Linden, 2007). Conversely, because peroxide absorbs UV light, adding too much peroxide will decrease the UV dose/energy delivered and may compromise photolysis (Sgroi, Roccaro, Oelker, et al., 2015). Finally, careful attention must also be given to the amount of quenching agent and/or chlorine that must be used to eliminate the 90% of peroxide still remaining after UV/AOP. Thus, careful design and validation must be used to determine the optimal dose of hydrogen peroxide or chlorine (in the case of UV/chlorine-based AOP systems) to achieve the desired formation of hydroxyl radicals and subsequent contaminant oxidation while minimizing chemical costs and impacts to downstream chlorine addition.

Additionally, UV lamps require some time to warm up to obtain full operational status, and also may also operate at reduced efficiency if there is a sudden increase or decrease in flow through the system. Normally these systems are flow controlled during start up, operation or shut down to ensure that water passing through the UV system receives the required UV energy. The monitors for operation of the UV

144 Water Environment & Reuse Foundation

system may include various combinations of power, UV fluence (or intensity), UV transmittance of the feed water, peroxide flow, feed water flow, and turbidity.

4.5.4.2 UV AOP Critical (Failure) Response Figure 4.11 illustrates the proposed Critical (warning) scenario for UV/H2O2 Advanced Oxidation Process. Treatment is provided both by UV light and hydrogen peroxide so the loss of hydrogen peroxide feed and/or the loss of operating lamps will reduce the treatment effectiveness. For these systems, loss of hydrogen peroxide results in the process becoming ineffective. Other failures may occur as a result of lamp failure or other mechanical issues. Specific response procedures would need to be developed for the exact sensor combinations used at a given facility, though this diagram will provide a basic overview of potential alarm response procedures.

145 Water Environment & Reuse Foundation

Figure 4.10. UV/AOP Alert (Warning) Response

146 Water Environment & Reuse Foundation

Figure 4.11. UV/AOP Critical Alarm (Failure) Response

147 Water Environment & Reuse Foundation

4.5.5 Maintenance Activities

The following items include the maintenance activities that should be followed to keep the instrumentation associated with the UV AOP process in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual (i.e., sleeve cleaning frequency), rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.). Weekly Validation Checks UVT monitors. UV sensors. Turbidimeters. Monthly Calibration Hydrogen peroxide feed systems. Hydrogen peroxide monitors. UVT monitors. UV sensors. Turbidimeters. Temperature sensors. Flow meters. Flow control valve and actuator maintenance.

4.5.6 Operations-Focused Design Considerations

The HACCP process performed as part of Water Environment & Reuse Foundation Project Reuse-13-03 identified the UV AOP as an important process for the removal of organic contaminants in the RO based treatment process train for direct potable reuse. Key operating parameters and associated alert levels and critical limits were identified for the process. For UV AOP, the critical control points were identified to be hydrogen peroxide dose, EED, and UV lamp failure. It should be noted that EED is also dependent on the water quality (i.e., flowrate, UV sensor, UVT) which is described further in the UV disinfection section. To provide the level of performance monitoring and control necessary to appropriately respond to alert level and critical limit situations, certain items should be included in the overall design of the UV AOP system. Most of these items are what would be required in a standard drinking water UV AOP design; however, it is important that the system parameters be monitored closely in the DPR situation to further reinforce the robustness of the process relative to public health protection.

Similar to other processes, multiple trains should be provided for operational flexibility and redundancy. For UV AOP, the redundancy should be considered as n+1. This will allow the system to maintain redundancy while one UV reactor is taken offline for any maintenance purposes (i.e., lamp replacement). Hydrogen peroxide dosing systems should be also considered as n+1 which includes the pumps and the injection system. This will allow the system to maintain redundancy while one pump or an injector is taken offline for any maintenance purposes. In the event of a critical limit condition, multiple reactors/pumps would allow the plant to continue to operate within the design flow and dose ranges when the problem reactor is automatically taken offline. Hydraulic flow splitting structures on the influent of the reactor trains would provide a simple, operator-friendly means of equally dividing the influent flow among the units. However, individual flow metering and a means of controlling the flow to the reactor in the event of a high flow alert would also be required. Effluent isolation devices interlocked with critical

148 Water Environment & Reuse Foundation

limit alarms are necessary to isolate the flow from any UV reactors experiencing critical limit situations (i.e., low hydrogen peroxide dose).

Hydraulics through a UV AOP reactor can directly impact the efficiency of the process. It is recommended that design engineers consider the use of computational fluid dynamic (CFD) modeling to optimize the mixing of the hydrogen peroxide prior to the UV reactor and hydraulic flow through the UV reactor. Inlet and outlet piping configurations of the UV reactor should be in accordance with the manufacturer’s recommendations (i.e., straight pipe lengths).

4.6 RO-Based Process – Stabilization

4.6.1 Process Overview

Chemical stabilization is commonly applied in reverse osmosis-based treatment plants in order to restore a sufficient balance of calcium hardness, alkalinity and pH to minimize corrosion of plant and distribution network infrastructure, in particular cement lining and copper services. While this dose is primarily concerned with protecting assets, from a health standpoint it is considered a critical control point to protect against the mobilization of lead and copper in distribution systems which may occur if water is not suitably stabilized.

Chemical stabilization can be achieved in several ways, including: Lime (calcium hydroxide) dosing Calcium carbonate contactor (calcite filter) Carbon dioxide dosing Mineral acid (sulfuric acid, hydrochloric acid)

The intent of these dosing regimens is to ensure that the balance of pH, alkalinity, ionic strength and in particular calcium hardness to manage chemical erosion of cement based materials or corrosion and mobilization of metallic substances. A number of different indices are used to target an appropriate balance of these parameters including the Langelier Saturation Index (LSI) and Calcium Carbonate Precipitation Potential (CCPP). In both cases, the most significant contributors to these indices is the pH and the level of calcium hardness. From a CCP standpoint, there is a health risk if the water is in a more aggressive state (i.e., more likely to solubilize metals) and as such controls will be in place to minimize the risks of extended periods at this condition. The use of online pH monitoring with a check on calcium hardness should be used as monitoring parameters. Calcium hardness is monitored as a dose flow (if lime is used) and supported by regular sampling and analysis for hardness.

The researchers note that where used, lime dosing is typically achieved by batching a slurry of dry hydrated lime (Ca(OH)2) with water into a suspension known as milk of lime. This suspension, which can contain various concentrations of lime, is either dosed directly or is fed to a lime saturator to remove impurities prior to where the clarified lime solution is dosed. A dose flow rate requirement will therefore need to be calibrated for the concentration of calcium that is dosed in the slurry liquid or supernatant.

Calcite contactors, sometimes called calcite filters, are contactors containing small chips or pellets of calcium carbonate. Water flows through these contactors to dissolve sufficient calcium hardness to achieve desired levels and stability indices. To target a required concentration, a portion of the water is bypassed. The response procedure has been written assuming a lime dosing/carbon dioxide or mineral acid dosing combination. However, this could also be used for calcite contactors with the lime dose flow monitor substituted for a flow to bypass flow ratio through the contactor.

149 Water Environment & Reuse Foundation

4.6.2 Key Operating Parameters

The key operating parameters are those that are used to calculate the targeted stability index. In the case of the CCPP, this includes: Calcium hardness (measured as mg/L of CaCO3). While on line hardness metering is available,

regular grab sampling should provide sufficient confirmation that operation is occurring within the required band. Daily grab sampling is typically acceptable with on-site analysis to provide sufficient accuracy.

Similarly, alkalinity should be measured daily. Temperature has a minor yet important effect on the CCPP calculation and should be monitored on line. pH is the most sensitive parameter for achieving desired stability indices. CCPP in particular is

sensitive to pH and control to within +/- 0.2 pH units is usually required.

In contrast for most of the selected CCPs, the health risk that is mitigated is a chronic chemical risk (i.e., lead and copper leaching). As a result, actions that are required to be taken need not be as immediate which is reflected in the response procedures. In this case, a daily 24 hour or similar average period can be taken to assess this parameter.

4.6.3 General Operating Procedures

The general operating procedures required for the management of water stability will depend to some extent on the treatment processes that are employed. This section will not provide an exhaustive review of different stabilization processes, but provide some general recommendations for some treatment process: Regular verification of pH analyzers (cross checks with grab samples) and regular calibration are an

important requirement for management of this system. A check on turbidity and solids residual, particularly in the case of lime dosing. This is especially true

if a lime saturation process is not utilized. Regular checks of chlorine contact tank and downstream pipework to review any lime solids or inert material that may have collected. At low tank levels this can cause turbidity plumes to occur, and can block instruments/analyzers.

Regular review of lime quality (if used for lime dosing) to minimize the levels of lime impurities (typically hydrated lime is provided at 90-95% purity).

4.6.4 Critical Control Point Response Procedures

4.6.4.1 Chemical Stabilization Alarm (Failure) Figure 4.12 illustrates the Alert procedures while Figure 4.13 illustrates Alarm (failure) procedures for the Chemical Stabilization step. Critical failure alarms for chemical stabilization processes are configured in a manner similar to the Alert (warning) status alarm. The primary difference is that a failure will result in an interruption to the process, whereas an alert will allow for operator intervention before a compliance issue arises. As for the warning alarm condition, as this is not an acute health risk, a longer time period prior to failure alarm will be provided. Table 4.5. Chemical Stabilization Alert and Alarm Summary

Monitoring Parameter Alert Level Critical (Failure) Level Notes

pH (24 hr moving average)

pH < 0.5 units from setpoint

pH < 1.0 units from setpoint

pH is the most sensitive parameter with respect to water stability. The setpoint for pH will be set at individual plants based upon final treated water chemistry and distribution network requirements.

Lime Dose (24 hr moving average)

25% below target dose level.

50% below target dose level. Lime dose can be either provided from a dose flow or

150 Water Environment & Reuse Foundation

Figure 4.12. Chemical Stabilization Alert (Warning) Response

pH Alert triggered?

Review SCADA trends (Flow, pH, lime dose

flow, CO2 or acid dose flow)

Check analyser(flow, chamber

cleanliness , signal, calibration)

Validate actual pH using handheld

analyzer

Alert is real?

Conduct risk assessment with supervisor and review corrective

actions

Normal Operations

Y

Notify as per Incident Response

Plan

Staff to carry out investigation and implement maintenance/corrective actions

Record Event

Parameter Alert level

pH24 hr moving average

pH < 0.5 units from setpoint

Lime Dose Flow24 hr moving average

flow < 25 % of setpoint

Plan analyzermaintenance/calibration

Remove analyzer from service

N

Level back to

normal ?

Y

N

Lime dose alert

triggered?

Y

Measure Ca Hardness,

Alkalinity, pH and TDS and calculate

stability index

N

NY

151 Water Environment & Reuse Foundation

Figure 4.13. Chemical Stabilization Alarm (Failure) Response

152 Water Environment & Reuse Foundation

4.6.5 Regular Maintenance Activities

The following lists the maintenance activities that should be followed to keep the instrumentation associated with the chemical stabilization system in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual, rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks pH monitors. Check of lime dosing system.

Monthly Calibration Chemical metering pumps. pH monitors. Temperature monitors. Bench-top hardness analysis. Delivered lime analysis.

4.7 RO-Based Process and Non-RO-Based Process – Chlorine CT

4.7.1 Process Overview

Chlorine is an oxidant and effective chemical disinfectant. The U.S. EPA provides up to 3-log inactivation credit for Giardia and 4-log inactivation credit for viruses for chlorine disinfection systems. The removal of Giardia and viruses is calculated by the determination of concentration (C) and contact time (CT). It is the result of multiplying the disinfectant residual concentration by the contact time. Since pH and temperature are also important factors for the efficacy of chlorine disinfection, these parameters are also needed to determine the removal of Giardia and viruses. The CT value along with pH and temperature are compared to values reported in the LT1ESWTR Disinfection Profiling and Benchmarking – Technical Guidance Manual to determine the log reduction values (LRV).

The CT requirement for the inactivation of Cryptosporidium with chlorine is a few orders magnitude higher than the removal of Giardia and viruses. Therefore, inactivation of Cryptosporidium is not considered within the chlorine contact basins.

From a process operation standpoint, the key to the chlorine disinfection process is to operate the chlorine contact basin within its targeted CT range which is the designed CCP. If the chlorine contact basin is operated within an established range of tested parameters, then the disinfection can be assumed to have achieved the pathogen inactivation associated with this range. CT, chlorine residual, flow rate, pH and temperature must be monitored continuously and used for the determination of LRV.

4.7.2 Key Operating Parameters

As noted above, the U.S. EPA provides up to 3-log inactivation credit for Giardia and 4-log inactivation of viruses for chlorine disinfection. As such, this is an important process in both treatment trains (membrane and non-membrane) to achieve the California DDW goal of 12-10-10 (virus-giardia-cryptosporidium) for reuse systems. The disinfection contact time and chlorine dose (CT) required to achieve the 3-log Giardia inactivation are the governing criteria for the chlorine disinfection. This is because the required CT to achieve Giardia is higher than that required for virus inactivation.

153 Water Environment & Reuse Foundation

CT values for 3-log giardia inactivation range from 24 to 552 mg-min/L based on water temperature and pH as reported in the Disinfection Profiling and Benchmarking – Technical Guidance Manual published by the U.S. EPA. In addition to CT tables for target pathogens, the Disinfection Profiling and Benchmarking – Technical Guidance Manual includes a discussion on procedures for the calculation of log-inactivation credit for various chlorine disinfection configurations. Utilities desiring to obtain the pathogen or virus inactivation credit for chlorine must utilize the method presented in the guidance manual for calculating the overall CT, unless the state has adopted alternative methods for calculation of CT. Validating CT U.S. EPA Disinfection Profiling and Benchmarking Technical Guidance Manual (US EPA 2003) requires drinking water disinfection systems to create a disinfection profile. In order to create a disinfection profile, systems are required to identify disinfection segments, collect required data for each segment, calculate CT, and calculate inactivation. Necessary data include peak hourly flow, residual disinfection concentration, temperature and pH (if chlorine is used).

The Guidance manual provides approaches for deriving Log Inactivation values for Giardia and viruses for free chlorine disinfection. The approaches involve calculating a value for “CTactual” based on measurements of chlorine residual concentration and exposure time. One approach is to collect data including disinfection residual concentrations and peak hourly flow rates through vessels. To determine the contact time (T10), the volume (V) of a vessel is divided by the peak hourly flow rate (Q) and then multiplied by the baffling factor assigned to the vessel:

𝑇10 =𝑉

𝑄× 𝐵𝑎𝑓𝑓𝑙𝑖𝑛𝑔 𝐹𝑎𝑐𝑡𝑜𝑟

The value for CTactual is then determined by multiplying T10 by residual disinfectant concentration for the disinfection segment:

𝐶𝑇𝑎𝑐𝑡𝑢𝑎𝑙 = 𝐶 × 𝑇10

A standard CT value, relating to a specific value of log inactivation is then usually obtained from a table for such CT values for various chlorine concentrations, temperature and (for free chlorine) pH. For Giardia, the standard CT is CT3-log, Giardia and for viruses, the standard CT is CT4-log, Virus:

𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝐿𝑜𝑔 𝐼𝑛𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑜𝑓 𝐺𝑖𝑎𝑟𝑑𝑖𝑎 = 3.0 × 𝐶𝑇𝑎𝑐𝑡𝑢𝑎𝑙

𝐶𝑇3−𝑙𝑜𝑔,𝐺𝑖𝑎𝑟𝑑𝑖𝑎

𝐸𝑠𝑡𝑖𝑚𝑎𝑡𝑒𝑑 𝐿𝑜𝑔 𝐼𝑛𝑎𝑐𝑡𝑖𝑣𝑎𝑡𝑖𝑜𝑛 𝑜𝑓 𝑉𝑖𝑟𝑢𝑠 = 4.0 × 𝐶𝑇𝑎𝑐𝑡𝑢𝑎𝑙

𝐶𝑇4−𝑙𝑜𝑔,𝑉𝑖𝑟𝑢𝑠

Most commonly, the standard CT values are obtained via an “Approximation Method” based on selecting conservative values of pH, temperature, and residual disinfectant concentration from established CT tables to estimate the CTs required for 3-log inactivation of Giardia and 4-log inactivation of viruses.

For a DPR facility, once this approximation value is known, operation is then a matter of maintaining a minimum treated water chlorine residual target, and maintain a flow below a maximum flow rate. The T can also be assured by operating at a minimum level within a contact tank. Varying chlorine concentration with different plant flows is possible (i.e., at lower flow rates dose lower chlorine to achieve the same CT) however is not recommended as it adds a level of complexity to operations. Far safer is to operate to a standard chlorine residual set point, related to a minimum chlorine level that will achieve required CT at maximum plant flow rate.

154 Water Environment & Reuse Foundation

Chlorine Residual Monitoring Continuous monitoring of free chlorine residual is required to ensure minimum CT. This monitor is a critical monitor in terms of the chlorine CCP. In RO permeate in the RO treatment based process, chlorine tends to be very stable due to the low levels of organics or other chlorine demand present in the water. However hydrogen peroxide injected upstream in the UV/H2O2 process can exhibit significant chlorine demand and must be accounted for. In the non-RO process, a similar low level of organics is anticipated to exhibit low chlorine demand and minimum fluctuation. Other possible reasons for chlorine level fluctuation can be changes in the quality of chlorine stock solution. Temperature Monitoring Temperature impacts the efficiency of the chlorine disinfection. Temperature should be monitored on line with alarm conditions provided if temperature moves outside of the acceptable range for CT. Typically, DPR treated water tends to maintain relatively stable temperatures due to the thermal buffering of the wastewater treatment processes, and so the risk of temperature impacting CT significantly is low. Nonetheless, temperature should be profiled and taken into account for the LRV calculation. Flow Flow rate allows the calculation of T for the CT calculation. As noted above for CT, the plant should operate below a maximum flow rate. pH pH also effects efficiency of chlorine disinfection, as it impacts the dissociation between hypochlorous acid and hypochlorite ion. Hypochlorous acid, being a stronger oxidant, provides a more efficient CT. pH at this point in the process is controlled to achieve final water stability indices, and as such is not controlled for CT. Nonetheless it must be monitored for use in determining the log reduction. Typically, pH remains stable within a relatively narrow band (plus or minus 0.2 of a pH unit) at this point in the process.

4.7.3 General Operating Procedures

Management of chlorine CT will require a relatively minimum number of operating procedures, as the contact time will be managed above a minimum based on treated water flow and minimum contact tank levels. These will be fully automated operations, set following design assessments for minimum chlorine residual required for chlorine concentration and applied baffling factors for the contact tank. Some key procedures that will be required include: Regular verification of pH and chlorine analyzers (cross checks with grab samples) and regular

calibration are an important requirement for management of this system. A regular review of operational trends, particularly chlorine residual and the chlorine dose rate

required should be regularly compared to determine if there is an increasing dose of chlorine required to achieve the target residual. This can be an indicator of either increasing chlorine demand (for example H2O2 overdose from UV/H2O2 systems) or loss of chlorine activity in chlorine stock solutions.

Regular check on chlorine stock solution (if using sodium hypochlorite). This chemical can degrade over time, particularly in warmer climates. Degradation will result in higher doses of chlorine, and in some cases unwanted byproducts such as chlorate. Chlorine stock solutions should be managed to ensure sufficient storage turn over to avoid build-up of these unwanted byproducts. In warm climates, consideration can be given to diluting solution on site which can slow down the degradation rate.

155 Water Environment & Reuse Foundation

4.7.4 Critical Control Point Response Procedures

4.7.4.1 Chemical Disinfection (Chlorine) Alert (Warning) Figure 4.12 illustrates the proposed chemical disinfection alert (warning) for the post-treatment chlorination step. Chemical disinfection processes are common to water treatment and re-use applications. Under most circumstances a disinfectant (e.g., chlorine) is added to the water and allowed to react for a period of time to inactivate any remaining target pathogens (e.g., Giardia, coliform bacteria and/or virus). The disinfection residual is normally, but not always, measured at the end of the process together with temperature, pH, and flow.

Disinfection is normally expressed as a function of the chemical residual concentration (C) multiplied by the contact time (T), or CT. Because pH and water temperature affect the disinfection process they are also monitored. Under some circumstances the volume of the disinfection structure may also be a variable and will require a methodology to calculate the compliance flow (e.g., min, avg. hour, peak hour, etc.) established by the regulatory authority to be used in the disinfection calculation.

It should be noted that many water treatment facilities calculate CT based upon peak hourly flow at the outlet condition. This approach correlates to an operator hand measurement and the calculation may be cumbersome to calculate using the PLC. Using this approach a scenario can exist where there is a gap between the availability of the calculation parameter (e.g., peak hourly flow) necessary to calculate the CT parameter. Thus it is suggested that alternative online measurement and averaging strategies may be more appropriate to monitor a system under continuous flow conditions. This will allow the calculation to per performed in real time.

The most common problem associated with this type of equipment is loss of an accurate analyzer signal due to drift, sample flow, loss of reagent or other issue with the instrumentation. Since disinfection structures are generally common to the overall facility, it may be advisable to locate redundant analyzers in a common structure. 4.7.4.2 Chemical Disinfection (Chlorine) Alarm (Failure) Figure 4.13 illustrates the Alarm (failure) procedures for the Chemical Disinfection step. Critical failure alarms for disinfection processes are configured in a manner similar to the Alert (warning) status alarm. The primary difference is that a failure will result in an interruption to the process, whereas an alert will allow for operator intervention before a compliance issue arises.

156 Water Environment & Reuse Foundation

Table 4.6. Chlorine Disinfection Alert and Alarm Level Summary Monitoring Parameter Alert Level

Critical (Failure) Level Notes

CT (mg/L.min) Specific targets below Specific targets below

CT is calculated according to the U.S. EPA disinfection guidance manual specific requirements for chlorine residual, flow rate, temperature and pH are managed as individual alerts as a part of this CCP.

Chlorine (mg/L) < 25% of target 15 min moving average

< 50% of target 15 min moving average

Target is set to be well above the minimum that is required to achieve chlorine CT for given range of temperature, flow and pH. In this case, it is set such that it is 2 x the requirement for minimum CT. This should be evaluated case by case at each facility.

Temperature (oC or oF)

Less than 110 % of minimum design temperature. 15 min moving average

< Design minimum temperature 15 min moving average

Alarm should be set at minimum temperature of design at which CT can be achieved based on EPA guidance manual.

Flow (gpm) 10% greater than maximum flow

20% greater than maximum flow

CT will be designed on a maximum flow rate at which chlorine dosing can achieve required dose. This is not a likely scenario, as the RO process

pH < 6.5 or > 8.7 < 6.0 or > 9.0 EPA CT figures are only valid for this pH range. pH outside this range would also likely trigger

157 Water Environment & Reuse Foundation

Figure 4.14. Disinfection System (Chlorine) Alert (Warning) Response

158 Water Environment & Reuse Foundation

Figure 4.15. Disinfection System (Chlorine) Alarm (Failure) Response

159 Water Environment & Reuse Foundation

4.7.5 Regular Maintenance Activities

The following lists the maintenance activities that should be followed to keep the instrumentation associated with the chlorination in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual, rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks Online chlorine residual analyzers. pH monitors.

Monthly Calibration Chemical metering pumps. pH monitors. Temperature monitors. Online chlorine residual analyzers. UVT monitors (if applicable). TOC analyzers (if applicable). Flow meters. Bench top chlorine analyzer. Flow control valve and actuator maintenance.

4.8 Non-RO-Based Process – Ozone

4.8.1 Process Overview

Ozone is a very strong oxidant and chemical disinfectant. The U.S. EPA (U.S. EPA Alternatives Disinfection provides up to 3-log inactivation credit for Cryptosporidium and Giardia and 4-log inactivation credit for viruses for properly operated and designed ozonation systems as outlined in the remainder of this section. The process is effective in oxidation of inorganic and organic compounds alike, and can effectively reduce the concentration of emerging contaminants of concern when coupled with downstream biologically active carbon (BAC). This assumes only carbon media is used in the downstream filtration process. Ozone will also provide the benefits of reduction of taste and odor (T&O) compounds and particle conditioning prior to filtration.

There are four primary components that make up the overall ozonation process train. These components are: Feed gas treatment Ozone generation Ozone contact Residual ozone destruction

Ozone gas must be generated on-site as it is inherently unstable and decomposes rapidly in aqueous solutions. The ozone gas is generated by passing dry air or oxygen between two electrodes. The feed gas treatment step conditions the incoming air or oxygen for optimum ozone formation, including removal of dust, nitrogen gas, water and oil. Of these parameters, the most critical items are removal of moisture and control of the gas temperature. Higher moisture content gas flows will require more power to generate the same amount of ozone, so most feed gas treatment systems target a moisture removal of approximately

160 Water Environment & Reuse Foundation

99.98%. A dewpoint temperature of 76°F is targeted as this maintains a low water content per thousand cubic feet of air.

In the ozone generation step, a high electrical potential (10,000 – 30,000 volts) is applied across a pair of electrodes, which converts a portion of the feed gas to ozone. There are three types of ozone generators – low, medium and high frequency generators – with low (50 to 60 Hz) and medium frequency (60 to 1,000 Hz) units being the most common. There are inherent advantages and disadvantages of each type that should be evaluated during design, including ozone generation efficiency, power requirements and generator cooling requirements. The former is very important as the majority of the energy applied to the electrodes in the ozone generator is given off as heat, so a cooling water system must be included to control the generator temperature.

In the ozone contact step, ozone gas is dissolved into the water to be treated. There are various ways in which the ozone gas can be delivered, but they are all targeted at optimizing the rate of mass transfer of ozone from the gas to the liquid phase. This is doubly important because of the low solubility of ozone in water; optimizing the rate of mass transfer can help to maximize the amount of ozone that makes it into the process stream to be treated.

Residual ozone destruction equipment is needed to reduce the concentration of ozone in the process exhaust air. Due to the limited water solubility of ozone and practical limits in mass transfer efficiency, between 5% and 10% of the added ozone remains in the off-gas/process exhaust. The residual ozone in the off-gas must be reduced to below ambient air quality standards and occupational health levels. Ozone destruct systems generally fall into the thermal destruct or catalytic destruct categories, each with its own inherent operation and maintenance requirements.

The use ozone in treatment can impact downstream treatment processes, which must be taken into account when designing ozone systems. The oxidation of organic matter leads to an increase in assimilable organic carbon (AOC). It can contribute to increased bacterial regrowth in distribution system piping. Therefore ozone is typically coupled with biologically active carbon (BAC) for AOC removal. Finally, while the required ozone dose is typically low for bacteria and virus inactivation (1 to 4 mg/L), higher background levels of natural organic matter and/or other organic compounds can require higher doses. Provision of upstream processes for coagulation and settling of organic matter and other particles in the water can help to minimize the ozone dose required for pathogen inactivation.

4.8.2 Key Operating Parameters

As noted above, the U.S. EPA provides up to 3-log inactivation credit for Cryptosporidium and Giardia and 4-log inactivation of viruses for ozonation systems. As such, this is an important process in the non-membrane train to achieve the California DDW goal of 12-10-10 (virus-cryptosporidium-giardia) for reuse systems. The disinfection contact time and ozone dose (CT) required to achieve the 3-log Cryptosporidium inactivation are the governing criteria for the ozone process. This is because the required CT to achieve Giardia and virus inactivation is nearly an order of magnitude lower than that required for the same level of inactivation of Cryptosporidium. CT values for 3-log Cryptosporidium inactivation range from 4.7 to 72 mg-min/L based on water temperature (30°F to 0.5°F, respectively) as reported in the Long Term 2 Enhanced Surface Water Treatment Rule (LT2ESWTR) Toolbox Guidance Manual published by the U.S. EPA.

In addition to CT tables for target pathogens, the LT2ESWTR Toolbox Guidance Manual includes a discussion on procedures for the calculation of log-inactivation credit for various ozone contactor configurations. This discussion includes, among other items, options for measurement and/or estimation of residual ozone concentration and calculation of disinfectant contact time within each chamber of an ozone contactor. Utilities desiring to obtain the pathogen or virus inactivation credit for ozone must utilize

161 Water Environment & Reuse Foundation

one of the methods presented in the guidance manual for calculating the overall CT, unless the state has adopted alternative methods for calculation of CT. A common theme among the options presented is the requirement to sample for residual ozone concentration at multiple points within the contactor and at the contactor effluent.

Figures 4.16 and 4.17 depict the reactor configuration and monitoring equipment required for calculation of CT in an ozone contactor. The figure is focused on the bubble-type ozone diffuser/contactor configuration, as these have been traditionally used in treatment facilities. Additionally, the figure considers the scenario where a utility does not have site-specific tracer data for calculating the hydraulic detention time through the contactor. For more details on bubble diffusers coupled with conventional contactors (chambers divide by baffle walls) and the associated CT calculation methodology accepted by the EPA, refer to Appendix B of the LT2ESWTR Toolbox Guidance Manual.

As noted in the previous section, use of side-stream injection has been increasing due to the higher ozone transfer efficiencies that can be achieved with this technology. Sidestream injection method also requires a downstream contact basin for residual ozone contact time and associated disinfection credit calculations (similar to bubble diffusers). However, sample collection points for ozone residual monitoring would have to be changed in order to accommodate differences of the side-stream injection method. One of the main differences associated with side-stream injection is the location of the first sampling port. Typically, a fraction of dosed ozone will be consumed rapidly due to instantaneous ozone demand in the water (i.e., metals). Once the immediate ozone demand has been satisfied, the rest of the contact basin can potentially be classified as a reaction basin for disinfection calculations. Therefore, care should be taken in locating the first ozone sample port in sidestream injection systems such that enough reaction time is allowed for the immediate ozone demand. For more details on side-stream injection and the associated CT calculation methodology, refer to Appendix B of the LT2ESWTR Toolbox Guidance Manual. Furthermore, some of the contact chambers, especially with side-stream ozonation, may be designed without the baffle walls and can be considered as non-conventional contact reactors. Appendix B of the LT2ESWTR Toolbox Guidance Manual also includes methods for non-conventional contact basins that they can be evaluated in a similar manner as conventional contactors.

162 Water Environment & Reuse Foundation

Figure 4.16. Reactor Configuration and Monitoring Equipment < 3 consecutive reaction chambers

Figure 4.17. Reactor Configuration and Monitoring Equipment > = 3 consecutive reaction chambers

163 Water Environment & Reuse Foundation

Given the regulatory requirements and the primary components of the ozonation process, a number of parameters were defined for process control and monitoring. Water Environment & Reuse Foundation project Reuse-13-03 identified critical control points and associated monitoring parameters for membrane and non-membrane based direct potable reuse treatment trains. The ozonation process was identified to be a critical control point in the non-membrane based treatment train, with the key monitoring parameter being the calculated CT. This parameter was selected because it is directly related to the log inactivation value that can be achieved by the process.

Ozone Dose – Ozone dose in the context of daily operations monitoring is intended to mean the ozone dose required to achieve a target residual ozone concentration in the ozone contactor. The target residual ozone concentration is the value developed during design of the ozone contactor to achieve the required 3-log Cryptosporidium inactivation. Ozone dose should be monitored on a daily basis because a change in ozone dose is indicative of potential upstream water quality changes. Water quality changes resulting in an increase in required ozone dose to achieve a target ozone residual could include an increase in natural organic matter (NOM) concentration or synthetic organic chemical (SOC) concentration, an increase in pH or temperature of the water to be treated, or an increase in carbonate or bicarbonate concentration.

Ozone Residual – This parameter is recommended for continuous monitoring because it factors directly into the calculation of disinfection CT, and therefore the log inactivation value that can be obtained for the process. A change in ozone residual could be attributed to issues with ozone demand (see ozone dose parameter discussion above), ozone delivery equipment within the contactor itself, or issues with ozone monitoring equipment.

Change in UVT – This parameter is recommended for daily monitoring because a change in ultraviolet light transmittance (UVT) is indicative of a change in the NOM concentration in the water being ozonated. Correlations between UVT and organic compounds of interest can be made, and the change in UVT across the ozone process can be attributed to oxidation of the target organic compound(s) of interest. If the change in UVT across the ozone contactor is plotted over time, trends in background water quality can also be observed. For example, if the ozone dose remains constant, and the change in UVT is reduced over time, it can be assumed that an increase in background ozone demand has occurred. This observation could be checked against the ozone residual concentration over time to see if a similar reduction occurred (as it should if ozone demand is increased and the ozone dose is held constant).

4.8.3 General Operating Procedures

The ozonation process should be operated to achieve the target CT for inactivation of pathogens and viruses. As noted above, the primary control parameters for the ozonation process are reactor flow, ozone dose and ozone residual. The discussion below assumes that the target ozone dose and contact time for the ozone contactor have previously been defined through bench-scale demand tests and tracer tests, respectively. The discussion is also focused on the operation of the ozonation process as a whole and is not focused on the depiction of operational procedures for each piece of equipment that comprises the ozonation process. Interested parties are directed to the equipment literature available from ozone equipment suppliers for detailed component operating descriptions.

Reactor flow is the portion of the total plant flow that is conveyed to a given ozone contactor, and is directly related to the disinfection contact time available in the reactor. The higher the reactor flow, the lower the disinfection contact time available, and vice versa. The design maximum reactor flow is the maximum flow through a contactor that still achieves the target CT at the design ozone dose and lowest operating temperature. For plants with multiple contactors, the total plant flow should be evenly split among the available ozone contactors.

164 Water Environment & Reuse Foundation

The plant’s automated control system (PLC, SCADA, etc.) should monitor the flow to each contactor to ensure that the design maximum reactor flow is not exceeded, particularly if one of the other ozone contactors is out of service for repair or maintenance. If the reactor flow exceeds the design maximum value, then the automated control system should take positive control steps to reduce the flow to the reactor or, if not possible, reduce the overall plant flow such that the target CT is always maintained. If the target ozone dose cannot be maintained for whatever reason, the control system should reduce the reactor flow to increase the disinfection contact time, which may also mean reducing the total plant flow. Process control alarms should be designed to alert operators to the events requiring adjustments in process flow.

Ozone residual is the other component of the disinfection CT calculation. The number of ozone residual monitoring points in a given ozone contactor is typically dictated by the method of CT calculation used by the plant. However, additional ozone residual analyzers can be provided to provide greater resolution of contactor performance. The LT2ESWTR Toolbox Guidance Manual published by the U.S. EPA provides the following regarding ozone residual monitoring points and CT calculations. Refer to Figure 4.16 presented previously in this section for graphical depictions of the contactor components associated with CT calculations. In ozone dissolution chambers (other than the first chamber in a contactor), CT credit is only allowed

if there is a measurable ozone residual in the influent to the given dissolution chamber. The CT calculation then uses the effluent ozone residual concentration from the given dissolution chamber.

In the reaction chambers (no ozone addition), CT credit calculations depend on the number of reaction chambers. o If there are less than two reaction chambers, then the CT calculation relies on the effluent ozone

residual concentration from each reaction chamber. o If there are three or more reaction chambers, then the Extended-CSTR Method can be used. This

requires at least three reaction chambers with a detectable ozone residual for the calculation.

Based on the guidance, operators must monitor the ozone residual concentration from at least four locations within each ozone contactor. For drinking water CT credit, the CT must be determined daily during maximum flow; however, it is recommended that the ozone residual be measured at least hourly for data trending purposes.

Ozone dose is the amount of dissolved ozone gas that is delivered to the water in the ozone contactor. The amount of ozone delivered to the ozone contactor must be greater than the ozone demand in the water to be treated in order to obtain a measurable ozone residual. Parameters that can increase ozone demand include natural organic matter concentration, bromide, and carbonate/bicarbonate concentration. Bromide also leads to the formation of brominated disinfection by-products and is a key health risk to be considered when considering the use of ozone for disinfection. As noted in the LT2ESWTR Toolbox Guidance Manual, if the source water feeding an ozone process has a bromide concentration in excess of 50 ppb, there is a high likelihood of forming elevated levels of bromate ion and brominated disinfection by-products. In this latter event, utilities should consider steps to reduce the bromide concentration, adjust the ozonation process to minimize the formation of the disinfection by-products, or consider processes other than ozone.

It is recommended that SCADA plots of operating data be generated for TOC concentration vs UVT to develop a correlation between site specific background organic matter concentrations and online UVT measurements. The latter allows more rapid and easy monitoring of trends impacting ozone dose. Online UVT and TOC monitoring is discussed at the end of this section. Operators should routinely monitor these parameters, as well as the pH and temperature of the water to be treated, and adjust the ozone dose accordingly to achieve the target ozone residual. If the maximum ozone generation and feed capacity of

165 Water Environment & Reuse Foundation

the system cannot overcome the background ozone demand, then corrective actions must be taken as described in the next section.

4.8.4 Critical Control Point Response Procedures

4.8.4.1 Ozone Low CT Alert (Warning) Response In some cases there may be one or two ozone injection points (one prior to BAC and possibly one prior to floc/sed). In this case, the CCP response procedures are identical for both injection points but the alert/alarm set points will need to be adjusted according to dose requirements and function of the process.

In general, ozone systems that are designed for disinfection us the “CT” concept similar to that explained for chlorine. The U.S. EPA has authored a guidance manual ( U.S. EPA, 2010) for the use of ozone for inactivation of pathogens and viruses, including the calculation of disinfection CT for various ozone contactor configurations. Utilities desiring to obtain the pathogen or virus inactivation credit for ozone in drinking water must utilize one of the methods presented in the guidance manual for calculating the overall CT, unless the state has adopted alternative methods for calculation of CT. In the absence of regulatory guidance from a direct potable reuse perspective, it is likely that this guidance will be used by states in that application as well. A common theme among the options presented in the guidance manual is the requirement to sample for residual ozone concentration at multiple points within the contactor and at the contactor effluent.

The LT2ESWTR Toolbox Guidance Manual provides the following considerations regarding ozone residual monitoring locations and CT calculations: In ozone dissolution chambers (other than the first chamber in a contactor), CT credit is only allowed

if there is a measurable ozone residual in the influent to the given dissolution chamber. The CT calculation then uses the effluent ozone residual concentration from the given dissolution chamber.

In the reaction chambers (no ozone addition), CT credit calculations depend on the number of reaction chambers.

If there are less than two reaction chambers, then the CT calculation relies on the effluent ozone residual concentration from each reaction chamber.

If there are three or more reaction chambers, then the Extended-CSTR Method can be used. This requires at least three reaction chambers with a detectable ozone residual for the calculation.

Based on this guidance, operators must monitor the ozone residual concentration from at least four locations within each ozone contactor, thus it is paramount that site-specific response procedures be developed. However, the generalized response procedures presented in this Chapter can be used as a starting place for developing customized response procedures.

Figure 4.18 and 4.19 illustrate the proposed Alert (warning) scenario for investigation of a low disinfection CT alert within the ozonation process. As disinfection CT is affected by both flow and ozone residual, and the latter by the applied ozone dose, there are three alert level response steps. The first step, as depicted in Figure 4.18, is to identify if the individual ozone contact flow is within the design range. Ozone contactor flow is a continuously available parameter and can be reviewed by operators and engineers via the plant SCADA system. Under normal operations, if the operator or engineer review the SCADA trends and observe that the flow is above the design rating of the reactor, the flow to the reactor should be reduced. This may either entail a redistribution of flow to the other online reactors, or a decrease in the overall DPR plant flow.

If the reactor flow is within the design range according to SCADA, or a reduction of contactor flow does not increase disinfection CT to the desired level, then the next step is to investigate the ozone residual

166 Water Environment & Reuse Foundation

measurements at each stage in the contactor. The first step is to review SCADA trends for ozone residual at each monitoring point to identify if any residual measurements are below the target value. At each identified low residual measurement point, staff should take a hand-held ozone residual analyzer and validate the low residual measurement at the point of sampling. If the online measurement is validated, then the ozone dose to the reactor should be increased and reasons for the increase in required ozone dose investigated. If the online measurement is shown to be inaccurate, the affected ozone residual analyzer(s) should be taken offline for recalibration or repair.

When online ozone residual analyzers are taken out of service for maintenance, the impact on the CT calculation needs to be assessed and the calculation method revised if needed. As noted previously in this section, CT credit for ozone dissolution chambers is only granted when there is a measurable ozone residual in the influent to the chamber, and the effluent from the dissolution chamber if used for the CT calculation. If one of the impacted analyzers taken out of service is the influent analyzer to a dissolution chamber, then the operator or engineer must exclude that particular chamber from the overall CT calculation. A similar revision to the CT calculation must be performed depending on the number of reaction chambers and associated online residual analyzers. If the target CT cannot be maintained when the calculation method is revised, then the contactor flow must be reduced in an attempt to achieve the target CT or the overall DPR plant flow reduced and the particular ozone contactor taken offline while the analyzers are recalibrated and/or repaired.

Figure 4.20 illustrates the final step in the ozone alert level response process, namely evaluation of root causes for the required increase in ozone dose. Ultraviolet light transmissivity (UVT) is an indirect measure of organic matter content in the influent to the ozone reactor. The lower the UVT, the greater the amount of organic matter present in the influent. This organic matter can increase ozone demand, requiring a higher ozone dose to achieve the target residual concentration in the ozone contactor. If a reduction in UVT is observed through SCADA, operations staff should discuss potential process performance issues with the upstream wastewater treatment plant operations staff to identify potential measures to improve upstream organics removal. In addition, staff should assess whether or not the ozone diffusers are functioning properly and, if they are not, incorporate the affected diffusers into the plant’s preventative maintenance program. This latter item can be verified by residual ozone concentration levels or pressure in the ozone feed line. The ozone concentration in the contactor headspace should be measured prior to the destruct units and trends developed. If the residual ozone concentration increases, then there is an issue with one or more diffusers. Likewise, it there is clogging or other issues with the diffusers themselves, the pressure in the ozone feed line will increase.

4.8.4.2 Ozone Low CT Critical Alarm (Failure) Response Figure 4.21 illustrates the Critical (Failure) response procedures for the ozonation system. The first step in the response is an automatic shutdown of the affected ozone contactor. Following shutdown of the contactor, plant staff should notify the plant manager and/or supervisor of the shutdown and review the alert level response procedures to confirm that the critical (failure) limit breach is real. If the breach is real, then staff should perform the following actions: Verify the calibration status of the flow meters and online ozone residual analyzers, and collect

confirmatory grab samples to validate the online residual analyzer values. Verify that no upstream process upsets have occurred that might impact ozone demand. Review SCADA trends for ozone dose and residual, flow, and influent UVT. Review ozone diffuser performance. Verify the CT calculation performed in the PLC by hand. Assess the water quality of the contactor effluent. Perform incident reporting to appropriate stakeholders.

167 Water Environment & Reuse Foundation

If the water quality was actually compromised during the critical limit breach, additional actions and interaction with external entities may be required.

Table 4.7. Ozone Low CT Alert and Alarm Summary

Monitoring Parameter Alert Level Critical (Failure) Level Notes

CT (mg/L.min) Overall monitor

Flow (gpm) (per contactor)

10 % greater than maximum flow for each contactor.

(15 min moving average)

20% greater than maximum flow for each contactor.

(15 min moving average)

This is performed on a per-contactor basis.

Ozone Residual (mg/L) (per contactor)

< 25% of target

15 min moving average

< 50% of target

15 min moving average

Target is set to be well above the minimum that is required to achieve chlorine CT. In this case, it is set such that it is 2 x the requirement for minimum CT. This should be evaluated case by case at each facility.

Delta UV Transmittance (%) (across ozone contactor)

Less than 110% of minimum setpoint.

(15 min moving average)

Minimum setpoint.

(15 min moving average)

Actual setpoint will be site-specific based on design and operating experience. The intent is the detection in a change to normal conditions.

168 Water Environment & Reuse Foundation

Figure 4.18. Ozone Low CT and Low Flow Alert Procedures

169 Water Environment & Reuse Foundation

Figure 4.19. Ozone Low CT and Low Ozone Residual Alert Procedures

170 Water Environment & Reuse Foundation

Figure 4.20. Ozone Low CT and Low UVT Alert Procedures

171 Water Environment & Reuse Foundation

Figure 4.21. Ozone Low CT Critical Failure Response

172 Water Environment & Reuse Foundation

4.8.5 Maintenance Activities

The following lists the maintenance activities that should be followed to keep the instrumentation associated with the ozonation process in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual, rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks Ozone residual monitors UVT monitors

Monthly Calibration Ozone gas monitors Ozone residual monitors UVT monitors Flow meters Bench top ozone analyzer Flow control valve and actuator maintenance

4.8.6 Operations-Focused Design Considerations

The HACCP process performed as part of Reuse-13-03 identified the ozonation process as a critical control point in the non-RO treatment process train for direct potable reuse. Through risk assessment procedures and Monte Carlo modeling, key operating parameters and associated alert levels and critical limits were identified. For ozone, this effort identified the target log reduction value (LRV) to be obtained through the process. To provide the level of performance monitoring and control necessary to appropriately respond to alert level and critical limit situations, certain items should be included in the overall design of the ozonation process. Some of these items may be greater than what is required in a standard drinking water ozonation process design; however, it is important that these additional items be provided in the DPR situation to further reinforce the robustness of the process relative to public health protection.

Multiple reactor trains should be provided for operational flexibility and redundancy. In the event of a critical limit condition, multiple contactors would allow the plant to continue to operate within the design flow and dose ranges when the problem contactor is automatically taken offline. Hydraulic flow splitting structures on the influent of the reactor trains would provide a simple, operator-friendly means of equally dividing the influent flow among the units. However, individual flow metering and a means of positively affecting the flow split in the event of an ozone dose/residual alert would also be required. Effluent isolation devices interlocked with critical limit alarms are necessary to isolate the flow from any ozone contactor trains experiencing critical limit situations.

Another aspect of the instrumentation design that is critical to the operation of the ozonation system is the online ozone residual analyzer. The sampling ports within the contactor should be located to obtain representative ozone residuals at each monitoring point, and the residual analyzers should be located in close proximity to each sampling port to minimize travel time between sample point and analyzer. The latter is important because ozone decays rapidly and the goal is to obtain an ozone residual measurement that is as close to what exists in the contactor as possible.

173 Water Environment & Reuse Foundation

While not required for meeting CT through the ozonation process, influent TOC and UVT and effluent UVT analyzers should be considered for networking to the plant SCADA system. As noted earlier in this section, changes in UVT can help to optimize the operation of the ozonation process, be used to identify required dosing for chemical oxidation, and provide information about how much chemical oxidation has occurred for a given ozone dose/water quality condition. Paired TOC and UVT analyzers on the influent to the ozone process will allow for monitoring of background organic matter concentration and potential ozone demand changes, and can also allow for a correlation to be developed between the two parameters. The influent instruments can be located on the overall influent flow to the ozonation process, but the effluent instruments should be installed on the end of each reactor train.

The system must allow for data logging and plotting all of the various monitoring results in SCADA. Analysis of performance trends is key to optimizing ozone contactor performance as well as early identification of potential problem situations (i.e., increasing ozone demand).

Depending on the source water bromide concentration, the design of the ozonation system may need to incorporate operational flexibility and/or alternative chemical feeds to address bromate ion and brominated disinfection by-product (DBP) formation. Research results reported in the literature indicate varying degrees of success using hydrogen peroxide or ammonia to control bromate and brominated DBP formation. Providing the ability to control the ozonation pH will allow operators to adjust the process to reduce the formation of either bromate ion or brominated DBPs (lower pH favors brominated DBPs, while higher pH values favor bromate ion formation). Hydraulic detention time in the ozone contactor also affects the formation of ozone-related DBPs, and can be an important part in the ability of ammonia feed to control the formation of those DBPs.

The hydraulics through an ozone contactor can directly impact the efficiency of the process. It is recommended that design engineers consider the use of computational fluid dynamic (CFD) modeling to optimize the hydraulic design of the contactor, including baffle wall locations, chamfering, and inlet and outlet conditions. Tracer tests should also be considered to develop site specific characterization of contactor hydraulics for use in CT calculations. The ozone guidance manuals referenced previously in this section provide further details on the benefits of tracer studies.

4.9 Biologically Active Carbon (BAC)

4.9.1 Process Overview

In a biologically active filter, the media acts simultaneously to support the growth of biomass and as a filtration medium to retain any particulate matter. The microbial growth attached to the filter media (biofilm) consumes some of the organic matter that would otherwise flow through the treatment plant and ultimately into the distribution system. The end products are carbon dioxide, water, biomass, and simpler organic molecules. Particle filtration takes place on the filter media as well as the biofilm. Granular activated carbon (GAC) is often used to provide the necessary surface to promote the development of the biofilm. Therefore, these systems are commonly referred as biologically active carbon (BAC). Accumulated solids are removed from the BAC through backwashing. Backwashing also removes some of the biofilm which would lead to clogging of the BAC if not controlled.

The main goals of BAC in the context of the non-membrane based DPR process train are to reduce the concentration of AOC after ozonation and to remove pathogenic cysts (e.g., Cryptosporidium and Giardia). Since BAC is essentially a filter bed, the U.S. EPA provides up to 2.5-log inactivation credit for Giardia and 2-log inactivation credit for viruses and Cryptosporidium for properly operated systems (with upstream coagulation processes) as reported in the National Primary Drinking Water Regulations and Long Term 2 Enhanced Surface Water Treatment Rule (LT2ESWTR) Toolbox Guidance Manual

174 Water Environment & Reuse Foundation

published by the U.S. EPA. While strict monitoring of turbidity is required, this credit is provided regardless of the water temperature or pH. As such, this is an important process in the non-membrane train to achieve the California DDW goal of 12-10-10 for reuse systems. The fact that BAC process can remove organic contaminants and achieve log reduction credit in a single process makes it a critical treatment process in the non-RO treatment process train for direct potable reuse. Two operation modes of BAC are discussed in this section which are ozone- and coagulant-BAC.

As described in the ozone disinfection section, pre-ozonation provides many benefits to the water treatment process (e.g., no formation of halogenated DBPs formation, color removal, iron and manganese removal, reduction of taste and odor, enhanced biological activity, etc.). However, pre-ozonation by-products are generally readily biodegradable and can lead to biogrowth in the distribution system. Therefore, ozone is typically coupled with BAC for AOC removal. In this configuration, pre-oxidation results in the formation of readily biodegradable dissolved organic carbon (BDOC) including aldehydes, volatile fatty acids, and keto-acids which are consumed by the biological activity in the process. This increases the biological stability of water, and reduces the rate of biofilm accumulation in downstream processes (i.e., distribution system). In addition, the process can lead to the complete removal of ozonation by-products that are of health concern and may be targeted for future regulations (i.e., short-chain aldehydes). BAC is also effective for eliminating synthetic organic chemicals (SOCs) and micropollutants such as benzene, toluene, and pesticides like atrazine which present health concerns. The process also can reduce the concentrations of taste- and odor-causing compounds. Additionally, BAC can minimize DBP formation via destruction of organic DBP precursors. Use of BACs have been shown to reduce the need for residual chorine addition and the level of combined organic chlorine. Finally, ozone coupled BAC can enhance the adsorption capacity of GAC for non- or slowly biodegradable compounds by eliminating substances that would otherwise compete for adsorption sites.

For the BAC to be an effective filter for the removal of suspended solids, the feed water to BAC would have to be conditioned. Most suspended solids in water possess have a negative charge and consequently repel each other. This repulsion prevents the particles from agglomerating, causing them to remain in suspension. Coagulation in wastewater can be an effective technique for suspended solids. The addition of coagulant (i.e., ferric chloride) destabilizes material by neutralizing the charges and form a gelatinous mass to trap (or bridge) thus forming a mass large enough to settle or be trapped in the filter. During this process the following components including suspended, dissolved organic and/or inorganic matter, as well as several biological organisms, such as bacteria, algae or viruses may be removed. Therefore, a coagulant feed system prior to the BAC is imperative. An optimized coagulation system would also decrease extend the operational time of BAC.

4.9.2 Key Operating Parameters

As noted above, the main goals of BAC in the context of the non-membrane based DPR process train are to reduce the concentration of AOC after ozonation and to remove pathogenic cysts (e.g., Cryptosporidium and Giardia). From a CCP perspective, however, the only real control mechanism in BAC is related to pathogen reduction. Thus, discussion within this section includes chemical removal which is ancillary to the BAC’s primary function as a filter for particle removal.

The U.S. EPA provides up to 2.5-log inactivation credit for Giardia and 2-log inactivation credit for viruses and cryptosporidium for properly operated systems. While strict monitoring of turbidity is required, this credit is provided regardless of the water temperature or pH. As such, this is an important process in the non-membrane train for direct potable reuse to achieve the California DDW goal of 12-10-10 for reuse systems. Given these purposes, several parameters were defined for process control and monitoring. Reuse-13-03 identified critical control points and associated monitoring parameters for membrane and non-membrane based direct potable reuse treatment trains. The BAC process was identified to be an important process in the non-membrane based treatment train for the removal of

175 Water Environment & Reuse Foundation

organics (particularly AOC) when coupled with upstream ozonation, and achieving inactivation credit for Giardia, Cryptosporidium and viruses when combined with an upstream coagulation process. For the ozone/BAC combination, the key monitoring parameters were identified as high ozone dose and low empty bed contact time (EBCT). For the pairing of coagulation and BAC, high filtrate turbidity and high coagulant dose were selected as the key parameters.

4.9.2.1 Ozone-BAC Ozone Dose – This parameter was selected for monitoring of the ozone-BAC process pair because a change in ozone dose is an indicative of potential upstream water quality changes that may influence the performance of the BAC. The target residual ozone concentration is the value developed during design of the ozone system to achieve the intended conversion of DOC to biodegradable organic carbon. Water quality changes resulting in an increase in required ozone dose to achieve a target ozone residual could include an increase in natural organic matter (NOM) concentration or synthetic organic chemical (SOC) concentration, an increase in pH or temperature of the water to be treated, or an increase in carbonate or bicarbonate concentration.

Empty Bed Contact Time – EBCT is a measure of how long the water to be treated stays within a BAC, and can be easily calculated by dividing the volume of the BAC contactor by the contactor flow rate. This parameter was selected for monitoring the ozone-BAC process pair because a low EBCT would directly impact the performance of this process. The use of EBCT in the context of this CCP monitoring parameter is intended to mean the contact time required to provide sufficient time for adsorption or assimilation/biodegradation of organic compounds. EBCTs for BAC process is typically within the range of 15-20 minutes. The plant automation system should track and record the per contactor flow rate as well as calculate the EBCT associated with that flow rate. Operators should routinely monitor these two parameters and compare them to the design values for the reactor. If the calculated EBCT is less than the design EBCT, then insufficient contact time is being provided in the BAC contactor for the removal of biodegradable organic carbon. In this situation, the flow to the individual reactor should be reduced until the design EBCT is achieved.

4.9.2.2 Coagulant-BAC Turbidity – This parameter was selected because it is a direct indicator of filtration efficiency and is used by regulators in the application of inactivation credits. LT2ESWTR states that the combined filter effluent turbidity must be less than or equal to 0.3 nephelometric turbidity units (NTU) in 95% of samples taken each month and must never exceed 1 NTU. This provides 2.5 LRV credit if combined filtered water and 3 LRV if an individual filter. An additional log reduction credit is allowable if turbidity is less than 0.15 NTU. However, the Partnership For Safe Drinking Water stipulates a 0.1 NTU goal. A high filtered water turbidity could be indicative of a dirty filter bed (requiring backwashing) or an un-optimized coagulation system.

Coagulant Dose – This parameter was selected because a change in the filter effluent turbidity can be directly related to a change in coagulant dose needs. This change is most often caused by variations in the influent quality (i.e., alkalinity, turbidity, or organic matter concentration), but it can also be a result of malfunctions in chemical metering pumps, chemical mixing, or other components of the coagulation process.

4.9.3 General Operating Procedures

As discussed in the previous section, BAC provides dual benefits via its coupling with both the ozone and coagulation process steps. BAC should be operated to achieve the target removal of pathogenic cysts (e.g., Cryptosporidium and Giardia) from the reuse process stream, but with an eye towards synergy with

176 Water Environment & Reuse Foundation

chemical removal. Accordingly, plant operators need to operate the BAC process to optimize the performance of both modes.

One of the main goals of BAC is to remove organic contaminants and to achieve that DOC must be converted into BDOC and AOC prior to the BAC. Therefore, operators should follow the ozone dose and residual periodically. Operators should maintain the ozone dosing system according to manufacturer’s recommendations. The calibration of the analytical instruments for tracking ozone consumption should also be done periodically.

The removal of organic contaminants is most often tracked by ensuring that the design EBCT is maintained. The EBCT is a measure of how long the water to be treated stays within a BAC media, and can be easily calculated by dividing the volume of the BAC media by the flow rate. The plant automation system should track and record the per contactor flow rate as well as calculate the EBCT associated with that flow rate. Operators should routinely monitor these two parameters and compare them to the design values for the reactor. If the calculated EBCT is less than the design EBCT, then insufficient contact time is being provided in the BAC media for the removal of biodegradable organic carbon. In this situation, the flow to the individual reactor should be reduced until the design EBCT is achieved.

In addition to EBCT, UVT should be measured in both the feed water and the treated water, and the delta UVT value calculated for each contactor (the use of this parameter was discussed in more detail above). While the delta UVT measurement is a rapid and easy monitoring method, it can be prone to interference from background constituents in the water matrix. Therefore, it is recommended that TOC removal data be collected together with UVT to validate the UVT measurements. It is also recommended that plots of operating data be generated for TOC vs UVT to develop a correlation between site specific background organic matter concentration and online UVT measurements.

In addition to the organics, BAC process is important for removing suspended solids and the pathogenic cysts (e.g., Cryptosporidium and Giardia) from the reuse process stream. Turbidity should be measured in the filtrate continuously as it is important to maintain the log reduction credit for Giardia, Cryptosporidium and viruses. If the turbidity of a BAC filtrate reaches pre-determined levels, it should be taken offline and backwashed. During the backwash period, the flow should be split evenly among the operating BAC’s or should be diverted to the stand-by BAC.

Due to biological growth, the pores between the activated carbon will be filled with cells and extra cellular polymeric materials. Clogging of the pores will increase the headloss through the filter over time. Consequently, if not controlled, flow through a BAC filter will decrease and possibly flow through other BAC will increase. Since this will adversely affect the EBCT, it indicates that controlling flowrate to each BAC is critical when operating multiple BACs. When the headloss through a BAC reaches pre-determined levels, it should be taken offline and backwashed. During the backwash period, the flow should be distributed among the operating BACs or should be diverted to the stand-by BAC.

Along with the headloss and turbidity, BAC operators should also monitor the coagulation system closely. Optimum coagulant dosing will result in lower headlosses and longer operation in between backwashes. To determine the optimum coagulant dose and coagulant dose level, operators should conduct jar tests periodically. Also, the chemical metering pumps should be regularly maintained following manufacturers suggestions. The calibration of the pumps should also be done periodically. A set of spare parts or a spare pump should be available for immediate response to a malfunctioning chemical metering pump.

177 Water Environment & Reuse Foundation

Operators should routinely monitor the parameters discussed above, as well as the pH and temperature of the water to be treated. If the BAC cannot provide removal of organic contaminants or meet turbidity requirements, then corrective actions must be taken as described in the next section.

4.9.4 Critical Control Point Response Procedures

4.9.4.1 Ozone/BAC Alert (Warning) Response The ozone/BAC (biologically active carbon) combined process and critical control point is intended to reduce the concentration of organic contaminants in the recycled water, including emerging contaminants of concern that may not be captured in the initial risk assessment. The process has two key operating parameters associated with it: ozone dose and empty bed contact time (EBCT). Ozone dose is the dose applied in the ozone contactor to oxidize the organic compounds, while EBCT is the time that the oxidized organic compounds in the water being treated reside within the carbon bed of the biologically active filter. The contact time allows for both adsorption of organic compounds to the open pore spaces in the carbon media or degradation of those compounds via biological activity. Sometimes dose is determined by (a) a desired residual concentration for CT credit, (b) a ratio of ozone to TOC, or (c) an observed change in UVT across the ozone contactors (Serna, Trussell and Gerringer, 2014; Snyder, Korshin, Gerrity, et al., 2013; Wert, Rosario-Ortiz and Snyder, 2009). All three dosing strategies have their merits and may be synergistically used together for multiple treatment objectives.

It should be noted here that in the case of an ozone-BAC combined process there are varying objectives, including disinfection (which is handled by the “ozone” CCP alone) and the oxidation and biodegradation of dissolved organic constituents in the water which is handled by the ozone/BAC CCP. In this case, other measures of process performance may be needed besides a simple CT calculation. UV transmittance has been shown to be a good surrogate for contaminant oxidation as well as disinfection, and provides a useful monitoring tool for ozone-BAC applications (Gerrity, Gamage, Holady, et al., 2011; Pisarenko, Stanford, Yan, et al., 2012; Snyder, Korshin, Gerrity, et al., 2013; Wert, Rosario-Ortiz and Snyder, 2009). Thus, there will be a need for separate site-specific alert (and alarm) procedures to be developed for UVT-based monitoring and control systems that may be applied in water reuse applications.

Figure 4.22 depicts the response procedures associated with alert (warning) level notifications for either of the two key operating parameters. In the case of a high ozone dose alert, the plant operator or engineer should follow the high ozone dose investigative procedures depicted previously in Figure 4.19 and described above. In the case of a low EBCT alert, plant staff should identify through SCADA if the flow to the biologically active carbon filter is outside of the design range. If the flow is too high to achieve the target EBCT, then the flow to the BAC filter should be reduced until the target is met. This can entail either a redistribution of flow to the online BAC filters or a reduction in the overall DPR plant flow. If the individual BAC filter flow is too high and cannot be reduced, then a critical (failure) condition exists and those procedures should be followed.

4.9.4.2 Ozone/BAC Critical (Failure) Response Figure 4.22 illustrates the Critical (Failure) response procedures for the ozone/BAC critical control point. The first step in the response is an automatic shutdown of the affected ozone contactor and/or BAC filter. Following shutdown of the contactor and/or filter, plant staff should notify the plant manager and/or supervisor of the shutdown and review the alert level response procedures to confirm that the critical (failure) limit breach is real. If the breach is real, then staff should perform the following actions: Verify the calibration status of the flow meters and online UVT analyzers, and collect confirmatory

grab samples to validate the online UVT analyzer values. Verify that no upstream process upsets have occurred that might impact ozone demand. Review SCADA trends for ozone dose, flow and influent UVT.

178 Water Environment & Reuse Foundation

Review ozone diffuser performance. Assess the water quality of the ozone/BAC process effluent. Perform incident reporting to appropriate stakeholders.

If the water quality was determined to have been compromised during the critical limit breach, additional actions and interaction with external entities may be required. Table 4.8. Ozone-BAC Alert and Alarm Summary

Monitoring Parameter Alert Level Critical (Failure) Level Notes

Ozone Dose (mg/L)

90% of maximum dose

(15 min moving average)

Maximum Dose

(15 min moving average)

Ozone dose is controlled to match inlet flow, and influent UVT. If ozone dose required is higher than maximum, this may be due to a high flow or higher UVT than design capability.

A loss of ozone dose will be captured as a part of the individual ozone disinfection CCP.

EBCT (Flow per filter) 10 % greater than maximum flow for each contactor.

(15 min moving average)

20% greater than maximum flow for each contactor.

(15 min moving average)

This is performed on a per-contactor basis.

4.9.4.3 Coagulant/BAC Alert (Warning) Response The coagulant/BAC combined process and critical control point is intended to provide control for the removal of pathogenic cysts (e.g., Cryptosporidium and Giardia) from the DPR process stream. This critical control point includes the processes of rapid mixing/coagulation, flocculation (perhaps) and settling and BAC filtration. The pre-treatment processes are required upstream of the filtration process to condition the cysts and other particles in the water for removal via interception on the individual carbon grains. The upstream coagulation process is also required by the US EPA to receive credit for filter performance in removing pathogenic microorganisms. Since it is difficult and expensive to quantify pathogenic cyst removal across a filter (and impossible to do so in real time), achieving a target filter effluent turbidity goal is used as a surrogate. As long as the effluent turbidity from the BAC filter is less than the target value (generally 0.1 NTU), then the pathogen log reduction value across the filter can be reasonably assumed to have been achieved.

Figure 4.24 depicts the alert (warning) level response process for this critical control point associated with a high filter effluent turbidity reading. The first step in the response process is for the plant operator and/or engineer to review the SCADA flow readings to verify that the processes are operating within their design loading rates. If not, then the process flow rate should be reduced to achieve the target rates and avoid stressing the unit processes beyond their design capacity. Next, staff should review the SCADA trends for individual and combined BAC filter effluent turbidity to identify if this is an isolated problem or plant-wide problem. Grab samples should be collected from each sampling point that indicates an elevated turbidity for validation on a bench-top turbidimeter. If the online turbidimeter readings are shown to be correct, then the affected BAC filter(s) should be taken out of service and placed into the backwash rotation for cleaning.

If backwashing the affected filter(s) does not resolve the high turbidity alert, then plant staff need to evaluate the coagulation process. Coagulation neutralizes the surface charges on the particles and cysts in

179 Water Environment & Reuse Foundation

the water so that they will effectively adhere to the individual BAC filter media grains. The first step in the coagulation assessment is to verify through SCADA that no changes in upstream pH have occurred. If they have, then a pH adjusting chemical needs to be fed to reduce the pH into the optimum range for proper coagulation to occur. Following this, the operations staff should perform jar tests to verify the amount of coagulant to be delivered to the process and adjust the coagulant dose full-scale. Staff should then continue to monitor SCADA trends to ensure that the issue is resolved, and should repeat the above steps again if needed to further resolve the issue. Other steps that can be taken to evaluate the alert include assessing background coagulant demand (via online UVT analysis), chemical dosing pump calibration and verification of the coagulant dose control algorithm in the plant’s PLC.

Finally, long-term monitoring of filter performance including filter run times, individual filter turbidity, head loss, filter profiles of turbidity breakthrough over time, unit filter run volumes (UFRVs), backwash settings and backwash turbidity, among others, can all be used to assist in the diagnosis and repair of filter inefficiencies or issues that may trigger an alert before they escalate into failure and alarm status. It is key that operations teams collect and review such data throughout the life of the plant to better learn from past experience and use that information to assist in decision making and asset maintenance/repair.

4.9.4.4 Coagulant/BAC Critical (Failure) Response Figure 4.25 illustrates the Critical (Failure) response procedures for the coagulant/BAC critical control point. The first step in the response is an automatic shutdown of the affected BAC filter(s) and incorporation of said filter(s) into the overall backwash queue. Following filter backwash, plant staff should notify the plant manager and/or supervisor of the process unit shutdown and review the alert level response procedures to confirm that the critical (failure) limit breach is real. If the breach is real, then staff should perform the following actions: Verify the calibration status of the flow meters and online turbidity analyzers, and collect

confirmatory grab samples to validate the online turbidimeter values. Verify that no upstream process upsets have occurred that might impact coagulant demand Review SCADA trends for coagulant dose, flow, and influent pH. Review filter backwash history and perform additional jar tests. Assess the water quality of the coagulant/BAC process effluent. Perform incident reporting to appropriate stakeholders.

If the water quality was actually compromised during the critical limit breach, additional actions and interaction with external entities may be required.

Table 4.9. Coagulation-BAC Alert and Alarm Summary

Monitoring Parameter

Alert Level Critical (Failure) Level Notes

Filtered Water Turbidity (NTU)

90% of maximum turbidity

(15 min moving average)

Maximum turbidity (typically 0.1 NTU)

(15 min moving average)

Laser turbidimeter can be considered for higher resolution of turbidity.

180 Water Environment & Reuse Foundation

Figure 4.22. Ozone/BAC Alert Response Procedures

181 Water Environment & Reuse Foundation

Figure 4.23. Ozone/BAC Critical Failure Response Procedures

182 Water Environment & Reuse Foundation

Figure 4.24. Coagulant/BAC Alert Response Procedures

183 Water Environment & Reuse Foundation

Figure 4.25. Coagulant/BAC Critical Failure Response Procedures

184 Water Environment & Reuse Foundation

4.9.5 Maintenance Activities

The following items include the maintenance activities that should be followed to keep the instrumentation associated with the BAC process in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual, rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks Turbidimeters. UVT monitors. COD or TOC monitors (if applicable).

Monthly Calibration Turbidimeters. UVT monitors. Flow meters. Flow control valve and actuator maintenance. COD or TOC monitors (if applicable). Bench top COD analyzer (if applicable). Bench top or online organic carbon analyzer (if applicable).

4.9.6 Operations-Focused Design Considerations

The HACCP process described in Reuse-13-03 identified the BAC filters as an important process for the removal of both organics and pathogenic cysts (e.g., Cryptosporidium and Giardia) in the non-membrane based treatment process train for direct potable reuse. Key operating parameters and associated alert levels and critical limits were identified for the process. For BAC filters, the critical triggers points were identified as high ozone dose and low EBCT for ozone-BAC, and high filtrate turbidity and high coagulant dose for coagulant-BAC. To provide the level of performance monitoring and control necessary to appropriately respond to alert level and critical limit situations, certain items should be included in the overall design of BAC filters. Most of these items are what would be required in a standard BAC filter design for a drinking water system; however, it is important that the BAC parameters shall be monitored closely in the DPR situation to further reinforce the robustness of the process relative to public health protection.

Similar to other processes, multiple trains should be provided for operational flexibility and redundancy. For the BAC filters, the redundancy should be considered as n+1, The additional filter however will run in operation to ensure that EBCT is met with one filter in backwash. This will allow the system to continue operation while one BAC is taken offline for backwashing. In the event of a critical limit condition, multiple BAC filters would allow the plant to continue to operate within the design flow and dose ranges when the problem contactor is automatically taken offline. Individual flow metering and a means of positively affecting the flow split in the event of an alert is required. As discussed previously, flowrate to each BAC should be monitored closely for two purposes: EBCT and headloss. Effluent isolation devices interlocked with critical limit alarms are necessary to isolate the flow from any BAC filter experiencing critical limit situations (i.e., turbidity).

UVT is a measure of the ratio of light entering and exiting water, and can be measured either via an online system or via a bench-top analyzer. UVT is inversely proportional to UV absorbance, and both can be correlated to the level of contaminants in the water. As noted earlier in this section, change in UVT is a good indicator of a change in BAC contactor performance; however, since it is a spectrophotometric

185 Water Environment & Reuse Foundation

method, it is prone to interference from background constituents in the water matrix. While not required, TOC analyzers should also be considered for monitoring the level of organics removal. Paired TOC and UVT analyzers on the influent and effluent of the BAC process will allow for monitoring of background organic matter concentration and potential exhaustion of BAC, and can also allow for a correlation to be developed between the two parameters. The influent instruments can be located on the overall influent flow to the BAC contactors, but the effluent instruments should be installed on the end of each individual contactor.

Turbidity is a measure of the scattering of light of a water sample and can be correlated to the level of colloidal material in water flow. Similar to UVT, turbidity can be measured via an online instrument or a bench-top analyzer. The former should be used on the influent and effluent of the BAC units to provide real-time monitoring of BAC performance, whereas the latter should be used to periodically validate the measurements of the online instruments. The influent instruments can be located on the overall influent flow to the BACs, but the effluent instruments should be installed on the end of each filter. The system must allow for data logging and plotting all of the various monitoring results in SCADA (i.e., pH, DOC, UVT, turbidity). These parameters can be used to develop correlations with different parameters in the feed water (i.e., turbidity, DOC).

4.10 GAC

4.10.1 Process Overview

Conventional water reclamation facilities (WRFs) have been designed for the removal of bulk substances including organic matter and the nutrients nitrogen and phosphorus. Although the majority of the organics are removed, the remaining organic content is still too high for potable water applications. The presence of high levels of effluent organic matter (EfOM) can lead to operational issues in the sequential treatment of reclaimed water, such as high oxidant demand, formation of disinfection by-products, and microbial growth in distribution systems, among other issues. Among the various types of EfOM, particular concern has been associated with the presence of “emerging contaminants”. These emerging contaminants, typically referred to as micropollutants, include but are not limited to pharmaceuticals and personal care products (PPCPs), endocrine disrupting chemicals (EDCs), plasticizers, and flame retardants. Furthermore, some micropollutants have very limited removal efficiencies in conventional WWTPs (<20%) (Kim et al., 2014). Some of the available technologies for the efficient removal of the more recalcitrant EfOM include membrane filtration, activated carbon adsorption, oxidation, and riverbank filtration.

Activated carbon is an adsorbent with a highly porous structure, with a broad range of pore sizes ranging from visible cracks and crevices down to molecular dimensions. Intermolecular forces cause molecules of dissolved contaminants to be accumulated on the adsorbent surface. Activated carbon is an ideal media as it provides a large surface area on which the contaminants can adhere (typical surface area >700 m2/g).

In water treatment, GAC is generally used in adsorption beds or columns. Contaminant molecules are adsorbed on the surfaces until the active sites are exhausted and the carbon can no longer adsorb any contaminant. At that point, the spent carbon must be replaced with a virgin or regenerated carbon. The efficiency of activated carbon adsorption depends on the accessibility of the target organic compounds to the adsorption sites, which in turn depends on the properties of both the adsorbates (the target compounds) and adsorbents (the activated carbon). Thus, the level of dissolved organic carbon (DOC), the amount of contact time between the contaminants and the GAC, and the type of GAC are major parameters influencing the effectiveness of activated carbon adsorption.

186 Water Environment & Reuse Foundation

In general, due to the high level of organics in the primary or secondary wastewater treatment effluent, the number of available adsorption sites in a GAC contactor will rapidly start decreasing after the feedwater starts flowing through the media. Consequently, GAC media has to be replaced or regenerated periodically to maintain a high adsorptive capacity for organics removal. However, in the context of the non-membrane based DPR process train, the GAC process would be installed downstream of ozone and BAC. The presence of the BAC process prior to GAC contactors will increase the process runtime before carbon replacement/regeneration is required. This is because the BAC step will remove the majority of the AOC created through ozonation of the EfOM as well as the biodegradable dissolved organic carbon (BDOC). The remaining organic content is expected to be relatively low; therefore, the second stage GAC contactors can act as a “polishing step” to provide an additional barrier for the remaining organic contaminants, including the micropollutants discussed above.

In addition to organic matter adsorption, if designed and operated properly, GAC contactors can also be classified as second stage filters. This type of application would qualify for additional microbial removal credit under the Long-Term 2 Enhanced Surface Water Treatment Rule (LT2ESWTR) (i.e., an additional 0.5-log reduction of Cryptosporidium) (U.S. EPA, 2010). However, in the context of this study and the overall non-membrane based DPR process train, only the removal of organics is considered for the GAC process.

4.10.2 Key Operating Parameters

As discussed in the previous section, the main goal of GAC adsorption is to remove EfOM and micropollutants from the reuse water flow. Given this purpose for GAC adsorption, a number of parameters were defined for process control and monitoring. Reuse-13-03 identified critical control points and associated monitoring parameters for membrane and non-membrane based direct potable reuse treatment trains. The GAC process was identified to be an important process in the non-membrane based treatment train for the removal of organics, with the key monitoring parameters being the change in UVT of the water (the “delta UVT”) and the remaining carbon life. These parameters were selected because they are directly related to the available capacity for organic matter adsorption as described further in the following paragraphs. Alert level and critical level monitoring parameters are summarized below.

Change in UVT – This parameter was selected because a change in UVT is indicative of a change in the organics concentration in the water being treated by GAC contactors. UVT is a measure of the ratio of light entering and exiting water, usually reported for a path length of 1 cm, and is the inverse of UV absorbance. As UV absorbance increases (by organics or suspended solids interfering with the light penetration), UV transmittance decreases. Suspended solids interference with the UV absorbance measurement is not anticipated to be a factor because the GAC contactors would be installed downstream of the BAC process. Therefore, during GAC treatment, a change in UVT would be associated with a change in organic matter concentration.

Correlations between UVT measurements and the concentration of organic compounds of interest can be made, and the change in UVT across the GAC contactors can be attributed to the adsorption of these organic compounds of interest. A fresh GAC media would remove the majority of the influent organic matter, thus the treated water would have a much higher UV transmittance and much lower UV absorbance than the feed water. As the GAC media becomes exhausted and less adsorption sites are available, some of the organic matter will pass through the contactor. Consequently, the UV transmittance will decrease and UV absorbance will increase, gradually approaching the feed water values. As the delta UVT decreases, it is an indicator that the activated carbon sites are being exhausted and some of the organic matter is not being removed. If the change in UVT across the GAC contactors is plotted over time, trends in background water quality can also be observed. These trends can be compared to the remaining carbon life to make informed decisions regarding carbon regeneration / replacement.

187 Water Environment & Reuse Foundation

Carbon Life – This parameter was selected because the effectiveness of a GAC contactors to remove organic matter is based on the availability of adsorption sites. GAC contactors cannot remove organic matter once their adsorption sites are exhausted by the organic contaminants.

There are different types of GAC available on the market, each with different characteristics. GAC media selection is typically based on performance testing of several carbons using the specific feed water to be treated to account for potential impurities competing with the target organic matter for adsorption sites. This GAC media performance testing can be used to aid in the selection of the appropriate GAC media and to estimate the expected carbon usage rate (carbon life). Isotherm testing and rapid small-scale column testing (RSSCT) are two such tests that can be conducted to determine design parameters. Both of these tests provide essential data used to evaluate the effectiveness of different activated carbon types, estimate the required empty bed contact time (EBCT) and activated carbon usage rate, develop a breakthrough profile for various contaminants, and identify the expected lifetime for the preferred type of GAC media.

As noted above, decreases in the delta UVT across a GAC contactor is indicative of activated carbons exhaustion. Due to fluctuations in feed water quality, the lifetime of GAC may vary but should generally be within the initial range determined by preliminary testing (RSSCT or pilot-testing). When a low delta UVT alert is triggered, if the current age of the GAC media is relatively close to or exceeds the pre-determined life from initial testing, then the activated carbon should be replaced or regenerated.

4.10.3 General Operating Procedures

GAC contactors should be operated to achieve the target removal of organic contaminants. This is most often tracked by ensuring that the design EBCT is maintained. The EBCT is a measure of how long the water to be treated stays within a GAC contactor, and can be easily calculated by dividing the volume of the GAC contactor by the contactor flow rate. The plant automation system should track and record the per contactor flow rate as well as calculate the EBCT associated with that flow rate. Operators should routinely monitor these two parameters and compare them to the design values for the reactor. If the calculated EBCT is less than the design EBCT, then insufficient contact time is being provided in the GAC contactor for organic matter adsorption. In this situation, the flow to the individual reactor should be reduced until the design EBCT is achieved.

In addition to calculating the EBCT, GAC contactor flowrates can be used to calculate the number of bed volumes treated by the GAC basin. One bed volume is equivalent to total volume of the part of the GAC contactor which contains the activated carbons. The number of bed volumes that a given contactor should be able to treat prior to exhaustion is typically determined through initial testing. While the number of bed volumes treated prior to exhaustion may vary somewhat in practice from the value determined through bench-scale values, this parameter is another good indicator of remaining GAC contactor treatment capacity. Therefore, as the number of bed volumes treated gets close to the maximum number of bed volumes that the GAC contactor can treat (as determined through bench or pilot testing), the utility should plan for the replacement of the GAC media.

As discussed above, the number of bed volumes treated cannot be used by itself due to fluctuations in the feed water quality. If possible, multiple methods for measuring organic matter removal effectiveness should be utilized. UVT should be measured in both the feed water and the treated water, and the delta UVT value calculated for each contactor (the use of this parameter was discussed in more detail above). While the delta UVT measurement is a rapid and easy monitoring method, it can be prone to interference from background constituents in the water matrix. Therefore, it is recommended that TOC removal data be collected together with UVT to validate the UVT measurements. It is also recommended that plots of operating data be generated for TOC vs UVT to develop a correlation between site specific background organic matter concentration and online UVT measurements.

188 Water Environment & Reuse Foundation

Operators should routinely monitor these parameters, as well as the turbidity, pH and temperature of the water to be treated. If the GAC contactors cannot provide the required removal of organic contaminants, then corrective actions must be taken as described in the next section.

4.10.4 Critical Control Point Resources

4.10.4.1 GAC Alert (Warning) Response The GAC contact process step / critical control point is intended to reduce the concentration of organic matter in the GAC contactor effluent. By using GAC as a sorption-based process for contaminant control, the GAC will lose its sorptive capacity over time and will need to be replaced or regenerated on a regular basis. The regeneration of GAC is typically triggered by observations of organic matter breakthrough beyond a specified setpoint (e.g., 1 mg/L TOC or 50% breakthrough of C/C0) that is site-specific and/or state-specific. The removal of organic matter can be evaluated in one of two ways:

By measuring the percent reduction (delta) in TOC concentration across the GAC contactor. By measuring the percent reduction in ultraviolet light transmissivity (UVT) across the GAC

contactor.

The former can be more expensive and/or time consuming to run from an analysis standpoint, while the latter can be rapidly performed on a bench-top spectrophotometer. Both can be monitored via a continuous, online analyzer with regular, scheduled verification using grab samples for laboratory analysis. While TOC is a direct measurement of organic matter concentration, UVT is more of a surrogate monitoring parameter. Common practice is to monitor TOC on a weekly basis, while using UVT for the more frequent analysis of organic matter removal. A site-specific correlation needs to be developed between TOC and UVT for this method to work, which requires more frequent, initial monitoring of TOC.

Figure 4.26 provides a depiction of the alert (warning) level response process to address a low TOC and/or low delta UVT alert. The first step in the response process is to review SCADA trends for changes in TOC and/or UVT removal over time, as measured by influent and effluent analyzers. If possible, any observed reductions in percent removal across the GAC contactor should be correlated with DPR plant or upstream wastewater treatment plant performance (i.e., increased TOC loading onto the GAC filters). Grab samples for GAC contactor influent and effluent should be collected to allow validation of online instrument readings on bench-top analyzers. If the organics removal alert is shown to be real, then plant staff should review the flow being sent to the GAC contactor to verify the EBCT is on target, and should review the GAC replacement history to identify the age of the carbon in the affected contactor(s). If the carbon life alert is triggered based on this review, the affected contactor(s) should be taken out of service and the media replaced with virgin or regenerated carbon.

If the EBCT and carbon life are acceptable but organics removal performance is still low, then a forensic evaluation of the GAC contactor itself should be performed. This should include an assessment of the adsorptive capacity remaining by sampling of the media. This filter condition assessment involves taking the GAC contactor offline and performing multiple evaluation steps to identify potential problems with the GAC contactor other than media life. Items to be evaluated include a sieve analysis to confirm that the installed media meet the effective size and uniformity coefficient design parameters for the contactor and a media profile analysis to ensure that the proper bed depth is provided and that there are no mudballs or other deleterious matter in the carbon bed. The historical filter run length should also be assessed and a chlorinated backwash performed to control any biological growth that might have occurred in the contactor. Any issues identified through the condition assessment should be performed prior to returning the contactor to service.

189 Water Environment & Reuse Foundation

4.10.4.2 GAC Critical (Failure) Response Figure 4.27 and Table 4.10 illustrate the Critical (Failure) response procedures for the GAC critical control point. The first step in the response is an automatic shutdown of the affected GAC contactor(s) and incorporation of said contactor(s) into the overall backwash queue. Following contactor backwash, plant staff should notify the plant manager and/or supervisor of the process unit shutdown and review the alert level response procedures to confirm that the critical (failure) limit breach is real. If the breach is real, then staff should perform the following actions:

Verify the calibration status of the flow meters and online TOC and/or UVT analyzers, and collect confirmatory grab samples to validate the online analyzer values.

Verify that no upstream process upsets have occurred that might impact organics loading to the process.

Review SCADA trends for contactor flow and influent and effluent TOC and UVT, and review carbon replacement history.

Perform filter condition assessment and implement corrective actions. Assess the water quality of the GAC contactor effluent. Perform incident reporting to appropriate stakeholders.

If the water quality was actually compromised during the critical limit breach, additional actions and interaction with external entities may be required.

Table 4.10. GAC Alert and Alarm Summary

Monitoring Parameter Alert Level Critical (Failure) Level Notes

Total Organic Carbon (mg/L)

0.4 mg/L

(1 hr moving average)

0.5 mg/L

(1 hr moving average)

Target for TOC taken from California Title 22 regulations.

Delta UVT % 80% of critical setpoint

(1 hr moving average)

Critical setpoint

(1 hr moving average)

Delta UVT% to be a site specific correlation determined from piloting or full scale operation.

190 Water Environment & Reuse Foundation

Figure 4.26. GAC Alert Response Procedures

191 Water Environment & Reuse Foundation

Figure 4.27. GAC Critical Failure Response Procedures

192 Water Environment & Reuse Foundation

4.10.5 Maintenance Activities

The following items include the maintenance activities that should be followed to keep the instrumentation associated with the GAC process in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual (i.e., backwash rates, loading rates, and adsorption capacity), rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks UVT monitors. COD or TOC monitors (if applicable).

Monthly Calibration UVT monitors. Flow meters. Flow control valve and actuator maintenance. COD or TOC monitors (if applicable). Bench top COD analyzer (if applicable). Bench top or online organic carbon analyzer (if applicable).

4.10.6 Operations-Focused Design Considerations

The HACCP process performed as part of Reuse-13-03 identified the GAC contactors as an important process for organics removal in the non-membrane based treatment process train for direct potable reuse. Key operating parameters and associated alert levels and critical limits were identified for the process. For GAC contactors, the critical triggers points were identified to be delta UVT and carbon life. To provide the level of performance monitoring and control necessary to appropriately respond to alert level and critical limit situations, certain items should be included in the overall design of the GAC process. Most of these items are what would be required in a standard drinking water GAC contactors design; however, it is important that the system parameters be monitored closely in the DPR situation to further reinforce the robustness of the process relative to public health protection.

Similar to other processes, multiple trains should be provided for operational flexibility and redundancy. For the GAC contactors, the redundancy should be considered as n+2. This will allow the system to maintain redundancy while one contactor’s carbon is being regenerated. In the event of a critical limit condition, multiple contactors would allow the plant to continue to operate within the design flow and dose ranges when the problem contactor is automatically taken offline. Hydraulic flow splitting structures on the influent of the reactor trains would provide a simple, operator-friendly means of equally dividing the influent flow among the units. However, individual flow metering and a means of positively affecting the flow split in the event of UVT alert would also be required. Effluent isolation devices interlocked with critical limit alarms are necessary to isolate the flow from any GAC contactors experiencing critical limit situations (i.e., delta UVT).

Flowrates to each contactor should be monitored closely for the GAC process for two purposes. First, it is important to maintain the pre-determined EBCT through the carbon beds. Second, flowrate measurements are needed to keep track of the exhaustion of the activated carbon and compare the treated bed volumes to the expected life of carbon.

UVT is a measure of the ratio of light entering and exiting water, and can be measured either via an online system or via a bench-top analyzer. UVT is inversely proportional to UV absorbance, and both can

193 Water Environment & Reuse Foundation

be correlated to the level of contaminants in the water. As noted earlier in this section, change in UVT is a good indicator of a change in GAC contactor performance; however, since it is a spectrophotometric method, it is prone to interference from background constituents in the water matrix. While not required, TOC analyzers should also be considered for monitoring the level of organics removal. Paired TOC and UVT analyzers on the influent and effluent of the GAC process will allow for monitoring of background organic matter concentration and potential exhaustion of GAC, and can also allow for a correlation to be developed between the two parameters. The influent instruments can be located on the overall influent flow to the GAC contactors, but the effluent instruments should be installed on the end of each individual contactor.

The carbon life is directly related to the concentration of contaminants in the feed water and the adsorptive capacity of the particular type of activated carbon. Selection of the appropriate GAC media should be done by conducting tests (RSSCT or pilot-testing) to determine the carbon usage rate for a given feed water and expected life of the carbon.

4.11 UV

4.11.1 Process Overview

Ultraviolet light (UV) disinfection process targets the inactivation of pathogens in the water. The mechanism of disinfection by UV light differs considerably from the mechanisms of chemical disinfectants such as chlorine and ozone. Chemical disinfectants inactivate microorganisms by destroying or damaging cellular structures, interfering with metabolism, and hindering biosynthesis and growth. UV light inactivates microorganisms by damaging their nucleic acid. The high energy associated with UV energy, primarily at 254 nm wavelength, is absorbed by cellular RNA and DNA. This absorption of UV energy forms new bonds between adjacent nucleotides, creating double bonds or dimers. Change in the RNA and DNA structure of bacteria and viruses prevents their replication abilities. If a pathogen cannot reproduce, then it cannot infect a host. Among the pathogens of interest in drinking water, viruses are most resistant to UV disinfection followed by bacteria, Cryptosporidium oocysts, and Giardia cysts. Since UV is a very effective disinfectant, the U.S. EPA provides up to 4-log inactivation credit for Cryptosporidium, Giardia, and viruses for properly operated and designed UV reactors as outlined in the remainder of this section.

From a process operation standpoint, the key to the UV disinfection process is to operate the UV reactor within its validated range. Since the process does not physically remove pathogenic cysts from the process flow, the measurement of pathogen inactivation must be evaluated via performance monitors. The performance monitors typically used in UV disinfection are flow rate, UV intensity, and UV transmittance (UVT). The values for these parameters are set during the full-scale validation (or testing) of UV reactors, where operating window is established for proper reactor performance. These parameters are then used in the calculation of UV dose delivered through the reactor. If the UV reactor is operated within this operating window of tested parameters, then the UV reactor can be assumed to have achieved the targeted pathogen inactivation via dosing delivery.

The use of UV disinfection in treatment does not have any impacts on downstream treatment processes; however, it is highly dependent on the feed water quality to the reactor. Water quality changes resulting in an inefficient pathogen inactivation typically involve a decrease in UV transmissivity due to high levels of organics, metals or anions, or an increase in turbidity. In the context of the direct potable reuse treatment, the water quality is expected to be of excellent quality as impact to UVT are not likely to occur. This expected UVT stability makes any observed change in UVT a good indicator of upstream process upset(s).

194 Water Environment & Reuse Foundation

4.11.2 Key Operating Parameters

As noted above, the U.S. EPA provides up to 4-log inactivation credit for Cryptosporidium, Giardia, and viruses for UV disinfection systems. As such, this is an important process in the non-membrane train to achieve the California DDW goal of 12-10-10 for reuse systems. UV disinfection cannot be determined directly and is usually calculated using a dose monitoring equation to estimate the UV dose based on the flow rate, UV intensity, and UVT, as measured during reactor operations. Based on the US EPA’s UVDGM (UVDGM-LT2ESWTR), the required UV dose to achieve log inactivation of pathogens is summarized in Table 4.11.

Table 4.11. UV Dose (mJ/cm2) Required to Achieve Pathogen Inactivation (U.S. EPA UVDGM)

Log Inactivation

Pathogen 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0

Cryptosporidium 1.6 2.5 3.9 5.8 8.5 12 15 22

Giardia 1.5 2.1 3.0 5.2 7.7 11 15 22

Virus 39 58 79 100 121 143 163 186

In operating UV disinfection systems, there are a number of parameters that must be monitored to ensure proper treatment is being provided (i.e., the target UV dose is being delivered). These monitoring parameters are described below.

UV Dose – UV dose in the context of daily operations monitoring is intended to mean the UV dose required to achieve a target inactivation of pathogens in the UV reactor. The target UV dose is the value developed during design of the UV reactor to achieve the required 4-log inactivation of Cryptosporidium, Giardia, and viruses. UV dose should be monitored continuously because a change in UV dose directly impacts a facilities ability to achieve the target log reduction of pathogens. Note that UV dose is a calculation including UV Intensity, flow and UV transmittance based on system validation.

Flowrate – This parameter was selected because a change in flowrate has a direct influence on the calculated UV dose. An increase in flowrate would decrease the hydraulic retention time in the UV reactor, leading to delivery of lower UV doses. UV reactors are typically sized to deliver the required UV dose under a specific range of flowrates. Flowrate to the UV reactor may vary over time; however, it should be within the UV reactor’s validating operating window.

Change in UVT – This parameter was selected because a change in UVT is indicative of a change in the quality of the water being treated in the UV reactors. UVT is a measure of the ratio of light entering and exiting water. As the concentration of organic matter, metals, and anions increases, the ability of UV light waves to propagate through the water decreases. This would be reflected in a lower UVT value. As UVT decreases, the light intensity throughout the reactor decreases, which reduces the dose the reactor delivers. UV reactors are typically sized to deliver the required UV dose under specified UVT conditions for the application. UVT may vary over time due to fluctuations in the feed water; however, it should be within the UV reactors validated range.

Turbidity – This parameter was selected because the effectiveness of UV disinfection can be adversely affected by the presence of particles in the water. Particles can shield pathogens from exposure to UV light and can also scatter UV light waves.

195 Water Environment & Reuse Foundation

4.11.3 General Operating Procedures

UV reactors should be operated to achieve the target removal of pathogens. This is most often tracked by ensuring that the design UV dose is met, which in turn involves ensuring UV intensity, UV transmittance and flow are within the validated range for a given reactor. UV intensity is most commonly determined by a UV sensor incorporated in the UV reactor design. The sensor(s) tracking the UV intensity should be maintained properly and calibrated periodically. Low UV intensity can be caused by failure of a UV lamp. UV lamps have a set lifespan, which is typically within the range of 8,000 to 14,000 hours. The plant SCADA system should keep track and record the use of each lamp. Any lamp failures should be replaced immediately and the SCADA system should be updated. During the operation of the UV reactors, quartz sleeves around the UV lamps should be periodically cleaned to prevent accumulation of any foulants. This can be done manually or through pneumatically or electrically actuated systems. The operation of the wiper system should also be tracked and recorded in the plant SCADA system.

As discussed previously, UV disinfection is dependent on the exposure duration of pathogen to UV source. Flowrate determines how long the water to be treated stays within a UV reactor, and can be easily tracked by the flowmeters of each UV reactor. The plant automation system should track and record the per reactor flow rate. Operators should routinely monitor this parameter and compare them to the design values for the reactor. If the flowrate is higher than the design capacity, then insufficient contact time is being provided in the UV reactor for the inactivation of pathogens. In this situation, the flow to the individual reactor should be reduced until the flowrate is within the design range.

Operators should also routinely monitor the reactor temperature to prevent overheating of the UV lamps. If the UV reactors cannot meet the target UV dose, then corrective actions must be taken as described in the next section.

4.11.4 Critical Control Point Responses

4.11.4.1 UV Disinfection Alert (Warning) Response The ultraviolet light (UV) disinfection process targets the inactivation of pathogens in the reuse process flow. If the UV reactor is operated within an established range of tested parameters (i.e., the “validated range”), then the UV reactor can be assumed to have achieved the pathogen inactivation associated with this range. The validated range consists primarily of a range of UV doses, as measured by UV intensity sensors in the UV reactor, and a range of flow conditions, as measured by effluent flow meters. In some cases, the UVT and turbidity of the reactor influent is also measured. These parameters are used to indicate how clean the water is, which can be attributed to how easily the UV light passes through the water to the target organisms. By monitoring all of these parameters (as appropriate to the particular model of UV reactor), plant operations staff can ensure that the process is operating within its validated range and that the required level of pathogen inactivation is being achieved. If values begin to approach the boundaries of the validated range a warning/alert should be indicated.

Figure 4.28 depicts the response procedures associated with alert (warning) levels for the UV disinfection process for UV dose, UV reactor flow, and influent UVT. If a low UV dose alert is triggered, the first step involves a review of existing SCADA data to compare what UV dose has been achieved over time versus the target inactivation dose. The calibration of each UV intensity sensor should be checked, and if out of calibration, the affected sensor(s) should be replaced with the spare calibrated sensors. If the UV intensity sensors are all reading properly, then the condition of the UV reactor equipment should be assessed. This can include reviewing the lamp age data, cleaning the quartz sleeves that protect the UV lamps, replacing aged lamps and/or planning for preventive maintenance on the UV lamp ballasts.

196 Water Environment & Reuse Foundation

In the event of a high reactor flow alert, plant operations and/or engineering staff should review SCADA data for individual reactor flows and adjust the per reactor flow until each reactor is operating within its validated range. This may necessitate reducing the overall DPR plant flow.

Finally, if a low UVT alert (or high turbidity alert) is triggered, plant staff should review the SCADA trend of influent UVT over time to identify if any specific observations can be made. Staff should simultaneously review the performance of upstream treatment processes to ensure that they are operating properly, such that organics and other compounds that absorb UV light are being removed prior to the UV disinfection step (i.e., GAC contactors). Grab samples should also be collected to validate that the online UVT (or turbidity) measurements are accurate. If not, the affected reactor(s) should be removed from service and the associated UVT (or turbidity) monitor(s) recalibrated.

4.11.4.2 UV Disinfection Critical (Failure) Response Figure 4.29 illustrates the Critical (Failure) response procedures for the UV disinfection critical control point. The first step in the response is an automatic shutdown of the affected UV disinfection reactor(s). Following reactor shutdown, plant staff should notify the plant manager and/or supervisor of the shutdown and review the alert level response procedures to confirm that the critical (failure) limit breach is real. If the breach is real, then staff should perform the following actions: Verify the calibration status of the flow meters, UV intensity sensors and online UVT and turbidity

analyzers, and collect confirmatory grab samples to validate the online analyzer values. Verify that no upstream process upsets have occurred that might impact organics loading to the

process that would result in a reduced UVT or increased turbidity in the reactor influent. Review SCADA trends for reactor flow, influent UVT, and UV dose, and review reactor maintenance

history (lamp age, ballast repair history, etc.). Assess the water quality of the UV reactor discharge. Perform incident reporting to appropriate stakeholders.

If the water quality was actually compromised during the critical limit breach, additional actions and interaction with external entities may be required.

197 Water Environment & Reuse Foundation

Figure 4.28. UV Disinfection Alert Response Procedures

198 Water Environment & Reuse Foundation

Figure 4.29. UV Disinfection Critical Failure Response Procedures

199 Water Environment & Reuse Foundation

4.11.5 Maintenance Activities

The following items include the maintenance activities that should be followed to keep the instrumentation associated with the UV disinfection process in good working order. The procedures are not as detailed as what would be provided in a vendor Operation and Maintenance (O&M) manual (i.e., sleeve cleaning frequency); rather they are intended to provide an overview of the components to be maintained and a general description of the level of effort (weekly, monthly, annually, etc.).

Weekly Validation Checks UVT monitors. UV sensors. Turbidimeters.

Monthly Calibration UVT monitors. UV sensors. Turbidimeters. Temperature sensors. Flow meters. Flow control valve and actuator maintenance.

4.11.6 Operations-Focused Design Considerations

The HACCP process performed as part of Water Environment & Reuse Foundation Project Reuse-13-03 identified the UV reactors as an important process for inactivation of pathogens in the non-membrane based treatment process train for direct potable reuse. Key operating parameters and associated alert levels and critical limits were identified for the process. For UV reactors, the critical control points were identified to be UV dose, high flow, and low UV transmittance. To provide the level of performance monitoring and control necessary to appropriately respond to alert level and critical limit situations, certain items should be included in the overall design of the UV disinfection system. Most of these items are what would be required in a standard drinking water UV disinfection design; however, it is important that the system parameters be monitored closely in the DPR situation to further reinforce the robustness of the process relative to public health protection.

Similar to other processes, multiple trains should be provided for operational flexibility and redundancy. For UV disinfection, the redundancy should be considered as n+1. This will allow the system to maintain redundancy while one UV reactor is taken offline for any maintenance purposes (i.e., lamp replacement). In the event of a critical limit condition, multiple reactors would allow the plant to continue to operate within the design flow and dose ranges when the problem reactor is automatically taken offline. Hydraulic flow splitting structures on the influent of the reactor trains would provide a simple, operator-friendly means of equally dividing the influent flow among the units. However, individual flow metering and a means of controlling the flow to the reactor in the event of a high flow alert would also be required. Effluent isolation devices interlocked with critical limit alarms are necessary to isolate the flow from any UV reactors experiencing critical limit situations (i.e., low UV dose).

Flowrates to each contactor should be monitored closely. As previously discussed, UV disinfection is dependent on the exposure duration of the target pathogen to UV light. Flowrate determines how long the water to be treated stays within a UV reactor and high flowrates can result in insufficient disinfection. Conversely, low flowrates can also be a concern for UV reactors as that can lead to overheating of the lamps. Furthermore, hydraulics through a UV reactor can directly impact the efficiency of the process. It is recommended that design engineers consider the use of computational fluid dynamic (CFD) modeling

200 Water Environment & Reuse Foundation

to optimize the hydraulic flow through the UV reactor. Inlet and outlet piping configurations of the UV reactor should be in accordance with the manufacturer’s recommendations (i.e., straight pipe lengths).

4.12 References AWWA. 1999. Water Quality & Treatment: A Handbook of Community Water Supplies, 5th Ed. New York, NY: McGraw-Hill, Inc.

AWWA and ASCE. 2005. Water Treatment Plant Design, 4th Ed. New York, NY: McGraw-Hill, Inc.

AWWA and ASCE. 2012. Water Treatment Plant Design, 5th Ed. New York, NY: McGraw-Hill, Inc.

Crittenden, J.C., et. al. 2005. Water Treatment Principles and Design, 2nd Ed. Hoboken, New Jersey: John Wiley & Sons, Inc.

Kawamura, S. 2000. Integrated Design and Operation of Water Treatment Facilities, 2nd Ed. New York, NY: John Wiley & Sons, Inc.

Kim M., Guerra P., Shah A., Parsa M., Alaee M., Smyth S.A., (2014) Removal of pharmaceuticals and personal care products in a membrane bioreactor wastewater treatment plant. Water Science and Technology, 69 (11), 2221-2229.

Metz, D.H., Reynolds, K., Meyer, M., and Dionysiou, D.D., 2011. The effort of UV/H2O2 treatment on biofilm formation potential. Water Research. 45. 497-508.

MWH. 2012. Water Treatment Principles and Design, 2nd Ed. Hoboken, NJ: John Wiley & Sons, Inc.

National Water Research Institute (NWRI), and Melin, G., 2000. Treatment Technologies for Removal of Methyl Tertiary Butyl Ether (MTBE) from Drinking Water. Second Edition.

U.S. EPA. 1989. 40 CFR Parts 141 and 142 Drinking Water; National Primary Drinking Water Regulations; Filtration, Disinfection; Turbidity, Giardia lamblia, Viruses, Legionella, and Heterotrophic Bacteria; Final Rule (54 FR 27486, June 29, 1989)

U.S. EPA. 1999. Alternative Disinfectants and Oxidants Guidance Manual. Office of Water, U.S. Environmental Protection Agency, Washington, D.C. (EPA 815-R-99-014).

U.S. EPA. 2007 Simultaneous Compliance Guidance Manual for the Long Term 2 and Stage 2 DBP Rules. Office of Water, U.S. Environmental Protection Agency, Washington, D.C. (EPA 815-R-07-017.

U.S. EPA. 2003. LT1ESWTR Disinfection Profiling and Benchmarking – Technical Guidance Manual. Office of Water, U.S. Environmental Protection Agency, Washington, D.C. (EPA 816-R-03-004).

U.S. EPA. 2009. Summary of National Primary and Secondary Drinking Water Regulations. Office of Water, U.S. Environmental Protection Agency, Washington, D.C. (EPA 816-F-09-004).

U.S. EPA. 2010. Long Term 2 Enhanced Surface Water Treatment Rule: Toolbox Guidance Manual. Office of Water, U.S. Environmental Protection Agency, Washington, D.C. (EPA 815-R-09-016).

Ultraviolet Disinfection Guidance Manual for the Final Long Term 2 Enhanced Surface Water Treatment Rule, U.S. EPA, Office of Water (4601), EPA 815-R-06-007, November 2006

201 Water Environment & Reuse Foundation

Chapter 5

Recommended Staffing

5.1 Approach to Determining O&M Staffing for Water Reuse Facilities

5.1.1 Overview

Developing initial operations and maintenance (O&M) staffing estimates for water utilities is not a perfect science. This is due to the many variables that exist between facility location, process flow schemes, process equipment, level of automation and operations and maintenance strategies. While estimates can be developed using certain empirical data including benchmarking studies and facility design information, there is no substitute for actual operating experience. Once in operation, staffing levels need to be evaluated and adjusted over time to reflect actual operating circumstances.

There are many factors and sources of information that need to be considered when developing initial O&M staffing estimates. In addition, the ability to develop estimates with a high level of confidence is dependent upon the stage of facility development. For example, a facility that is in the planning, conceptual stage will not have sufficient, available information to determine specific operations and maintenance tasks and responsibilities. At this stage estimates are considered “high level” or rough estimates and are typically used for initial O&M budget planning purposes. By contrast, a facility that is in the design stage will include a site plan and specific information on selected unit processes, equipment and control strategies and draft equipment O&M manuals, all of which are valuable inputs towards detailed O&M duties and responsibilities. Estimates drawn from this information typically provides initial staffing estimates that are sufficient for facility startup and initial operations.

In order to establish levels with a higher level of confidence requires actual operating experience that reflects site specific O&M idiosyncrasies and needs. O&M staffing levels will always remain somewhat dynamic and will require careful review and occasional refinement at least annually. This is due to ever changing conditions, such as future facility modifications that may alter operating strategies, aging infrastructure that may require greater maintenance attention and existing staff abilities and tenure.

The objective of this study is to identify various methods of determining reasonable O&M staffing level estimates for water reuse facilities at various stages of facility development including planning, design, startup and operations.

202 Water Environment & Reuse Foundation

5.1.2 Approaches

Many factors directly influence the ability to develop O&M staffing estimates, all of which are dependent upon available information at a certain point in time in the development of the facility. The following table provides a listing of information that is typically available at each stage of facility development:

Table 5.1. Available Information during Stages of DPR Facility Development DPR Facility Stage Available Information

Master Planning Very limited information – conceptual approach, estimated capacity and water quality requirements. A DPR facility will have a general process outline in terms of likely critical control point (CCP) processes.

Preliminary design Fundamental details – facility location, site plan, hydraulic profile, process flow scheme, unit processes including type and size, preliminary control strategy (monitoring & automation), initial O&M management plans. CCPs will be known at this time.

Final design Same as above but at a higher level of detail and acceptance. CCP response procedures will have been developed.

Pre- Startup Installed equipment, equipment performance testing data, draft standard operating procedures, O&M manuals, O&M training

Startup Functioning equipment, operating experience, troubleshooting data

Operations Historical performance data, real time O&M experience, recorded adjustments.

Because O&M staffing represents a significant portion of the overall annual O&M budget, estimating staffing levels are an important component to the overall process of planning, design and startup. While each stage of facility development provides varying levels of information, initial estimates can be established at each level and then refined as the process progresses. There are certain approaches that can be considered when estimating O&M staffing levels at various stages of facility development including: Database benchmarking (wide database comparison of water utilities with an estimated accuracy of

65-75%. Partner benchmarking (comparison of similar local facilities with an estimated accuracy of 75-85%. Zero-Based Assessment (detailed calculation of site specific staffing needs with an estimated

accuracy of 85-95%. Post Startup (actual operating experience with an estimated accuracy of 95-98%)

The following provides a general approach and guidelines for estimating O&M staffing levels at each stage of Planning, Design and startup and Operations.

5.1.3 Planning Stage – Benchmarking Assessment

Because there is little information on facility specifics, planning stage estimates can be drawn from Database Benchmarking and Partner Benchmarking. Database benchmarking includes a large pool of staffing comparisons. Common sources are provided by AWWA and WEF which include data on staffing levels related to a certain class and facility size. One example is the AWWA Performance Indicators for Water and Wastewater Utilities. However, this information may be somewhat limited in its use for DPR facilities which employ Full advanced treatment technologies such as membranes, activated carbon or advanced oxidation while the majority of data sample points in these databases are derived from

203 Water Environment & Reuse Foundation

traditional surface water treatment facilities and processes such as coagulation, sedimentation, filtration and disinfection.

Partner benchmarking can provide estimated staffing levels at greater levels of detail and confidence. This typically includes a comparison of three to six water organizations of similar size and complexity within the same general geographic region to the host facility. Information obtained from these comparisons typically include design capacity, process flow train, numbers and types of O&M, laboratory and administrative positions, compensation ranges and O&M budgetary data.

While information obtained from these comparisons are not considered suitable for establishing exact staffing numbers for organizational planning and eventual employee recruitment, they do provide a reasonable basis for initial budgeting and planning purposes.

As a part of a benchmarking exercise, three utilities operating advanced water recycling plants were surveyed to provide benchmark data on staffing numbers. These were:

1. Orange County Water District, Groundwater Replenishment Scheme, Fountain Valley, California. This facility is a large scale IPR system that utilizes an RO based advanced treatment process consisting of chloramination, microfiltration, reverse osmosis and advanced oxidation (UV-H2O2). The operations team from Orange County Water District (OCWD) operate the advanced portion of the plant, with water delivered from the neighboring wastewater plant operated by Orange County Sanitation District (OCSD). The process outline is shown in the diagram below:

Figure 5.1. Process Flow Diagram OCWD GWRS (Courtesy OCWD)

204 Water Environment & Reuse Foundation

2. West Basin Municipal Water District, El Segundo, California. The West Basin treatment facilities are owned by the West Basin Municipal Water District, with operation by Suez (formerly United Water Services). West Basin purchases secondary effluent from the City of Los Angeles’ Hyperion Treatment Plant (Hyperion) and treats it at the Edward C. Little Water Recycling Facilities ECLWRF) to meet disinfected tertiary recycled water Title 22 requirements prior to distributing the recycled water to its customers and satellite treatment facilities. The satellite treatment plants provide supplemental treatment for customers that require better water quality for their business processes. In total, West Basin produces five separate types of recycled water at four recycling facilities. Current production typically ranges between 40 and 50 million gallons per day (mgd) of recycled water. El Segundo, California is West Basin’s main treatment facility (see attached process flow diagram—“Exhibit A”). All of the secondary effluent is collected, treated, and distributed from this facility. The ECLWRF produces several qualities of recycled water, including: Disinfected tertiary treated Title 22 recycled water – Includes coagulation, flocculation,

clarification, filtration, and disinfection and currently treats up to 40 million gallons per day (mgd). This disinfected tertiary treated Title 22 recycled water (Title 22) serves as a supply to the three satellite facilities.

Seawater barrier injection water – Produced for the West Coast Basin Barrier (Barrier) and uses ozone as a pre-treatment to microfiltration (MF), MF, reverses osmosis (RO) and advanced oxidation (hydrogen peroxide and ultra-violet light disinfection) to obtain high quality water and currently has a capacity of 17.5 mgd.

High pressure and low pressure boiler feed water – This water is used at the Chevron Refinery as feed water for boilers and includes ozone as a pre-treatment to MF, MF, RO, and a second pass of RO for a portion of the flow and currently has a capacity of 4.8 mgd.

The Exxon Mobil Water Treatment Facility is a satellite treatment facility treating water from ECLWRF (Title 22 recycled water) with MF and RO to produce boiler feed water. In addition, this facility employs a nitrification treatment process of the Title 22 water utilizing a biological aerated filter (BAF) to produce nitrified water for cooling tower make-up.

The Carson Regional Water Recycling Plant is a satellite treatment facility treating water from ECLWRF (Title 22 recycled water) with MF and RO to produce boiler feed and cooling tower make-up water. In addition, this facility employs a nitrification treatment process of the Title 22 water utilizing BAF to produce nitrified water for cooling tower make-up.

The Chevron Nitrification Plant is a satellite treatment facility that only produces BAF treated nitrified water for cooling tower makeup.

3. Gwinnett County F. Wayne Hill Water Resources Center, Buford Georgia. Gwinnett County’s Fort Wayne Hill Water Resources Center includes ozone and biofiltration

followed by a second ozone biological activate carbon process. The plant does not currently provide DPR water, though they produce IPR water that is returned to Lake Lanier through an ozone-BAC based treatment process.

205 Water Environment & Reuse Foundation

Figure 5.2. Processes – West Basin Recycled Water Treatment System (Courtesy WBMWD)

Figure 5.3. Existing and Proposed Process Trains, Gwinnett County F. Wayne Hill Facility, GA.

The survey results are shown in the tables on the following pages. What is most instructive in these results, is that there are significant differences between each of the three plants in terms of overall technology used, and combinations of technology.

206 Water Environment & Reuse Foundation

Table 5.2. Advanced Reuse Staffing Benchmark Survey – RO-Based Treatment Facilities

Utility Name Orange County Water District Groundwater

Replenishment System West Basin MWD Location Fountain Valley, California El Segundo, CA Process Flow Description MF, RO UV/H2O2 Multiple recycled water trains including:

O3/MF/RO/UV AOP/chlorine Two stage RO system for boiler feed water Three satellite treatment facilities including biological filtration, MF and RO.

Design Flow (mgd) 100 mgd 47 mgd of treatment overall. 17.5 mgd RO permeate for groundwater barrier injection. 2.6 mgd single pass RO permeate for low pressure boiler feed. 2.6 mgd second pass RO permeate for high pressure boiler feed. 8.6 mgd RO for refinery boiler feed (satellite facility) 11 mgd biological filtration for refinery cooling towers. 5 mgd irrigation water.

Daily, Annual Avg. Flow 100 mgd 30 mgd produced approximately. Annual Operating Budget $26 million $50 M for entire recycled water program. Number Operations Staff (including shift operators and operations supervisors)

20 19

Number Maintenance Staff (including maintenance supervisors)

16 11

Number Laboratory Staff (Noting if this is separate from treatment plant staff)

Lab staff is separate from treatment plant staff and work on analysis for groundwater and retailers not just treatment plant samples. 28 total persons

6

Number Mgt/Admin Staff In the Water Production department that runs the plant (includes Ops, Maintenance, I&E) there are 5 Operations supervisors, 1 process manager, 1 instrumentation/electrical manager, 1 maintenance manager, 1 director, and 1 admin assistant for a total of 10

13

Total Staff 61 in water production department 58 Other Specialty Staff Operations chemist, material & chemical management,

process specialist (3) 5

Description of any Outsourced Services None Facility O&M is contracted to private operator

207 Water Environment & Reuse Foundation

The OCWD and West Basin’s programs contained almost the same number of operational staff overall, however with just less than half of the recycled water production. However, a comparison of flow rate alone is not sufficient as there are other significant differences including: Differences in the treated water quality that is produced (West Basin provides six different water

qualities with a more complex set of treatment trains) relative to a single water quality for GWRS.

GWRS has one single plant location, whereas West Basin has one major site with three remote satellite facilities.

These two examples of RO based treatment systems highlight the difficulty of providing benchmark comparisons for these facilities and technologies. While this is not an exhaustive survey, there are a relatively small number (although increasing) of these type of facility in operation within the US, and each facility has unique aspects that provide differentiation which can challenge staff benchmarking.

The IPR treatment system at the F Wayne Hill facility is integrated into the overall wastewater treatment facility. Staff operate across both biological wastewater treatment processes as well as advanced treatment processes, and as such tasks and roles are shared across all of the facility.

Similarly, the Terminal Island Water Reclamation Plan operated by the City of Los Angeles, has the RO based treatment facility (MF/RO/UV-H202) incorporated as a part of the overall wastewater treatment operations. Operators rotate between process operations from the biological treatment through to advanced treatment. Maintenance staff including mechanical, instrument and electrical support are also shared across the facility. As a large organization, the City of Los Angeles provides utility wide support for laboratory services, asset management and other operational functions. Table 5.3. Survey Results – Non-RO-Based Process Utility Name Gwinnett County F. Wayne Hill Water Resources Center Location Buford, GA Process Flow Description See process flow diagram. Design Flow 60 mgd average day max month (40 mgd max to Lake Lanier and 20 mgd to

Chattahoochee River) Daily, Annual Avg. Flow 31.5 mgd Annual Operating Budget $15.2M Number Operations Staff 17 operators including shift supervisors and foremen Number Maintenance Staff 16 preventative maintenance and small corrective, 2 housekeeping Number Laboratory Staff Operations lab run by operators listed above. Another GCDWR environmental

lab runs compliance and special samples. Number Mgt/Admin Staff At F Wayne Hill - 2 operations managers, 1 superintendent, 1 admin assistant

At department level - 1 manager over all 3 ww plants, 1 deputy director over all water and wastewater facilities, 1 director DWR

Total Staff 39 located at Fort Wayne Hill Other Specialty Staff Corrective maintenance staff work on all facilities (not included in this list). Description of any Outsourced Services

Calibration and maintenance of certain analytical instruments.

208 Water Environment & Reuse Foundation

These results provide some information for benchmarking, although must be carefully reviewed as to the context of: Utility size – Does the utility operate more than one facility? If so, are many operational functions

centralized and shared across the entire facility? Is the facility operated by a private operator with off-facility support services?

Integration – Is the DPR facility integrated into the wastewater treatment operations (Gwinnett County, Terminal Island) or is it a separate entity?

Complexity – Is the DPR facility a single process train operated to produce a drinking water supply only, or are other reuse technologies and water qualities also produced (such as at West Basin’s facilities?)

Other specifics such as local industrial agreements and local permit requirements must also be considered.

5.1.4 Design Stage – Zero-Based Assessment Zero Based Assessments are the most comprehensive process when estimating staffing levels. This involves an objective, clean slate assessment of staffing needs based on available, empirical data coupled with certain staff availability and productivity assumptions. The process involves a detailed due diligence of expected tasks to be performed by each O&M position from shift operator and maintenance trades to supervision. It is also important at this stage of assessment to identify which O&M services are planned for outsourcing such as specialized or support services. These can include everything from routine janitorial services to periodic, specialized maintenance procedures. Planned outsourced services should be identified and budgeted upfront and removed from the staffing estimate process.

The first step in the zero based assessment process is to estimate the actual work time that will be available from each employee. This includes a situation specific review of the theoretical 2080 paid hours per employee per year less time that is allocated for employee benefits and other institutional needs.

The following is an example of average available municipal staff days that are not available for work over a 365-day period: Table 5.4. Average Available Days Per Year – Municipal Staff

Event Days/Year

Weekends 104 Holidays 12 Vacation 14 Training 10 Sick Leave 5 Miscellaneous 3 Daily break/lunch time 27 Subtotal – unavailable days/year 175 Total available days/yr. (365 days – 175 days) 190 days or 1520 hours/yr.

209 Water Environment & Reuse Foundation

5.1.4.1 Operations Staffing Operator duties typically include routine tasks of monitoring, inspecting, adjusting, cleaning and reporting. In some cases these duties are performed during a single day shift or on a 24/7 basis. While there may be periods of non-routine tasks, such as process and equipment troubleshooting, operator duties are normally considered predictable and stable.

Relevant Information: Facility site plan (distances between processes and equipment) Control strategy (monitoring, automation and control plan) Draft O&M manuals (equipment type, size, operating modes) Standard Operating Procedures (equipment and process startup, shutdown and operating procedures)

Once a review of all available information is conducted, consideration needs to be given to the amount of time that will be required for: Process monitoring (time to staff control stations) Field tasks such as equipment/process inspections, adjustments and instrument calibration Safety requirements such as confined space entry Record keeping and reporting Housekeeping Routine preventive maintenance tasks (if included)

While the direct time and effort to perform each of these individual duties and tasks can be estimated, it is also helpful to conduct a virtual walkthrough of the entire facility for each operating shift. This includes an estimate of time required for an operator to physically go through the motion of inspections, adjustments, recording and maintaining the facilities on an assigned shift. Additional time should also be factored (estimated 10%) for non-routine circumstances that may require additional unplanned effort. Once all tasks and estimated real time are developed, a total number of operating staff can be determined by dividing the total real time hours by the number of available hours per staff (1520 hour example).

Example: total real time hours/year of 4888 (as determined through assessment) divided by 1520 available hours/staff x 10% contingency = 3.5 operators

5.1.4.2 Maintenance Staffing In contrast to operator assignments, maintenance duties typically involve greater variables especially as it relates to corrective or breakdown maintenance. This includes situations that require equipment troubleshooting and tear down in order to determine cause and course of action. Below are approaches to determining both preventative (predictable) and corrective (less predictable) maintenance staffing estimates.

Relevant information: Facility site plan (distances between equipment and processes) Description of installed equipment (size, type and manufacturer) Equipment O&M manuals (needed to determine prescribed preventative maintenance) – Where

manuals are not available, comparable equipment maintenance schedules and tasks or staff experience can be considered.

RS Means – Optional for identifying maintenance task times for similar, related equipment

210 Water Environment & Reuse Foundation

The next step is to identify each maintenance discipline that is to be evaluated. Typical disciplines include: Mechanical Electrical Instrumentation and Control (I&C) Heating Ventilation Air Conditioning (HVAC) Structural

5.1.4.3 Preventive Maintenance (PM) Preventive maintenance staffing is determined by identifying the specific time that is required to perform prescribed PM tasks. These tasks are those typically recommended by equipment manufacturers (equipment O&M manuals), other relevant reference material or experienced maintenance trades staff.

The following table provides an example of instrumentation and control (I&C) maintenance staff time required to perform specific PM tasks. The same evaluation process is conducted for all other maintenance trades and PM tasks. Table 5.5. Example – Instrument and Control Maintenance Staff Time Required

Equipment – pH Analyzer Trade

No. of Units Freq./yr No. of Staff Hrs/Task Staff Hrs/Yr

I&C

Clean Sensor 40 12 1 0.25 120

Remove sensor from process and check with buffer solution

4 1 0.50 80

Check reference electrode

4 1 0.25 80

Calibrate analyzer 12 1 0.50 240

Clean transmitter window

4 1 0.25 40

Total Hrs./yr. 560

Once estimated PM tasks and time requirements to perform each task are determined for each maintenance trade, an allowance for employee productivity needs to be considered. This includes “non-wrench time” such as traveling from the shop to the field, organizing task specific tools and parts, work schedule coordination and paper work.

Because there are many factors that influence maintenance productivity such as age and condition of assets, geographic area to service, in-house expertise, etc., an 80% productivity factor can be considered for PM tasks as a starting point:

Example Preventive Maintenance Estimate: total real time hours/year of 560 (as determined through assessment) divided by 1520 available hours/staff @ 80% productivity = .46 or ½ I&C position for PM

5.1.4.4 Corrective Maintenance (CM) As discussed above, corrective maintenance efforts inherently involve more variables than operations and preventive maintenance duties due to unknown time associated with troubleshooting and determination of cause and effect of equipment breakdown. Therefore, estimated time required to perform specific CM

211 Water Environment & Reuse Foundation

tasks is not available to the degree it is with PM. In the absence of actual onsite experience with specific assets, initial CM efforts and associated staffing can be estimated based on a standard ratio of PM/CM where 70% of maintenance efforts are applied to PM and 30% applied to CM. This ratio has been widely considered reasonable for a well-managed maintenance program.

Another major difference between PM and CM is the productivity of staff to carry out their duties. Again, this is due to additional time associated with CM troubleshooting and repair unknowns, parts ordering, etc. Therefore, and for the purpose of this approach, CM productivity is considered at 65%.

PM = 70% effort @ 80% productivity

CM = 30% effort @ 65% productivity

Using the above example for a typical pH analyzer requiring 560 hours PM annually and applying the 70/30 PM to CM ratio along with the 65% CM productivity factor, the following estimate of CM effort would apply:

Example: 1. 560 base PM hours represent 70% and 240 hours of CM represent 30% for a total 800 base

hours annually 2. 240 base CM hours divided by 1520 available hours/staff @ 65% productivity = .24 I&C

technician time for CM 3. Total zero based time annual estimate, including PM and CM, to maintain pH analyzers = 0.7

I&C technicians time

This assessment process can be duplicated for each maintenance discipline in order to estimate total maintenance staff needs.

5.1.4.5 Management, Supervisory and Administrative Support Because each organization is built on and supported by individual needs, it is difficult to provide specific estimates on the level of management, supervisory and administrative support needs. Major factors influences range from available management systems and procedures to number and type of O&M staff to institutional requirements such as personnel policies and procedures, labor agreements, to organizational cultures. For general planning purposes, and in line general business practices, an estimate of 20% in addition to the O&M staff estimates is considered reasonable for planning purposes.

5.1.4.6 Post Start-up Adjustments Following facility startup when sufficient time is available to assess steady operations (minimum of one year), all staffing levels should be reevaluated and refined to reflect actual needs. Any realignment or adjustment should be based on both empirical data gained through operating experience coupled with an assessment of reasonable business risks.

5.2 Zero-Based Assessment Example

5.2.1 Example – RO Membrane-Based DPR

By way of example, an estimate of operations and maintenance staff was conducted using a zero based assessment (task based) approach as described in Section 5.1 of this report. This example is provided as staff assessment guidance.

212 Water Environment & Reuse Foundation

In this case, an arbitrary 10 mgd advanced treatment portion of a DPR system was evaluated, using the RO membrane based technology of chloramination/MF/RO/UV-H2O2 and final chlorination. The process flowchart below outlines the process that was considered in this exercise.

Figure 5.4. RO-Based Advanced Treatment Process – For Staffing Evaluation

Basis of the estimate included:

10 x 1.3 MGD MF Units at 35 GFD ~ 70 MF Elements per Rack

5 x 2 MGD RO at 12 GFD at 400 ft2 elements = 416 elements, 60 pressure vessels per train.

4 Advanced Oxidation UV Reactors

5 x 2.5 MGD Distribution Pumps

Auto Strainer

MF Feed Pump

Feed Storage

Membrane Filtration

Break Tank

Influent from Wastewater

Treatment Plant

Carrier Pump

Ammonium Sulfate

Sodium Hypochlorite

Cartridge Filters

LP Feed Pump

HP Feed Pump

Reverse Osmosis

Packaged MF Train (x10) Packaged RO Train (x5)

UV Reactors

Hydrogen Peroxide

Product Storage

Product Water Pumps

Product Water to Blending

RO Concentrate to Outfall

Drain Box

Return Flow to Wastewater Treatment Plant

Spent Membrane Wash Water from

MF and RO

ERD

Antiscalant

Sulfuric Acid

Sodium Bisulfite

Lime

Sodium Hypochlorite

213 Water Environment & Reuse Foundation

5.2.2 Estimation of Hours Available per Full-Time Equivalent. The following operating hours per day for a water treatment plant operator were calculated based upon a real example for a water utility in California, USA (name of utility withheld). Total hours for paid holidays, vacation, training and other non-operating tasks were subtracted from the total paid hours per year. An additional hour per day was assumed for breaks, meals and showers.

Total Hours per Year

Pay Periods 26

Hours Per Pay Period 80

Total Paid Hours per Year 2080

Non-Available Hours

Paid Holidays 68

Vacation 160

Floating Holidays 40

Sick Leave 96

Technical Training 40

Mandatory Training 52

Subtotal 456

Days Available (above/8hrs/d) 203

Break/Meal/Shower (1 hrs /d) 203

Operating Hours Available 1421

214 Water Environment & Reuse Foundation

5.2.3 Task-Based Analysis – Operations

An assessment of operating tasks was broken down into two types of tasks; specific tasks (Table 5.6) related to specific equipment or process item and a list of general tasks (Table 5.7). This was done on a monthly basis, with estimates of the frequency and duration of each task. This information is taken from experience at other operational plants. These tables may be refined as design knowledge improves, or with ongoing plant experience. Table 5.6 provides an example for the DPR system reviewed: Table 5.6. Specific Task Assessment – RO-Based DPR Process

Specific Tasks Frequency Frequency/Month Task Duration (hrs) Hours Per Month RO Cartridge Filter Replacements Six Monthly 0.2 8 1.3

RO Feed SDI Check Weekly 4 1 4

Ammonium Sulfate Dosing Pump Calibration Monthly 1 0.5 0.5

Sodium Hypochlorite Dosing Pump Calibration (2 sets) Monthly 2 0.5 1

Sodium Bisulfite Dosing Pump Calibration Monthly 1 0.5 0.5 Antiscalant Dosing Pump Calibration Monthly 1 0.5 0.5

Sulfuric Acid Dosing Calibration Monthly 1 0.5 0.5

Hydrogen Peroxide Dosing Pump Calibration Monthly 1 0.5 0.5

Lime Dosing Pump Calibration Monthly 1 0.5 0.5 Lime System Operation Daily 30 1 30

Chemical Delivery - Sodium Hypochlorite Monthly 1 1 1

Chemical Delivery - Ammonium Sulfate Monthly 1 1 1 Chemical Delivery - Sodium Bisulfite Monthly 1 1 1

Chemical Delivery - Sulfuric Acid Monthly 1 1 1

Chemical Delivery - Hydrogen Peroxide Monthly 1 1 1

Chemical Delivery - Antiscalant and Cleaning Chemicals Six Monthly 0.2 1 0.2

pH Analyzer (x 3) Verification Daily 30 0.5 15

Chlorine (x 4) Verification Weekly 16 0.5 8

Chloramine Weekly 4 0.5 2

MF Cleaning (10 units) Monthly 10 2 20

RO Cleaning Four Monthly 1.25 4 5 Turbidity ( x 4) Weekly 16 0.5 8

TOC Analyzer Weekly 4 2 8

Check Neutralized waste Weekly 4 1 4

Total Specific Tasks

Total Hours/Month 114.5

215 Water Environment & Reuse Foundation

Table 5.7. General Task Assessment – RO-Based DPR Process

General Tasks Frequency Frequency/Month Task Duration

(hrs)

Hours Per Month

(Single Site)

Total Specific Tasks From table above 114.5

Process/Equipment Integrity Daily 30 2 60

Equipment/Building Review Daily 30 1 30

Process inspections (on site) – filling out site logs Daily 30 1 30

Sampling and Analysis Daily 30 3 90

Operator Lock Out Tag Out and troubleshooting Weekly 4 3 12

Process review ( SCADA) Daily 30 1 30

Shift reporting Daily 30 1 30

Admin/Reporting Weekly 4 2 8

Subtotal 404.5

Environmental Health and Safety 20.2

General Contingency (15%) 63.7

Total hrs per month 488.4

Operator Hrs per year 5861.2

Total Full Time Equivalent

Total Hours/1421 hrs/y 4

It should be noted that the full time equivalent calculation above is based on tasks required to be completed as a part of operations. It does not include a review of supervisory or senior management and does not include a review of specific staffing regulatory requirements (such as 24/7 attendance requirements). It does not include for finance, human resources or any other head office type activities. It is assumed that these would be managed differently from utility to utility. Some specific responsibilities that should be considered for DPR facilities are included in Section 5.3.

216 Water Environment & Reuse Foundation

5.2.4 Task-Based Analysis – Maintenance As for the operator assessment above, an analysis was conducted for maintenance activities required at the plant. This was similarly based on a task analysis, with a set of estimates for maintenance requirements on specific components within the plant. This was broken down into categories of major equipment, instruments/analyzers, valves, electrical, plumbing and HVAC. This database of maintenance hours is estimated based on a combination of maintenance experience and vendor recommendation. Typically, at the start of operations maintenance requirements will be consistent with vendor recommendations to maintain equipment warranties, however as experience is gained with equipment this is often optimized. Methodologies such as Reliability Centered Maintenance can assist in this endeavor. An exhaustive list of estimated maintenance hours has not been included in this report, however as an example, the maintenance hours required for major pumps at a facility has been included as an example in Table 5.8.

217 Water Environment & Reuse Foundation

Table 5.8. Example Estimate of Equipment Maintenance Requirements – Major Pumps

Staf

f R

equ

ired

PM

CM

Fre

q. p

er

Ye

ar

No

. of

Staf

f

Ho

urs

/ ta

sk

N/A

To

tal

Elec

tric

al T

ota

l

I&C

Ho

urs

Mec

han

ical

To

tal

Plu

mb

ing

Tota

l

HV

AC

To

tal

Op

erat

or

Tota

l

PM

Ho

urs

CM

Ho

urs

Un

it H

ou

rs p

er

Ye

ar

Co

mm

en

ts

Major Pumps

1) Clean and inspect pump assembly. MECH X 4 1 0.5 0 0 0 2 0 0 0 2 0 2

Assume Horizontal Split Case

2) Maintain oil in bearings. MECH X 1 1 1 0 0 0 1 0 0 0 1 0 1

3) Lubricate driver and coupling. MECH X 1 1 0.5 0 0 0 0.5 0 0 0 0.5 0 0.5 From USEM

4) Repack stuffing box. MECH X 2 1 2 0 0 0 4 0 0 0 4 0 4

5) Disassemble and inspect pump bodies. MECH X 1 2 4 0 0 0 8 0 0 0 8 0 8

6) Pump overhaul. MECH X 0.2 2 8 0 0 0 3.2 0 0 0 0 3.2 3.2

7) Pump reassembly. MECH X 1 2 4 0 0 0 8 0 0 0 0 8 8

Subtotal Task Hours 27

218 Water Environment & Reuse Foundation

For each equipment item, hours have been estimated for the particular maintenance sub tasks. A breakdown of tasks is also made between planned maintenance (PM) and corrective (or breakdown) maintenance (CM). A comparison of PM/CM ratios are useful metrics in the review of maintenance tasks as noted in Section 5.1.1.4 With a list of estimated hours per year per item of equipment, the task then simply becomes a listing of all equipment, and count of numbers of each item, to provide an overall assessment of maintenance hours required. This can be broken down by discipline, to provide an estimate of particular trade/skill qualifications required. For many facilities, maintenance will be managed by a mix of operator tasks, in-house trade professionals, and outsourced expertize. The two charts below (Figures 5.5 and 5.6) provide an estimate of maintenance full time equivalents for the 10 mgd DPR plant with breakdowns for process area as well as for trade/skill discipline.

Figure 5.5. Maintenance Estimate FTE by Plant Process – 10 MGD RO-Based Process

2.0

1.6

0.2

0.2

0.2

SITE MAINTENANCE MANPOWER REQUIREMENTS, FULL TIME EQUIVALENTS

Pretreatment

Desalination

UVAOP

Stabilization and Disifection

Distribution

219 Water Environment & Reuse Foundation

Figure 5.6. Maintenance Estimate FTE by Trade/Skill Discipline – 10 MGD RO-Based Process

5.3 Minimum Operator Certification Requirements The State of California outlines specific requirements for the certification level of operations staff required for water treatment plant operation. This requires that each water supplier will nominate at least one Chief Plant Operator and at least one shift operator for each water treatment facility utilized by the water system for each operating shift. The Chief Plant Operator must have an operator certificate grade equal to or greater than the Plant Classification (e.g., Class T3 WTP requires that CPO has a valid, unexpired T3 Certificate). Plants are classified according to capacity, technology used, and water quality risks. Wastewater treatment plants have a similar requirement.

It is anticipated that DPR systems will incur a similar requirement, and this must be taken into consideration for plant staffing. This, including a more thorough discussion of operator certification and training is included in Chapter 6.

0.4

0.4

1.80.0

0.0

1.6

OVERALL MAINTENANCE BREAKOUT BY TRADE, FULL TIME EQUIVALENTS

ElectricalI&CMechanicalPlumbingHVACOperator

220 Water Environment & Reuse Foundation

5.4 Important Roles and Responsibilities In addition to the operational and task based analysis provided earlier in this chapter, there are a number of areas of responsibility that require specific attention for DPR facilities. For large installations (say > 10 mgd) these roles may be assigned as a full time role, or as a significant part of a role. For smaller installations, this will present a challenge to operational budgets. These roles include: CCP Manager. As noted in Chapter 3, the HACCP process requires the assignment of HACCP lead

to manage and champion the overall CCP process. This would involve managing source water quality risk assessments, reviewing CCP reporting and regular reviews. This role would fit best with a water quality manager, chemist, and process engineer or experienced lead operator.

Water Quality Manager. This responsibility involves managing of sampling and analysis schedules, management of internal laboratories, external laboratories, water quality data management and reporting. This role may also incorporate the CCP manager responsibilities.

SCADA and Controls. DPR plants have a high level of controls and automation. SCADA support is critical in the long term to support PLC program debugging, update of software, management of hardware, data management and reporting software. This role can be managed by internal staff for large utilities, and for smaller utilities and facilities is commonly outsourced to contract staff.

Instrumentation and Analyzers. DPR plants have a high level of instrumentation and analyzers. In particular, they may include somewhat complex analyzers that require extensive training of internal staff to calibrate and troubleshoot. In these cases, outsourcing requirements for either specific instrument management, or entire instrument inventories is an option for utilities. Again, for smaller utilities and facilities this may be the preferred option.

Operational Interface Manager. As noted in Chapter 3, the interface of operations between wastewater to advanced treatment, and advanced treatment to drinking water or distribution requires effective communication and collaboration. A key responsibility that should be included in the role of staff at each facility is that of Operational Interface Manager. This role would require the management of operational interface protocols, communication and collaboration.

Public Relations/Communications. Effective communications to stakeholders and the public is key to the success of a DPR facility. This role would generally be managed by public relations/communications staff at a utility headquarters. In some cases, this may be outsourced to a specific public relations firm.

221 Water Environment & Reuse Foundation

Chapter 6

Operator Certification and Training

6.1 Introduction Operator certification and training programs are used across the United States in order to provide a minimum standard of operational skill and knowledge for the operations of wastewater treatment plants, water treatment plants, and the management of drinking water distribution systems. Currently, potable reuse does not have its own certification curricula, but rather utilities rely on these existing wastewater and water certifications from which the pool of operations staff is drawn. While this covers a number of important elements for potable reuse, there currently remain gaps for both some of the technologies applied, as well as some of the operational tasks and methodologies.

The water and wastewater curricula have been developed in order to cover the requirements for the majority of water and wastewater treatment operators. Recycled water is currently produced by a small percentage of utilities overall, and while rapidly increasing in its application, IPR and DPR remain a relatively small enterprise in comparison to the vast number of water and wastewater treatment plants. As a result, IPR and DPR have not seen significant coverage in existing operator training curriculum or examinations.

Direct potable reuse necessitates the application of a variety of relatively advanced water treatment technologies in order to meet water quality requirements. Some of these technologies are covered to an extent in the existing curriculum. For example, reverse osmosis is applied not only in reuse applications but also for groundwater desalting and seawater desalination for drinking water applications and as a result it has some, albeit limited, coverage in the existing drinking water certification curriculum. Similarly, microfiltration and UV disinfection are relatively minor components of the curriculum. Other technologies however, such as advanced oxidation, are not currently covered at all.

IPR and in particular DPR are perceived as a higher risk to public health relative to normal drinking water production, due to the nature of the source water employed. In addition to the gaps in training for specific technologies employed, a key focus on the management of this higher risk to health (whether perceived or real) is imperative to the success of operations. More intensive requirements for water quality sampling and analysis; specific requirements for instrument calibration and verification; critical operational monitoring, reporting, and effective operational responses must all be considered in the development of a training and certification process to meet the requirements of DPR.

6.2 California – Existing Training and Certification Programs In California, training and certification programs exist for both drinking water and wastewater treatment operators. Wastewater and water treatment operations are managed separately, and until recently had been handled by separate organizations when the administration of the Drinking Water Program (DWP) was transferred from the Department of Public Health (DPH) to the State Water Resources Control Board (SWRCB) in July 2014. While they remain as two separate certification tracks, there are nonetheless a number of commonalities between the two programs including:

An operator grade level from 1 to 5, based on a combination of experience and the passing of examinations. For wastewater operators this is referred to as a grade (1 - 5), for drinking water it is referred to as T1 - T5. The higher number refers to a more experience and a higher level of coursework and certification.

222 Water Environment & Reuse Foundation

A classification system for permitted facilities, based on size and complexity. This classification demands a minimum requirement for operator grade certification for chief plant operator and shift operator.

Some commonality of curriculum material, noting the use of some treatment technologies in both water and wastewater programs.

Recycled water from wastewater origin is handled under the jurisdiction of the California SWRCB and contained within the wastewater curriculum. This covers recycled water at tertiary filtration levels (commonly referred to as Title 22 water) - although does not include significant content for IPR and none for DPR.

6.2.1 Drinking Water Training and Certification Laws and regulations governing certification of potable water treatment facility operation were enacted in California in 1971. These established the level at which water treatment facilities should be manned, the minimum qualifications for testing and criteria for renewal and revocation of operator certificates.

Water treatment plant operator certification is managed by the SWRCB since the transfer of these functions from the California Department of Public Health in 2014. It is currently managed by the Office of Operator Certification within the Division of Financial Assistance. The certification requirements are specified in Title 22 California Code of Regulations (CCR) Sections 63900 and 63905 for water treatment operators and distribution operators respectively.

Operator certification progresses through 5 levels, from T1 to T5, with each level requiring a demonstration of increased operational expertise and experience. Each level requires an educational pre-requisite and successful passing of an examination based on the knowledge, skills and abilities set forth in the regulation. Educational pre-requisites include such qualifications as high school diplomas or General Educational Development (GED), and also courses specific to water treatment. Examinations are offered twice per year. From level T3 upwards, the demonstration of applicable operating experience is also required. Certificates must be renewed every three years, with proof of continuing education required at all levels.

The required knowledge, skills, and abilities of each certification level are developed based on job analyses conducted by subject matter experts, who are typically water treatment system operators and managers with extensive field experience. There is an ongoing validation process to ensure that examination questions are representative of operator duties and responsibilities, with workshops attended by the subject matter experts to validate existing exam questions and to write new questions [Operator Certification Annual Report for State Fiscal Year 2013-2014]. Table 6.1 provides a summary of the educational and qualifying experience requirements for each level of drinking water certification in California.

223 Water Environment & Reuse Foundation

Table 6.1. Water Treatment Plant (WTP) Operator Certification Requirements California (1) WTP OPERATOR CERTIFICATION REQUIREMENTS TABLE

per Title 22 CCR, Division 4, Chapter 13 sections 63775 and 63800

PATH EDUCATION REQUIREMENT QUALIFYING EXPERIENCE T1

1 H.S. diploma or GED N/A

2 Successful completion of the “Basic Small Water System Operations” course provided by the Department.

N/A

3 One year as an operator of a facility that required an understanding of chemical feeds, hydraulic systems, and pumps.

N/A

T2

1 H.S. diploma or GED AND Successfully completed at least one course of specialized training covering the fundamentals of drinking water treatment.

N/A

2

Successful completion of the “Basic Small Water System Operations” course provided by the Department AND Successfully completed at least 1 course of specialized training covering the fundamentals of drinking water treatment.

N/A

3

One year as an operator of a facility that required an understanding of chemical feeds, hydraulic systems, and pumps AND Successfully completed at least 1 course of specialized training covering the fundamentals of drinking water treatment.

N/A

T3

1

H.S. diploma or GED AND Successfully completed a total of at least two courses of specialized training that includes at least one course covering the fundamentals of drinking water treatment

and At least one year of operator experience working as a certified T2 operator for a T2 facility or higher AND At least one additional year of operator experience working as a certified treatment operator.

A bachelor of science or a master of science degree AND the completion of a comprehensive operator training program may be used to fulfill the "initial" operator experience requirements.

A degree earned at an accredited academic institution may be used to fulfill the "additional" experience requirements as follows: * An Associate degree or certificate in water or wastewater technology that includes at least 15 units of physical, chemical, or biological science may be used to fulfill one year of operator experience. * A Bachelor's degree in engineering or in physical, chemical, or biological sciences may be used to fulfill 1.5 years of operator experience. * A Master's degree in engineering or in physical, chemical, or biological sciences may be used to fulfill two years of operator experience.

A certified operator may substitute on a day-for-day basis the "additional" experience requirements with experience gained while working with lead responsibility for water quality related projects or research. Experience gained as a certified waste water treatment plant operator may be used to fulfill up to two years of the "additional" operator experience requirements. Each two months of experience as a waste water treatment plant operator shall be considered equivalent to one month of water treatment facility operator experience.

224 Water Environment & Reuse Foundation

T4

1 Successfully completed at least 3 courses of specialized training that includes at least 2 courses in drinking water treatment

and

A valid Grade T3 operator certificate AND At least one year of operator experience working as a shift or chief operator, while holding a valid T3 operator certificate, at a T3 facility or higher AND At least three additional years of operator experience working as a certified treatment operator.

A bachelor of science or a master of science degree AND the completion of a comprehensive operator training program may be used to fulfill the "initial" operator experience requirements. A degree earned at an accredited academic institution may be used to fulfill the "additional" experience requirements as follows: * An Associate degree or certificate in water or wastewater technology that includes at least 15 units of physical, chemical, or biological science may be used to fulfill one year of operator experience. * A Bachelor's degree in engineering or in physical, chemical, or biological sciences may be used to fulfill 1.5 years of operator experience. * A Master's degree in engineering or in physical, chemical, or biological sciences may be used to fulfill two years of operator experience. Experience gained as a certified waste water treatment plant operator may be used to fulfill up to two years of the "additional" operator experience requirements. Each two months of experience as a waste water treatment plant operator shall be considered equivalent to one month of water treatment facility operator experience

T5

1

Successfully completed at least four courses of specialized training that includes at least two courses in drinking water treatment.

A valid Grade T4 operator certificate AND At least two years of operator experience working as a shift or chief operator, while holding a valid T4 operator certificate, at a T4 facility or higher AND At least three additional years of operator experience working as a certified treatment operator.

A degree earned at an accredited academic institution may be used to fulfill the "additional" experience requirements as follows: * An Associate degree or certificate in water or wastewater technology that includes at least 15 units of physical, chemical, or biological science may be used to fulfill one year of operator experience. * A Bachelor's degree in engineering or in physical, chemical, or biological sciences may be used to fulfill 1.5 years of operator experience. * A Master's degree in engineering or in physical, chemical, or biological sciences may be used to fulfill two years of operator experience. Experience gained as a certified waste water treatment plant operator may be used to fulfill up to two years of the "additional" operator experience requirements. Each two months of experience as a waste water treatment plant operator shall be considered equivalent to one month of water treatment facility operator experience.

To provide preparation for examinations, the SWRCB provides a course for the first two levels of certification “Basic Small Water System Operations”. Beyond this, there is a recommended list of reading materials and external courses which best map the requirements for examination preparation.

Chief among these is the Office of Water Programs at Sacramento State University. There are several correspondence and on line courses available that are specifically designed to support operator certification.(2) http://www.owp.csus.edu/courses/drinking-water.php. At the core of these courses are two text book volumes that are used as de facto core material for water treatment plant operator examination preparation:

225 Water Environment & Reuse Foundation

"Water Treatment Plant Operations, Volume 1". "Water Treatment Plant Operations, Volume 2."

These books are commonly referred to as the "Ken Kerri" books, as they were developed by Kenneth D. Kerri at the Office of Water Programs. The books are designed for use with a correspondence course, provide accepted continuing education units (CEUs) as well as material to pass the examination. In addition, the school has developed “Small Water System Operation and Maintenance” which is recommended reading for T1 operators, but does not provide CEUs.

Additional material is also recommended, including material from the American Water Works Association (AWWA), New York State Department of Health and other sources.

Plant Classification The state of California requires that the size and complexity of water treatment systems is matched to an appropriate skill and experience level of operator. Across the state, water treatment plants are classified based on a points system that is based on the water source, specific elements of influent water quality, the treatment processes employed and the plant capacity.

The criteria and scoring system in the development of plant classification is shown in the table below:

Table 6.2. Water Treatment Plant Classification – California (3) Classification of Water Treatment Plants Based on Total Points

Title 22 CCR, Division 4, Chapter 15 - Domestic Water Quality and Monitoring, Article 2

Area Criteria Points

Section (Title 22 CCR Div 4,

Chapter 15)

Source of Water

Groundwater and/or purchased treated water meeting primary and secondary drinking water standards, as defined in section 116275 of the Health and Safety Code

2

64413.1.b.1

Water that includes any surface water or groundwater under the direct influence of surface water

5

Microbiological water quality of influent

MPN less than 1 per 100 mL 0 64413.1.b.2

MPN 1 through 100 per 100 mL 2

MPN greater than 100 through 1,000 per 100 mL 4

MPN greater than 1,000 through 10,000 per 100 mL 6

MPN greater than 10,000 per 100 mL 8 Water Turbidity of Influent (for facilities treating surface water or groundwater under the direct influence of surface water)

Less than 15 NTU 0 64413.1.b.3

15 through 100 NTU 2

Greater than 100 NTU 5

Nitrite/Nitrate in Influent

Less than or equal to the maximum contaminant level (MCL), as specified in Table 64431-A

0 64413.1.b.4

Greater than the MCL 5

Contaminants with primary MCLs in Influent (for all contaminants)

Less than or equal to the MCL 0 64413.1.b.5 Greater than the MCL 2 5 Times the MCL or greater 5

Treatment Processes used at a surface water treatment plant for Conventional, direct, or inline filtration 15

64413.1.b.6

226 Water Environment & Reuse Foundation

compliance (Sum of points from each applicable line item)

Diatomaceous earth filtration 12 Slow sand, membrane, cartridge, or bag filter 8 Backwash recycled as part of process 5

Treatment process used to reduce the concentration of one or more contaminants for which a primary MCL exists. Each process that was not previously counted

10

64413.1.b.7 Treatment process used to reduce the concentration of +1 contaminants for which a secondary MCL exists. Each process that was not previously counted

3

64413.1.b.8 Treatment process used for corrosion control or fluoridation. Each process that was not previously counted

3 64413.1.b.9

Disinfection Process used for Log Inactivation Credit

Ozone 10 64413.1.b.10 Chlorine and/or chloramine 10 Chlorine dioxide 10 Ultraviolet (UV) 7

Disinfection/Oxidation Treatment without Inactivation Credit

Ozone 5 64413.1.b.11 Chlorine and/or chloramine 5 Chlorine dioxide 5 Ultraviolet (UV) 3 Other oxidants 5

Other treatment process that alters the physical or chemical characteristics of the drinking water and that was not included above

3

64413.1.b.12

(Maximum rated) Flow Capacity per MGD (maximum of 50 pts)

2 x MGD Flow 64413.1.b.13

Total Points Minimum Chief Operator Certificate Grade

Less than 20 T1

20 through 39 T2

40 through 59 T3

60 through 79 T4

80 or more T5

The state requires that each water supplier will nominate at least one Chief Plant Operator and at least one shift operator for each water treatment facility utilized by the water system for each operating shift. The Chief Plant Operator must have an operator certificate grade equal to or greater than the Plant Classification (e.g., Class T3 WTP requires that CPO has a valid, unexpired T3 Certificate).

6.2.2 Wastewater Training and Certification Wastewater treatment operator certification is also managed by the SWRCB.

Similar to drinking water, plants are classified and operator requirements set to meet those classifications which are outlined in Title 23, Chapter 26, Division III. These requirements are intended to ensure that

227 Water Environment & Reuse Foundation

operators meet a minimum level of competence and are matched to the size and complexity of the wastewater treatment plant that they will be operating.

Currently, water recycling plants are covered under this regulation and consequently water recycling plants require that operators are certified as wastewater treatment operators. This is reflective of the many recycled water plant operations including those providing water for irrigation as well as more advanced technology systems such as groundwater recharge.

Wastewater operator certification progresses through 5 levels, from Grade 1 to Grade 5 with each level a demonstration of increased operational expertise and experience. Similar to drinking water, each level requires an educational pre-requisite and successfully passing of an examination based on the knowledge, skills and abilities set forth in the regulation. Examinations are offered twice per year. In contrast with drinking water, demonstration of applicable operating experience is required for all levels of certification, not just from level three onwards. Certificates must be renewed every three years, with proof of continuing education required at all levels.

The required knowledge, skills, and abilities of each certification level are developed based on job analyses conducted by subject matter experts, who are typically wastewater treatment system operators and managers with extensive field experience. There is an ongoing validation process to ensure that examination questions are representative of operator duties and responsibilities, with workshops attended by the subject matter experts to validated existing exam questions and to write new questions.

Table 6.3 provides a summary of the educational and qualifying experience requirements for each level of certification in California.

228 Water Environment & Reuse Foundation

Table 6.3. WWTP Operator Certification Requirements California (4) WWTP OPERATOR CERTIFICATION REQUIREMENTS TABLE

Title per 23 CCR, Division 3, Chapter 26 section 3688

PATH EDUCATION

QUALIFYING EXPERIENCE GRADE I

1 H.S. diploma or equivalent and 6 educational points1 and 1 year of full-time qualifying experience

GRADE II

1 H.S. diploma or equivalent and 9 educational points and 18 months of full-time qualifying experience as a Grade I operator

2 H.S. diploma or equivalent and 12 educational points and 2 years of full-time qualifying

experience

3 Associate’s degree, a higher degree, or a minimum of 60 college semester units, including a minimum of 15 semester units of science courses

and 1 year of full-time qualifying experience

GRADE III

1 H.S. diploma or equivalent and 12 educational points and 3 years of full-time qualifying experience as a Grade II operator

2 H.S. diploma or equivalent and 18 educational points and 4 years of full-time qualifying experience

3 Associate’s degree or a minimum of 60 college semester units, including a minimum of 15 semester units of science courses

and 2 years of full-time qualifying experience

4 Bachelor’s degree or a higher degree, including a minimum of 30 semester units of science courses

and 1 year of full-time qualifying experience

GRADE IV

1 H.S. diploma or equivalent and 32 educational points and 6 years of full-time qualifying

experience

2 Associate’s degree or a minimum of 60 college semester units, including a minimum of 15 semester units of science courses

and 4 years of full-time qualifying experience

3 Bachelor’s degree or a higher degree, including a minimum of 30 semester units of science courses

and 3 years of full-time qualifying experience

4 Valid registration as a chemical, civil, or mechanical engineer issued by the California Board for Professional Engineers and Land Surveyors or by another state, territory, or Indian tribe

and 2 years of full-time qualifying experience

GRADE V

1 H.S. diploma or equivalent and 48 educational points and 10 years full-time qualifying

experience

2 Associate’s degree or a minimum of 60 college semester units, including a minimum of 15 semester units of science courses

and 6 years of full-time qualifying experience

3 Bachelor’s degree or a higher degree, including a minimum of 30 semester units of science courses

and 5 years of full-time qualifying experience

4 Valid registration as a chemical, civil, or mechanical engineer issued by the California Board for Professional Engineers and Land Surveyors or by another state, a territory, or an Indian tribe

and 4 years of full-time qualifying experience

(Note 1: Educational points are granted based on the completion of wastewater related courses or approved continuing education units. These are specified as:

(1) One completed three-unit semester course which is directly related to wastewater treatment and which is part of the curriculum of an accredited college or university is equal to eight educational points. Completed courses which result in more or less than three units or which are quarter units rather than semester units will be credited with educational points on a proportional basis.

229 Water Environment & Reuse Foundation

(2) All other courses will be assigned educational points at the rate of one educational point per 10 hours of completed classroom instruction. Subjects which are directly related to wastewater treatment shall be assigned full credit for educational points. Subjects which are indirectly related shall be given one half credit.

(3) One Continuing Education Unit which is directly related to wastewater treatment is equal to one educational point.)

Wastewater treatment plants are classified from lowest (I) to highest (V) based on treatment processes employed and the overall plant’s design flow. Table 6.4 outlines this classification:

Table 6.4. Wastewater Treatment Plant Classification Table (5)

A Chief Plant Operator (CPO) must be certified at the same level of the plant, or higher. The designated operator in charge of a shift must be certified at no more than one grade below the plant classification with the exception that a Class I plant requires a Grade 1 operator or higher, and a Class V plant may use a Grade III operator. Class IV and V plants must have over 50% of operators at Grade II or higher.

As noted above, recycled water plant operations are currently included under the wastewater certification umbrella. Additionally however, a certified water treatment plant operator may also operate a recycled water plant at a grade level appropriate for the class of wastewater plant being operated. This allowance opens the door to both certified water and wastewater treatment plant operators at water recycling

230 Water Environment & Reuse Foundation

facilities. Figure 6.1 provides a graphical representation of the operator certification requirements for water and wastewater operators within the California regulatory framework.

Figure 6.1. Existing Water and Wastewater Operator Certification Framework – California

6.2.3 Example Operator Certification at Advanced Reuse Facilities in California. Orange County Water District’s Groundwater Replenishment System (GWRS), in Fountain Valley California, utilizes wastewater treatment operators with only a few exceptions. This plant is an IPR facility with processes that include microfiltration, reverse osmosis, and UV with advanced oxidation treatment steps to produce water for groundwater injection and surface spreading. The selection of wastewater-certified operators is a preference of the facility managers who are of the view that wastewater operators often have a range of experience with different treatment processes and are more adept at the challenges of the relative complexity of the system. In addition, Orange County Water District does not operate any other water or wastewater facilities (the wastewater facility is operated by the Orange County Sanitation District – a distinct entity), and as a result there are no other facilities, either drinking water or wastewater from which operators are transferred to the GWRS.

The history of advanced recycled water treatment at this facility extends to the mid-1970s when reverse osmosis was first employed for recycled water as a means to control salinity, and as a result there is a great deal of organizational knowledge and institutional memory with some key operational staff still engaged in operations from that time. (6)

For GWRS, operator certification is considered to provide the minimum knowledge to work at the facility, and operator training is supplemented with over 150 hours of unique training which has been developed by consulting engineers, equipment manufacturers and operational staff. The Chief Plant Operator requires that operators pass an examination, developed in-house, for promotion which is geared to operations specifically at this facility.

Similar to OCWD, the City of Los Angeles Bureau of Sanitation (LABOS) Terminal Island recycled water facility, with processes that include microfiltration and reverse osmosis, also utilizes wastewater certified operators for their facility. This facility places a high value on certification and has also developed substantial site-specific training and lesson plans. (7)

In contrast, the Santa Clara Valley Water District’s Silicon Valley Advanced Water Treatment Plant, also a microfiltration, reverse osmosis, and ultraviolet light disinfection (with potential upgrade to advanced

Grade 5

Grade 4

Grade 3

Grade 2

Grade 1

Wastewater Operator

T 5

T 4

T 3

T 2

T 1

Water OperatorE

xper

ienc

e

Exp

erie

nce

Exam

Exam

Exam

Exam

Exam

Title per 23 CCR, Division 3, Chapter 26

Title 22 CCR, Division 4, Chapter 13

Trai

ning

Trai

ning

231 Water Environment & Reuse Foundation

oxidation) utilizes primarily drinking water certified operators. This is primarily driven by opportunities for operator career advancement at other operated drinking water treatment facilities within the district.(8)

There is, however, a challenge with this approach in that water treatment operators working at a reuse facility can only accrue water treatment operator-qualified experience at half time because the plant is not technically a potable water treatment facility. This reduced credit for experience may be a barrier to operator advancement for the operator who runs significantly more advanced treatment systems than most drinking water facilities and is yet penalized due to this requirement. Therefore, there is a need for a unified certification system that allows for career advancement and lateral moves between types of operating facilities.

6.3 Certification in Other States/Countries In addition to California, a review of a number of additional state operator certification frameworks was conducted. The intent was to determine firstly if direct potable reuse had been considered, how indirect potable reuse was considered, and also to review any alternative framework approaches that may be of benefit to a new certification program. A review of six states, along with a review of the Australian approach, were conducted as a part of this review.

The findings provided some useful additional options that could be adopted to enhance a future DPR certification program.

6.3.1 Washington Drinking water operator certification in Washington State is managed by the Washington State Department of Health, and Wastewater is managed by the State Department of Ecology. Both of these departments have developed standards for reclaimed water use and jointly administer a reclaimed water program, however operator certification is managed distinctly by each organization.

There are five levels of wastewater operator certification and four levels of water treatment plant operator certification. Similar to California, an operator in responsible charge must be certified at a level that is equal to or greater than the classification of the wastewater treatment plant. Progression through different certification has a minimum educational and experience requirement and requires the applicant to pass an examination.

6.3.2 Virginia The Board for Waterworks and Wastewater Works Operators and On Site Sewage System Professionals regulates individuals who operate water and wastewater treatment facilities and alternative on-site water reclamation systems. This board is comprised of two ex-officio members including the Director of the Office of Water Programs of the State Department of Health and the Director of the Department of Environmental Quality, a faculty member of a state university or college whose principal field of teaching is management or operation of water/wastewater works, and a citizen member representing operators.

The Virginia Department of Professional and Occupational regulation administers water and wastewater certification programs. As for other states, operators must hold certificates equal to or greater than the facility classification. Water facilities are classed from 1 through to 6 (with 1 being highest and 6 being no treatment). Wastewater facilities are classed from 1 through to 4.

6.3 3 Texas Water and wastewater operator certification is managed by the Texas Commission on Environmental Quality (TCEQ). Drinking water operations has four classes, with A the highest and D the lowest. Additionally, there is a distinction between licenses geared to specific water sources including surface

232 Water Environment & Reuse Foundation

water and groundwater. The class of operator required for a particular drinking water system is outlined based on the number of system connections served, types of treatment and type of disinfection used.

Effective in September of 2016, a specific requirement for reverse osmosis and nanofiltration membrane systems will require that operators complete an approved training course for operations and maintenance of those systems. (9)

Wastewater treatment operators also have four classes, from A to D. In this case the chief operator of each facility must possess a license equal to or higher than that of the category of treatment facility. Wastewater categories are determined by type of treatment and plant capacity. (10)

Texas recently had two temporary operating DPR systems. The Colorado River Municipal Water District’s Raw Water Production facility in Big Spring, Texas treats secondary municipal effluent utilizing microfiltration, reverse osmosis, and UV Advanced oxidation treatment. For this plant, TCEQ require that a Class B surface water (drinking water) operator be employed by the CRMWD reclamation facility, with class C surface water operator able to operate the facility if the Class B operator is on call. Water and wastewater licensed operators may also operate the facility, however there is a prohibition on operators from the wastewater facility operating at the raw water production facility during the same shift to prevent introduction of contaminants. (11)

6.3.4 Arizona Reuse of suitably treated wastewater effluent (reclaimed water) has been practiced in Arizona for over 50 years. Originally, the predominant method of reuse was direct non-potable reuse for irrigation of crops and turf grass. Beginning in the mid-1980s, Arizona utilities began what amounted to indirect potable reuse (IPR) projects involving the recharge of water supply aquifers using reclaimed water. With the state’s water conservation standards requiring safe-yield from groundwater, reclaimed water recharge is a widely-used method of generating credits to enable the pumping of an equivalent volume of groundwater back out of the ground for consumption. Since most municipalities in Arizona operate combined water/wastewater operations, the same entity that seeks recharge credits has control of reclaimed water that can be used to generate those credits. Over the past several decades, there has been a steady trend of decentralization in the direction of a greater number of smaller, local facilities developed by individual municipalities to optimize the resources as they best see fit to meet their individual needs. It has been estimated (12) that approximately 95% of the wastewater treated in the Phoenix metropolitan area is converted to reclaimed water and reused for some beneficial purpose.

As is the case elsewhere in the nation, the potential for direct potable reuse (DPR) of reclaimed water now has begun to draw attention in Arizona. As an outgrowth of a larger effort to plan for Arizona’s water future (13), the Steering Committee for Arizona Potable reuse (SCAPR) was initiated in March 2013, with its stated purpose being “to guide Arizona water interests in mitigating impediments to potable reuse (real or imagined) within industry standards of practice.”(14) One element of the SCAPR’s work has been to define potential treatment trains that might be feasible alternatives to the “industry-standard” full advanced treatment (FAT) train. Because of Arizona’ inland location, disposal of reverse osmosis brine has emerged as a major impediment to the deployment of RO treatment facilities in the state. At present, there are three alternative treatment trains (A, B, and C) proposed for consideration by the SCAPR: A: WWTP – ozonation – membrane biofiltration – advanced oxidation – blend – WTP B: WWTP – ozone – biologically active carbon – microfiltration – advanced oxidation – blend – WTP C: WWTP – ozone – biologically active carbon – microfiltration – GAC – blend – WTP

In Arizona, reclaimed water used in existing IPR efforts generally is produced simply as an extension of a typical wastewater treatment plant (WWTP), and these facilities are typically referred to as “water

233 Water Environment & Reuse Foundation

reclamation facilities” (WRFs). The principal processes to implement IPR at Arizona WRFs are biological denitrification, tertiary filtration, and a high level of disinfection (but not advanced oxidation) in addition of to what otherwise would be considered conventional wastewater treatment facilities. In most cases the reclaimed water has the opportunity to percolate downward for several hundred feet through the vadose zone before reaching the aquifer, thus providing the potential for soil-aquifer treatment as well.

Some Arizona utilities do provide advanced treatment (beyond filtration and denitrification) to water being reclaimed for reuse. The most widely-known example is the City of Scottsdale Water Campus, which includes microfiltration, reverse osmosis, and advanced oxidation processes to produce a very-high quality reclaimed water; some of this is delivered directly to reuse customers, principally golf courses, and the remainder is recharged into the local aquifer by vadose-zone recharge drywells. The Fountain Hills Sanitary District has a stand-alone microfiltration facility that further treats denitrified and filtered reclaimed water from the District’s wastewater treatment plant. This water is then recharged at an elevation below the water table (i.e., injected) into the local aquifer which at times serves as a back-up water supply to the community. Several communities, including Peoria, Bullhead City, Lake Havasu City, and Chino Valley have constructed membrane bioreactors to produce very-low turbidity reclaimed water but that water still is either recharged or delivered for non-potable direct reuse.

In each of these cases, the advanced treatment facilities provided for reuse are operated by wastewater treatment plant operations staff simply as part of the overall treatment facility. The operators of the WWTP are responsible for the added processes to produce reclaimed water for non-potable reuse, IPR, and potentially DPR.

In Arizona, the Arizona Department of Environmental Quality (ADEQ) has regulatory authority to certify operators of water and wastewater systems. ADEQ’s operator certification program includes certifications in four categories: Water Treatment, Water Distribution, Wastewater Collection, and Wastewater Treatment. Within each of the four certification categories in Arizona are four levels of certification, ranging from a nearly entry-level certification at Level 1 to advanced operator standing at Level 4. Operators must progress through each of the levels beginning with Level 1. ADEQ’s certification requirements include required periods of employment in the field of operations along with examination requirements.

There is no certification program in Arizona specifically for potable reuse (and at present direct potable reuse (DPR) remains prohibited). As discussed above, as a practical matter, reclaimed water in Arizona is produced at wastewater treatment facilities. The operators responsible for producing reclaimed water generally are wastewater treatment operators, even though unit processes relevant to potable reuse conceptually fall under both Water Treatment and Wastewater Treatment.

For approximately the past five years, Arizona has used the standard national operator certification examinations developed by the Association of Boards of Certification (ABC). Arizona is now one of over 40 states that use the operator certification exams provided by ABC. ADEQ contracts with ABC to provide the state with the examinations for the various categories and grade levels, and contracts with the Phoenix-area Gateway Community College (GCC) to administer the registration process and proctor the exams state-wide. The exams consist of 100 questions; 70% is the passing score.

The ABC examinations’ designs are based on “need to know criteria” in several categories, with the required depth and breadth of knowledge increasing with the level of certification being sought. ABC makes guidance documents available that summarize the design and content of the examinations, but they do not provide training per se or study materials for exam preparation. Questions on the examinations fall into six areas of “core competencies.” The number of questions per category varies only slightly between certification levels. Within each of the core competency category specific potential topics are listed in the

234 Water Environment & Reuse Foundation

ABC need to know criteria. Those are also listed below, along with a summary total of the other potential topics included in the competency categories. Table 6.5. ABC Water and Wastewater Core Competencies

Water Treatment No. questions Wastewater Treatment

No. questions

Evaluate Characteristics of Source Water Bacteriological Biological Chemical 6 other topics

5 - 6 Evaluate Physical Characteristics of Wastestream (8 categories)

5 - 6

Monitor, Evaluate, and Adjust Treatment Processes Chemical addition Chlorine dioxide Chlorine gas Ozonation Ultraviolet 4 other subtopics Coagulation and flocculation (3 subtopics) Clarification and sedimentation (5 subtopics) Filtration Membranes 7 other subtopics Residuals disposal (8 subtopics) Additional treatment tasks (12 subtopics)

Monitor, Evaluate, and Adjust Treatment Processes Preliminary treatment (5 subtopics) Primary treatment: clarifiers Secondary treatment (14 subtopics) Tertiary treatment Advanced waste treatment, chemical recovery, carbon regeneration Biological or biological/chemical advanced waste treatment Chemical/physical advanced waste treatment following secondary Ion exchange Media filtration RO, ED or other membrane filtration techniques Disinfection Chlorination Dechlorination Hypochlorination Ozonation UV irradiation Chemical Addition (dry, gas, or liquid) Effluent discharge For Reuse For Discharge Solids Handling (14 subtopics)

32 - 35

Operate and Maintain Equipment Evaluate Operation of Equipment (6 subtopics) Operate Equipment (17 subtopics) Perform Maintenance (20 subtopics)

24 - 27 Operate Equipment (25 categories)

16

Comply With Drinking Water Regulations: 19 EPA rules

10 - 12 Evaluate and Maintain Equipment Evaluate equipment (10 subtopics) Perform Preventive and Corrective Maintenance (34 subtopics)

27 - 28

Laboratory Analyses (22 categories)

11 - 13 Laboratory Analysis (33 categories)

7

Perform Security, Safety, and Administrative Procedures (5 categories)

13 - 18 Perform Security, Safety, and Administrative Procedures (6 categories)

10 - 11

As can be seen from Table 6.5, reuse-related topics are included but only to a limited extent on the existing ABC water and wastewater certification exams used in Arizona. With 100 questions on the examination, and many more potential categories and topics to be covered, it is clear that the existing water and wastewater certification examinations only scratch the surface in terms of measuring operator applicants’ capabilities relevant to potable reuse.

235 Water Environment & Reuse Foundation

There is a widely noted lack of relevant training offered in Arizona to assist operators in preparing for water and wastewater certification exams in general. ADEQ offers some training classes, but they typically focus on ADEQ rules, administrative matters and safe drinking water topics. Most of ADEQ’s funding for training comes from the U.S. EPA safe drinking water program and therefore cannot be used for “wastewater” topics. There are several private companies that offer some training programs, but in general those get mixed reviews from personnel who have taken the classes.

6.3.5 North Carolina North Carolina requires the certification of water and wastewater operators to ensure competency in treatment as well as water distribution and wastewater collection. The certification of operators is the responsibility of the NC Department of Environment and Natural Resources (NCDENR). Certified operators have been required since 1971 under the purview of NCDENR and their respective Certification Boards.

Drinking Water Treatment certification is handled under the Water Treatment Facility Operators Board of Certification. Wastewater certification is overseen by the Water Pollution Control System Operators Certification Commission. NCDENR is responsible for training of operators in association with educational institutions and other private or public entities.

Drinking Water Systems are classified into general categories of surface water, groundwater (or well systems) and distribution systems. Certification for cross connection control is also part of the Drinking Water certification program.

Drinking Treatment facilities with surface or groundwater are classified based on a point system and designated as Class A, B, C or D. The points are assigned based on a range of parameters such as population served, type of source water, processes used, chemicals applied in treatment, and quality control complexity and associated analytical needs. Twenty three (23) different categories, processes and systems are scored (see attached -Subchapter 18D – Water Treatment Facility Operators).

Examples of scoring include: Population Served: 1 point per 1,000 served = 50 points max. Plant Capacity: 1 point per 1 MGD capacity = 25 points max. Sedimentation: Standard rate = 5 points; Tube Settlers = 3 points; Pulsators and plates = 5 points Disinfection: Gas Cl2 = 10 points; Hypochlorite = 7 points; Ozone = 13 points; ammonia and chlorine

= 12 points. Reverse Osmosis = 15 points Filtration: Sand = 10; Anthracite/GAC = 12; with surface wash or air scour= 2

Class A systems with large capacity and/or complex treatment are categorized as “A” and score over 110 points; Class “B” systems are in the 51-110 range, etc.

Drinking Water Distribution certification (again Class A through D) is simply based on the number of service connections. A Class A system would have over 3300 service connections.

Grades of drinking water operator certification mirror the A, B, C and D system classification. All treatment systems and distribution systems must have an “operator in responsible charge” that is licensed with the appropriate Grade of A,B,C, or D. In order to become certified, individuals must complete educational and experience requirements, pass examinations, as well as continuing education credits on an annual basis. Operator schools and examinations are conducted throughout the State on an annual

236 Water Environment & Reuse Foundation

basis. Experience requirements in order to achieve a higher grade generally range are from six months to a year and increase with higher levels.

Water Pollution Control Systems (i.e., WWTPs) include certification for treatment systems, collection systems, surface irrigation, land application, and subsurface water pollution control. Treatment is generally categorized as Biological Water Pollution Control Systems with Grades I thought IV. Grade I is basically passive systems such as septic, lagoons or wetlands. Higher Grades include activated sludge or fixed growth processes as well as more sophisticated processes are classified by daily flows in mgd (Grade IV is achieved at > 2.5 mgd). Physical/Chemical systems for such activities such as groundwater remediation also are classified as Grade I or II and have individual certifications. Collection systems are based on population served with Grade IV being the highest with > 50,000 served.

Grades of Water Pollution Control Operator Certification mirror the Grade I, II, III, IV system classification. All treatment systems and distribution systems must have an “operator in responsible charge” that is licensed with the appropriate Grade of the facility or collection system.

In order to become certified, individuals must complete educational and experience requirements, pass examinations, as well as continuing education credits on an annual basis. Operator schools and examinations are conducted throughout the State on an annual basis. Experience requirements in order to achieve a higher grade certifications generally range are from six months to three years, and increase with higher levels of certification. College education from both two and four year institutions with relevant course work can substitute for some experience time. Continuing education requirements are required as well. Reciprocity is possible from States that have programs that meet or exceed NC requirements.

While DPR in North Carolina was legislatively allowed as of July 31, 2014 (NC Senate Bill 163, State Legislation 2014-113, “Reclaimed Water as a Source Water”), there is no clear path or requirement for operator certification. Likewise, the legislation does not provide specific treatment or water quality requirements (other than meeting state and federal drinking water standards), nor does it indicate the need for a separate advanced water treatment facility. Therefore it is likely that operation of the specific processes used for DPR would fall under the supervision of the operators staffing that facility (i.e., wastewater operators would operate any processes located at their facility while drinking water operators would operator any additional treatment processes at the DWTP). However, process-specific training will be required for advanced treatment that has not been typically used at the DWTP or WWTP.

6.3.6 Wisconsin All of the previous states noted, including California, use a similar approach of education and work experience to qualify for an examination which when passed will allow the operator to gain a license and progress to subsequent higher license levels. One of the disadvantages of this approach is that the certification is more generalist, and not necessarily representative of the competencies required to perform particular tasks. For example, an operator that manages a trickling filter will be required to answer questions on activated sludge processes, or an oxidation pond operator may be tested on centrifuge operation and anaerobic digestion.

Wisconsin offers a different approach to wastewater plant classification. Certification categories for both operators and wastewater facilities changed from four levels (prior to 2015) to two levels (basic and advanced). There is also a Grade T level, or operator-in-training. Subclasses are assigned to the wastewater treatment plants that correspond to the processes used at each plant. To become a certified wastewater treatment plant operator, both a general examination and at least one subclass examination must be passed, which should be relevant to the plant at which the operator is engaged, or may be relevant to a plant that the operator will work in the future. The categories of subclasses are shown in Table 6.6.

237 Water Environment & Reuse Foundation

Table 6.6. Wisconsin Wastewater Operator Subclass Categories (15)

Category Subclass Name Description

Biological Treatment Suspended Growth Processes Activated Sludge and Variants

Attached Growth Processes Trickling filters, RBCs and biotowers

Recirculating Media Filters

Ponds, Lagoons and Natural Systems

Anaerobic Treatment of Liquid Waste

High strength liquid waste treatment system.

Solids Separation Biological Solids/Sludge Handling, Processing and Reuse

Aerobic and anaerobic digestion, thickening, dewatering, land application.

Nutrient Removal Total Phosphorus

Total Nitrogen

Disinfection Disinfection Chlorination, ultraviolet radiation, ozone.

Laboratory Laboratory Registered or certified on-site laboratories.

Special Unique treatment systems Unique, special treatment plants that use biological, chemical or physical methods.

Collection System Sanitary Sewage Collection System

The Wisconsin approach to wastewater operator certification provides the advantage of a general examination that achieves the goal of having all operators aware of rule, regulations, general treatment technologies and troubleshooting; while subclass examinations provides for a focused detail on the processes that those operators are responsible for operating.

6.3.7 Australia Australia has been developing a national drinking water operator certification scheme to provide a consistent criteria that defines and recognizes the minimum level of competency and capability required of operators who treat and/or sample drinking water. A national framework has yet to be adopted, however a trial of the proposed system is in operation in two Australian states (Queensland and New South Wales) while a longer established system is being used in the state of Victoria.

Water treatment plants are ranked not according to plant capacity or technologies employed, but rather a level of microbial risk. Each water treatment facility is required to undertake a water supply public health (microbial) risk classification. The rationale is that the greater the microbial risk, the more technologically

238 Water Environment & Reuse Foundation

complex the water treatment system becomes to manage that risk. A relatively simple scoring table provides facilities with a ranking mechanism for each of those plants:

Table 6.7. Scoring Table for Water Supply System Public Health (Microbial) Risk Classification (16) Item Max Points Comments Raw water sources (rating based on public health significance) Catchment:

Protected (0 points) Unprotected (60 points)

60 A fully protected catchment is one where the entire catchment is protected from watershed to water treatment facility. Where this is not the case the catchment is considered to be unprotected.

Raw Water Source: Seawater/saltwater (0 points) (assumes reverse

osmosis treatment) Groundwater (confined aquifer) (0 points) Surface Water

o Reservoir with greater than 30 days detention time under normal operating conditions (5 points)

o Reservoir with less than 30 days detention time under normal operating conditions (10 points)

o River or stream (30 points)

30 Surface waters (either reservoirs or river/streams) score more highly than other sources because of the risk of contamination. Streams score more highly than reservoirs since there is no holding period for the water.

Raw Water Quality Average raw water quality variation

Little or no variation (0 points) Minor variations: ‘high quality’ surface or

groundwater source (1 point) Moderate variations: during variations in raw water

quality, coagulant dose (or pH adjustment chemicals dose) changes are made monthly (3 points), weekly (4 points), or daily (5 points).

5

Rainfall event raw water quality variation Minor variations: during rainfall events the increase

in raw water turbidity is less than 50% of the dry weather figure, or raw water turbidity remains < 25 NTU during rainfall events (10 points)

Moderate variations: during rainfall events the increase in raw water turbidity is between 50% and 100% of the dry weather figure, or the raw water turbidity is between 25 and 100 NTU during rainfall events (20 points)

Severe variations: during rainfall events the increase in raw water turbidity is greater than 100% of the dry weather figure, or raw water turbidity is greater than 100 NTU during rainfall events (30 points)

30 Rainfall events have been identified as major hazard and significantly increase the risk of pathogen breakthrough.

Pollutant input to raw water Raw water source subject to:

o Agricultural or septic tank inputs (20 points)

o Sewer overflows during rainfall events (15 points)

o Treated effluent from sewage treatment plants (10 points)

o Light industrial waste (5 points) o None of the listed inputs (0 points)

50 Point source discharge, in particular any source that may be contaminated with fecal material increases the risk to public health.

Other raw water characteristics critical to treatment processes 30 Various attributes of raw water have an impact on the treatment processes and if

239 Water Environment & Reuse Foundation

Presence of tasted and/or odor compounds for which treatment process adjustments are routinely made (2 points)

Presence of cyanobacteria and possible toxins (4 points)

Iron and/or manganese > Australian Drinking Water Guideline limits (4 points)

1 point per average mg/L DOC to maximum of 20 mg/L

not managed may have an adverse impact on the quality of water or on the management of the distribution system.

Filter Ripening Period Water produced during filter ripening to waste (0

points) Water produced during filter ripening not sent to

waste (5 points)

5 Filter ripening (the period immediately after backwash) is characterized by high turbidity water which in turn represents a risk to consumers.

Residuals management Sludge supernatant/backwash water not returned to

head of plant (0) Sludge supernatant/backwash water treated with

ozone or UV and returned to raw water storage (0 points)

Clarified and/or settled sludge supernatant/backwash water, treated with ozone or UV and returned to head of plant prior to coagulation point (5 points)

Unsettled and/or untreated sludge supernatant/backwash water returned to head of plant prior to coagulation points (10 points)

Unsettled and/or untreated sludge supernatant/backwash water returned to head of plant after coagulation point (30 points)

Return of backwash water or sludge supernatant to the head of the plant carries with it a risk of returning viable protozoan pathogens to the influent. Wherever any recycled stream is returned to the head of plant, best practice is that the flow is continuous and less than 5% of inflow and that coagulation dosing is flow paced and includes an allowance for changes in flow.

The microbial risk classification is calculated by adding the individual points and expressing them as a percentage of the total possible. The levels of each water treatment facility is then based on the following criteria:

Table 6.8. Microbial Risk Plant Classifications (16)

Score Microbial Risk Classification 70% and above Level 4

Between 50% and 70% Level 3

Less than 50% Level 2

240 Water Environment & Reuse Foundation

For each of these plant classifications, a responsible person should be qualified to a minimum level of certification. The responsible person is defined as the staff member who has day-to-day operational responsibility for a particular water treatment facility. The level of certification is noted in the following table. The structure of certification for the Australian system includes a number of certification levels including:

Table 6.9. Minimum Qualifications, Experience, and Refresher Training Requirements for the Responsible Person at Water Treatment Systems per Classification (16)

Microbial Risk Classification Certification Level Experience Refresher Training Level 2 Certificate II in Water Operations At least nine months

in a water treatment or water quality role

Refresher Training Required

Level 3 Certificate III in Water Operations Two years responsibility for a Level 2 facility or above, or two years assisting in the operation at a Level 3 facility

Refresher training required, plus a mandatory safe drinking water issues update course, during every three year period.

Level 4 Certificate IV in Water Operations Two years responsibility for a Level 3 facility

Refresher training required, plus a mandatory safe drinking water issues update course, during every three year period.

An important differentiator of the Australian system of water operator certification, relative to systems in the U.S., is that certification is not gained via an examination and experience, but it is achieved by mastering and demonstrating specific modules of competence. There are both core competencies that must be mastered for each specific level of certification, as well as a number of elective competencies, in which the operator can focus on those areas that are most relevant to the plant or plants at which they work.

Competency-based training is an approach to vocational education and training that places emphasis on what a person can do in the workplace as a result of completing a program of training or based on workplace experience and learning. Progress within a competency-based training program is not based on time. As soon as students have achieved or demonstrated the required competency, they can move to the next competency.

241 Water Environment & Reuse Foundation

By way of example, the Certificate III requirements in water operations require that the candidate demonstrate competency in eleven units of competency, comprising three core and eight elective units, taken from the following list:

Table 6.10. Certificate III Water Operations Competency Requirements (Water Industry Training Center NWP -07 Training Handbook)

Core (These units must be completed)

NWP301B Implement, monitor and coordinate environmental procedures

BSBWOR301B Organize personal work priorities and development

BSBOHS303B Contribute to OHS hazard identification and risk assessment

Electives – (8 of these electives must be completed).

NWP300B Provide and promote customer service

NWP345B Monitor, operate and control water treatment processes

NWP346B Monitor, operate and control wastewater treatment processes

NWP347B Monitor, operate and control coagulation and flocculation processes

NWP348B Monitor, operate and control sedimentation and clarification processes

NWP349B Monitor, operate and control sedimentation and incineration processes

NWP350B Monitor, operate and control trickling filter processes

NWP351B Monitor, operate and control activated sludge processes

NWP352B Monitor, operate and control dissolved air flotation processes

NWP353B Monitor, operate and control anaerobic bioreactor processes

NWP354B Monitor, operate and control granular media filtration processes

NWP355B Monitor, operate and control membrane filtration processes

NWP356 Monitor, operate and control ion exchange processes

NWP357 Monitor, operate and control reverse osmosis and nanofiltration processes

NWP359 Monitor, operate and control nutrient removal processes

NWP360 Monitor, operate and control dewatering processes

NWP361 Monitor, operate and control gas scrubber processes

NWP362 Monitor, operate and control reclaimed water irrigation

242 Water Environment & Reuse Foundation

NWP363 Monitor performance and control maintenance of treatment plant assets

NWP364 Perform laboratory testing

NWP365 Identify and confirm blue green algae outbreaks

NWP366 Monitor, operate and control chloramination disinfection process

NWP367 Monitor, operate and control activated carbon adsorption process

NWP368 Respond to blue green algae incidents

NWP369 Monitor, operate and control lagoon processes

NWP370 Perform industry calculations

Each of the individual modules can be provided by attendance at specific training courses (often delivered by technical colleges, private training providers or on line). The training organization that delivers the competency module will organize an assessment through a combination of specific examination of knowledge components as well as an assessment of practical application in the workplace. This may be done in a simulated or real situation in the workplace.

Assessors of competencies may be representatives that provide the training courses, or often are supervisory and/or more experienced staff at the water treatment facility itself.

Example of a competency module – NWP355B Monitor Operate and Control Membrane Filtration Processes.

243 Water Environment & Reuse Foundation

Table 6.11. Example of a Competency Module - NWP355B Monitor Operate and Control Membrane Filtration Processes – Australian Government Industry Skills Panel NWP07 Water Training Package (17) Unit Descriptor This unit of competency describes the outcomes required to monitor, operate and control membrane filtration plant, including micro and ultra-filtration; and to measure and report on system performance and process quality control. The ability to identify faults, determine and apply technical adjustments, conduct chemical dosing procedures and produce technical reports are essential to performance. Application of the Unit This unit supports the attainment of skills and knowledge required for operational staff with a specific responsibility for ensuring that membrane filtration processes in treatment plants conform to organizational standards and comply with statutory requirements. Elements and Performance Criteria

ELEMENT PERFORMANCE CRITERIA

1 Monitor membrane filtration plant performance.

1.1 Monitor test results and processes to maintain the parameters of operation. 1.2 Identify and report process faults and the operational condition of plant according to organizational and statutory requirements. 1.3 Correctly select, fit and use required safety equipment, including personal protective equipment.

2 Control chemical use. 2.1 Use, handle and store chemicals according to organizational procedures and statutory requirements. 2.2 Determine chemical dosing according to plant processes and organizational procedures and statutory requirements. 2.3 Maintain chemical supply and usage records according to statutory requirements.

3 Operate and control membrane filtration processes.

3.1 Carry out routine plant inspections according to organizational and plant requirements 3.2 Conduct and analyze process tests and determine performance against plant operational requirements. 3.3 Make integrated process adjustments to improve system performance according to organizational and statutory requirements. 3.4 Collect, interpret and record process data according to organizational and plant requirements.

Required Skills and Knowledge This describes the essential skills and knowledge and their level, required for this unit. Required skills:

identify and correct operational problems produce reports and logs use safety and personal protective equipment interpret plans, charts and instructions interpret policies, procedures and standards communicate with employees and customers use communication equipment give and receive instructions determine chemical dosing requirements perform system calculations operate computerized equipment identify control system faults sample and test products

244 Water Environment & Reuse Foundation

Evidence Guide The Evidence Guide provides advice on assessment and must be read in conjunction with the Performance Criteria, Required Skills and Knowledge, the Range Statement and the Assessment Guidelines for the Training Package. Critical aspects for assessment and evidence required to demonstrate competency in this unit

The candidate should demonstrate the ability to monitor, operate and control membrane filtration processes, including:

monitoring test results and processes identifying and reporting faults conducting routine plant inspections taking samples and performing basic tests preparing and applying chemical dosing making basic process adjustments according to

instructions collecting data and completing required documentation

Context of and specific resources for assessment

Access to the workplace and resources including: documentation that should normally be available in a

water industry organization relevant codes, standards, and government regulations

Where applicable, physical resources should include equipment modified for people with disabilities. Access must be provided to appropriate learning and/or assessment support when required. Assessment processes and techniques must be culturally appropriate, and appropriate to the language and literacy capacity of the candidate and the work being performed. Validity and sufficiency of evidence requires that:

competency will need to be demonstrated over a period of time reflecting the scope of the role and the practical requirements of the workplace

where the assessment is part of a structured learning experience the evidence collected must relate to a number of performances assessed at different points in time and separated by further learning and practice

a decision of competence only taken at the point when the assessor has complete confidence in the person's competence over time and in various contexts

all assessment that is part of a structured learning experience must include a combination of direct, indirect and supplementary evidence

where assessment is for the purpose of recognition (RCC/RPL), the evidence provided will need to be authenticated and show that it represents competency demonstrated over a period of time

assessment can be through simulated project-based activity and must include evidence relating to each of the elements in this unit

In all cases where practical assessment is used it will be combined with targeted questioning to assess the underpinning

245 Water Environment & Reuse Foundation

knowledge. Questioning will be undertaken in a manner appropriate to the skill levels of the operator, any cultural issues that may affect responses to the questions, and reflecting the requirements of the competency and the work being performed.

Range Statement The range statement relates to the unit of competency as a whole. It allows for different work environments and situations that may affect performance. Bold italicized wording, if used in the Performance Criteria, is detailed below. Add any essential operating conditions that may be present with training and assessment depending on the work situation, needs if the candidate, accessibility of the item, and local industry and regional contexts. Processes may include: micro filtration

ultra-filtration

Organizational and statutory requirements may include:

by-laws and organizational policies standard operating procedures Australian Drinking Water Guidelines National Water Quality Management strategy environment protection occupational health and safety, including the use of personal

protective equipment chemicals dangerous goods lifts and cranes Environment Protection Authority regulations World Health Organization standards licensing agreements electrical standards

Routine plant inspections may include:

the use of equipment, including: electronic monitoring and metering systems chart recording systems basic hand tools sampling and laboratory testing equipment computerized equipment on- and off-road vehicles communication equipment personal protective equipment interaction and communication with other employees, other

authorities and the general public visual observation implementation of reporting procedures that may also include

procedures for the implementation of by-laws, organizational policies and statutory requirements

Tests may include: turbidity

color pH transmembrane pressure

246 Water Environment & Reuse Foundation

6.4 Summary of Existing Systems In the cases of all of the states within the United States reviewed as a part of this study, there is a heavy emphasis on a combination of an examination, and years of experience in the field in order to progress through different levels of operator certification. Drinking water and wastewater treatment operator certification is managed separately, with recycled water often considered a part of the wastewater treatment portfolio. There is no separate certification for reuse operators, and in particular nothing specific for IPR or DPR. Operators in existing IPR reuse systems are drawn from both wastewater and drinking water operations certifications, depending upon the history and circumstances of their particular plant and other municipal operations. Each level of certification is matched to a rating of the treatment plant at which the operator works – with a focus on ensuring a minimum level of competence particularly for the Chief Plant Operator and any supervisors. This rating is based typically on treatment processes employed, raw water quality and plant capacity.

Figure 6.2 Process of Water and Wastewater Treatment Plant Operator Certification (Courtesy CA/NV AWWA)

An interesting approach is offered by the system used in Wisconsin, in which a number of electives that are tailored to process selection enable plant operators to focus on processes that are consistent with the plants that they operate.

By contrast, the Australian system provides an alternative approach of using competency-based training and assessment – ensuring that operators can demonstrate their competence of specific tasks that are important to the delivery of their jobs. As systems in the US are heavily invested in the examination approach, it is not anticipated that a switch to a wholly competency-based approach is realistic. However, consideration of elements of this approach are nonetheless worthwhile either in terms of adding more specific requirements for the operator experience component, or for use in site specific training programs that are in addition to any operator certification.

One of the challenges of operator certification currently, especially noted in California, is the ability of operators to progress to higher certification levels, due to the requirement of operating experience at water or wastewater treatment facilities of a sufficient certification level. Many smaller utilities may not have these plants within their portfolio, and consequently operators within that utility may be unable to progress without moving to another utility. It is common practice for some utilities to share higher level operators between themselves in order to cover absence of an operator for sickness, vacation, or while a new recruit is to be hired. Given that the number of DPR systems (and likely IPR systems) will be relatively low compared with the number of water and wastewater facilities, consideration will need to be given as how to both: Gather sufficiently trained staff from the existing operating pool. Ensure experience can be gathered sufficiently to progress through certifications.

247 Water Environment & Reuse Foundation

In addition, consideration of the existing pool of IPR system operators should be taken into account, with an acknowledgement of their capability through some kind of grandfathering to any future certification program. The requirements of operator certification for any DPR system cannot be too onerous as to discourage operational staff from investing their time and effort. At the same time, , the operational certification process should be sufficiently portable back to water and wastewater operations such that the certification process does not preclude that person from making lateral moves or career-advancing moves to facilities other than IPR or DPR plants. As a summary, from this review an operator certification program for DPR (and IPR) should take the following into consideration: How to leverage experience and content from existing water and wastewater certification programs. Consideration of the use of elective modules specific to the operating plant (as per the Wisconsin

Model) in order to tailor to the treatment technologies used in IPR/DPR systems. Consideration of competency modules, to more clearly define some of the operational experience

requirement for certification. Ensure key technologies used for IPR/DPR are included in the curriculum

The team’s proposed recommendation for the operator certification considers these important aspects.

6.5 Proposed Operator Certification IPR/DPR Curriculum One of the most important elements of a workable operational certification program is to ensure that the curriculum best matches the technology mastery and skills that will be required for the operator. While this project was not specifically designed to develop the content for DPR operator certification, a review of existing curricula for water and wastewater certification has been conducted, along with a preliminary gap analysis to assess what additional material will be required.

6.5.1 Treatment Processes for Consideration The treatment processes under consideration for this project are consistent with those that were assessed under the Critical Control Point Assessment to Quantify the Robustness and Reliability of Multiple Barriers of a DPR Scheme (Reuse-13-03). These processes differ slightly from those provided in the project brief as they have both undergone critical control point assessment, and additional barriers where required have been included. The first process is a reverse osmosis (RO) membrane-based treatment train consisting of membrane filtration (either microfiltration or ultrafiltration), reverse osmosis, UV advanced oxidation and final chlorination (Figure 6.3). Based on the critical control point selection work conducted in project Reuse-13-03, pre-chloramination and stabilization have also been included as necessary components of the treatment system.

Figure 6.3. RO Membrane-Based Treatment Train

248 Water Environment & Reuse Foundation

The second process is an ozone-BAC based treatment train consisting of ozone, biological activated carbon, GAC, UV disinfection, and final chlorination (Figure 6.4). As with the RO based train, additional processes were included as a result of the critical control point selection from Reuse-13-03. This includes the addition of a coagulation/settling process ahead of the ozone and biological activated carbon.

Figure 6.4. Ozone-BAC-Based Treatment Train

Based on these treatment options and the unit processes contained therein, a detailed review of the existing California water operator and wastewater treatment operator curriculum was conducted in order to determine what content and requirements are already suitable for DPR operations, and to conduct a gap analysis to determine what additional material will be required.

6.5.2 Existing Water and Wastewater Curricula in California Water and wastewater examination content was reviewed, along with supporting reference material, as a key to the curriculum material that is covered in the examinations. Actual copies of past examinations could not be obtained, as this material is considered sensitive and will not be released for fear of compromising the examination process. A gap analysis was then conducted reviewing requirements for DPR both in terms of technologies employed and also additional operational requirements considered critical to the success of DPR operations.

Wastewater treatment plant examination content is outlined as per the specific regulation (Section 3701 Title 23. Waters Division 3. State Water Resources Control Board and Regional Water Quality Control Boards Chapter 26. Classification of Wastewater Treatment Plants and Operator Certification)(18). Table 6.12 details the content, along with the grade of certification for which exam it is relevant.

249 Water Environment & Reuse Foundation

Table 6.12. Wastewater Treatment Plant Operator Examination Content – California

Wastewater Treatment Plant Operator Examination Content per Section 3701

Examinations shall test the applicant’s knowledge of: Grade I

Grade II

Grade III

Grade IV

Grade V

Basic safety practices and hazards related to wastewater treatment plant operation X X X X X Wastewater constituents including simple and routine sampling and analysis procedures X X X X X Procedures involved in operating and maintaining preliminary and primary treatment facilities including sludge digestion and disinfection X X X X X

Specifics regarding the operation of stabilization ponds X X X X X State regulations regarding wastewater treatment plant classification, waste discharge requirements, and operator certification X X X X X Commonly used processes for preliminary, primary, and secondary treatment including disinfection, sludge handling, and digestion X X X X Routine sampling and analysis procedures for evaluation of process and overall wastewater treatment plant performance X X X X Basic supervision responsibilities X X X X Limitations, controls, and performance calculations for primary and secondary treatment and sludge-handling processes X X X

Basic principles of tertiary treatment processes X X X State regulations regarding water recycling X X X Public health issues X X X Limitations, controls, and performance calculations for tertiary treatment processes X X Requirements and practices for water reclamation and reuse X X Supervision and management responsibilities including energy management, safety program development and control, operator training, and budget development and control. X X

Knowledge of the components of the Grade IV exam (see above) as applied in more difficult and complex situations X

250 Water Environment & Reuse Foundation

The water treatment plant operator certification provides a greater breakdown of curriculum material that is required for the examination. A summary of content is provided below:

Table 6.13. Drinking Water Treatment Exams Expected Range of Knowledge

Exam Content Number of questions

Grade T1 T2 T3 T4

Source Water 25 25 20 15

Watershed Protection, Wells/Groundwater, Surface Water/Reservoirs, Raw Water Storage, Clear well Storage

Water Treatment Processes 25 25 35 20

Coagulation/Flocculation/Sedimentation, Filtration, Disinfection, Demineralization, Corrosion Control, Iron and Manganese removal, Fluoridation, Water Softening, BAT (Best Available Technology)

Operation and Maintenance 20 20 15 15

Chemical Feeders, Pumps and Motors, Blowers and Compressors, Water Meters, Pressure Gauges, Electrical Generators, Safety, SCADA Systems

Laboratory Procedures 15 15 15 15

Sampling, General Lab Practices, Disinfectant analysis, Alkalinity analysis, pH analysis, turbidity analysis, specific conductance, hardness, fluoride analysis, color analysis, taste and odor analysis, dissolved oxygen analysis, algae count, bacteriological analysis

Regulation/Administrative Duties 15 15 15 35

Planning, organizing, directing, controlling, staffing, implementing regulations, record keeping, safe drinking water act and amendments, surface water treatment rule and amendments, primary contaminants, secondary contaminants, lead and copper rule, fluoride regulations, operator certification regulations

For both water and wastewater operator examination content, a detailed gap analysis was conducted to determine what elements of for IPR/DPR were covered, what was partially covered, and what additional material will be required for DPR. In this assessment, each of the components of the water treatment operations examination expected knowledge is detailed and is consequently a large list. summary gap analysis is included in Table 6.14.

251 Water Environment & Reuse Foundation

Table 6.14. Water and Wastewater Operations Certification Curriculum – Gap Analysis

Topic Area

Existing Curriculum Helpful for DPR Additional Requirements for DPR Drinking Water Operations Wastewater Operations

Source Water/Water Quality (Groundwater/Surface Water/Raw Water Storage)

Provides basic knowledge of water quality assessment and characteristics, including a knowledge of microbial contamination. This will provide a solid base for operators moving to a future DPR certification. Some of the important items include:

Water quality characteristics. Source water assessment Ability to recognize abnormal

conditions. Microbial contamination. Interpretation of water quality

reports Flow and flow measurement. Calculation of chemical dose. Measuring pH

Relatively small amount of information on sewer shed and source control.

Additional information for sewershed management and source control. Understanding of industrial waste contributions and other source contaminants that may risk treatment processes. The source water for DPR treatment processes is municipal wastewater. For drinking water operators, a basic knowledge of wastewater processes. In addition,

Understanding source water risks from wastewater source.

Ability to develop and manage a water quality risk register.

Understanding process changes and impacts from wastewater treatment processes.

Understand important key process monitoring parameters at inlet of advanced treatment plant.

Treatment Processes Coagulation/Flocculation/Sedimentation Provides a thorough coverage of these

processes. This knowledge is likely adequate for DPR operators where this technology is employed (non RO based treatment train)

Some knowledge of clarification processes for primary sedimentation and secondary clarification.

No additional curriculum required for DPR.

Filtration Provides a thorough coverage of conventional media filtration, with a limited coverage of granular activated carbon.

Minimal information. Conventional filtration appears well covered, however granular activated carbon and biologically carbon will require significantly more coverage. Membrane filtration (MF/UF) will require substantial coverage.

Disinfection Provides a thorough knowledge of chlorination practices including analysis of free and total chlorine, calculation of CT, calculation of chemical dose and a knowledge of breakpoint chemistry.

Provides some knowledge of calculating chlorine demand, operation and maintenance procedures for disinfection, calculating disinfection usage.

Will require additional detail for chloramine dosing, which is used for membrane disinfection. Additional content required includes:

Knowledge of chloramine control.

252 Water Environment & Reuse Foundation

Topic Area

Existing Curriculum Helpful for DPR Additional Requirements for DPR Drinking Water Operations Wastewater Operations

Includes some material for ozone, UV disinfection and chloramines.

Protection of RO membranes from chlorine.

Ozone also requires additional information including:

Basic understanding of ozone chemistry.

Basic ozone generation management. Knowledge of UV absorbance analyzer

calibration. Ozone residual analyzer management Ozone dose-control strategies

Demineralization (RO, NF and Ion Exchange Treatment)

This contains some water quality analysis content, knowledge of electrical conductivity and total dissolved solids analysis. There is subject matter relating to ion exchange but none for reverse osmosis.

Not included Reverse osmosis is a core technology for the RO based treatment train. Content for this process is required, including:

Operation and maintenance of membranes

Measurements of process performance and membrane integrity

Monitoring and measurement of chemical rejection and log reduction of pathogens

Corrosion Control Useful basic knowledge of corrosion, including health effects from lead and copper.

Not included Will provide useful basis for additional chemical stabilization process required for the RO based treatment train.

Iron and Manganese Removal Thorough knowledge of iron and manganese removal.

Not included Not specific to DPR treatment processes, however will provide some useful knowledge for RO system membrane scaling and fouling.

Fluoridation General knowledge of fluoridation processes.

Not included Not specific to DPR treatment, unless fluoridation is required in a DPR system that operates directly to distribution.

Softening Knowledge of water hardness chemistry, hardness removal and softening processes.

Not included Not directly related to DPR, but some value to RO treatment processes. May also be important for non-RO membrane-based treatment

Wastewater Treatment Technologies Not included Material that is focused on the main wastewater treatment processes including:

Preliminary treatment (screening, grit removal)

Primary treatment processes. Anaerobic sludge digestion Stabilization ponds

Valuable knowledge for DPR treatment to understand impacts upstream of advanced treatment. It is also important to understand the difference between monthly compliance goals/environmental impacts versus continuous water quality goals for DPR and drinking water quality goals.

253 Water Environment & Reuse Foundation

Topic Area

Existing Curriculum Helpful for DPR Additional Requirements for DPR Drinking Water Operations Wastewater Operations

Secondary processes including trickling filters and activated sludge

Sludge handling and solids thickening

Tertiary treatment Overall process control.

Best Available Technology Knowledge of waterborne pathogens, best available technologies for removal, adverse health effects from regulated contaminants and knowledge of emerging contaminants.

Not included Specific knowledge of technologies / BATs used in drinking water treatment (and DPR) is required.

Operations & Maintenance (O&M) Requires knowledge of key process plant mechanical components including:

Chemical feeder. Pumps and motors Blowers and compressors Water Meters Instruments and analyzers SCADA components and on

line analyzers Calibration of some key

instruments.

Thorough review of some maintenance requirements including:

Electrical equipment Motors Pumps Valves

Specific knowledge of treatment process maintenance requirements including items such as membrane management, UV lamps, ozone generation and lime/CO2 systems for stabilization. The understanding of detailed instrument verification and calibration for multiple analyzers is an important addition for DPR.

Laboratory Thorough requirements for laboratory analysis, chains of custody, sampling and analysis requirements as well as detail on a number of specific, common water quality analyses.

Thorough review of sampling and analysis requirements for wastewater treatment applications.

Some additional knowledge for management of numerous water quality analysis parameters, including knowledge of sampling and sample management for complex contaminants.

Safety General knowledge of safety, safe working practices, lock out-tag out procedures and some first aid.

General knowledge of safety, including importance of hygiene, lock out tag out, safe work practices and specific safety requirements for wastewater technologies.

Suitable for DPR, with a focus on safety requirements included for technologies specific to DPR.

Administration Covers a broad range of administrative requirements including organization, monitoring and reporting requirements, reviewing and transcribing data, review of overall plant performance, review of reports, evaluating facility performance.

Covers a broad range of administrative requirements including staffing, financial management, capital planning, and data management.

Additional requirements include: Critical control point methodology Critical control point response

procedures and communication protocols

Critical control point incident investigation and follow up action methodology

254 Water Environment & Reuse Foundation

Topic Area

Existing Curriculum Helpful for DPR Additional Requirements for DPR Drinking Water Operations Wastewater Operations

Regulations Knowledge of key regulatory requirements including disinfection requirements, knowledge of MCLs, consumer confidence reports, Surface Water Treatment Rule, development of operations plans, disinfection requirements and other regulatory aspects.

Some regulatory content including classification of wastewater treatment plants and operator certification regulations, and requirements for reclamation and reuse (although not focused on IPR/DPR).

Additional knowledge of future DPR regulatory requirements must be included. In addition, any specific reporting and communication protocols for regulators must be included. An important aspect will be the comparison and contrast with water and wastewater regulations. A specific example will be how water quality treatment requirements will likely be a single maximum target, rather than monthly averages or means as is common in wastewater treatment.

Math Specific calculations for major water treatment operations including:

Flow rate calculation Volume calculation Chemical dosing rates. Detention times Backwash rates Production rates CT calculations

Specific calculations for wastewater treatment operations including:

Removal efficiencies Overflow rates Hydraulic loading Solids loading Chemical dosing Evaluation of specific

processes.

Additional calculations will be required for specific unit processes not covered in the existing curricula but required for DPR.

Communication Effective communication is critical for the success of DPR. Operators must understand the importance of timely communication within their operating facility to assist in rapid and effective operational responses to issues. They must also understand the importance of clear communication across operational interfaces, and to external stakeholders including regulators and the public.

Management of analyzers and instruments. There is a high reliance on analyzers and instruments for successful IPR and DPR plant operation. Specific curriculum material that covers the importance of regular instrument verification, calibration and key maintenance requirements for important instruments is required.

SCADA, reporting and alarm management

Covering important SCADA management and reporting with a focus in particular on alarm management and operator response.

Operational Interfaces General knowledge of water treatment General knowledge of wastewater Knowledge of requirements at operator

255 Water Environment & Reuse Foundation

Topic Area

Existing Curriculum Helpful for DPR Additional Requirements for DPR Drinking Water Operations Wastewater Operations

processes. treatment processes. interfaces between wastewater treatment and advanced treatment, and advanced treatment and drinking water treatment is required. For some utilities, the full suite of treatment may be operated by a single entity. For others, there will be different organizations operating these entities. An understanding of process and treatment at these interfaces is required.

Critical Control Point and the HACCP Process An understanding of the critical control point approach, including specific critical control points for DPR processes, water quality risk management and operational responses.

256 Water Environment & Reuse Foundation

The existing curriculum content for both wastewater and water treatment operator certification is relevant to DPR system operation and will provide a strong basis for many of the aspects of both RO-based and non-RO-based treatment trains. There are, however, gaps as noted in the table above, there are specific requirements for process technology that will require additional coverage including the general content described in Table 6.14 and the specific material listed in Table 6.15.

Table 6.15. Availability of Training Content for DPR Processes

Process Technology Training Content Availability

Chlorine Disinfection Good Content Available

Coagulation/Settling Good Content Available

Chloramines (Pre-MF or Pre-RO Biofouling Control) Some Content Available

Microfiltration Some Content Available

Reverse Osmosis Membranes Some Content Available

Chemical Stabilization (Corrosion Control) Some Content Available

Ozone Some Content Available

Granular Activated Carbon Some Content Available

UV Disinfection Some Content Available

Biological Activated Carbon/ Biofiltration Minimum Content Available

UV Advanced Oxidation Processes Minimum Content Available

257 Water Environment & Reuse Foundation

Table 6.16. Required Additional Material for DPR Processes.

RO-Based Treatment Train Non-RO-Based Treatment Train

Chloramination Basic understanding of chloramine chemistry. Ability to measure chloramine in water Ability to measure ammonia in water Importance of chemical dosing location Protection of RO systems from excess

chlorine or chloramine

Ozone Basic understanding of ozone chemistry Basic ozone generation management Knowledge of UV absorbance analyzer

calibration Ozone residual analyzer management Calculation of ozone CT Ozone system monitoring Ozone system safety

Membrane Filtration (Microfiltration and Ultrafiltration)

Membrane terminology Membrane system configurations MF system monitoring including TMP and

permeability MF system flux maintenance and cleaning MF membrane integrity monitoring MF membrane troubleshooting and repair Management/Monitoring of turbidity

analyzers

Ozone-BAC Filtration theory and importance of upstream

pre-treatment (i.e., coagulation) Basics of biological activated carbon Understanding of maturation time for organic

absorption Management/monitoring of turbidity

analyzers Know how to calculate empty bed contact

time BAC system monitoring BAC system safety

UV/Advanced Oxidation UV reactor basic operation Management/monitoring of UV transmittance

analyzer UV lamp care, maintenance and replacement UV lamp ballast care, maintenance and

replacement Handling and management of hydrogen

peroxide Basics of advanced oxidation Sampling and analysis of NDMA, 1,4 dioxane

GAC Basics of granular activated carbon Understanding different carbon types and uses Know how to calculate empty bed contact

time How to determine contaminant breakthrough Managing filter media replacement. GAC system monitoring GAC system safety

Stabilization Understand how to measure hardness Understanding management of pH analyzer Calculation of stability indices (LSI, CCPP,

Ryznar Index) Understanding carbon dioxide dosing systems Understanding lime dosing systems Understanding calcite filters Managing hardness and alkalinity Understanding the risks from lead and copper

in distribution systems. Lime and CO2 system safety

UV Disinfection UV reactor basic operation Management/monitoring of UV transmittance

analyzer UV lamp care, maintenance and replacement UV lamp ballast care, maintenance and

replacement Calculating UV Dose UV system monitoring UV system safety

258 Water Environment & Reuse Foundation

6.5.3 Existing Sources for Additional Curriculum

Membrane Treatment Processes (Microfiltration and Reverse Osmosis) There are commercially available courses for reverse osmosis membrane treatment systems that provide a level of certification. An example is David H Paul www.dhptraining.com which provides four levels of reverse osmosis treatment certification for treatment plant operators. These courses provide a combination of classroom and hands-on training covering most aspects of reverse osmosis operation and trouble-shooting. These certificates are not recognized by any regulatory agency as directly operator certification, however is a recommended provider of training for some state agencies including TCEQ (Texas Commission for Environmental Quality) where the course can provide continuing education units CEUs.

A Membrane Operator Certification (MOC) has also been developed by the South East Desalting Association based in Florida. This provides training in reverse osmosis and membrane filtration technologies (microfiltration and ultrafiltration). As of 2015, other regional membrane associations including the South Central Membrane Association and the South West Membrane Operator’s Association have developed MOCs to cover both membrane filtration and membrane bioreactor technologies. These courses are often conducted by the respective organization at an operating plant or at technical colleges.

These membrane associations, along with the national American Membrane Technology Association (AMTA) also provide a number of operator focused training workshops and conferences which provide training opportunities for operators focused on membrane technologies.

The American Water Works Association (AWWA) has also developed several manuals of practice for membrane systems including:

Manual of Practice M46 – Reverse Osmosis and Nanofiltration. Manual of Practice M53 – Microfiltration and Ultrafiltration Membranes for Drinking Water.

UV There is some content covering UV as a disinfection system in the Californian Water Operator certification curriculum, with a portion of the Disinfection Chapter of “Water Treatment Plant Operation” by Ken Kerri et al.

The AWWA has also developed “The Ultraviolet Disinfection Handbook” as a technical guide for disinfection of drinking water.

Otherwise, UV training is often provided by UV system equipment vendors and engineers on new plant installations, and can be offered by many providers as a stand-alone training service for existing installations.

Advanced Oxidation Advanced oxidation is a less common water treatment technology that has applications in advanced water recycling, taste and odor control for surface water treatment systems, and as for treatment of groundwater for some organic contaminants. Training is currently provided almost exclusively by equipment vendors, with some additional material provided by engineering consultants. This particular process area will require a significant curriculum development.

259 Water Environment & Reuse Foundation

Ozone As with UV, ozone dosing for disinfection is also briefly covered in the Disinfection chapter of “Water Treatment Plant Operation” Kerri et al, 2007. In addition, a recently published book Ozone in Drinking Water Treatment: Process Design, Operation, and Optimization, Raknes, 2015 is available.

Ozone/BAC Ozone/BAC processes, and biofiltration in general, are not currently covered in existing curriculum material. While there are numerous journal articles and conference papers that cover this material, it has not yet been established as training content for certification programs and we could not find in our review any operationally-focused reference material. There is some material covering the use of ozone/BAC covered in Activated Carbon, Solutions for Improving Water Quality, Chowdhury et al., 2013, AWWA.

6.6 Proposed Operator Certification Framework for DPR A successful operator certification framework for DPR must be one that is workable for operations staff and utilities. It must provide value in developing ensuring the capacity of operational staff and underpin safety and reliability for this supply of drinking water.

As described in the previous sections, a DPR operator certification must provide: Adequate coverage of DPR technologies. Understanding of source water risks and risk management. Incorporation of the critical control point methodology. Specific DPR regulatory requirements. Management of operational responses.

However, not only must the certification framework account for this important curriculum content, it must also be a framework that operators and utilities alike find workable and that does not provide substantial impediments for finding and developing operator staff for facilities. On the contrary, it must incentivize operators and utilities to obtain and be recognized for the skills and knowledge required to manage DPR.

At the outset, it must be acknowledged that future DPR facilities will be operated by staff that are drawn from the existing pool of certified water and wastewater operators. Consideration must therefore be given to how to integrate DPR certification into the existing system, and how to leverage from the existing certification programs. It must also be acknowledged that DPR facilities will likely remain a small fraction of the overall water and wastewater treatment facilities in any given state, and therefore a difficult or cumbersome operational framework may be a major disincentive for operational staff and provide challenges for utilities to staff their new facilities. While this project does not intend to water down the requirements considered necessary for a DPR certification, developing a program that is accommodating given these realities is nonetheless important to its success.

In order to determine a preferred program, a number of options are discussed below.

6.6.1 Option 1 – Specific DPR Certification Curriculum This option considers the development of a stand-alone operator certification for DPR that draws from a base of water and wastewater certified operators (Figure 6.4). In this case, for lower operator grades (1 and 2), an operator would progress through either water or wastewater certification, with an additional DPR examination module to cover some elements of DPR technology and other important introductory curriculum.

For these two lower grades, the operators would remain certified as water or wastewater operators, but would have an additional DPR accreditation. This additional accreditation should cover core elements of

260 Water Environment & Reuse Foundation

the advanced technologies utilized at their facility, along with other key elements of DPR (detailed in Table 6.17).

Figure 6.4. DPR Operator Certification Framework Option 1

In terms of operating experience, consideration could be given to allow water and wastewater plant operation for those grades to be counted toward total experience. This would allow operators from non-DPR facilities to begin preparation to work at a facility that is being built for DPR, or allow them to transfer from a water or wastewater facility to a DPR facility. This provision would likely increase the available pool of operators for new facilities which will be important during the time where the number of DPR facilities will be limited relative to the operations pool of resources.

From a grade 3 or T3 onwards, in this model the operator certification would transition to a specific DPR certification. From this point the examination would focus on DPR technologies and experience and will be required for operating a DPR facility.

The examination would contain a common core of important DPR elements, along with elective technology modules, in order for staff to tailor the technology component to the plant at which they are working, or are considering working. Table 6.17 outlines proposed content.

Grade 5

Grade 4

Grade 3

Grade 2

Grade 1

Wastewater Operator DPR

T 5

T 4

T 3

T 2

T 1

Water Operator DPR

DP

R E

xper

ienc

e

DP

R E

xper

ienc

e

DPR Exam

DP

R T

rain

ing

DP

R T

rain

ing

DPR

DPR

DPR

DPR

Operator DPR

DPR Exam

DPR Exam

DPR Exam

DPR Exam

261 Water Environment & Reuse Foundation

Table 6.17. General Curriculum for DPR Certification Grade Levels.

Operator Grade DPR Curriculum Summary General content/curriculum material. 1 Technology modules (MF, RO, UV-H202, GAC, Ozone-

BAC, Stabilization), Source water risks. Critical Control Point Process and response procedures. Safety for DPR. DPR regulations. Sampling and analysis for DPR. Analyzer management.

Focused on providing additional content for DPR, leveraging from water and wastewater curriculum.

2 Technology modules (MF, RO, UV-H202, GAC, Ozone-BAC, Stabilization) – (more advanced) Source water risks. Critical Control Point Process and response procedures. Safety for DPR. DPR regulations. Sampling and analysis for DPR. Analyzer management.

3 Technology modules (MF, RO, UV-H202, GAC, Ozone-BAC, Stabilization) – (more advanced) Source water risk management and responses. Critical Control Point Process and response procedures. Safety for DPR. Sampling and analysis for DPR. DPR regulations. Analyzer management. Laboratory management for DPR. Wastewater and drinking water treatment interface management. Importance of effective communication with stakeholders.

Content focused specifically on DPR requirements, with greater emphasis on managing specific DPR requirements.

4 Technology modules (MF, RO, UV-H202, GAC, Ozone-BAC, Stabilization) – (more advanced). Managing water quality risk assessment. Critical Control Point Process management, response procedures. Safety for DPR. Sampling and analysis management for DPR. Analyzer management. DPR regulations. Communication with stakeholders, management and regulators.

Greater focus on operational management aspects and communication requirements.

5 DPR system management. DPR regulations. Critical control point system management. Water quality and operational risk management. Communications with stakeholders, regulators and the public.

Greater operational management focus.

262 Water Environment & Reuse Foundation

This approach provides the advantage of a clear pathway for DPR, drawing from both the water and wastewater pool of operators. However there are some significant disadvantages including: A specific operator certification for DPR may restrict operators to a relatively small pool of operating

facilities, unless they become transferable for water or wastewater certification.

Operators that are already grade 3 or grade 4 may be required to drop to lower levels to cover curriculum, experience and certification requirements specific for the DPR curriculum. Consideration would need to be given in particular to grandfathering certified operators that are currently operating IPR systems if those systems become upgraded to DPR. Alternatively consideration of a phase-in period to allow transfer of highly qualified operators from water and wastewater plants to take on senior operating roles of new DPR facilities.

6.6.2 Option 2. DPR “Add On” to Existing Certification Frameworks

Figure 6.5. DPR Operator Certification Framework 2

This option considers the use of the existing operator and certification framework for both water and wastewater, with an additional module for DPR included at that level of certification. For this approach, an additional examination would be required to cover elements that are particular to DPR. An operator would progress as a water or wastewater operator, but would also be required to obtain an additional certification for DPR.

In this case, there would be a major benefit in taking advantage of the existing operator certification curriculum, and adding on this basis, as this would increase the pool of available operators for plant operation substantially. As an add-on, it would provide greater incentive for operators as they would be working to add material specific to their roles, but would also continue with an existing certification path and not become stranded in DPR if they wanted to move to other non IPR/DPR facilities.

As for option 1, there would be a set of core material for DPR, however tailored to build on existing curriculum material and act to bridge gaps. Elective modules would also be included to allow operators to focus on specific technology.

Experience could be gained either from a water or wastewater plant, however consideration would need to be given as to whether specific DPR experience is required beyond a certain grade of certification (as for

Grade 5

Grade 4

Grade 3

Grade 2

Grade 1

Wastewater Operator

T 5

T 4

T 3

T 2

T 1

Water Operator

DP

R E

xper

ienc

e

DP

R E

xper

ienc

e

DP

R T

rain

ing

DP

R T

rain

ing

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR Overall

Technology Module

Technology Module

Technology Module

Technology Module

Technology Module

263 Water Environment & Reuse Foundation

Option 1). While DPR-specific experience would be desirable, it must be noted that the relatively small number of DPR facilities may limit the ability for new and transfer operators to achieve certification. A possible work-around would be the consideration of identifying specific experience requirements for various process components (for example focus on some technologies in application outside of a DPR environment, such as reverse osmosis treating groundwater for drinking for example).

The California/Nevada AWWA has currently commissioned an Advanced Water Treatment Certification Committee, which is focused on developing operator certification for advanced technologies – including those used for DPR, but also used in IPR, groundwater treatment and other applications. This development may assist in the development of technology modules that could be appended to the existing certification framework.

6.6.3 Option 3 – DPR “Add On” to Existing Certification Frameworks Covering Water/Wastewater Certification Gaps

Figure 6.6. DPR Operator Certification Framework 3

This option is similar to Option 2, but also acknowledges that there is material in the curriculum from either water or wastewater certification curriculum that is important for operations. That is, it is important for water derived DPR operators to have knowledge of wastewater operations and vice versa.

This framework would operate in a manner analogous to Option 2, however the DPR core examination material would be different for water-derived and wastewater-derived operators to assist in covering gaps in each other’s knowledge.

6.6.4 Consideration of Competency Based Curriculum As noted for each of the three options above, an examination and experience component would be required for each progressive level of operator certification. One of the challenges faced in gaining certification for operators from the existing pool of water and wastewater will be in obtaining specific experience in DPR if they are not working at a DPR facility. In addition, even if operators are working at a DPR facility, it will be critical to have a methodology for quantifying and validating specific experience.

Grade 5

Grade 4

Grade 3

Grade 2

Grade 1

Wastewater Operator

T 5

T 4

T 3

T 2

T 1

Water Operator

DP

R E

xper

ienc

e

DP

R E

xper

ienc

e

DP

R T

rain

ing

DP

R T

rain

ing

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR

DPR for Water

Technology Module

Technology Module

Technology Module

Technology Module

Technology Module

Technology elective modules

DPR for Wastewater

264 Water Environment & Reuse Foundation

A competency-based assessment program may assist in providing greater structure and provide important validation for the experience component of operator certification. By placing emphasis on demonstrating specific skills and tasks in the field, we can have greater certainty that the operator has the necessary skills to operate the facility. There will be surety that the operator can perform important tasks – not just pass an examination. This competency-based approach could also allow for many elements of the experience to be conducted at non-DPR facilities, if the requirements are similar enough to DPR. Further, by the use of competency-based assessments, there may be an ability to reduce the amount of material that is required for an examination, and instead replace this with competency based modules.

A key concern raised for competency-based approaches is ascertaining the veracity of site testing, and the scrutiny of the examiner or reviewer. It is true, that there is a higher risk of subjectivity relative to a direct examination. However, if well-developed, a testing of in-field skills can provide a greater mitigation of operating risk than passing an exam. For example, demonstrating the completion of a response procedure for a critical control point, rather than simply answering an exam question about it, will provide great certainty that the operator understand the material and can apply it accordingly.

It is highly recommended that facilities consider this approach for operations staff, even if this remains an internal utility arrangement independent of state certification.

6.6.5 Educational Requirements and Continuing Education Credits Overall, it is recommended that educational requirements and educational units be consistent with those currently required for water and wastewater operator certification. It is recommended that continuing education units be pursued at relevant venues for both DPR technologies, and for reuse itself. These can be achieved from external training opportunities (for example the membrane association MOC schools, regional reuse conferences and training seminars) or from in-house training sessions that receive certification for education credits. As DPR systems become more prevalent, consideration of training programs developed by regional AWWA, WEF, and WateReuse organizations should be considered.

6.6.6 Overall Recommendation It is recommended that a DPR operator certification be a system that is appended to the existing water and wastewater certification, consistent with Option 3 noted above.

This will ensure that there is a substantial pool of operators that can be drawn from for new DPR facilities, and that any operator that chooses to take on DPR will still have opportunities for transfer and promotion to other, non-DPR facilities in the future. This system will also leverage from existing operator certification programs for water and wastewater and thus attention can be focused on developing the additional curriculum that is required, rather than starting a system afresh. Recommendations on the required additional curriculum have been included in the sections above.

It is also recommended that consideration be given to a competency-based approach to the experience portion of certification required. A competency based approach will provide some certainty that operations staff not only have the knowledge, but are truly competent at specific, required tasks.

265 Water Environment & Reuse Foundation

References: (1) California Code of Regulations Title 22, Division 4, Chapter 13 sections 63775 and 63800. (2) Office of Water Supply Programs Sacramento State University http://www.owp.csus.edu/courses/drinking-water.php (3) Title 22 CCR, Division 4, Chapter 15 - Domestic Water Quality and Monitoring, Article 2 (4) WWTP OPERATOR CERTIFICATION REQUIREMENTS TABLE Title per 23 CCR, Division 3, Chapter 26 section 3688. (5) (http://www.swrcb.ca.gov/water_issues/programs/operator_certification/wwtp.shtml) (6) Workshop discussion, M Patel, T Neely, Orange County Water District GWRS. (7) Los Angeles Bureau of Sanitation Workshop, Terminal Island Facility. (8) Santa Clara Valley Water District, Pam Johns. (9) http://texreg.sos.state.tx.us/public/readtac%24ext.TacPage?sl=R&app=9&p_dir=&p_rloc=&p_tloc=&p_ploc=&pg=1&p_tac=&ti=30&pt=1&ch=290&rl=46 (10) www.tceq.texas.gov (11) Revision to the Previously Granted Exception to Use Membrane-Treated Reclaimed Wastewater from the Big Spring Wastewater Treatment Plant as a Raw Water Source for Public Drinking Water Systems Colorado River Municipal Water District – PWS ID No. 1140038, Howard County Texas (Letter – April 11, 2013). (12) U.S. Department of the Interior, Bureau of Reclamation, Colorado River Basin Stakeholders - Moving Forward to Address Challenges Identified in the Colorado River Basin Water Supply and Demand Study: Phase 1 Report, p. 3-20, 2015. (13) State of Arizona, Blue Ribbon Panel on Water Sustainability, Final Report, November 30, 2010, p. 28. (14) Presentation by Tim Thomure, Arizona Steering Committee to Advance Potable reuse, “Potable Reuse in Arizona An Update on the Statewide Initiative” presented at AZ Water Association monthly meeting, April 14, 2015. (15) http://dnr.wi.gov/regulations/opcert/wastewater.html. (16) Victorian framework for water treatment operator competencies Best practice guidelines. Victorian Department of Health and Victorian Water Industry Association. (17) Australian Government Industry Skills Panel NWP07 Water Training Package Water Industry Training Center NWP -07 Training Handbook. (18) Section 3701 Title 23. Waters Division 3. State Water Resources Control Board and Regional Water Quality Control Boards Chapter 26. Classification of Wastewater Treatment Plants and Operator Certification

Water Environment & Reuse Foundation 1199 North Fairfax Street Alexandria, VA 22314-1177

Phone: 571-384-2100 Fax: 703-299-0742 Email: werf@werf.org www.werf.org

WE&RF Stock No. Reuse-13-13 WE&RF ISBN: 978-1-94124-255-1

December 2016