automated photo enforcement audit - edmonton

Automated Photo Enforcement

Audit

August 19, 2013

EDMONTON 13358 – Automated Photo Enforcement

Office of the City Auditor

The Office of the City Auditor conducted this project in accordance with the

International Standards for the Professional Practice of Internal Auditing.


Office of the City Auditor Page i

Automated Photo Enforcement Audit

Table of Contents

1. Introduction .............................................................................................................. 1 2. Background .............................................................................................................. 1

2.1. Automated Photo Enforcement Program ........................................................... 1

2.2. Traffic Safety Program Resources ..................................................................... 2

2.3. Program Authority, Oversight, and Administration ............................................. 3

3. Objective, Scope, & Methodology ............................................................................ 5

3.1. Objective ............................................................................................................ 5

3.2. Scope ................................................................................................................. 5

3.3. Methodology....................................................................................................... 5

4. Observations and Analysis ....................................................................................... 5

4.1. Speed-on-green Review Processes ................................................................... 6

4.1.1. Violation Photo Review Processes .............................................................. 8

4.2. Testing and Maintaining Equipment ................................................................. 15

5. Conclusions ........................................................................................................... 18


Office of the City Auditor Page ii

This page is intentionally blank.


Office of the City Auditor Page 1

Automated Photo Enforcement Audit

1. Introduction

At the request of the Transportation Services Department, the Office of the City Auditor (OCA) included an audit of the City’s Automated Photo Enforcement program in its 2013 Annual Work Plan. The Automated Photo Enforcement program includes obtaining violation data, reviewing violations and issuing tickets using two types of technology. The Intersection Safety Devices (ISDs) capture photos of vehicles that speed through 50 permanently monitored intersections (speed-on-green). They also capture photos of red light violations at those intersections. The Automated Photo Enforcement program also includes mobile speed enforcement vans and trucks (commonly referred to as photo radar). In November 2009, the City started issuing speed-on-green tickets. In June 2010, the City issued an invalid speeding ticket. It was issued because an ISD malfunctioned and the violation photo reviewers did not detect the error. In January 2011, the Chief Crown Prosecutor requested that the Court of Queen’s Bench quash all speed-on-green tickets that had been issued by the ISDs. The request was based on concerns about the ISD’s ability to accurately capture speed-on-green violations. The motion was granted. The Court’s decision covered tickets issued between the start of the program in November 2009 and January 2011. The Province of Alberta and the City of Edmonton refunded the applicable speed-on-green ticketed violations. The City refunded $12.3 million for over 100,000 tickets. During 2011 and 2012, the City made several changes to its processes to avoid invalid tickets. We focused our audit on speed-on-green violations to determine whether the process changes that the City has made since January 2011 have been effective.

2. Background

2.1. Automated Photo Enforcement Program The Transportation Services Department’s Office of Traffic Safety (OTS) endeavours to make Edmonton’s roads and communities safer by supporting effective management and enforcement of local traffic programs. OTS is one of the functional areas in the Transportation Operations Branch. OTS conducts research, develops programs designed to encourage drivers to minimize risky driving behaviours, and delivers safety programs to reduce speeding, deter risky driver behaviour, and reduce collisions. OTS deploys and maintains automated photo



enforcement equipment, defines criteria for accepting or rejecting violation photos, issues tickets, and trains assigned peace officers. OTS uses its Automated Photo Enforcement program to help achieve its objectives. This program requires specialized equipment including ISDs and mobile photo radar units to capture photos of vehicles violating speed limits or running red lights. OTS uses specialized violation review software to review the photos and process violations of the Traffic Act of Alberta into tickets. The City of Edmonton has used automated photo enforcement equipment to support traffic enforcement since 1993. There are currently over 700 approved photo enforcement locations where photo radar (mobile enforcement vehicles) can be used and another 50 sites where ISDs are permanently installed. Table 1 outlines the recent events associated with the City’s Automated Photo Enforcement program.

Table 1 – Recent Automated Photo Enforcement Events in Edmonton Year Event Description

2007 The City decided to assume responsibility for the Automated Photo Enforcement program from the contracted service provider.

2008 The City extended its existing contract for processing violation photos until it could take over both equipment operations and violation processing.

2011 In January, the Court of Queen’s Bench directed the City to cease issuing speed-on-green tickets and to refund fines paid on speed-on-green tickets issued between November 2009 and January 2011.

2012 In April, the Edmonton Police Service and OTS formalized and clarified each party’s roles and responsibilities for the City’s Automated Photo Enforcement program. In August, OTS assumed responsibility for all aspects of violation processing for mobile photo radar

1 and ISDs.

2012 to present

OTS developed new processes to ensure the proper functioning of ISDs and the accuracy of resulting speed-on-green and red light violation notices.

2.2. Traffic Safety Program Resources Table 2 below shows the operating and capital budgets for OTS’ photo enforcement programs. In addition to the traffic safety programs carried out by OTS and other areas of the Transportation Services Department, the Edmonton Police Service also conducts traffic safety programs. The City receives funds from the Automated Photo Enforcement program through transfer from the Province, which collects fines generated by OTS’ photo enforcement programs. The funds are received as general revenues and used to fund City operations. The City’s practice is to use those funds to offset the cost of operating its own traffic safety programs and those of Edmonton Police Service. Revenue from the program is directed to equipment needs, education programs, and engineering support. The City provides a portion of this revenue to the Edmonton Police Service each year to help support its traffic safety programs. Table 2 summarizes the revenues and financial supports for traffic safety programs.

1 In May 2013, the City expanded its mobile photo violation processing to include photo laser violation

photos.



Table 2 –Traffic Safety Programs Financial Information ($ in thousands)

Description 2011

Actual 2012

Actual 2013

Budget

Automated Photo Enforcement Fines Revenue (note 1) 15,937 20,684 30,100

Automated photo enforcement expenditures (ISDs and photo radar combined)

7,774 8,192 6,511

Other traffic safety program expenditures [approximately 27 positions outside the automated enforcement program] (note 2)

6,651 2,848 4,108

Capital equipment expenditures - - 2,377

Repayment of prior equipment acquisitions - - 1,404

Total City Administration - Traffic Safety Program Expenses 14,425 11,040 14,400

Edmonton Police Service - Traffic Safety Program Expenses 1,512 9,644 15,700

Total Traffic Safety Program Expenditures 15,937 20,684 30,100

Net Revenues - - -

Notes: 1 – The low revenues in 2011 resulted from the Court’s direction in January 2011 to refund tickets issued

for speed-on-green violations. 2 – The decrease in “Other traffic safety program expenditures” in 2013 was the result of Transportation

Services Department’s removal of street sanding from the program in 2012.

Table 3 shows the positions that are supported by fines revenues produced by OTS’ traffic enforcement programs. Table 3 – Positions Supported by the Automated Photo Enforcement Program Revenues

Location of Positions 2011

Actual 2012

Actual 2013

Budget

Automated Photo Enforcement Positions

Photo Enforcement Operations (Office of Traffic Safety positions)1 10 13 13

Violation Reviews (Treasury Management Section positions)

2 4

4 9

Information Systems Support 1 1 1

Digital Print Centre (Materials Management Branch position) 0 1 1

Total Automated Photo Enforcement Positions 15 19 24

Other City Traffic Safety Program Positions (other Transportation Services Department programs and information systems support, not including Edmonton Police Service programs)

27 27 27

Total Traffic Safety Program Positions 42 46 51

Notes: 1 – Office of Traffic Safety added three positions in 2012 to replace positions that were previously staffed

by Edmonton Police Service. 2 – The Treasury Management Section of Financial Services & Utilities Department has had nine

positions budgeted in 2011 – 2013, but only four of those positions were filled in 2011 and 2012. Eight positions are filled as of mid-2013. The Section conducts two of the five violation photo reviews.

2.3. Program Authority, Oversight, and Administration Three organizations are involved in the Automated Photo Enforcement program:

The Province: The Province governs automated photo enforcement programs in Alberta. The Province’s Automated Traffic Enforcement Technology Guidelines



and Automated Traffic Enforcement Training Guidelines provide the standards for using, maintaining, and monitoring the equipment and for training equipment maintainers and operators. The Province collects fine payments from registered vehicle owners and transfers 73 percent of those revenues to the City. The Ministry of Justice provides registered vehicle owners the right to dispute tickets and fine amounts.

The Edmonton Police Service: The Province assigns responsibility for providing oversight to the City’s automated photo enforcement operations to the Edmonton Police Service, including approving automated photo enforcement equipment locations.

The City of Edmonton, Office of Traffic Safety: The Edmonton Police Service has delegated responsibility for administering the City’s Automated Photo Enforcement program to OTS. That delegation gives OTS the legal right to access vehicle owner data2 to be used in the Automated Photo Enforcement program. OTS is legally required to use, manage, and protect vehicle owner data in accordance with provincial privacy legislation. Similarly, automated photo enforcement violations are offences under Alberta’s Traffic Safety Act, so Crown Prosecutors prosecute the violations if a vehicle owner appeals a ticket. The Peace Officer Act and the Government of Alberta Automated Enforcement Guidelines provide the authority for OTS’ peace officers to test ISD equipment, approve or reject ISD violations, and give evidence related to these activities.

OTS is responsible for conducting the Automated Photo Enforcement program. It does so by dividing accountabilities among three of its functions:

1. Administer Program: This function: a. Manages contracts with automated photo enforcement equipment vendors, b. Obtains Edmonton Police Service approval for new sites, c. Designs and constructs new sites for automated photo enforcement

equipment, d. Manages the equipment inventory, and e. Administers operational agreements with the Province and Edmonton Police

Service and maintains Peace Officer contracts. 2. Process Violations: This function reviews violation photos captured by the

automated photo enforcement equipment, processes violations into tickets, and distributes tickets. This function operates under the oversight of OTS, but two of the five levels of review are conducted by staff in the Treasury Management Section of Financial Services & Utilities Department.

3. Test and Maintain Equipment: This function supports the overall integrity and accuracy of automated photo enforcement equipment, including equipment testing and maintenance activities.

2 Vehicle owner data contains “personal information” as defined in Alberta’s Freedom of Information and

Protection of Privacy Act, section 1(n).



3. Objective, Scope, & Methodology

3.1. Objective To evaluate the adequacy and effectiveness of OTS’ control processes related to ISDs.

3.2. Scope We focused our audit on determining whether the changes made by OTS to avoid recurrence of the issues identified from 2010 have been adequate and effective. We did this by evaluating the processes OTS uses to:

1. Review the validity of violation photos produced by the ISDs. 2. Test, monitor, and maintain the ISDs.

Since January 2011, the control processes around violation photo processing have changed. During that time, the City has made the following changes:

Moved the violation photo processing in-house,

Increased the frequency of its testing and maintenance processes,

Added two new levels of violation photo review,

Added speed reference lines painted on the roadway to assist reviewers with confirming the actual speed of vehicles passing through the intersections, and

Added electronic stop line and speed reference line overlays superimposed over violation images to help confirm the actual vehicle speed during nighttime and/or adverse weather conditions where the painted lines are not visible.

We focused our testing on OTS’ testing and maintenance processes and reviewing violation photos uploaded to the violation processing system between September 7, 2012 (about 1 month after the City began its in-house processing) and March 31, 2013.

3.3. Methodology As we planned our audit, we worked closely with OTS. They identified the root causes that led to the June 2010 invalid ticket and provided documents that described the changes they made in 2011 and 2012 to address those root causes. We focused our work on evaluating the adequacy and effectiveness of the process changes that OTS has implemented since January 2011. We also identified process changes that will further strengthen the City’s Automated Photo Enforcement program.

4. Observations and Analysis

The root cause of the June 2010 event and the subsequent refund of $12.3 million in 2011 was a combination of an equipment error and human errors during the violation photo review process.



4.1. Speed-on-green Review Processes

We assessed the City’s violation photo review processes to determine whether they support the ongoing accuracy, timeliness, and completeness of information obtained from ISDs. We found that the City has made good progress in improving consistency of the violation photo review processes.

In-house violation processing

In 2007, the City decided to conduct all automated photo enforcement ticket processing in-house. Starting in 2008, the City purchased the photo enforcement hardware and developed its own violation photo review software. In-house ticket processing started in August 2012. From 1993 to 2012, there were three levels of violation photo review. Before restarting the speed-on-green violation process in August 2012, OTS added two new review levels. The additional levels of review were added to avoid having the June 2010 event happen again. Each review level has one or more reviewers and a supervisor. Reviewers can choose to accept or reject each violation photo based on criteria defined by OTS. All rejected violation photos are automatically diverted for review by the assigned supervisor, while accepted violation photos flow into the next reviewer’s queue. The software is designed to prevent a reviewer from conducting a later review of the same violation photo. Figure 1 (on the next page) provides an overview of the violation photo review process.

Process flow and reliability

The first violation photo review focuses primarily on the data associated with the images. It is designed to ensure that only valid violations are passed on to the next level of review. The second and third reviews independently confirm the accuracy of the violation data, image quality and licence plate data. In the fourth review, Peace Officers review image quality, violation data, and license plate data and must satisfy themselves that there are reasonable and probable grounds to believe that the violation occurred and all related data are correct. The fifth review focuses on ensuring that the electronic version of the ticket is valid and ready for printing. Most of the violation photo review criteria are applied at multiple review levels. We believe that the first four reviews are critical to the reliability and accuracy of the speed-on-green program. The fifth review was originally established because some violation notices and tickets were not printing completely. Since that time, additional responsibilities have been assigned to that review level, including managing printing, monitoring court date assignments and processes, and conducting specified violation and systems validity reviews. We believe that there are opportunities to automate at least some of the tasks that are conducted in the first review (the primarily technical review level). One of the key



elements of that review is to ensure that violation photos are not released from the queue if an ISD site is flagged with an error condition. At the time of our review, that was an entirely manual process. There is an opportunity to automate at least a portion of that activity in a future violation photo review system update. Automation could both lower the risk that violation photos could be released into reviewers’ queues in error and improve operating efficiencies (Recommendation 1).

Figure 1 – Violation Photo Review Process



Recommendation 1 – Improve Review Process Flow

The OCA recommends that the Office of Traffic Safety evaluate the costs and benefits of at least partially automating the first level of review and minimizing the duplication between review levels. Management Response and Action Plan

Accepted Action Plan:

1. OTS has addressed the automation of the first review. The violation photo review system now automatically identifies the IDRIS and SIT test status for each site and reports this information on the daily report. We intentionally stopped the programmers from allowing the system to release violations. That final authority remains with the first review personnel who use the results from the daily report, the issue table, and on occasions where required, conversations with technicians, systems analysts, and ISD team to confirm the status of an ISD location as it relates to potential issues that would not be identified by the internal systems or SIT test results.

2. The first review level has taken away some review responsibility from the second and third levels of review related to control sheet review and matching. The second and third reviews have in turn taken on responsibility for some data and image quality review processes from the first review level.

3. The second review level no longer has responsibility for approving and rejecting license plate frame criteria, eliminating the duplication of this review step by assigning responsibility to the third and fourth review levels.

Planned Implementation Date: Ongoing Responsible Party: Executive Director, Office of Traffic Safety

4.1.1. Violation Photo Review Processes

We evaluated our sample of violation photos (see Figure 2 for an example of a violation photo) against established criteria. We agreed that every violation photo that was processed into a ticket was processed correctly. We randomly selected our sample of 461 photos from the 65,000 violations that occurred between September 7, 2012 and March 31, 2013. We found no errors in the 290 tickets that were issued. Therefore, we concluded that there is little3 possibility that violation photos will be processed into tickets because of human error. In the event that human error does occur, OTS has an internal appeals process for registered owners to appeal the ticket. We concluded that OTS’ enhanced review processes have significantly lowered the risk of another invalid ticket similar to the June 2010 event.

3 Our sample size allowed us to conclude with 99 percent confidence that less than 1 percent of violation

photos could be mistakenly processed into tickets.



Figure 2 – Example of an ISD Violation Photo

Inconsistent application of review criteria

In our sample of 461 violation photos, we observed that 171 (37 percent) were rejected and not processed into speeding tickets. There are a number of valid reasons for rejecting violation photos, including severely out-of-focus photos, dirt obstructing the camera lens or the licence plate, out-of-province licence plates, emergency vehicles, etc. OTS indicated that the normal rate of rejected violation photos ranges from about 20 percent in the summer to about 40 percent in the winter months (because of more days of inclement weather), so our sample was within the normal range. We found 36 violation photos in our sample of 461 that we believe could have been processed into speeding tickets. Although we needed to use the system’s enhancement tools to clarify some photos, most were clear enough they did not need enhancing. The reviewers could have rejected those images because they were being overly cautious or they could have just applied the criteria inconsistently (see Recommendation 2). When we reviewed the inconsistencies in our sample with OTS, they agreed that those 36 photos should also have been processed. We used the violation photo review software to enhance 17 of the 36 photos, but 19 of them were readable without enhancement. If reviewers used the enhancement tools more often, it would take them a bit more time to process the more difficult photos. OTS advised us that each reviewer processes up to 1,200 violation photos per day (about 20-30 seconds per photo). More frequently enhancing photos could impact reviewers’ productivity slightly, but would result in increased enforcement. This would, in turn, further contribute to the program goal of safer streets.



Of the 36 violation photos that we believe should have been processed into tickets (8 percent of our sample of 461 violation photos), reviewers did not consistently apply OTS’ review criteria. According to the violations database, those violation photos were rejected because of photo clarity, shadows, partial obstruction, etc. However, other violation photos in our sample with comparable photo clarity (both before and after the criteria were updated in October 2012) were processed into tickets. At least some of the inconsistencies can be attributed to the reviewers’ amount of experience, subjective judgements regarding image quality, the volume of reviews conducted each day, and reviewers adapting to the new criteria adopted in late October 2012. Most of the observed inconsistencies occurred in September and October (prior to the criteria updates). Between November and March, the ratio of inconsistent reviews dropped to about 4 percent. There were approximately 65,000 speed-on-green violation photos processed from September through March. If reviews were conducted on those violations today, we estimate that about 2,500 additional violation photos should have been processed into tickets (4 percent of the 65,000). The average fine amount for the tickets in our sample was about $110. If those tickets had been processed, approximately $280,000 in additional fines would have been issued over the seven months included in our review. Those additional tickets would have further supported the program’s goals of improving the overall safety of the City’s streets (Recommendation 2).

Supervisory review of violation photos

OTS advised us that during the first month after they started processing violation photos, they noticed inconsistencies in how different reviewers applied the review criteria. OTS worked closely with the Crown Prosecutors to reach a consistent set of review criteria that would be accepted by the Court. The resulting criteria included both sample photos and descriptions to better guide reviewers in applying the new review criteria. We confirmed that during the early weeks after the in-house violation review system went live, there was more inconsistency in applying the criteria than there was for the rest of our sample period (see Figure 3). The decision to accept or reject violation photos is often somewhat subjective. However, with regular training, supervisory input, and experience, subjectivity becomes less of a factor.



Figure 3 – Number of Times Criteria Were Applied Inconsistently by Month

We also observed that the processes in place at the time of our audit only required supervisory review of rejected violation photos. We believe that having supervisors conduct reviews of both accepted and rejected violation photos would better serve OTS’ mandate. Those reviews should be used as training opportunities to encourage more consistency in applying violation photo review criteria. Recommendation 2 – Improving Reviewer Consistency

The OCA recommends that the Office of Traffic Safety set up a process to regularly review the work of both supervisors and review staff to improve consistent application of the established review criteria. Management Response and Action Plan

Accepted Action Plan: The Automated Enforcement Validation Supervisor is now responsible to perform quality control review of both approved and rejected violations. The Feedback will be provided to supervisors, teams, and where appropriate, individual reviewers to aid in improving consistent application of review criteria. During the months of July and August 2013, 100 random violations will be reviewed for each automated enforcement violation type from reviews conducted each Wednesday (the day when all OTS reviewers are scheduled to be working). Commencing in September 2013, the quality control process will switch to a random sampling of 200 violations from each violation type once a month on an ongoing basis. Planned Implementation Date: July 1, 2013 Responsible Party: Executive Director, Office of Traffic Safety



System dependability

To assess whether or not ISDs were appropriately handling system malfunctions, we analyzed the recorded speeds for all the violation photos that were processed. The distribution of all speeds captured with violation photos ranged from the speeding violation threshold to one vehicle detected at 147 km/hr in a 70 km/hr zone. The highest recorded speed was similar to recent extreme speeding offences identified in a recent media release issued by Edmonton Police Service. We found that nearly all violations (93 percent) occurred at less than 25 km/hr over the posted limit. After the court-ordered refund in January 2011, the City enhanced the way in which the ISD system outputs are monitored. The ISD systems now include hardware and software enhancements that are designed to capture any error conditions. The built-in system check and reporting functionality was not in place at the time of the June 2010 incident. Because of that fundamental change in ISD system design, we asked OTS to show us how the current ISD systems manage extreme speed readings. They provided us data showing recorded speeds for every vehicle passing through an ISD-monitored intersection at a recorded speed of 150 km/hr or higher for the period August 3, 2012 through April 9, 2013. There were about 550 records of vehicles recorded at speeds of 150 km/hr or higher where the system did not generate a violation. Every one of those records was marked by the ISD system as an invalid speed record. At the time of the June 2010 incident, this process did not exist. Each ISD system is now programmed to report its functionality each day. For example, as the system is activated by vehicles travelling through the monitored lanes, the system will analyze the data and determine whether or not the violation is valid. If specific criteria are not met, no violation is generated but a record is kept in the event data table and reports. This means that during the seven months covered by our review, the on-board computer systems in the ISDs performed as expected. All records associated with errors were filtered into a special queue and diverted from the normal processing queues. Based on our testing, there was no evidence of ISDs functioning outside of operating specifications.

System enhancement

We observed that four violation photos in our sample were rejected because the first photo was not usable, but the second photo was clear. The violation processing system is currently configured to allow reviewers to capture the licence plate photo only from the first photo. In addition, if the plate photo is captured before that violation image is completely loaded, it will be too blurry to read, resulting in an unnecessary rejection by the next reviewer. The violation photo review software should be enhanced to ensure that plate photos: 1) cannot be captured prematurely, 2) can be replaced as needed, and 3) can be captured from the second photo. We would expect that this change would result in fewer violation photos being rejected. This would increase the program’s contribution to OTS’ overall traffic safety mandate.



Recommendation 3 – System Enhancements

The OCA recommends that the Office of Traffic Safety implement system enhancements that allow licence plate photos to be captured from either photo and to be replaced if the licence photo is blurry. Management Response and Action Plan

Accepted Action Plan:

1. During the last update, the violation photo review system was programmed to allow for the plate capture to be redone at the third review level. However, it is not functioning properly and has been identified as a required fix for an upcoming release.

2. OTS will consult with the software developer to determine the feasibility of obtaining a plate capture from either image and having the system amend the court package affidavits to identify which image the capture was taken from.

Planned Implementation Dates:

1. This is already programmed, but needs to be debugged. The process is currently underway.

2. Discussions will occur with the software developer for programming updates prior to September 30, 2013 and updates will be scheduled for a future release.

Responsible Party: Executive Director, Office of Traffic Safety

Data transfer

The violation photo review database contains two major data tables. The first is the event table, which contains records of every vehicle that passes through an ISD-monitored intersection (about 23 million records from the two-month period we assessed). The second is the violation table, which contains records of every vehicle that was speeding through one of those intersections. If not all violation records were being processed, then the violation table would be incomplete, which could cast suspicion on the overall system integrity. We assessed the completeness of the violation table using our audit software. We identified two missing records in the violation table of about 65,000 violation records. Working with OTS, we confirmed that the two missing violation records still existed in the master event table. OTS investigated those two violation records and confirmed the violation photo review system had started to import those two records, but timed out before the import was complete. The end result was that records were not imported for violation review even though the violation sequence numbers were used. We also assessed the completeness of the master event table and found no missing event records.



Recommendation 4 – Violation Record Completeness

The OCA recommends that the Office of Traffic Safety implement system and/or process controls to ensure that all violation event data is transferred to the violation review software. Management Response and Action Plan

Accepted Action Plan: OTS implemented an interim solution when the OCA notified us of the issue. We are now generating a reconciliation report that allows us to identify any import failures, view the reasons for import failures, and re-import any violation that failed. Once we have established the reasons for import failures, we will modify the violation photo review software to avoid any issues in the future. Planned Implementation Date: December 31, 2013 Responsible Party: Executive Director, Office of Traffic Safety

Organizational structure

As shown in Figure 2, OTS conducts the first, fourth, and fifth reviews, while Treasury Management Section conducts the second and third reviews. Edmonton Police Service has delegated the authority to conduct the City’s Automated Photo Enforcement program to OTS. Edmonton Police Service’s delegation gives OTS the legal right to access vehicle owner data to be used in the Automated Photo Enforcement program. The Executive Director of the Office of Traffic Safety is responsible for program administration and management, violation photo review consistency, and is accountable for appropriate use of vehicle owners’ personal information. Although the Executive Director has overall responsibility for operating the program, he does not have full authority over all the reviewers. Combining the two groups into a single unit would improve accountabilities and consistency of reviews at all five levels. In addition, it would allow more effective use of resources. The original program design model presented to Council proposed that the violation photo review process would be managed within a single organizational unit.4 Since the violation photo review software incorporates appropriate segregation of duties, we do not believe that the reviewers need to be organizationally separate. There is an opportunity to increase the overall accountability of the review process by combining the two groups into a single group. That group would be responsible for the entire violation photo review process.

4 Report to City Council, “Program Delivery Model for the Automated Traffic Enforcement Program,”

Report 2007COG003, June 21, 2007, pages 13 and 22.



Recommendation 5 – Evaluate Organizational Structure

The OCA recommends that the General Managers of Transportation Services and Financial Services & Utilities Departments evaluate the opportunity to improve process accountability by consolidating the two groups of reviewers. Management Response and Action Plan

Accepted Action Plan: 1. Regardless of the organizational structure, it is important that all persons reviewing

the tickets avoid the rejection of acceptable tickets. The General Managers of Transportation Services and Financial Services & Utilities will meet with supervisory staff in the Office of Traffic Safety (OTS) and Treasury Management Section to determine why standards and expectations for the processing of tickets may not have been consistently applied during the five reviews conducted on each ticket.

2. Given that the Executive Director of OTS is solely accountable for the Automated Enforcement Program, that person must be able to direct all staff involved in the processing of tickets. The General Managers of Transportation Services and Financial Services & Utilities will meet with supervisory staff in the Office of Traffic Safety (OTS) and Treasury Management Section to determine:

if the current organizational structure (two teams) can be strengthened through a clear definition of standards and expectations, and clarity on a single point of direction from the Executive Director of OTS or

if a new organizational structure (one team) provides a best the option to ensure clarity in standards, expectations, direction and accountability.

Planned Implementation Date: October 15, 2013 Responsible Party: General Managers of Transportation Services and Financial Services & Utilities

4.2. Testing and Maintaining Equipment We assessed OTS’ ISD testing and maintenance procedures to determine whether they are sufficient to ensure the ongoing functionality and accuracy of the ISDs. We confirmed that OTS’ current processes surpass both legislative and ISD manufacturer’s testing requirements. We believe the current processes are adequate to ensure that equipment errors will be effectively managed and will not result in invalid tickets.

System Testing and Maintenance

Following is a description of the four different equipment tests that OTS conducts and the testing we conducted to confirm that the testing meets legislative and manufacturer's requirements.

1. Speed Integrity Test – This test ensures the ongoing accuracy of the speed measurements and reliability of the system by comparing readings from the



Intersection Safety Device against readings from a laser speed gun. We confirmed that OTS’ actual cycle time is about 21 days, while the Province requires testing at least every 30 days. In 2011, the Province audited OTS’ testing procedures and determined that OTS complies with Provincial standards. We observed the following opportunities to enhance OTS’ procedures:

When conducting speed integrity testing, technicians capture computer screen shots that show the speed and time data captured by the ISD. They then complete a checklist with both the speed and time recorded by the ISD and the speed captured by the laser speed gun. At the time of our audit, OTS was not validating technicians’ handwritten ISD speed and time records against the computer screen shots. We observed 14 instances in which there was a slight difference between the ISD data on the screen capture and the data the technician wrote on the checklist. Since the screen captures are a part of the test records, the validity of the records is not in question. However, validating the checklist records against the screen shots would improve the completeness of the testing records in the event that they are required as evidence when a ticket is appealed (Recommendation 6).

During our audit, we observed that the technicians tested the laser speed gun at the beginning and end of each day of testing as recommended by the manufacturer. However, OTS’ speed integrity test manual did not require technicians to perform the test. Adding this test to OTS’ manual would ensure that it is a standalone reference guide (Recommendation 6).

2. Preventive Maintenance Test – This test ensures that the Intersection Safety

Device and related equipment (camera unit, sensor loops, master controller box, and cabinet) pass physical and electronic tests. We confirmed that OTS conducts its preventive maintenance procedures on a 21-30 day cycle. We also confirmed that preventive maintenance and speed integrity testing do not normally occur in the same week. We did not observe any issues with the performance of this test.

3. Remote Monitoring Test – We confirmed that OTS system analysts review the

results of the ISDs’ system tests twice a day. However, we observed that the system analysts did not document and record the results of this review process (for example, using a checklist). Without this kind of record it would be more time-consuming for the OTS to demonstrate that a particular site was functioning normally on a given date. (Recommendation 6).

4. Virtual Review – This is a daily review of all ISD status reports to ensure that

each ISD’s status as logged in the violation photo review system is consistent with test results. We confirmed that this review takes place as scheduled.

Table 4 shows the results of our review of ten randomly-selected ISD locations. We assessed whether or not OTS was complying with the Provincial and manufacturer’s standards and its own procedure manuals for testing and maintaining the ISDs.



Table 4 – OTS Compliance with Testing Standards

OTS Compliance

Test OTS

Procedures Provincial Standard

ISD Manufacturer

Standard

Speed Integrity Yes Yes Yes

Preventive Maintenance Yes n/a Yes

Remote Monitoring Yes n/a Yes

Virtual Review Yes n/a n/a

As shown in Table 4, OTS is conducting its testing procedures and ISD maintenance work in accordance with Provincial and the ISD manufacturer’s standards. We confirmed that any equipment malfunctions are being identified and recorded to avoid having violation photos processed into invalid tickets.

Testing and maintenance records

OTS maintains the test records in binders and/or spreadsheets. Those test records are used to maintain the status of each ISD location in OTS’ violation photo review software. The ISD status for each location is used to ensure that violation photos are processed only when the ISDs are functioning normally. OTS currently uses an “issues log” in a spreadsheet that contains over 5,000 rows. This log maintains records of issues noted at specific ISD locations. The logs are used to track an ISD’s condition from the time an issue is identified to its final resolution. For example, if the sensors malfunction, the location will be taken off line until they are repaired. We observed that it is cumbersome to maintain the log and there is a risk that an issue may be missed. Automating the issues log to allow reviewers to display only issues that are not resolved would reduce the time required to monitor site status and the risk that essential data could be missed (Recommendation 6). Recommendation 6 – Procedure Manuals and Record-keeping

The OCA recommends that the Office of Traffic Safety:

Modify its technical procedure manuals to enhance the completeness and accuracy of site testing, monitoring, and maintenance records.

Ensure that its issues logs are maintained in a manner that optimizes operational efficiency and effectiveness.

Management Response and Action Plan

Accepted Action Plan: When the OCA raised these issues with OTS, we began working to resolve them. Our progress and plans are as follows:

1. To date we have modified our testing procedure manuals, developed a checklist to demonstrate completion of remote monitoring tests for each ISD location, and implemented a process step to verify the numbers the technicians record on the speed integrity test forms against the numbers in the computer screen captures.

2. We are investigating options for implementing an easier to use issue tracking



system, including further enhancements to our violation photo processing software, spreadsheet enhancements, and off-the-shelf ticketing systems.

Planned Implementation Dates:

1. Implementation complete 2. December 31, 2014

Responsible Party: Executive Director, Office of Traffic Safety

5. Conclusions

We were invited by Transportation Services Department to conduct this audit of the Automated Photo Enforcement process. The Department’s express intent was to ensure that operational and system changes it had implemented (added review levels and process improvements) adequately addressed the system failures in 2010 that triggered the refund of $12.3 million in fines. The risk that an invalid ticket can be processed can never be completely eliminated. Based on our testing and analysis, however, we believe that the changes the Office of Traffic Safety have made since January 2011 have effectively and significantly lowered the level of risk. In addition, the equipment and procedural changes that OTS implemented ensure that any equipment malfunctions that affected ISD violation photos can be isolated. That capability was not in place in June 2010. Instead of having the validity of all violation photos questioned because of an equipment malfunction, OTS can now isolate any malfunction to a particular day at a particular location. This further compartmentalizes the risk that equipment malfunction could result in a mass refund. We believe that the changes made to system testing procedures effectively addressed the risk that equipment failure would not be noticed. We also found that the increase in levels of review and improved criteria definitions have reduced the risk that reviewers would accept an invalid violation photo. We did not find any examples of violation photo reviews in which a ticket was issued in error. We did, on the other hand, find 36 examples of violation photos that were rejected because of inconsistent application of review criteria. Most of those inconsistencies were during the first two months of our sample period while the violation photo review criteria were being updated. If the rate of inconsistencies for the entire seven months had been similar to what we observed in the last five months of our review period, the City could have issued about 2,500 additional speed-on-green tickets. Since the average ticket amount was $110, this would have resulted in about $280,000 in additional fines over a period of about seven months. We believe that further increasing consistency in applying the review



criteria will enhance the goals of the City’s traffic safety programs, of which the Office of Traffic Safety is one element. We made a total of six recommendations to further enhance the overall performance of the Office of Traffic Safety and help it advance its goals of facilitating safer streets. We thank the staff of Transportation Services Department, Financial Services & Utilities Department, and Edmonton Police Service for their cooperation and assistance with completing this audit.

automated photo enforcement audit - edmonton

Documents