audit of investigation practices office of the ... · this includes roles and responsibilities, the...

Audit of Investigation Practices Office of the Commissioner of Official Languages

Audit Report March 30, 2012

© Deloitte & Touche LLP and affiliated entities. Office of Commissioner of Official Languages i Audit of Investigation Practices March 30, 2012

Table of contents

1 Executive Summary .......................................................................................................................... 1

1.1 Background and Context............................................................................................................ 1

1.2 Summary of Observations.......................................................................................................... 1

1.3 Conclusion ................................................................................................................................. 3

2 Audit Objective, Scope and Approach .............................................................................................. 4

2.1 Overview of the Investigation Process ....................................................................................... 4

2.2 Audit Objective ........................................................................................................................... 5

2.3 Audit Scope ................................................................................................................................ 5

2.4 Audit Approach ........................................................................................................................... 6

3 Findings and Recommendations ...................................................................................................... 7 3.1 Investigations Process Monitoring ............................................................................................. 7

3.2 Documentation and Understanding of Investigation Processes ................................................ 9

3.3 Completeness, Quality and Timeliness of Investigations Files ................................................ 12

3.4 Complaints Information Management System (CIMS) ............................................................ 15

3.5 Public Awareness ..................................................................................................................... 16

© Deloitte & Touche LLP and affiliated entities. Office of Commissioner of Official Languages 1 Audit of Investigation Practices March 30, 2012

1 Executive Summary

1.1 Background and Context As an agent of Parliament and agent of change, the Office of the Commissioner of Official Languages (OCOL) is a relatively small federal government organization that has a mandate to promote the Official Languages Act and oversee its full implementation, protect the language rights of Canadians and promote linguistic duality and bilingualism in Canada. The Commissioner ensures that the three key objectives of the Act are achieved and takes all necessary measures in this respect. These objectives are:

• the equality of English and French in Parliament, the Government of Canada, the federal administration and the institutions subject to the Act;

• the development and vitality of official language minority communities in Canada; and, • the equal status of English and French in Canadian society.

As documented in the Internal Audit Plan 2010-2013, investigation practices were deemed to be a high audit priority, considering the lack of past audit coverage, the recent structural changes at OCOL impacting investigations (i.e. Commissioner’s Ombudsman role) and the importance of investigation activities with regards to OCOL’s mandate.

The objective of the audit is to determine if OCOL has effective management practices for investigations. This includes roles and responsibilities, the complaint reception center (and related intake and prioritization process), how investigations are conducted and files maintained, and how information derived from investigations is ultimately leveraged throughout OCOL. Detailed audit criteria can be found in Appendix B.

1.2 Summary of Observations The key observations with regards to the audit are provided below.

Summary of Strengths Noted 1. In March 2011, the Compliance Assurance Branch (CAB) commissioned a review to address

significant gaps identified in workflow management and tracking processes. Recommendations were based on extensive consultations within the organization and with other key ombudsman institutions. CAB management prepared an implementation plan for these recommendations, and implementation started during the 2011-2012 fiscal year and will continue through 2012-2013.

2. The increased monitoring of investigation files, especially for the facilitated resolution process, has led to a significant increase in the percentage of facilitated resolution files being completed within 90 days; from 24% in July 2011 to 72% in January 2012.

3. Efforts made by the Compliance Assurance Branch (i.e. blitzes to close investigation files) have contributed to reducing the backlog of incomplete investigations. In July 2011, there were 378 files in the backlog, and that number was reduced to 313, as of January 2012 (17% reduction).

4. A Quality Assurance function was recently added to the organizational chart to oversee the quality of the investigation process. The organisation is currently in the process of staffing this position permanently.


5. While it has not yet been implemented, it was noted that a thorough professional development program was prepared by the Compliance Assurance Branch to favour professional development and mobility of Analysts, to structure the training approach for employees, and to provide access to required tools and support. This initiative is part of a corporate-wide professional development program to promote staff retention and career development within CAB.

Summary of Recommendations

1) We recommend that the monthly Dashboard be modified to track the percentage of formal investigations completed within 180 days and the quality of investigation files. Data and data sources used to produce the monthly Dashboard should also be reviewed to ensure the quality and integrity of information provided to the Executive Committee for monitoring purposes.

2) A formal process should be implemented to perform follow-ups on the implementation of investigation recommendations and commitments to allow OCOL to measure the effectiveness of both the formal and facilitated investigation processes on a regular basis.

3) We recommend that reports on complaints received from parliamentarians, and investigations considered to be “hot” or “priorities” be produced on a regular basis to ensure that OCOL is able to quickly and effectively identify investigations that might be sensitive.

4) We recommend that key procedural documents (i.e. Investigation Procedures Manual, Practice Notices, guidelines for the reception of complaints, letter and report templates) supporting the investigation process be updated and approved. These documents should also be shared with all staff members of the Compliance Assurance Branch, and others as needed, through a centralized repository, such as the intranet or one common shared drive. This repository should be kept up-to-date and its contents should be communicated to all staff, on a need to know basis.

5) (a) We recommend that training be provided to Analysts responsible for investigations with regards to the facilitated resolution process, the adequate documentation of investigation files, and the requirement to obtain consent from the complainant to use the facilitated resolution process.

(b) We also recommend that regional staff be provided with training or guidance with regards to the reporting capabilities of the Complaints Information Management System (CIMS).

(c) For all training provided, we recommend that attendance be monitored to ensure that staff are provided with required training on a regular basis.

6) We recommend that the current quality assurance review of investigation files be enhanced to ensure that the completeness of investigation files is properly assessed. This process should also include the review of the quality of investigation files in order to properly measure the effectiveness of the process.

7) We recommend that mechanisms be considered by the Compliance Assurance Branch to provide more flexibility with regards to assigning investigation files to analysts across portfolios.

8) (a) We recommend that the delegation of authorities for the facilitated resolution process be updated to ensure that all investigation files prepared by Analysts are formally approved by an Assistant Director before final findings or conclusions are sent to the complainant.

(b) We also recommend that file reviews performed by Assistant Directors before final reports or letters are sent be formally documented, for both formal and facilitated processes.

9) As part of the process to replace the current Complaints Management Information System (CIMS), we recommend that OCOL considers a system that provides employees with enhanced reporting capabilities and pertinent information required to perform their duties (e.g. Assistant Directors and analysts should be able to quickly have a snap shot of their ongoing investigations, the Commissioner’s Representatives in regional offices should be able to quickly obtain status updates


for their region, etc.).

10) We recommend that OCOL’s website be updated to provide the public with: a. Definitions and information on both investigation processes (i.e. Formal Investigation and

Facilitated Resolution); b. Information on OCOL’s service standards (e.g. A facilitated resolution process is

expected to be completed in 90 days, and a formal investigation is expected to be completed in 180 days);

c. An online application form to streamline the complaint filing process; and d. A feedback form to provide management with additional information from complainants

and federal institutions to help assess the timeliness and quality of investigations.

1.3 Conclusion Based on the aforementioned observations and overall scope of the audit, OCOL has significant issues related to the effectiveness of the management practices for investigations. The recommendations included in this report are intended to help OCOL management strengthen the effectiveness of the organization’s management control framework to govern and manage investigations.

Based on our professional judgment as auditors, sufficient and appropriate audit procedures have been conducted in accordance with the Treasury Board (TB) Policy on Internal Audit, and evidence gathered supports the accuracy of the conclusions contained in this report. The conclusion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed to with management. The evidence has been gathered to provide senior management with reasonable assurance of the accuracy of the conclusions drawn from this audit. This report and audit were conducted for OCOL management purposes. Use of this report for other purposes may not be appropriate.


2 Audit Objective, Scope and Approach

2.1 Overview of the Investigation Process As part of its mandate, OCOL conducts investigations of complaints against institutions subject to the Official Languages Act. Under OCOL’s current organizational structure, investigations fall under the jurisdiction of the Compliance Assurance Branch. The Compliance Assurance Branch (CAB) is structured in three portfolios: Security and Transport, Social and Cultural, and Economic and International. These three portfolios report to a Director, who reports to the Assistant Commissioner, Compliance Assurance. Each portfolio is managed by an Assistant Director and is supported by 5 to 11 Analysts (investigators) (See Section 3.3 for further details). This structure was designed to help ensure that Analysts focus their efforts on a limited number of institutions, gain a better understanding of these institutions and therefore develop a better working relationship with the institutions. Finally, the CAB also has responsibility for the Complaints Reception Center, comprised of 2 receptionists/assistants and 2 Complaint Reception Officers that report to the Supervisor of Branch Support Services.

OCOL receives complaints from the public, from federal government employees and from parliamentarians by mail, telephone or fax. These complaints are received by the Complaints Reception Center and assessed for admissibility. The Complaint Reception Center sends a confirmation of reception to the complainant, collects information about the complaint, verifies if it falls under OCOL’s mandate, performs research on related or similar complaints and forwards the complaint to the Compliance Assurance Branch Senior Analyst assigned to the institution that is the subject of the complaint. This information is captured in OCOL’s Complaints Information Management System (CIMS) by a Complaint Reception Center Officer or Assistant.

Once the complaint is received by a Senior Analyst, its complexity level is assessed to determine if a Senior Analyst (for complex files) or an Analyst (for less complex files) should be leading the investigation. Once the decision is made, the Senior Analyst or the Analyst determines if the complainant wishes to proceed with a facilitated or formal process (see definitions below) and performs the investigation by gathering information from various sources such as the complainant, the target institution and other sources, as needed. While most Senior Analysts and Analysts are based in Ottawa, some are located in regional offices. This may change as OCOL is currently reviewing the roles and organizational structure of the Compliance Assurance Branch and regional offices.

As previously noted, OCOL provides complainants the option of a “facilitated” or a “formalized” process to investigate their complaint. Regardless of the selected process, both require: strategic thinking and planning; collecting and collating facts; consultations; analysis; documenting; and a final assessment.

A facilitated resolution process means that the Office of the Commissioner’s primary objective is to effectively resolve the issue brought forth by the complainant, without drawing a conclusion as to whether the complaint is founded or unfounded. In this process, OCOL attempts to facilitate and assist the parties in resolving the complaint without the Commissioner needing to determine who is right or wrong. This process is the preferred method for resolving complaints, unless the formal investigation process is desired or considered the most appropriate in the circumstances. Investigations conducted under the facilitated resolution process are expected to be completed within 90 working days. As of January 2012,


64% of all opened investigations are being conducted under this process, and 72% of facilitated resolution investigations between July 2011 and January 2012 were completed within the service standard of 90 working days. It should be noted that the service standards were only formally communicated to staff in March 2012.

A formal investigation process enables the Commissioner to reach a decision on the validity of an admissible complaint and relies on the results of the investigation to reach conclusions and make recommendations as appropriate. A formal investigation means that the Commissioner will formally conclude on whether the complaint is founded or unfounded (rights based) by means of a report to the Deputy Head of the Institution that is the subject of the complaint. Investigations conducted with the formal investigation process are expected to be completed within 180 working days. For the formal investigation process, the success rate for that specific service standard is not tracked, but the monthly Dashboard prepared for the Executive Committee indicates that 122 formal complaints had been opened for more than 180 days in July 2011; this number grew to 616 in January 2012 (including 437 complaints related to 1 specific incident).

Based on data provided monthly to the Executive Committee, OCOL received 438 complaints from April 2011 to January 2012, 64% of which were investigated through the facilitated resolution process. The following summary table highlights OCOL’s performance with regards to both investigation processes:

# of complaints

in 2011-2012

Investigation Process Chosen

Service Standard to

complete the investigation

Service Standard Met # of files in backlog

July 2011

January 2012

July 2011

January 2012

438 Facilitated: 64% 90 days 24% 72% 256 133

Formal (and other): 36%

180 days Data is not tracked

Data is not tracked

122 616* (180)

*Includes 437 complaints for 1 specific event

For both facilitated and formal processes, details are documented in OCOL’s Complaints Information Management System (CIMS) and through paper files; however, the final and official file is always the paper file archived at OCOL’s headquarter in Ottawa. Finally, the Commissioner reviews and approves all final reports that contain recommendations, issued through the formal investigation process. Reports issued through a facilitated resolution process are reviewed and approved by the Commissioner only if they pertain to complaints received from parliamentarians, or other issues considered to be sensitive. Other reports are reviewed and approved by Analysts, the Director or the Assistant Commissioner, CAB, as needed, before being finalized and closed.

On the human resources level, the CAB is undergoing a restructuration of its organizational structure to ensure that its activities are streamlined and that investigations are conducted as effectively and efficiently as possible. This new structure will be in line with A-Base review recommendations indicating that the CAB had too many management-level positions.

Finally, it was noted through this audit that OCOL’s Complaints Management Information System (CIMS), and its underlying systems, don’t meet the organization’s needs. As such, the organization is reassigning internal resources towards the update of supporting systems and infrastructure, and, at the time of writing this report, had submitted a Treasury Board submission for additional funds in order to replace its critical systems, including CIMS.

2.2 Audit Objective The objective of the audit is to determine if OCOL has effective management practices for investigations.

2.3 Audit Scope The scope of this audit includes investigation management practices of the Compliance Assurance


Branch. This includes roles and responsibilities, the activities of the complaint reception center (and related intake and prioritization process), how investigations are conducted and files maintained, and how information derived from investigations is ultimately leveraged throughout OCOL.

2.4 Audit Approach The approach and methodology used for the audit were consistent with the Internal Audit standards as outlined by the Institute of Internal Auditors, and were aligned with the Internal Audit Policy for the Government of Canada.

OCOL strives to maintain effective management practices for investigations that are reflective of industry leading practices. Consequently, the following control frameworks were being leveraged for the audit:

• Framework of Core Management Controls and Audit Criteria (CMC – May 2010) established by the Office of the Comptroller General of Canada (OCG);

• Management Accountability Framework (MAF VIII) that sets out the Treasury Board's expectations of senior public service managers for good public service management; and,

• Other criteria were also included to ensure appropriate coverage of the aforementioned audit scope.

A risk-based audit program was developed (see Appendix B) to provide more details on how the various audit areas were addressed. Various audit procedures were used to conduct this audit, including:

• Review of plans, objectives and strategies related to the investigation process, and the Compliance Assurance Branch’s operational plans;

• Review of a sample of investigations files for completeness and compliance with policies and procedures;

• Review of policies, procedures, tools and templates related to the investigation process; • Review of management meeting minutes, job descriptions and investigation reports; and, • Interviews with targeted individuals in all sectors (the list of interviewees can be found in

Appendix A).

The audit was conducted within the following timelines: • Planning Phase : February - May 2011 • Examination Phase: As the Compliance Assurance Branch was implementing a series of

initiatives to restructure and streamline its activities, the examination phase was performed in August 2011 - February 2012

• Reporting Phase: March 2012 • Presentations to the Executive Committee and the Audit and Evaluation Committee: June 2012

The audit’s observations and recommendations were made in accordance with the rating table described below:

Higher Priority Significant deficiencies related to the objective of the audit; should be dealt with in the short-term.

Moderate Priority Moderate deficiencies related to the objective of the audit; should be dealt with in the medium term.

Lower Priority Deficiency related to the objective of the audit is considered minimal, and/or this represents a best practice consideration.


3 Findings and Recommendations

3.1 Investigations Process Monitoring


We expected to find that appropriate performance measures for the investigations process have been defined; that performance is being tracked based on these measures; that management is informed of the results of monitoring and evaluation activities; and that the investigations process is modified as needed. We also expected to find that a process is in place to consistently identify publicly sensitive complaints and promptly communicate them to management.

We found that performance measures for the investigations process are identified in the Performance Measurement Framework (PMF), and that results are presented in the organization’s Departmental Performance Report. The PMF is in the process of being reviewed. Currently, OCOL management’s tool to measure performance with regards to some of the identified indicators is a monthly report on statistical data (the Dashboard) for the investigation process. It was noted that this report doesn’t include specific data with regards to some of the indicators. Here are the five performance indicators identified for the investigations process:

1. Percentage of recommendations related to compliance for which a follow-up took place in the following areas: audits, annual reports, investigations, performance report cards.

2. Percentage of OCOL responses to complaints, requests for intervention and inquiries delivered as per service standards.

3. Quality of the investigation process (based on a review of a sample of files). 4. Percentage of completed investigations. 5. Percentage of recommendations related to compliance for which a follow up took place in the

following areas: Audits, Commissioner’s annual reports, Investigations, and in Performance measurement of institutions (report cards).

Through the monthly Dashboard, the Executive Committee is also provided with many other data elements, such as the number of investigations in progress and completed, the number of files in backlog, the percentage of facilitated resolution process investigations completed within 90 days, etc. The monthly Dashboard does not track the percentage of formal investigations completed within 180 days and the quality of investigation files (PMF indicators 2 and 3). A review of the monthly Dashboard provided to the Executive Committee revealed that some of the information might not be accurate (e.g. # of files in the backlog, % of files that are deemed complete, etc.).

Also, there is currently no formal process in place to ensure that follow-ups on the implementation of investigation recommendations and commitments are performed on a regular basis. Management is therefore not able to report on 2 of its 5 performance indicators (PMF indicators 1 and 5).

With regards to publicly sensitive complaints, we found that a weekly report on received complaints is prepared and provided to the Compliance Assurance Branch (CAB) Management Team (i.e. CAB Assistant Commissioner, CAB Directors, CAB Assistant Directors and CAB Special Advisor) and the


Executive Committee. The report provides a summary of all the complaints received during the previous week. This ensures that senior management is informed of all the new investigations being started and provides management with an opportunity to flag potentially sensitive files.

Also, as of December 2011, another monthly report is produced to highlight complaints received from parliamentarians. The report is provided to the CAB Management Team and the Commissioner. Other reports are expected to be prepared monthly, as of April 2012, to cover “Hot and Priority” files. As these reports (complaints from parliamentarians; hot and priority files) were recently implemented, or will be in 2012, their effectiveness was not assessed as part of this audit, but they were considered to be important measures for effective risk management and monitoring purposes.

The lack of sufficient data with regards to the investigation process increases the risk that performance indicator targets will not be met, and that corrective measures will not be identified and implemented. Also, the current lack of sufficient reports to monitor complaints or investigations considered to be “hot” or “priorities” increases the risk that publicly sensitive files will not be adequately monitored by the management team.

Recommendations

1) We recommend that the monthly Dashboard be modified to track the percentage of formal investigations completed within 180 days and the quality of investigation files. Data and data sources used to produce the monthly Dashboard should also be reviewed to ensure the quality and integrity of information provided to the Executive Committee for monitoring purposes.

2) A formal process should be implemented to perform follow-ups on the implementation of investigation recommendations and commitments to allow OCOL to measure the effectiveness of both the formal and facilitated investigation processes on a regular basis.

3) We recommend that reports on complaints received from parliamentarians, and investigations considered to be “hot” or “priorities” be produced on a regular basis to ensure that OCOL is able to quickly and effectively identify investigations that might be sensitive.

Management Response

1.

Management accepts Recommendation 1.

A statistical table on files completed by the formal investigation process will be added to the April 2012 monthly Dashboard. It was planned that this statistical table would be added to the monthly Dashboard for fiscal year 2012-2013. Note that since the implementation of the new service standards (on April 1, 2012), the service standard for the formal investigation process is 175 working days to complete the final report (close the case).

Regarding the quality of files, Table 9 (Completeness of paper files for completed investigations) is already incorporated into the monthly Dashboard. We will change the title of the table so it is clearly associated with the quality of investigation files, starting with the April 2012 report.

The data and data sources used to produce the monthly Dashboard are currently being revised. Also, the activities planned for phase 1 of the IEMS project (modules 1 and 2) will provide an opportunity to better identify or clarify the data used in producing the monthly Dashboard, the gradual implementation of which is expected to be complete by March 31, 2013.

The methods and data will be revised by the Quality Assurance Officer. An staffing process for the position of Quality Assurance Officer is currently underway.


2.


In the new complaint management system, there will be specific fields for recommendations and follow-ups. This will enable management to follow up on the implementation of commitments and recommendations related to investigations in 2013-2014.

On a temporary basis, it is also planned for the Professional Practice officer to train all employees on sections 7.3.3 (commitment verification form), 7.3.5 (recommendation form) and 7.3.6 (follow-up form) of the Investigation Manual. A staffing process for the position of Professional Practice Officer is currently underway. In the meantime, the management team will follow-up with staff regarding sections 7.3.3, 7.3.5 and 7.3.6. This will be done by the fall of 2012.

3.


The report on active complaints from parliamentarians has been generated monthly since July 2011.

The report on hot files had been generated monthly since April 2012.

The report on priority files will be generated monthly starting in fall 2012.

Note that the production of these different reports is spread out over a period of about 18 months to ensure normal progression of analysts’ workloads. First, we introduced the production of the report on active complaints to parliamentarians, then the hot file report, and next we will introduce the priority file report.

3.2 Documentation and Understanding of Investigation Processes


We expected to find that priorities, objectives, roles and responsibilities with regards to the investigation process are clearly defined, documented and communicated through policies, guidelines, job descriptions, tools and templates.

We found that the introduction of the facilitated resolution process in 2009 created some confusion amongst employees of the Compliance Assurance Branch (CAB) with regards to the two investigation processes. These processes are identified and documented in the Investigation Procedures Manual and in training decks, but they are still not well understood across the organization. For example, some analysts indicated that they don’t always ask for the complainant’s approval before undertaking a facilitated resolution process, as complainants get confused by the two different processes.

We also found a number of elements that might be contributing to that confusion with regards to OCOL’s investigation processes:

• The first element is that the Investigation Procedures Manual is currently under review and the updated version has not been communicated internally yet. Guidance with regards to the facilitated resolution process has been provided to all Analysts through training sessions and practice notices, but Analysts are still expecting further details in the updated version of the manual. It was mentioned that a completely updated manual would only be available in approximately 2 years (2013 or 2014).

• The second element is that several versions of tools and templates (e.g. letters and reports sent to complainants and targeted institutions, templates to follow-up on commitments, etc.) are being


used. Through time, Analysts developed their own templates as they don’t have access to one central repository containing the latest templates. They currently have to consult 3 different sources (i.e. intranet and 2 different shared drives) that are not kept up-to-date.

• The third element is that roles and responsibilities for which investigation staff and other OCOL staff are held accountable are defined in multiple documents; job descriptions, the Investigation Procedures Manual, guidelines, the Compliance Assurance Branch Performance Framework, in service standards and in practice notices; despite this, it was noted through discussions with some staff members that roles, responsibilities and accountabilities are not understood the same way by all Compliance Assurance Branch employees. For example, expectations for Complaint Reception Officers, Analysts and Senior Analysts are not the same across all 3 portfolios.

• Finally, we were informed that training with regards to the facilitated resolution process and the formal investigation process was provided to new employees joining the organizations, as well as all analysts already involved with investigations in June and October 2009; December 2010; and February and October 2011. Training is also offered for one-on-one sessions as needed. Unfortunately, attendance for these training sessions was not documented, and could not be tested as part of this audit. It was also noted that some staff members were not familiar with the reporting capabilities of the Complaints Information Management System (CIMS) and that further training would be beneficial. Also, the Compliance Assurance Branch recently developed a new professional development program, but it was not implemented, and therefore we could not test its effectiveness as part of this audit.

The lack of clearly defined, communicated and understood priorities, objectives, roles and responsibilities with regards to the investigation process increases the risk that investigations will not be conducted in a consistent fashion by all Analysts; therefore having a potential negative impact on the quality of investigations and on OCOL’s reputation.

Recommendations

4) We recommend that key procedural documents (i.e. Investigation Procedures Manual, Practice Notices, guidelines for the reception of complaints, letter and report templates) supporting the investigation process be updated and approved. These documents should also be shared with all staff members of the Compliance Assurance Branch, and others as needed, through a centralized repository, such as the intranet or one common shared drive. This repository should be kept up-to-date and its contents should be communicated to all staff, on a need to know basis.

5) (a) We recommend that training be provided to Analysts responsible for investigations with regards to the facilitated resolution process, the adequate documentation of investigation files, and the requirement to obtain consent from the complainant to use the facilitated resolution process.

(b) We also recommend that regional staff be provided with training or guidance with regards to the reporting capabilities of the Complaints Information Management System (CIMS).

(c) For all training provided, we recommend that attendance be monitored to ensure that staff are provided with required training on a regular basis.

Management Response

4.


Four practice notices were created during fiscal year 2011-2012. They were posted in the CAB intranet page and will eventually be posted on the IEMS. These notices formally update certain sections of the Investigation Procedures Manual while we await its revision.

Furthermore, a work plan for updating the Manual was been prepared in November, 2011. The revision of


the Manual has begun and will be completed during 2013-2014. As the chapters are reviewed and approved, they will be disseminated to staff. They will also be posted on the intranet. Note that the two chapters dealing specifically with investigation procedures (chapters 5 and 6) will be revised and approved in 2012-2013. There will also be an information session when the revised chapters are distributed.

The workflow and instructions for the Complaint Reception Centre will be completed and approved in June 2012 and posted on the intranet.

The templates (models) for letters and reports are currently being revised. They are expected to be completed during summer 2012. They will be posted on the intranet and eventually on the new IEMS.

5(a)

Management accepts Recommendation 5 (a).

Management confirms that all analysts have received training on the facilitated resolution process (FRP). However, in order to clarify the principles of FRP, the director of investigations will meet with all analysts during team meetings, by the fall of 2012.

Tools on the facilitated resolution process exist and have been distributed to all the analysts. They will be posted on the CAB intranet page in order to be more readily available. These tools will be revised and updated, and then presented to the analysts at team meetings.

Regarding adequate documentation of investigation files, the checklist will be revised. The list will include a component to verify that consent to use the FRP has been obtained. A statement of practice will be prepared to this effect and distributed to CAB personnel during summer 2012.

Note that it is already an established practice that analysts obtain consent from complainants to use the FRP. Procedures require that this information be recorded in a mandatory CIMS field (mandatory for the user, but not required by the system). This field has existed since the FRP was introduced. Once IEMS is implemented, this field will be mandatory and required by the system.

The quality assurance officer will ensure that this practice is correctly applied.

Finally, as mentioned in the response to recommendation 10(d), a client satisfaction survey will be administered to complainants. One of the survey questions will concern consent to the use of the FRP, so that application of this practice can be measured.

5(b)

Management accepts Recommendation 5 (b).

The Assistant Commissioner, CAB, will work jointly with the Assistant Commissioner, PCB, to offer training for the Commissioner’s representatives. This training will be held in fall 2012.

5(c)

Management accepts Recommendation 5) (c)

Management will take attendance to ensure that staff receive the required training.


3.3 Completeness, Quality and Timeliness of Investigations Files


We expected to find that investigations are conducted in a consistent and appropriate fashion, with sufficient review and approval, and with the correct information being kept on file and provided to the complainant and institution as required. We also expected to find processes to review outstanding investigations and ensure they are resolved in a timely manner.

With regards to completeness and quality of investigation files, we found that OCOL’s investigations group is structured by portfolio, each headed by an Assistant Director and supported by Analysts. The portfolios oversee investigations for specific federal institutions. The current implementation of the portfolio approach contributes to a siloed approach to the investigation process, as Analysts are usually not made available to other portfolios to help deal with varying demands between portfolios. This reduces the organization’s ability to adjust to unpredictable fluctuations in the level of complaints and help ensure a proper distribution of the workload, therefore affecting the timeliness of some of these investigations. A review of data included in the monthly Dashboard for January 2012 reveals that human resources allocated to portfolios are somewhat reflective of the varying level of complaints received by each portfolio.

Portfolio Name #

employees (approx.)

# Active Formal Complaints

# Active FRP Complaints

# Complaints in Backlog

Social and Cultural 5 (22%) 125 (44%) 61 (32%) 104 (36%) Economic and International 7 (30%) 41 (14%) 34 (18%) 44 (15%) Transport and Security 11 (48%) 557* (120) (42%) 94 (50%) 580* (143) (49%) Total 23 723* (286) 189 728* (291)

*Includes 437 complaints for 1 specific event

We also found that OCOL implemented a process to review a monthly sample of investigation files to ensure their completeness. Results of this independent review process are communicated to Analysts, Assistant Directors and the Executive Committee. As it currently stands, this process is focused on the completeness of investigation files and does not include an assessment of their quality. The monthly Dashboard for January 2012 reveals that 86% of investigation files are considered to be complete (i.e. they include key documentation). Our own review of a sample of 25 investigation files revealed that only 48% of investigation files included these same key documents (e.g. letters sent to complainants or the federal institution, final reports, etc.). A review of other criteria not included in OCOL’s quality review process revealed that investigations are not conducted in a consistent fashion across all 3 portfolios and that required information is not always kept on file or provided to the complainant. For example:

• Each Analyst communicates with complainants and federal institutions by using his/her own letter and report templates; therefore not providing the same kind of information or level of detail;

• Different approaches are used to document investigation files, and some files contain very detailed information, whereas other files barely contain any information at all;

• It was noted that in all 25 cases, the selected investigation process (i.e. formal vs. facilitated resolution) aligns with applicable selection criteria but it was also noted that:

o 16 of the files did not meet all the established service standards (e.g. 90 business days to complete a facilitate resolution process, 180 business days to complete a formal investigation process, etc.); and,

o 18 of the files did not have any evidence of being reviewed by an Assistant Director before being closed.

• With regards to delegation of authority, it was noted that:


o For all 25 files, the acknowledgement of receipt of a complaint was sent by the Complaints Reception Center, as per delegated authority;

o For 24 of the files, the Notice of Investigation was sent by the Analyst (for the facilitated resolution process) or the Commissioner (for the formal investigation process), as per delegated authority. 1 exception was noted in this area;

o For 23 of the files, the final reports or letters were approved by the Analyst (for the facilitated resolution process) or the Commissioner (for the formal investigation process), as per delegated authority. 2 exceptions were noted in this area; and,

o Although it seems to be common practice for Analysts to have their investigation files and final reports reviewed by a Senior Analyst or an Assistant Director, the matrix of delegated authority reveals that Analysts have formal authority to conduct, conclude and send out final communications to the complainant and the federal institution for the Facilitated Resolution Process, except for complaints received from parliamentarians and complaints considered to be sensitive. A note at the bottom of the delegation of authority matrix indicates that all investigation files should be reviewed by an Assistant Director, but as noted above, 18 of the reviewed files did not have any evidence of this review.

With regards to the timeliness of investigations, OCOL has recently implemented a monitoring process designed to ensure that investigations that are taking more time than the applicable service standards are being followed-up on a weekly basis. Assistant Directors are informed of the number of days for which each investigation has been opened in their portfolio. They are then able to follow-up with the Analysts to ensure that mitigating measures are implemented, as needed. It must be noted that this is a relatively new and informal process, and we were therefore not able to assess its effectiveness as part of this audit. Nevertheless, an analysis of data provided through the Dashboard to the Executive Committee on a monthly basis revealed that performance is improving for the facilitated resolution process, but further improvements are needed to eliminate the backlog and ensure that formal investigations also meet the service level standards on a regular basis. (See the summary table in section 2.1 of the report.)

As a result, the lack of a standardized approach to document, review and approve investigation files and to communicate with complainants and federal institutions increases the risk that errors will be made, therefore affecting the completeness and quality of investigation files.

Recommendations 6) We recommend that the current quality assurance review of investigation files be enhanced to ensure

that the completeness of investigation files is properly assessed. This process should also include the review of the quality of investigation files in order to properly measure the effectiveness of the process.

7) We recommend that mechanisms be considered by the Compliance Assurance Branch to provide more flexibility with regards to assigning investigation files to analysts across portfolios.

8) (a) We recommend that the delegation of authorities for the facilitated resolution process be updated to ensure that all investigation files prepared by Analysts are formally approved by an Assistant Director before final findings or conclusions are sent to the complainant.

(b) We also recommend that file reviews performed by Assistant Directors before final reports or letters are sent be formally documented, for both formal and facilitated processes.

Management Response

6.


Regarding adequate documentation of investigation files, the checklist will be revised. The list will include


a component to verify that consent to use the FRP has been obtained. A statement of practice will be prepared to this effect and distributed to CAB personnel during summer 2012.

In recent months, there has been a certain instability in the branch regarding staffing measures for the position of Professional Practice Officer (AS-06). The work description for this position has been revised and a staffing process is currently underway to fill the position indeterminately.

The branch reviewed its structure in July 2011 and has added a Quality Assurance Officer position (AS-06). The work description was developed in winter 2011-2012, and a process to staff the position is underway. The Officer, an expert in the field, will ensure a rigorous quality assurance approach to the branch’s activities. A rigorous methodology and quality assessment criteria will be developed.

The Planning, Audit and Evaluation Directorate is currently reviewing the organization’s performance measurement framework. More specifically, the compliance component is being revised and required indicators will be developed.

7.


The time study project (work measurement), carried out in two phases – April to August 2011 and January to May 2012 – sought to determine how much time is spent on each activity in an investigation process. The results and conclusions of this project will enable CAB to study mechanisms to provide more flexibility with regards to work distribution (investigation files) by the fall of 2012.

An additional weekly meeting of the Assistant Directors, Operations will be held to discuss the distribution of newly-received complaints. This meeting has been added to the schedule since May 2012.

8(a).

Management accepts Recommendation 8) (a).

Regarding the CAB delegation of signing authorities chart, Note (b) states “In the case of notice of intention to investigate or closing, the document must be reviewed and approved in advance by the Assistant Director, Operations.” This statement refers to the formal investigation process and the FRP. However, to clarify approval roles and requirements, the chart will be changed to clearly show that the Assistant Director, Operations, will review and approve all files. Also, a note will be added stating that a review/approval delegation to an analyst will be possible and that a note must be added to the file to document such delegation.

It is expected that the Quality Assurance Officer will ensure that this practice is correctly applied.

The chart is dated March 18, 2010 (approved by EXCOM) and it is scheduled to be updated during 2012-2013.

8(b)

Management accepts Recommendation 8) (b).

Quality control will be documented by the Assistant Directors for all investigation files (FRP and formal investigation process) regarding sending letters of closure and final reports.

The Quality Assurance Officer will ensure that this practice is correctly applied. Quality assurance is a distinct function to be performed by someone who is independent of the investigation process.


3.4 Complaints Information Management System (CIMS)

Moderate Priority

Moderate deficiencies related to the objective of the audit; should be dealt with in the medium term.

We expected to find appropriate information systems to support and monitor the investigation process.

We found that OCOL’s current Complaints Information Management System (CIMS) is not configured to optimize internal processes and monitoring exercises. Most employees of the Compliance Assurance Branch (CAB) indicated that CIMS is outdated, unstable, not user friendly and doesn’t provide sufficient reporting functions. The replacement of CIMS was identified as one of the organization’s most important risks in its 2012 Corporate Risk Profile and the Corporate Management Branch is responsible for the implementation of mitigating measures, such as the development of a new replacement system for CIMS.

As mentioned in the overview of the investigation process in section 2.1 of this report, internal resources have been reassigned to ensure the update of underlying systems supporting CIMS to provide better system stability, and, at the time of writing this report, a Treasury Board submission had been submitted for additional funds to change the organization’s key systems, including CIMS. Most employees noted that CIMS does not provide adequate reporting functions for ongoing monitoring purposes. For example, current system access to CIMS’ reporting function is very limited. As such, the Assistant Director of Investigations Professional Practice is the only one that can extract data from the system with regards to the number of days gone by since each investigation was started. He therefore prepares regular e-mail updates with this information and provides it to the Assistant Directors, which is then relayed to Analysts, at the discretion of the Assistant Directors. Assistant Directors and Analysts are therefore not able to quickly and efficiently monitor their own investigation files whenever needed as they have to wait for statistics to be provided by e-mail.

Another example is that investigations-related data needed to support internal research and studies, or the analysis in trends for performance monitoring of federal institutions usually cannot be obtained through CIMS reports. As such, required raw data has to be extracted from CIMS by an IT specialist and then manually reviewed and sorted by internal staff in order to extract the required information. This process is time consuming and reduces the organization’s data analysis capabilities.

Finally, it was noted that the Commissioner’s Representatives in regional offices have access to system generated reports to review information about regional complaints (e.g. number and type of complaints, summaries of progress, trends, etc.), but that the process to generate reports is time consuming and can be somewhat confusing. The Commissioner’s Representatives are expected to build good working relationships with local organizations and respond to their needs, but the lack of easily accessible information with regards to regional complaints has a negative impact on their ability to do so effectively.

As a result, the lack of access to timely and relevant system-generated reports on investigation files has a negative impact on the organization’s ability to efficiently and effectively monitor its activities, and might also have a negative impact on the public’s perception of the relevance of the Official Languages Act.

Recommendations 9) As part of the process to replace the current Complaints Information Management System (CIMS),

we recommend that OCOL considers a system that provides employees with enhanced reporting capabilities and pertinent information required to perform their duties (e.g. Assistant Directors and analysts should be able to quickly have a snap shot of their ongoing investigations, the Commissioner’s Representatives in regional offices should be able to quickly obtain status updates for their region, etc.).


Management Response 9.


The case management system renewal is currently underway. This project is managed by the Corporate Management Branch (CMB). CAB commits to contributing to the project by sharing all its operational and report production needs.

The Assistant Commissioner, Corporate Management Branch, confirms that the new integrated solution will offer improved reporting capacity. The new functionalities will enable the organization to produce reports.

The integrated solution is composed of three modules 1-Electronic Document Management, 2-Case Management and 3-Web Content Management and on-line submission. However, it should be noted that the implementation of this integrated solution is progressing as funding permits. The intent of Module 1 is to position OCOL to be fully compliant with the IM/IT Policy Framework, and therefore be ready for the transition and meet the requirement for the Record Keeping Directive which requires managing life cycle of electronic documents.

Module 1 will offer to all employees new functionalities such as “metadata” which enables users to identify information using keywords, and retrieve pertinent information from the system in the form of reports.

Module 2 – Case Management, will offer more advanced functionalities to track and record case data and information, with more sophisticated reporting tool, and allow employees and management to measure and report on all case data and information in a set.

3.5 Public Awareness

Moderate Priority

Moderate deficiencies related to the objective of the audit; should be dealt with in the medium term.

We expected to find that formal communication processes / mechanisms exist and support sharing of timely, relevant and reliable information to the public about the investigations process, and that client service standards are established, communicated to the public and monitored by OCOL management.

We found that some formal communication processes / mechanisms (e.g. annual report, website, Analysts’ discussions with complainants, etc.) exist to support the timely sharing of relevant and reliable information to the public about the investigation process. It was noted, however, that the OCOL website does not provide any details with regards to services standards or the distinction between a formal investigation and the facilitated resolution process. Also, complaints can be filed by telephone, mail or fax, but complainants cannot file a complaint online through OCOL’s website. CAB employees indicated that the public is generally not well aware of OCOL’s investigation processes, and that they are currently not asking complainants for feedback once investigations are completed. Finally, it was also noted that OCOL’s 2010-2011 Departmental Performance Report doesn’t mention that two different processes are used to conduct investigations (facilitated resolution process and formal investigation process), and that they have different service standards.

As a result, the lack of clearly communicated investigation processes and service level standards increases the risk of an inefficient or under-utilized investigation process.

Recommendations 10) We recommend that OCOL’s website be updated to provide the public with:


a) Definitions and information on both investigation processes (i.e. Formal Investigation and Facilitated Resolution);

b) Information on OCOL’s service standards (e.g. A facilitated resolution process is expected to be completed in 90 days, and a formal investigation is expected to be completed in 180 days);

c) An online application form to streamline the complaint filing process; and

d) A feedback form to provide management with additional information from complainants and federal institutions to help assess the timeliness and quality of investigations.

Management Response 10(a)

Management accepts Recommendation 10 (a).

The definitions and information on the two processes will be updated on the OCOL Web site by March 31, 2013.

10(b)

Management accepts Recommendation 10 (b).

The information on service standards will be updated on the OCOL Web site by March 31, 2013.

10(c)

Management accepts Recommendation 10 (c)

This recommendation will be presented to EXCOM by March 31, 2013 for approval of the next steps and commitment of the other branches so that the implementation of this recommendation will be a priority.

CAB will develop the content of the form.

The Corporate Management Branch Assistant Commissioner confirms that the intent of Module 3 – Web Content Management and On-line Submission, is to offer the content management functionality and allow a complainant to file a complaint on-line via the OCOL Web site, in compliance with security requirements. It is understood that discussions need to be held on how CAB would like to streamline the existing process to ensure that governance instruments can be updated accordingly and that implementation take place in the new system by activating the functionality.

10(d)

Management accepts Recommendation 10 (d).

There is already an activity to this effect in the management action plan (Implementation Framework), as part of the workflow management project. More specifically, Recommendation 18 of this document states:

“That, as a first quality assurance exercise, an approach be developed for a Client Satisfaction Survey to eventually be conducted with a sampling of recent complainants, in view of improving the services CAB offers to Canadians”.

This activity will be completed by March 31, 2013.

As indicated above, (in response to Recommendation 6), a process to staff the Internal Quality Assurance Officer position is underway.


Appendix A – Interviewees

The following individuals were interviewed as part of the audit process:

• Carole Beauvais – Director, Investigations Directorate – Compliance Assurance Branch • Sylvie Carpentier – Assistant Director, Security and Transport Portfolio – Compliance Assurance

Branch • Ghislaine Charlebois – Assistant Commissioner – Compliance Assurance Branch • Lise Cloutier – Assistant Commissioner – Corporate Management Branch • Diane Côté – Assistant Director, Social and Cultural Portfolio – Compliance Assurance Branch • Patrick Desrochers – Senior Analyst, Strategic Interventions – Compliance Assurance Branch • Graham Fraser – Official Languages Commissioner • Sylvain Giguère – Assistant Commissioner – Policy and Communications Branch • Claude Haché – Senior Analyst, Atlantic Region – Compliance Assurance Branch • Henri Lahaie – A/Assistant Director, Investigations Professional Practice – Compliance Assurance

Branch • Deni Lorieau – Commissioner’s Representative for the Alberta, British Columbia, Northwest

Territories, Yukon and Nunavut Region • Eva Ludvig – Commissioner’s Representative for the Quebec Region • Johanne Morin – Assistant Director, Economic, Scientific and Central Agencies Portfolio –

Compliance Assurance Branch • André Nault – Analyst, Economic, Scientific and Central Agencies Portfolio – Compliance Assurance

Branch • Mathieu Ouellette – Complaint Reception Officer – Compliance Assurance Branch • Paul Sébestyen – Analyst, Social and Cultural Portfolio – Compliance Assurance Branch • Carole Seguin – Senior Analyst, Security and Transport Portfolio – Compliance Assurance Branch • Julie Story – Analyst, Security and Transport Portfolio – Compliance Assurance Branch • Johane Tremblay – Director and General Counsel – Legal Affairs Branch • Christine Wright – Complaint Reception Officer – Compliance Assurance Branch


Appendix B - Audit Criteria

The audit criteria within this audit program define the expectations against which the management practices for investigations were assessed. The audit criteria were designed to enable an assessment of key practices, procedures and controls in place within OCOL. The criteria have been linked to the Framework of Core Management Controls and Audit Criteria (CMC – May 2010) and the Management Accountability Framework (MAF VIII).

Audit Criteria Audit Sub-Criteria Frameworks References

1. Planning and Management

1.1 Operational plans and objectives, related to the investigations process, have been established and communicated, and are linked to OCOL’s mandate and identified priorities.

a. Objectives have been identified and communicated for the investigations process and are intended to enable the achievement of OCOL’s priorities.

b. The operational plan has adequately considered risks and mitigation measures related to the achievement of objectives, and has been communicated.

c. The operational plan has appropriately considered the corporation services requirements (HR, IM, IT, Finance) to support the investigation process activities through an integrated planning process.

CMC: G-3, G-4, CFS-1, CFS-2, RM-1, RM-5, RM-7, ST-1, RP-1 MAF VII: 2.1, 2.3, 2.4, 3.1, 3.2, 9.3

1.2 Roles and responsibilities for the investigations process are well defined, clearly communicated and effective.

a. Responsibilities for which investigation staff are held accountable are effective, formally defined and clearly communicated through job descriptions, policies, service standards, procedures, performance appraisals or other means.

b. The investigations process, plans, priorities and ongoing investigations are communicated to the audit group and the communications group to ensure proper coordination, to minimize duplications and avoid possible contradictions.

CMC: G-2, PPL-5, AC-1, CFS-1 MAF VII: 9.1

1.3 The organizational structure for the investigations process is appropriate and conducive to the achievement of OCOL objectives.

a. Human resource capacity and capabilities are assessed on a regular basis, and corrective measures are implemented as appropriate.

b. Lines of reporting in the organizational structure are clear, maintain an appropriate balance between managerial and operational positions, and are conducive to the achievement of OCOL’s objectives.

CMC: PPL-1, PPL-2, AC-3, LICM-4 MAF VII: 3.2



2. Policies and Procedures

2.1 A comprehensive policy framework has been established for the investigations process, and is supported by appropriate tools and a training and awareness program.

a. Policies have been developed to address the proper administration of the investigations process based on legislative and other requirements. These policies are regularly assessed for their relevance and are updated as required.

b. Appropriate guidelines, templates and tools are provided to investigation staff to ensure the investigations process requirements are meet.

c. Appropriate training is provided to investigation staff and is consistent with their roles and responsibilities.

CMC: PP-2, PP-4, PPL-4, G-4, AC-1 MAF VII: 10.4, 20.1, 20.2

2.2 Investigations are resolved in an efficient and effective manner, addressed consistently and according to policy and procedures, across intake channels and between investigation staff.

a. There is a triage process that can identify complaints that are relevant and not-relevant to OCOL’s mandate.

b. Criteria for the choice between the facilitated process and the formal process is clear, logical, applicable, documented and followed.

c. Investigations are conducted in a consistent and appropriate fashion, with the correct information being kept on file and provided to the complainant.

d. There is a process in place to review outstanding investigations and ensure they are resolved in a timely manner.

e. Key deliverables (i.e. final investigation reports) are properly reviewed and approved prior to being sent to the complainant and the targeted institution.

CMC: PP-4, PPL-4, CFS-1, CFS-2, RM-5

2.3 OCOL has made the complaint filing process well known to the public and easy to navigate and understand.

a. Formal communication processes / mechanisms exist and support sharing of timely, relevant and reliable information to the public about the investigations process.

b. Individuals are able to access the complaint filing process through multiple service delivery channels that are easy to understand and navigate.

c. Client service standards are established, communicated to the public and monitored by OCOL management.

CMC: CFS-1, CFS-2, CFS-4, CFS-5 MAF VII: 20.1, 20.2

3. Monitoring and Evaluation

3.1 Performance standards are defined and there is an overall quality assurance and continuous improvement process related to the

a. Appropriate performance standards for the investigations process have been defined, performance is being tracked based on these measures, and corrective measures are taken as appropriate.

CMC: G-1, G-6, ST-17, CFS-2, RP-2, RP-3, RP-4



investigations process. b. The investigations process is monitored on a regular basis to ensure the quality and consistency of the process.

c. Management is informed of the results of monitoring and evaluation activities and the investigations process is modified as needed.

MAF VII: 2.1, 2.3, 2.4, 7.1

3.2 A “no-surprise” environment allows OCOL to be prepared for media coverage.

a. A process is in place to consistently identify publicly sensitive complaints and promptly communicate this to management.

b. Clear criteria exist to support identification of publicly sensitive complaints.

c. An effective reporting process and tools are in place (media lines).

d. Management has access to regular and ad-hoc reports to monitor the investigations process.

CMC: G-1, G-5, G-6, PPL-4, CFS-1 MAF VII: 9.2, 20.1, 20.2

www.deloitte.ca Deloitte, one of Canada's leading professional services firms, provides audit, tax, consulting, and financial advisory services through more than 8,000 people in 56 offices. Deloitte operates in Québec as Samson Bélair/Deloitte & Touche s.e.n.c.r.l. Deloitte & Touche LLP, an Ontario Limited Liability Partnership, is the Canadian member firm of Deloitte Touche Tohmatsu Limited.

Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms, each of which is a legally separate and independent entity. Please see www.deloitte.com/about for a detailed description of the legal structure of Deloitte Touche Tohmatsu Limited and its member firms.

© Deloitte & Touche LLP and affiliated entities.

http://www.deloitte.com/about

audit of investigation practices office of the ... · this includes roles and responsibilities, the...

Documents