how to prepare for a virtual federal audit – and survive it!!
TRANSCRIPT
2013 NACTEI CONFERENCE
HOW TO PREPARE FOR A VIRTUAL FEDERAL AUDIT – AND SURVIVE IT!!
Sarah GalliherKY DOE, Office of Career and Technical Education
2020 Capital Plaza Tower500 Mero StreetFrankfort KY(502) [email protected]
Sarah
• Has worked with student data reporting for 24 years
• Has worked with Perkins accountability reporting (state level) 21 years
• Worked with programmer to develop/implement/maintain web-based system currently used to collect all secondary and postsecondary CTE data for Kentucky
• Consolidated Annual Report (CAR)• Local Applications• Plans for Improvement• Perkins Allocations• LEA Audits• Prepared for federal audits• Career Readiness
STEPS IN AUDIT PROCESS:
• (Mock Audit – not required but very beneficial)
• Notification of audit (date, type of audit etc.)
• Assemble state audit team members and assign tasks (preparation and participate in audit interviews)
• Collect/organize supporting documentation
• Post supporting documentation to sharepoint
• Federal audit team initial web meeting
STEPS IN AUDIT PROCESS continued:• Federal audit team interviews state
team members and selected LEAs• Respond to federal audit team’s
questions or requests for additional supporting documentation• Federal audit team exit interview with
preliminary findings• Formal notification letter of findings
from federal audit team• State responds to findings• Final notification letter from federal
audit team
This document outlines the evidence that each State needs to compile for its on-site visit by the Division of Academic and Technical Education (DATE) monitoring team pursuant to the Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV). Below are a few important notes regarding this document and the on-site monitoring visit.
This document includes the minimum evidence each State will need to produce for its monitoring visit. The DATE monitoring team may ask for additional pieces of evidence while on-site.
FROM THE LETTER FROM DATE/OVAE CONCERNING THE AUDIT:
INTRODUCTION
Unless otherwise specified, evidence should pertain to the current program year. For example, a monitoring visit conducted in October 2009, would require documents pertaining to the State’s Program Year (PY) 2009 grant award, covering the period from July 1, 2009, through June 30, 2010.
As part of the review of State’s programs of study, up to two DATE monitoring team members will conduct local site visits. Further information and expectations for local visits will be provided to the State at least one month prior to the visit.
Help us achieve a “green” visit. To the extent possible and practicable, provide access to original State documents and files, or to documents available electronically. If you have questions pertaining to any aspect of the monitoring visit, please contact Edward Smith, Program Administration Branch Chief, at [email protected] or Len Lintner, Perkins Monitoring Team Lead, at [email protected].
Kentucky Virtual Audit Visit / Virtual Review Details
Team Members/Module Responsibility:Marilyn Fountain: (A) Programs of Study; (B) Special Populations (Program Compliance and Quality)Denise Garland: (A) Accountability; (B) Special Populations (Student Outcomes and Performance)Andrew Johnson: Fiscal ResponsibilityAllison Hill: (A) State Administration; (B) Local Applications—(Team Lead)
From the letter, continued
Logistics:The team lead, along with David Preve, will meet with the Kentucky State Director or Acting Director to discuss technology and electronic delivery of the visit—the meeting participants will determine the best method for submitting documentation to our office. (Week of April 16-20, 2012.)
OVAE monitoring team will meet to discuss plans/logistics for the virtual visit (Week of April 30-May 4 or May 7-11, 2012)
We will secure the VTC room for the opening session, to be held May 21—time, TBA. (Awaiting feedback from David Preve)
We will secure the VTC room for the closing session, to be held May 25—time, TBA. (Awaiting feedback from David Preve)
(Optional, though highly recommended…) The team members will meet with the Kentucky State staff to discuss the gathering of evidence/virtual site visit logistics (Week of April 30-May 4 or May 7-11, 2012)
• The Virtual visit will be conducted within OVAE to conduct the review for the week of May 21-25, 2012.
• Each team member will be given a list of Kentucky staff members overseeing their respective packages at the State level. During the week, individual team members will be responsible for developing schedules to conference with State team members throughout the course of the week, as necessary.
• The OVAE team lead and Kentucky State Director will develop a general schedule for the week of the virtual review.
Week of Visit:
The opening session will be held the morning of May 21st. Post the opening session, OVAE monitoring team members will sequester external to OVAE in order to focus on the week-long review.
OVAE monitoring team members will, individually, conference with Kentucky State team members throughout the week, as needed—team members must schedule with respective State leads.
Also, OVAE team members will be personally responsible for securing and additional documentation during the week of the visit—PDF/FAX/etc.
Week of Visit:
The monitoring team lead will meet with team members on Thursday, May 24, 2012, 1:00 pm to discuss potential findings and outstanding issues regarding the review.
The OVAE team lead will conference with the Kentucky State Director on Thursday, May 24, 2012, 3:00 pm to discuss findings and outstanding issues.
The OVAE monitoring team will conduct the closing session on the morning of May 25, 2012.
KENTUCKY AUDIT TEAM MEMBERS• State CTE Director• Commissioner (Office of Career and
Technical Education)• Those who work with:• State administration• Fiscal Programs• Local Applications• Programs of Study• Tech Prep Programs• Special Populations• Accountability• State Plan• Time/Effort Reporting
Areas reviewed:•Administration•Fiscal Program Responsibility•Local Application•Programs of Study•Special Populations•Accountability
State Administration•Eligible Agency Responsibilities•State Plan Responsibilities•State Plan Development•State Leadership Activities
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
1. Section 121(a) -Eligible Agency Responsibilities
1.1 State policies and procedures for coordinating the development, submission, and implementation of the State plan.
1.2 State policies and procedures for coordinating the evaluation of the program, services, and activities within the State plan.
1.3 Documentation of eligible agency consultation on the State plan with the Governor and other agencies listed in section 121(a)(2).
1.4 Copies of agendas for meetings of the eligible agency (no less than 4 times per year and as consistent with State law and procedure for the conduct of such meetings).
Part A: State Administration
Fiscal Program Responsibility
•Within State Allocation
•Matching Requirement
• Reserve
• LEA Allocation Procedures
• Area Career and Technical Education Schools
• Consortia Requirements
• Postsecondary Allocation Procedures
• Redistribution of Unobligated Funds (Carry Forward)
•Maintenance of Fiscal Effort
• Audit and Fiscal Requirements
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
1. Section 112(a) – Within State Allocation
1.1 Grant award notifications (GANs) for Title I and Title II funds.
1.2 Ledger of expenditures for Title I and Title II grants that reconcile to the most recent “final” financial status reports (FSRs) submitted to OVAE the previous December 31. (Note: These expenditures should cover the entire 27 month grant period).
1.3 Interim and final FSRs that were submitted to OVAE the previous December 31.
Part B: Fiscal Program Responsibility
Local Applications•Requirements for Local Applications•Required Use of Local Funds•Permissible Use of Local Funds•EDGAR
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
3. Section 134(b)(1-12) - Requirements for Local Applications
1.1 Copy of local application form(s) for eligible recipients.
1.2 Copy of the form (checklist) used by the eligible agency to review local applications.
1.3 State policies and procedures governing the use of funds by eligible recipients to improve career and technical education programs (if contained in a document other than the local application itself).
Part C: Local Applications
Programs of Study• State Plan Contents
• Local Application Requirements/ Required Local Uses of Funds
• Permissible Uses of Funds
• Additional Material (State legislation/policies, partnerships, statewide articulation agreements, guidance programs, professional development, student assessments, evaluation/ accountability, local implementation
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
1. Section 122(c)(1)(A)(i-iv) – State Plan Contents
1.1 Documentation of the eligible agency’s career and technical programs of study that may be adopted by eligible recipients that—
a. Incorporate secondary education and postsecondary education elements.
b. Include coherent and rigorous content aligned with challenging academic standards and relevant career and technical content in a coordinated, non-duplicative progression of courses that align secondary to postsecondary education.
c. May include opportunity for secondary education students to gain postsecondary education credits through dual or concurrent enrollment programs or other means.
d. Lead to an industry-recognized credential or certificate at the postsecondary level or an associate or baccalaureate degree.
Part D: Programs of Study
Part E: Tech Prep
Kentucky rolled its Tech Prep funds in with its Basic Grants funds and so had no documentation to submit to the federal auditing team.
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
1. Section 203 (c) - Requirements for Consortia Applications
1.1 Copy of the application (form) for local consortia.
1.2 Copy of the form (checklist) used by the eligible agency to review consortia applications.
1.3 State policies and procedures governing the use of funds by eligible consortia (if contained in a document other than the application).
Part E: Tech Prep Programs
*Note: This section is applicable only if the State opts to retain all or a portion of its Title II tech prep funds for activities under Title II of the legislation.
Special Populations
•State Plan•State Leadership Activities•Local Plan / Local Uses of Funds•State Report / State Improvement Plan•Local Report / Local Improvement Plan•Additional Material (State-level initiatives/ activities)
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
1. Section 112(c) – State Plan
1.1 Documentation of programs that will prepare career and technical education students academically and technically for opportunities in postsecondary education or entry into high skill, high wage, or high demand occupations, including students who are—a. Members of the special population categories listed in
section 3(29).b. Enrolled in alternative education programs.c. Being served in State correctional institutions.
Part F: Special Populations
*Note: Most of the items of evidence in this section overlap with items requested elsewhere in this checklist. In those instances, the State is not required to duplicate evidence for this part.
Accountability
State Performance MeasuresGeneral population of studentsRace / GenderProgram AreasEach of the 7 Special Populations
State Plan
Permissive Uses of State Leadership Funds
State Report / State Improvement Plan
Local Performance Measures
Local Plan
Local Report / Local Improvement Plan
Management Controls (State Policies & Procedures, Training, Perkins data used for State and local program
improvement)
Accountability continued
SECTION EVIDENCE YES/NO COMMENTS/CONCERNS
1. Section 113(b) – State Performance Measures
1.1 Documentation of input from eligible recipients in establishing performance measures for the State for the core indicators of performance (core indicators) listed in section 113(b)(2)(A-B) and additional indicators pursuant to section 113(b)(2)(C).
1.2 Copy of student definitions and measurement approaches for the core indicators and additional indicators, along with any annual revisions made to the definitions or measures.
1.3 Documentation of State-adjusted levels of performance for each of the core indicators and additional indicators.
Part G: Accountability
The OVAE audit team appreciated that Kentucky had their documentation arranged and identified by the order in the ‘Checksheet for Monitoring Grants’ document.
A master folder was created for each of the 7 categories (State Administration, Fiscal Program Responsibility etc.).
Subfolders were created within each major category for each subcategory (1.1, 1.2 etc..)
The folders were named to match the numbering system used in the Checksheet.
n.
• Official results of last federal audit, with findings and corrective actions taken.
• The audit team will review the results from the last audit, checking to see if the State corrected any findings or action items
• Official State Plan document
• State Perkins Policy and Procedure Manual
• Samples of communication with LEAs, such as informational emails, web postings etc.
• Samples of technical assistance given to LEAs, such as Word documents, emails etc.
Documentation To Provide to the Audit Team
The following screenshots show how Kentucky organized their documentation on a shared network drive.
Each team member posted the documentation into the appropriate folder for each check-sheet item for which he/she was responsible.
The audit team leader could then monitor progress toward completing the documentation or review documents as they were posted.
Master folder with subfolders for each of the 7 major categories. Each major category folder contains subfolders for each item.
Subcategory for Part G: Accountability with subfolders for each item on Checksheet
Subcategory 9.1b of Part G: Accountability with documents posted in folder
Example of Accountability Measures summary report for a school chosen by the federal audit team for interviews
The federal audit team member responsible for Part G: Accountability requested access to our state CTE data reporting system (which was given).
Once documentation is complete, each folder was moved to a sharepoint that the federal audit team and the state audit team could access.
And then the audit began!!!
The initial meeting took place by video. The state audit team met in one room and were situated so the federal audit team could see everyone. The federal audit team did the same.
After initial introductions and greetings, the agenda for the audit was reviewed and questions answered.
Kentucky’s State Director then did a 20-minute presentation about CTE in Kentucky, highlighting the current activities and projects for improving CTE in our state.
This is your opportunity to showcase your activities.
Individual members of the federal audit team arranged to interview corresponding members of the State audit team as well as individuals at the selected LEAs. Questions were answered and additional documentation provided when requested.
The leader of the federal audit team met with Kentucky’s State Director to review the preliminary results of the audit, including any findings.
The exit interview was also by video. All members of both the State and federal audit teams were in the meeting.
‘Stellar’ documentation – well organized and well orchestrated
‘Kentucky set the bar high’ as the first state to have a virtual audit.
General Comments:
Exit Interview Results
AdministrationFINDINGS: NONE
‘Impressed’ with CTE partnerships with agencies, business & industry, secondary to postsecondary, advisory committees and included in CTE reform
‘Impressed’ with the program assessment process
‘Impressed’ with CTE professional development
Kentucky is meeting all requirements for local eligible agencies and State plan
Administration Strategies:
Provide an electronic policies & procedures guide with all documents in one place (website)
FINDINGS: NONE Approved of using the First-in/First-out
accounting system Accommodations for spreadsheet and overall
MOE Liked documentation on matching funds and
may use as model for other statesFiscal Program Responsibilities Strategies
• Develop a financial policies & procedures manual. Incorporate into State policies & procedures manual.
• Incorporate ‘rural, high % of CTE students or high # of CTE students’ into policies and procedures documentation for reserved fund
• Second calculation on Maintenance of Effort using per student spending as alternate possibility for meeting required level of spending
Fiscal Program Responsibility
FINDINGS: NONE Continue opportunity for local recipients to use % of
funds for important CTE initiatives Control amount used for travel to allow more $ to
benefit programs Review/document how local application
programmatic piece ties into the policies and procedures OCR review and facilities review.
Checklists used in monitoring needs back-up documentation in the policies and procedures document.
Local Application Strategies Set floors/ceilings for local application (use % of
funds for certain initiatives) Refine the definition of size/scope/quality in P& P
documentation
Local Applications
FINDINGS: NONE
‘Very impressed’, ‘wonderful’
10 components - Kentucky went well beyond required
Especially liked partnerships with employers, dual credit
Commended Kentucky Occupational Skills Standards Assessment (KOSSA), administered as an end of program assessment to all preparatory senior CTE students as it ensures quality in CTE programs
Share with all states how to offer all clusters at all sites
Programs of Study
FINDINGS: NONE
Liked the inclusion of the special populations in all CTE activities
Missed the 90% on some secondary/ postsecondary performance measures
Accommodation for special populations training included in NTI
Special Populations Strategies:
Provide a web portal where best practices could be shared
Link program accountability to performance
Add ‘including special populations; on narrative of local application questions
Special Populations
FINDINGS: NONE Consolidated Annual Report: ‘Exceptional job
with performance measures’ reports
Need reason for difference in TEDS reports and data submitted to CAR by EDEN files. (Some school numbers not identified in EDEN prevented data from being included in EDEN files submitted for the CAR.)
‘Fine and dandy’
Accountability
Kentucky paid Brustein & Manasevit to perform a ‘mock’ audit to help us prepare for our federal audit. Several areas were identified that would likely cause ‘findings’ during the actual audit.
Each area identified in the ‘mock’ audit had been corrected or was in the process of being corrected before the federal audit began.
As the previous slides demonstrated, it was well worth the cost.
The following screens show the action items recommended to correct the findings from the ‘mock’ audit.
Kentucky’s ‘Mock Audit’
‘MOCK’ AUDIT INFORMATION AND RESULTSPREPARATION FOR MONITORING VISIT
Select main contact for visit: responsible for all logistical communications w/OVAE (documentation, setting up interviews, etc.)
Request an entrance/exit meeting with OVAE
Identify key personnel
Monitoring documents and evidence must be easily accessible and organized.
Be prepared to address areas of noncompliance.
Review previous monitoring findings and recommendations.
Prepare subrecipients selected for monitoring so that appropriate documentation is available and the LEA has reviewed the monitoring tool.
Potential Areas of Noncompliance:
‘In my review of Kentucky’s administration of the Perkins grant program, I found many commendable career and technical education activities and services taking place within the State. I was particularly impressed by Kentucky’s data management system, the Technical Education Database System (TEDS).’ Administrative Costs:Kentucky must discontinue the practice of taking indirect costs out of the leadership set-aside, unless those costs are included in the State administration allocation and count toward the cap. Timely Obligation of FundsWhile, technically Kentucky is not out of compliance as all funds are spent within the period of availability, high carryover balances are considered a risk factor by the U.S. DOE’s Risk Management Service and indicative of poor internal controls. Kentucky should budget to ensure funds are spent within the period of obligation to reduce high carryover amounts.
Time and Effort ReportingKentucky should review the job descriptions of employees paid for in whole or in part with administrative funds to ensure that all activities performed are allowable under the administrative set-aside. Any employee that performs non-administrative activities works on multiple cost objectives and therefore must keep a PAR. Further, OCTE should have a sample PAR prior to OVAE monitoring and should continue to develop comprehensive time and effort policies and procedures. All staff should be trained on the policies and procedures. Finally, Kentucky must monitor subrecipients to ensure that subrecipients are also keeping compliant time and effort records.
Monitoring of Eligible RecipientsOCTE should draft monitoring procedures that clearly define the process to be used and the responsibilities for all agencies involved. The procedures should indicate which agencies are responsible for the fiscal and programmatic monitoring of all subrecipients, how often subrecipients will be monitored, and the monitoring process including follow-up procedures. OCTE must determine whether onsite monitoring visits will be part of the policy or whether the State’s monitoring will include only desk reviews. OCTE should develop a monitoring tool to be used in Perkins monitoring.
Reserve
While it appears that Kentucky’s use of the reserve funds is compliant, Kentucky should consider adding a sentence to next year’s request for proposal that specifically states that the funds are to be used for activities in (1) rural areas; (2) areas with high percentages of CTE students; and (3) areas with high numbers of CTE students.
Consortia
OCTE should review its budget histories of consortia members to ensure that funds to consortium members are being used for joint purposes and are not be subgranted to individual members based on the formula-generated amount. OCTE should review its policies and procedures, and if needed, make revisions, to ensure that the policies and procedures are compliant with this requirement. OCTE or KDE should monitor consortiums to ensure that funds to consortia are not being further subgranted to individual members based on their formula-generated amount.
If your state will be having a virtual federal audit, please feel free to contact Kentucky if you have any questions. Each of our audit team members would be glad to help. Address your inquiries to me and I will pass them to the appropriate person.
Sarah GalliherKY DOE, Office of Career and Technical Education
2020 Capital Plaza Tower500 Mero StreetFrankfort KY(502) [email protected]
Questions?
?
?
??
?
??
