services rfp - ohio · web viewthe ohio department of job and family services is requesting...

43
NOTICE This opportunity is being released to Professional Technical Services Contractors pre-qualified as a result of the Open Market RFP #0A1183. ONLY Contractors pre-qualified in Category 6.0 the Native Mobile App for iOS and Android Category are eligible to submit proposal responses AND to submit inquiries. The State does not intend to respond to inquiries or to accept proposals submitted by organizations not pre-qualified in this Technology Category. An alphabetical listing of Contractors pre-qualified to participate in this opportunity follows: Advocate Consulting Group Conduent State & Local Solutions Page 1 of 43

Upload: dongoc

Post on 29-Jun-2018

213 views

Category:

Documents


0 download

TRANSCRIPT

NOTICEThis opportunity is being released to Professional Technical Services Contractors pre-qualified as a result of the Open Market RFP #0A1183.

ONLY Contractors pre-qualified in Category 6.0 the Native Mobile App for iOS and Android Category are eligible to submit proposal responses AND to submit inquiries. The State does not intend to respond to inquiries or to accept proposals submitted by organizations not pre-qualified in this Technology Category.

An alphabetical listing of Contractors pre-qualified to participate in this opportunity follows:

Advocate Consulting Group

Conduent State & Local Solutions

Page 1 of 33

0A1183 Task Order – OMJ Registration Mobile Application – PTJFS-18-06-003

DATE ISSUED: September 1, 2017

The Ohio Department of Job and Family Services is requesting proposals for work leveraging Contract 0A1183 Professional Technical Services: System Operations and Maintenance for Employment Services Systems. The Terms and Conditions of Contract 0A1183 apply to this Task Order Solicitation:

PROJECT: OMJ Mobile Registration Application

POST DATE: September 1, 2017

PROPOSAL DUE DATE: September 22, 2017, at 1:00 p.m.

OPENING LOCATION: Department of Job and Family Services Office of Contracts and Acquisitions 30 East Broad Street, 31st Floor Columbus, Ohio 43215

Organization. This Task Order Solicitation consists of an executive summary, four (4) attachments and two (2) supplements. The attachments and supplements are listed below. The requirements for a Proposal's contents and formatting are contained in this Task Order Solicitation. We will also be using the evaluation and award process outlined in this Task Order Solicitation.

Attachments:Attachment One Evaluation CriteriaAttachment Two Work Requirements and Special ProvisionsAttachment Three Task Order Response

Submission RequirementsAttachment Four Personnel Profile Summary

Supplements:Supplement One OMJ System Registration Screen-Shots and Flow ChartsSupplement Two Research Demographic Information

Page 2 of 33

EXECUTIVE SUMMARYThe implementation of OhioMeansJobs (OMJ) by the Ohio Department of Job and Family Services (ODJFS) resulted in significant changes in the way that the State processes job seeker registration, and in the way job seekers, employers and staff interact. The Rapid Response staff cannot meet the demands of all the workplace downsizing events, and the Office of Workforce Development (OWD) needs to expand the use of technology to be more efficient in implementing Workforce Programs.

The existing OMJ website is built using the following tools and technology platforms:• Java 2 Enterprise Edition.• IBM WebSphere Application Servers.• IBM HTTP Web Servers.• Load Balancing.• Novell e-Directory Authentication.• Database – Oracle • Integrated Development Environment - Rational Application Developer.• Source Control: Dimensions.

The infrastructure information provided above is informational. The solution proposed must use the KONY platform for building the OMJ Registration Web Applications. The native/hybrid web application must be available for use no later than June 30, 2018.

Objectives. Successfully design, develop, test, and implement a native/hybrid web application solution for smart phones to enable State of Ohio staff to quickly register job seekers in OMJ by getting key information from the individual’s driver’s license or state issued ID.

Calendar of Events. The schedule for the Task Order Solicitation process and the Work is given below. The State may change this schedule at any time. If the State changes the schedule before the Proposal due date, it will do so through re-release of the Task Order Solicitation to the appropriate Contractors pre-qualified for the Category 6.0 the Native Mobile App for iOS and Android Category. After the Proposal, due date and before the award of the Task Order, the State will make schedule changes through the Task Order Solicitation amendment process. It is each prospective Pre-Qualified Contractor’s responsibility to keep up with current information regarding this Task Order Solicitation and its Calendar of Events through award of the Work.

Dates:Task Order Solicitation Issued: September 1, 2017Inquiry Period Begins: September 1, 2017Inquiry Period Ends: September 15, 2017Proposal Due Date: September 22, 2017, at 1:00 p.m.Award Date: October 1, 2017 (estimated)Work Begins: October 16, 2017 (estimated)

Page 3 of 33

Inquiry ProcessPre-Qualified Contractors may make inquiries regarding this Task Order Solicitation anytime during the inquiry period listed in the Calendar of Events. To make an inquiry, Pre-Qualified Contractors must use the following process:

Access the State’s Procurement Website at http://procure.ohio.gov/; From the Quick Links menu on the right, select “Bid Opportunities Search”; In the “Document/Bid Number” field, enter the Task Order Solicitation ID number found on the first page

of this Task Order Solicitation; On the Opportunity Search Results page, click on the hyperlinked Bid Number; On the Opportunity Details page, click the “Submit Inquiry” button; On the document inquiry page, complete the required “Personal Information” section by providing:

o First and last name of the prospective Pre-Qualified Contractor’s representative (the Pre-Qualified Contractor Representative) who is responsible for the inquiry,

o Name of the prospective Pre-Qualified Contractor, o The Pre-Qualified Contractor Representative’s business phone number, and o The Pre-Qualified Contractor Representative’s email address;

Type the inquiry in the space provided including: o A reference to the relevant part of this Task Order Solicitation, o The heading for the provision under question, and o The page number of the Task Order Solicitation where the provision can be found;

Enter the Confirmation Number at the bottom of the page Click the “Submit” button.

A Pre-Qualified Contractor submitting an inquiry will receive an acknowledgement that the State has received the inquiry as well as an email acknowledging receipt. The Pre-Qualified Contractor will not receive a personalized response to the question nor notification when the State has answered the question.

Pre-Qualified Contractors may view inquiries and responses on the State ’s Procurement Website by clicking the ”View Q & A” button on the document information page.

The State usually responds to all inquiries within three business days of receipt, excluding weekends and State holidays. But the State will not respond to any inquiries received after 8:00 a.m. on the inquiry end date.

The State does not consider questions asked during the inquiry period through the inquiry process as exceptions to the terms and conditions of this Task Order Solicitation.

Page 4 of 33

ATTACHMENT ONE: EVALUATION CRITERIA

Scored Criteria. In the technical evaluation phase, the State will rate the technical merits of theProposals based on the following requirements and the weight assigned to each requirement as defined in this Task Order Solicitation. For this Task Order Solicitation, a reference project will be of similar size and scope if it delivers functionality across Android and iOS mobile platforms, has a need to connect real- time to a live transactional based system, and has more than five (5) screens.

Personnel criteria that are to be scored are named individuals that the Pre-Qualified Contractor must provide for key positions of the proposed team. Identified key staff may, at the Pre-Qualified Contractor’s discretion, fulfill more than one project role, dependent upon sufficient and appropriate experience.

Scored Criteria Weight DoesNot Meet Meets Exceeds Greatly

ExceedsPersonnel Requirements

PROJECT MANAGER - RequirementsMust have at least thirty-six (36) months of project management experience on a project of similar size and scope.

5 0 5 7 9

Must have thirty-six (36) months of experience or three (3) projects with project management responsibilities for native/hybrid web applications.

5 0 5 7 9

Mobile User Experience (UX) Lead – RequirementsMust have twenty-four (24) months of experience as a mobile user experience (UX) design lead on at least two (2) projects of similar size and scope.

5 0 5 7 9

Must have at least three (3) years of experience in designing/architecting mobile user interfaces.

5 0 5 7 9

Mobile Test Lead – RequirementsMust have twenty-four (24) months of experience as a mobile test lead on at least two (2) projects of similar size and scope.

5 0 5 7 9

Must have at least thirty-six (36) months of experience in testing mobile user interfaces.

5 0 5 7 9

Mobile Application Lead – RequirementsMust have twenty-four (24) months of experience as a mobile application development lead on at least two (2) projects of similar size and scope.

5 0 5 7 9

Must have at least thirty-six (36) months of experience in designing/building native/hybrid web applications for smartphones and mini-tablet

5 0 5 7 9

Page 5 of 33

Solution Criteria:

Scored Criteria Weight DoesNot Meet Meets Exceeds Greatly

ExceedsProposed Solution

The proposal must describe how the Pre-Qualified Contractor will use KONY web application framework to build the mobile application and explain the advantages/disadvantages for

15 0 5 7 9

The proposal must include an approach to make existing system users aware of the mobile web solution, which must include, but not be limited to, establishing Quick Response (QR) Code, and approaches for making the application available for state staff.

5 0 5 7 9

The proposal must describe how the proposed solution will prevent unauthorized application and data access.

10 0 5 7 9

The proposal must include an approach for collecting web analytics to get the visibility into use of the native/hybrid web application.

5 0 5 7 9

The proposal must describe how the Pre-Qualified Contractor will create an intuitive, flexible, and interactive native/hybrid web application that is easy to navigate, visually appealing, easy to maintain, and easy to administer.

10 0 5 7 9

The proposal must describe the approach for thoroughly testing the native/hybrid web application for compatibility with major mobile platforms (iOS and Android) and mobile devices (smartphones, mini-tablet and tablets).

10 0 5 7 9

The proposal must describe how the mobileweb application will use the industry best practices for data security and privacy protection (NIST Special Publication 800-53Revision 3).

10 0 5 7 9

The proposal must describe how knowledge transfer to ODJFS will be addressed, and highlight the approach to transition including tools and technologies that are involved.

5 0 5 7 9

The proposal must describe how the native/hybrid web application will support 24/7/365 availability, how it will leverage the current OMJ registration interface (Supplement One and Two), and feature set to ensure as, to the extent possible, be a single common application base for all technology.

10 0 5 7 9

Price Performance Formula. The evaluation team will rate the Proposals based on the following criteria and respective weights.

Page 6 of 33

Criteria PercentageTechnical Proposal 70%

Cost Summary 30%

To ensure the scoring ratio is maintained, the State will use the following formulas to adjust the points awarded to each perceptive Pre-Qualified Contractor.

The Pre-Qualified Contractor with the highest point total for the Technical Proposal will receive 700 points. The remaining Pre-Qualified Contractors will receive a percentage of the maximum points available based upon the following formula:

Technical Proposal Points = (Pre-Qualified Contractor’s Technical Proposal Points/Highest Number of Technical Proposal Points Obtained) x 700

The Pre-Qualified Contractor with the lowest proposed total cost for evaluation purposes will receive 300 points. The remaining Pre-Qualified Contractors will receive a percentage of the maximum cost points available based upon the following formula:

Cost Summary Points = (Lowest Total Cost for Evaluation Purposes/ Pre-Qualified Contractor’s Total Cost for Evaluation Purposes) x 300

Total Points Score: The total points score is calculated using the following formula:

Total Points = Technical Proposal Points + Cost Summary Points

Submission Instructions & Location

Each Pre-Qualified Contractor must submit five (5) complete, sealed and signed copies of its Proposal Response and each submission must be clearly marked “OMJ Registration Mobile Application” on the outside of its package along with Pre-Qualified Contractor’s name. A single electronic copy of the complete Proposal Response must also be submitted with the printed Proposal Responses.

Each Proposal must be organized in an indexed binder ordered in the same manner as the response items are ordered. The State wants clear and concise proposals, but Pre-Qualified Contractor must answer questions completely and meet all the Task Order’s requirements. If there is a discrepancy between the hard copy and the electronic copy of the proposal, the hard copy will control, and the State will base its evaluation of the offeror’s proposal on the hard copy. Each proposal must contain an identifiable tab sheet preceding each section of the proposal. Proposal Response must be good for a minimum of 60 days.

The State will not be liable for any costs incurred by any Pre-Qualified Contractor in responding to this Task Order Solicitation, even if the State does not award a Task Order through this process. The State may decide not to award a Task Order at the State’s discretion. The State may reject late submissions regardless of the cause for the delay. The State may also reject any submissions that it believes is not in its interest to accept and may decide not to do business with any of the Pre-Qualified Contractors responding to this Task Order Solicitation

Proposal Responses MUST be submitted to the State Agency ’s Procurement Representative:Jay EasterlingOhio Department of Job and Family ServicesOffice of Contracts and Acquisitions

Page 7 of 33

30 East Broad Street, 31st Floor Columbus, Ohio 43215

Proprietary information

All Proposal Responses and other material submitted will become the property of the State and may be returned only at the State's option. Proprietary information should not be included in a Proposal Response or supporting materials because the State will have the right to use any materials or ideas submitted in any quotation without compensation to the Pre-Qualified Contractor. Additionally, all Proposal Response submissions will be open to the public after the Task Order has been awarded.

The State may reject any Proposal if the Pre-Qualified Contractor takes exception to the terms and conditions of the Contract.

Waiver of Defects

The State has the right to waive any defects in any quotation or in the submission process followed by a Pre-Qualified Contractor. But the State will only do so if it believes that is in the State's interest and will not cause any material unfairness to other Pre-Qualified Contractors.

Rejection of Submissions

The State may reject any submissions that is not in the required format, does not address all the requirements of this SOW Solicitation, or that the State believes is excessive in price or otherwise not in its interest to consider or to accept. The State will reject any responses from companies not pre-qualified in the Technology Category associated with this SOW Solicitation. In addition, the State may cancel this SOW Solicitation, reject all the submissions, and seek to do the work through a new SOW Solicitation or other means.

Page 8 of 33

ATTACHMENT TWO: WORK REQUIREMENTS AND SPECIAL PROVISIONS

PART ONE: WORK REQUIREMENTSThis attachment describes the Work and requirements of the Pre-Qualified Contractor. It also describes what the Pre-Qualified Contractor must deliver as the completed Work (the "Deliverables").

Background. Although specific work reviews, and meetings with State staff will need to be performed at ODJFS locations, ODJFS prefers the development team is offsite. Only those Pre-Qualified Contractor staff needing constant contact with the business or State staff are candidates to be housed at the ODJFS site.

The normal working hours for the ODJFS staff are 8:00 a.m. to 5:00 p.m., Monday through Friday, with a one-hour lunch period for a total of eight (8) working hours per day. Pre-Qualified Contractor may have to work under unusual working conditions which may include operation of a computer terminal for long periods of time, working in excess of eight hours per day, working on Saturdays, Sundays, and State holidays.

Overview. ODJFS seeks a Pre-Qualified Contractor to develop a Native or Hybrid Web Application that must supplement the current registration capabilities of the Pre-Qualified Contractor following list is a starting set of the requirements. During the Agile development lifecycle, the Pre-Qualified Contractor must refine requirements.

The infrastructure information provided above is informational. The solution proposed must use the KONY platform for building the OMJ Registration Web Applications. The native/hybrid web application must be available for use no later than June 30, 2018.

The native/hybrid web application must: include the design, development, testing and implementation of the OMJ registration native/hybrid web

application solution. include barcode scanning upload ability that can lift information from either the driver’s license or the state

identification card presented to the State Staff member at a rapid response event. use the existing OMJ match/create seeker logic’s “client inbound process.” allow for fast loading and easy access to functionality by the user over a variety of popular platforms (for example,

iOS and Android), device form-factors (e.g., smartphone, mini-tablet, and tablet), and screen resolutions. upload benefit qualifying documents or other paperwork to qualify for benefits delivered to Ohio's Workforce Case

Management System (OWCMS) for storage and retrieval, where OMJ users can access the documents uploaded for each seeker.

be device agnostic and optimized for the end point device with rich user experience. support all ODJFS and State of Ohio information security policies and standards. support all of ODJFS and State of Ohio design policies and standards. Bar code scan driver’s license/state ID card from any state. For each seeker registered via the solution, the solution must create a seeker service for this rapid response event.

It must include the Rapid Response Event ID that is specific to the employer’s staff reduction. This must be accomplished by the Pre-Qualified Contractor providing modification to an existing web service interface on OWCMS.

make use of the camera on the device that drives the solution to either a native application or a hybrid application. Other strategies (including a mobile web application) that meet the requirements are acceptable.

allow users the interaction necessary to log-in to the OWCMS system and to confirm that the registration was

Page 9 of 33

successful. During requirements elicitation, the Pre-Qualified Contractor must work with OWD to determine if they need a “Data Errors” screen

supplement the data from the driver’s license/state ID card must be done to maximize the usefulness of the registration.

make use of an email address, which is a useful addition beyond the information available on the driver’s license itself.

include within the proposal an expense of a manual input interaction to collect email address and phone number(s). As the design solidifies, the State may have options for data lookups rather than manual input. Supplements One and Two provides the current OMJ registration screen-shots and data that support these tasks as well as documentation for the existing system to system registration interactions.

After the Task Order award, the Pre-Qualified Contractor must conduct an initial consultation with ODJFS. The Pre-Qualified Contractor must complete a review of the current OWCMS web site, OMJ website and example web sites from other states and organizations as further input to the design process. The Pre-Qualified Contractor must confirm audience, objectives, graphic look and feel, and content to migrate from current user interface, navigation, website banner, website branding, website marketing, technology issues and assumptions by discussion with the State review team. The Pre-Qualified Contractor must study the other websites and recommend what parts of these sites must be reused in this effort.

To ensure the successful transition of the native/hybrid web application solution to state staff for ongoing support, ODJFS will assign two (2) technical staff to this project to ease the technical ownership transition once the solution is transitioned to production. These staff members must be integrated into the Pre-Qualified Contractors development team for knowledge transfer.

Pre-Qualified Contractor Responsibilities and Deliverables. The Pre-Qualified Contractor must meet all Task Order Solicitation requirements for the Work and complete all Work satisfactorily.

Task 1 – Project Management

Maintain Work Plan. The Pre-Qualified Contractor must update the Work Plan submitted with its Proposal and submit the detailed Work Plan, in electronic and paper form, to the Project Representative for approval within ten (10) business days after the State issues a purchase order under the Contract. The Work Plan must include the project schedule that includes milestones and dates.

Additionally, the Work Plan must allow sufficient time for the State’s staff to review all Work. The State will determine the number of business days it needs for such reviews and provide that information to the Pre-Qualified Contractor after award and update of the Work Plan.

Kick Off. The Pre-Qualified Contractor and the State will conduct a kick-off meeting within ten (10) business days of the purchase order being issued.

Meeting Attendance and Reporting Requirements. The Pre-Qualified Contractor 's management approach to theWork must adhere to the following meeting and reporting requirements:

• Immediate Reporting - The Project Manager or a designee must report any staffing changes for the Work to the Project Representative within 24 hours.

• Attend Status Meetings - The Project Manager and other Work team members must attend status meetings with the Project Representative and other people deemed necessary to discuss Work issues. The Project Representative must schedule these meetings, which must follow an agreed upon agenda and allow the Pre-Qualified Contractor and the State to discuss any issues that concern them.

Page 10 of 33

• Provide Status Reports - The Pre-Qualified Contractor must provide written status reports to the ProjectRepresentative at least one full business day before each status meeting. The Pre-Qualified Contractor's proposed format and level of detail for the status report is subject to the State’s approval.

• Prepare Monthly Status Reports - During the Work, the Pre-Qualified Contractor must submit a written monthly status report to the Project Representative by the fifth business day following the end of each month. The weekly status reports, monthly status reports and project schedules do not require a Deliverable review cycle. At a minimum, monthly status reports must contain the following:

o Updated overall status of the Work in terms of the approved Work Plan (schedule and cost);o Updated Work schedule;o The plans for activities scheduled for the next month;o The status of any Deliverables;o Time ahead or behind schedule for applicable tasks;o A risk analysis of actual and perceived problems; ando Strategic changes to the Work Plan, if any.

Pre-Qualified Contractor’s Deliverables. The deliverables to be produced by the Pre-Qualified Contractor for the Project

Management Task must include the following:1. Updated Work Plan; and2. Status Reports as required.

Task 2 – Analysis, Design, and DevelopmentThe major objectives of the Analysis, Design, and Development tasks are as follows:

• Ensure that the Pre-Qualified Contractor has a thorough understanding of the Rapid Response events and Ohio Workforce Case Management system registration;

• Confirm and refine requirements specified in this Task Order Solicitation and supporting documents;• Elaborate and document the requirements of the native/hybrid web application solution;• Support and participate in requirements management;• Document the analysis of the native/hybrid web application solution;• Design the native/hybrid web application solution; Conformance to the following standards (where

applicable): State of Ohio IT Standard, ITS- SEC-02, Enterprise Security Controls Framework. ( http:/ / www.p ri v a c y .o h i o . g o v / L i n k C li c k . a s p x ? f il e t i c k et = 3a F M 0P z 7 O eU % 3 d & t a b i d = 4 0)

• Develop and test the native/hybrid web application solution.

The acceptance of the native/hybrid web application solution design must be based, in part, on the following criteria:

• Inclusion of ODJFS in the iterative process of the native/hybrid web application solution design and the acceptance of comments and feedback on development and refinements from State staff. All deliverables and test results must be reviewed and approved by ODJFS.

o The native/hybrid web application solution design must also be reviewed and approved by the ODJFS Office of External Affairs to ensure compliance with State of Ohio and ODJFS branding standards and compliance.

• Art, images and text that become a part of the native/hybrid web application solution must be clear from copyright or other usage constraints. All creative material needs to be made available to ODJFS for use in implementation. All constraints that arise that would limit complete free reuse of this material on this or other projects must be raised for ODJFS review and approval before inclusion in the design.

• An informative, intuitive, flexible, and interactive native/hybrid web application solution that is easy to update and maintain.

• A native/hybrid web application solution that facilitates easy and intuitive access to information of interest to State of Ohio staff.

• A native/hybrid web application solution designed with an intuitive and maintainable information

Page 11 of 33

structure, navigation support, and application map.• A native/hybrid web application solution designed with standardized page templates that are easy to

use, consistent, and aligned with the State brand.• A native/hybrid web application solution that clearly conveys the OMJ, State of Ohio and OWCMS

brands that are visually appealing (similar to native look and feel), has a consistent look and feel, has a consistent design, and is easy to navigate.

Pre-Qualified Contractor Responsibilities. The Pre-Qualified Contractor must complete activities consistent with its proposed methodology to accomplish the task objectives and meet all Task Order Solicitation requirements. At a minimum, completion of this task must include the following activities:

• Development of a prototype of the proposed native/hybrid web application solution and supporting documentation (wire frames, brand graphs, etc.), and screen flow;

• Templates which define the standardization of the style sheets, and the overall look and feel of the native/hybrid web application solution.

• Design document that address the security, business logic and overall design of this native/hybrid web application solution. This artifact must cover the architectural views (use case, logical, process, deployment, and implementation views) including security and data perspectives;

• A test strategy and plan addressing both functional and performance testing;• Development of a detailed plan that can be used to make existing system users aware of the mobile

web solution. The plan must include, but not be limited to, establishing Quick Response (QR) Code and approaches for making the application available for the state staff.

• As a part of this task, in addition to the other tasks outlined herein, the Pre-Qualified Contractor must provide system requirement details regarding the high availability (99.0% available) application recommendation provided in its proposal. The System design, must be based on the Pre-Qualified Contractor’s recommendation, and outline the hardware, software and network needs to provide high availability for their native/hybrid web application solution; and

• Verification the native/hybrid web application solution is ready for testing.

Pre-Qualified Contractor Deliverables. Deliverables to be produced by the Pre-Qualified Contractor for this task include:

1. Delivery of a prototype for review by ODJFS;2. Templates showing the standardized look and feel of the solution;3. Design document;4. The native/hybrid web application and complimentary OWCMS changes;5. Test strategy plan;6. Awareness approach plan;7. System requirement details, based on the Pre-Qualified Contractor’s recommendation; and8. Verification stating the native/hybrid web application is ready for testing.

Task 3 – Testing

The native/hybrid web application solution must be subjected to integration, system testing, and user acceptance testing. Testing must occur in an established test environment that mirrors the production environment. The Pre-Qualified Contractor must function as users during Pre-Qualified Contractor testing phases and the State will evaluate all test outcomes. The Pre-Qualified Contractor must direct testing and operate the native/hybrid web application solution in accordance with the testing plans. The Pre-Qualified Contractor must provide all error resolution and other technical support as required. The Pre-Qualified Contractor must correct all defects.

• The Pre-Qualified Contractor must complete a comprehensive integration test prior to approval for moving

Page 12 of 33

into the User Acceptance Test (UAT) phase. All issues identified must be tracked and corrected prior to the start of User Acceptance Testing.

• Additionally, the application must be system tested with documented test results on testing to maximum data rate needs, testing on at least three different commercial wireless networks and a test of 200 simultaneous users with industry standard -- average response times below 1000ms

Pre-Qualified Contractor Responsibilities. At a minimum, the Pre-Qualified Contractor must perform the following activities during this phase:

Integration Test Plan. The Pre-Qualified Contractor must develop the integration test plan with the state reviewing and approving. The Pre-Qualified Contractor must test a collection of components within the context of the system/sub-system in which they function.

The Pre-Qualified Contractor must test modifications within the context of the integrated system/sub-systems (the collection of interconnected systems or sub-systems) in which it functions. Integration testing helps ensure that a defined set of interconnected systems/sub-systems must perform as designed after additions and modifications to the components. The testing must also ensure that interfaces with external systems are exchanging data correctly.

Additionally, the Pre-Qualified Contractor testing must include:

Multiple device testing, to include smartphones, mini-tablets, and tablets; Multiple platforms: iOS, Android; Integration with existing application environment; and Security testing compliant with applicable security controls as defined within the NIST Special Publication 800-53

Revision 3 - Recommended Security Controls for Federal Information Systems and Organizations.

System Test Plan. The Pre-Qualified Contractor must provide a system test plan that will be reviewed and approved by the State. The Pre-Qualified Contractor must develop test cases, scripts and scenarios for testing in this task. The plan must clearly set forth how the system test will fully test the native/hybrid web application solution and its features. The plan must identify the inputs to the test, the steps in the testing process and the expected results. The plan must also identify any software tools used during testing and any State resources needed.

The plan must provide detailed descriptions of the test environment, methods, workflow, training required, and the management of the system testing process and the defect identification and resolution processes to be executed during the system test. The Pre-Qualified Contractor must take responsibility for the ultimate implementation of the plan.

Test Results Document. The Pre-Qualified Contractor must test all native/hybrid web application solution functionality to demonstrate its performance characteristics prior to UAT. The occurrence of defects and their resolutions must be reported in the test results document. The system test results document must include enough information to permit the State to validate that the test has been successfully executed in accordance with the approved system test plan. The Pre-Qualified Contractor must validate that the tests performed adequately meet the requirements of the Task Order Solicitation.

User Acceptance Testing (UAT). The user acceptance testing must demonstrate that the native/hybrid web application solution is ready to perform all functions required by the Task Order Solicitation. The UAT must verify the full functionality and technical usability of the native/hybrid web application solution. The native/hybrid web application solution must be tested by ODJFS before implementation.

At the completion of the UAT, the Pre-Qualified Contractor must prepare a UAT report that certifies that the native/hybrid web application solution is ready for implementation. The State will review the Pre-Qualified

Page 13 of 33

Contractor 's report and determine whether to proceed with the implementation. If necessary, the user acceptance testing must be repeated until the system and user requirements are met.

The Pre-Qualified Contractor responsibilities for UAT include tracking and supporting the user acceptance testing. At a minimum, the activities of this task must include the following:

Develop UAT Plan. The Pre-Qualified Contractor must work with the State to develop a UAT plan that covers, at a minimum:

o Documentation of UAT scripts, procedures, timelines and processes;o Training of designated State UAT staff before the actual testing;o Working with the State to determine the scope of the tests and expected

outcomes for functionality and manual procedures; ando Working with the State to develop methods for reporting, reviewing, and

correcting discrepancies identified during UAT.o Demonstrating the completeness of the test by ensuring all functions, features, and

operations of the mobile application are included in the UAT

Develop Pilot Plan. The Pre-Qualified Contractor must work with the State to develop a Pilot plan that covers, at a minimum:

o Documentation of Pilot participants, timelines and processes;o Training of designated State staff before the actual pilot;o Working with the State to determine the scope of the pilot and expected

outcomes for functionality and success criteria;o Working with the State to develop methods for reporting, reviewing, and

correcting discrepancies identified during pilot.o Final pilot report

Train UAT Staff. The Pre-Qualified Contractor must train all appropriate State staff in the functions, features, and operations of the native/hybrid web application solution for successful execution of User Acceptance Testing.

Execute, Monitor and Support the UAT Processes. The Pre-Qualified Contractor in support of the State must execute and monitor the progress and ensure the level of quality of the UAT processes.

Log, Track and Resolve Problems. The Pre-Qualified Contractor must track all defects throughout UAT, and repair the defects throughout the UAT process. All defects and corrections must be reported to the State Project Management Team.

Produce UAT Final Report. The Pre-Qualified Contractor must prepare reports, based on the elements of the accepted UAT plan, describing the activities and outcomes of the UAT. The Pre-Qualified Contractor must summarize the results of the UAT in the UAT Final Report. The Pre-Qualified Contractor must also include a certification letter, in writing, that UAT was successfully completed.

Certify System is ready for Implementation. The Pre-Qualified Contractor must provide a certification letter, in writing, that the native/hybrid web application solution is ready for production.

Pre-Qualified Contractor Deliverables. Deliverables to be produced by the Pre-Qualified Contractor for this task include the following:

1. Test Plans (this includes Integration and System);2. Test Results Document;3. Pilot Plan;4. UAT Plan;5. Issue Log;

Page 14 of 33

6. UAT Final Report; 7. Final requirements document that is up to date and address type and version level for devices supported8. Letter certifying that the system is ready for production;9. Knowledge transfer document.

Task 4 – Implementation

During this task, implementation of the native/hybrid web application solution must occur. The acceptance of the implemented native/hybrid web application solution must be based on the criteria outlined under task 2.

Pre-Qualified Contractor Responsibilities. During implementation, the Pre-Qualified Contractor must be responsible for the successful operation of the native/hybrid web application solution in the production environment until the implementation task has been completed and accepted. The final acceptance of the native/hybrid web application solution must occur after a 90-day performance interval as well as successful transition to State staff for ongoing support. This 90-day interval must start upon successful implementation of the native/hybrid web application into Production. The Pre-Qualified Contractor agrees to repair all identified application defects discovered during the warranty period. At a minimum, the activities of this task include the following:

Implementation Plan. The implementation plan must demonstrate to the State how the Pre-Qualified Contractor must deploy the native/hybrid web application solution into the production environment. The Pre-Qualified Contractor must identify the path for transition from implementation to support through the warranty period.

Implementation Certification. The Pre-Qualified Contractor must provide an Implementation Certification Letter that certifies that the native/hybrid web application solution is ready for implementation. The Certification letter must confirm:

• Training task is complete;• All testing is complete;• The production environment has been prepared in accordance with the

Pre-Qualified Contractor’s requirements and accepted by the State; and• All user and system supports are in place.

Support of Help Desk. Although ODJFS expects Tier 1 support to come through its current 800 number, the Pre-Qualified Contractor must also include 90-days of Tier 2 support and training for Ohio call center staff.

Execute, Monitor and Support the Pilot Plan. The Pre-Qualified Contractor in support of the State must execute and monitor the progress and ensure the level of quality of the Pilot Interval.

Transition of the Product to JFS. The Pre-Qualified Contractor must deliver all the source material for the application to JFS for ongoing maintenance and enhancement support.

Present Project to the State for Final Acceptance. Upon completion of the Implementation, which includes successful operation of the native/hybrid web application solution in the production environment and successful completion of the performance period as defined in Attachment 4, Standards of Performance and Acceptance, the Pre-Qualified Contractor must present the native/hybrid web application solution to the State for acceptance. The native/hybrid web application solution presented for final acceptance must account for all required functionality.

Documentation. The Pre-Qualified Contractor must develop and provide all documentation at the time the native/hybrid web application solution is presented for final acceptance. The Pre-Qualified Contractor must provide

Page 15 of 33

electronic and hard copies of the documentation for the native/hybrid web application solution. The documentation must include:

User Documentation. The Pre-Qualified Contractor must develop user documentation that describes each feature of the native/hybrid web application solution, and assists the user in executing these features. User documentation must be written and organized so that users, not trained in data processing, can learn from reading the documentation and determine how to access the windows/screens and perform all other user functions.

Operations Documentation. The Pre-Qualified Contractor must develop a complete operations document that must include overviews of the native/hybrid web application solution application, system structure, major processing, and required interfaces. This documentation must include any required periodic maintenance tasks and must also describe the overall batch or background process schedule, which includes dependencies, sequencing, and timing. This documentation must be updated as required throughout the project.

Systems Documentation. The Pre-Qualified Contractor must produce complete system documentation that documents the native/hybrid web application solution application software and its architecture (e.g., implementation view of the application architecture). This includes all native/hybrid web application solution system source code, programs, and executables. The Pre-Qualified Contractor must maintain this documentation to reflect changes made throughout the project.

Additionally, the Pre-Qualified Contractor must document how the application complies with established controls already in place and in use at ODJFS to remain compliant with any applicable security controls as defined within the:

•National Institute of Standards and Technology (NIST) Special Publication 800-53Revision 4 - Recommended Security Controls for Federal Information Systems and

Organizations (https://www.nist.gov/publications/summary-nist-sp-800-53-revision-4-security-and-privacy-controls-federal-information ) •App Quality Alliance (AQuA) Best Practices Guidelines for Developing Quality Mobile Applications http://www.appqualityalliance.org/AQuA-best-practice-guidelines

Pre-Qualified Contractor Deliverables. Deliverables to be produced by the Pre-Qualified Contractor for the implementation task must include the following:

1. Implementation Plan;2. Signed Implementation Certification Letter; and3. Documentation (includes User, Operations, System and Security).

Task 5 – Transition

Knowledge Transition Plan. The Pre-Qualified Contractor must be prepared to initiate a knowledge transition plan including information on tools and technologies that were involved in the native/hybrid web application solution design and development. Once the Task Order is awarded, the knowledge transition plan must be reviewed and approved by the receiving development organization.

Page 16 of 33

ATTACHMENT THREE: Task Order Response Submission RequirementsResponse Format, Content Requirements

An identifiable tab sheet must precede each section of a Proposal, and each Proposal must follow the format outlined below. All pages, except pre-printed technical inserts, must be sequentially numbered.

Each Proposal must contain the following:

Cover Letter Agile Process Description Staffing Plan & Schedule ODJFS Support Expectations Assumptions Contingency Plan Risk Management Plan Quality Management Plan Proposed Solution Cost Summary

Cost Proposal (separately sealed)

Include the following:

1. Cover Letter

a. Must be in the form of a standard business letter;b. Must be signed by an individual authorized to legally bind the Pre-Qualified Contractor;c. Must include a statement regarding the Pre-Qualified Contractor’s legal structure (e.g. an Ohio

corporation), Federal tax identification number, and principal place of business; please list any Ohio locations or branches;

d. Must include a list of the people who prepared the Proposal, including their titles; ande. Must include the name, address, e-mail, phone number, and fax number of a contact person

who has the authority to answer questions regarding the Proposal.

2. Agile Process Description

The desire is that the work in this Task Order will be accomplished using agile processes. Describe your Software Development LifeCycle (SDLC) process definition and flows. Describe how the process will provide for the required phase gates to ensure design and scope quality. Describe your recommended iteration (sprint) duration and rationale for it. Describe how each iteration will deliver potentially deployable user accepted code that conforms to documented standards.

Describe the methodology overview to be provided to ODJFS SMEs and Product Owners.

Describe how your methodology will provide the required artifacts by providing examples of the following:a. Groomed product backlog; b. Sprint planning documents;c. Requirement documentation;d. Status report in MS word including project risks, issues, and change orders;e. Burn charts or other reports representing work completed, estimated work to-do vs. time

remaining;

Page 17 of 33

f. Sprint Show and Tell Presentation; and g. Sprint Retrospective Report.

3. Staffing Plan & Schedule

a. Identify your team composition, number of team members, roles and role definitions, and physical team member location.

b. Identify team’s ramp-up period and roll-off plan (by name or role, if applicable).c. Include the activities and schedule that you will execute during the first 30 days.d. Identify roles to be filled by employees or subcontractors.e. Provide names and resumes of key named staff.f. Provide a filled-in Candidate Education and Training Form for each key staff as outlined in

Attachment 4.g. Provide a filled-in Candidate Experience and Qualifications Form for each key staff as outlined in

Attachment 4. h. Provide documentation verifying how the team member(s) meet the outlined education and

certification qualifications.

4. ODJFS Support Expectations

The offeror must describe the support it needs from ODJFS for execution of this Task Order. Specifically, the offeror must address the following:

a. Nature and extent of State support required in terms of staff roles, percentage of time available, and so on;

b. Assistance from State staff and the experience and qualification levels required;c. Any additional or special expectations that the offeror has of ODJFS staff and resources during

the initial startup;d. Software development or project management tools; e. Team space expectations (if applicable: offeror may propose on-site, a virtual team, or some

combination); andf. Other support expectations.

5. Assumptions

The Pre-Qualified Contractor must list all assumptions the Pre-Qualified Contractor made in preparing the Proposal. If any assumption is unacceptable to the State, the State may at its sole discretion request that the Pre-Qualified Contractor remove the assumption or choose to reject the Proposal. No assumptions may be included regarding the outcomes of negotiation, terms and conditions, or requirements. Assumptions should be provided as part of the Pre-Qualified Contractor response as a stand-alone response section that is inclusive of all assumptions with reference(s) to the section(s) of the Solicitation that the assumption is applicable to. The Pre-Qualified Contractor should not include assumptions elsewhere in their response.

6. Contingency Plan

Identify and provide a Contingency Plan should the Contractor and Subcontractor staff fail to meet the Project Schedule, Project Milestones or fail to complete the deliverables according to schedule. Include alternative strategies to be used to ensure project success if specified risk events occur.

Page 18 of 33

7. Risk Management Plan

Describe the Risk Management Plan requirements including the risk factors, associated risks, and assessment of the likelihood of occurrence and the consequences for each risk. Describe your plan for mitigating selected risks and plan for keeping people informed about those risks throughout the project.

8. Quality Management Plan

Describe your quality policies, procedures, and standards relevant to the project for both project deliverables and project processes. Define who is responsible for the quality of the delivered project artifacts and deliverables.

9. Proposed Solution

This section of the Pre-Qualified Contractor’s proposal must, at a minimum, address each of the nine (9) Proposed Solution Scored Criteria located on page 6 of this Task Order Solicitation document.

COST SUMMARYThe Pre-Qualified Contractor’s Fee Structure. The Task Order award must be for a not-to-exceed fixed price, payable in accordance with the schedule below:

THE WORK COST

Payment for the Acceptance of Required Mobile WebApplication Prototype

25% of the Total Not to Exceed Fixed Price $

Payment for the review and acceptance of a successful Pilot 20% of the Total Not to Exceed Fixed Price $

Payment for the Final Acceptance of RemainingRequired Deliverable(s)

55% of the Total Not to Exceed Fixed Price $

Total Not to Exceed Fixed Price: $

ATTACHMENT FOUR: PERSONNEL PROFILE SUMMARY

Page 19 of 33

CANDIDATE REFERENCES

Duplicate this form for each candidate.

Position:

Candidate:

References. Provide three references for which the proposed candidate has successfully demonstrated meeting the requirements of the Task Order on projects of similar size and scope in the past five years. The name of the person to be contacted, phone number, company, address, brief description of project size and complexity, and date (month and year) of employment must be given for each reference. These references must be able to attest to the candidate’s specific qualifications.

The reference given should be a person within the client’s organization and not a co-worker or a contact within the offerors organization.

If less than three references are provided, the offeror must explain why. The State may disqualify the Proposal if less than three references are given. Duplicate this form as necessary.

Client Name: Client Contact Name:

Address: Phone Number:

Project Name: Beginning Date of Experience Month/Year:

Ending Date of Experience

Month/Year:

Description of services provided by the candidate that are similar as those to be provided as part of this Project.

Description of client project size and complexity.

PERSONNEL PROFILE SUMMARY

CANDIDATE EDUCATION AND TRAINING

Page 20 of 33

Duplicate this form for each candidate.

Position:

Candidate:

Education and Training. This section must be completed to list the education and training of each proposed candidate and must demonstrate in detail the proposed candidate’s ability to properly execute the Contract based on the relevance of the education and training to the requirements of the Task Order.

EDUCATION AND TRAINING MONTHS/YEARS WHERE OBTAINED DEGREE/MAJOR YEAR EARNED

College

Technical School

Other Training

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

Page 21 of 33

(Experience and Qualifications)

Position #1 – Project Manager

Candidate’s Name:

Requirement: Must have at least thirty-six (36) months of project management experience on a project of similar size and scope.

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTSPage 22 of 33

(Experience and Qualifications)

Position #1 – Project Manager

Candidate’s Name:

Requirement: Must have thirty-six (36) months of experience or three (3) projects with project management responsibilities for native/hybrid web applications

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

Page 23 of 33

(Experience and Qualifications)

Position #2 – Mobile User Experience (UX) Lead

Candidate’s Name:

Requirement: Must have twenty-four (24) months of experience as a mobile user experience (UX) design lead on at least two (2) projects of similar size and scope.

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

Page 24 of 33

(Experience and Qualifications)

Position #2 – Mobile User Experience (UX) Lead

Candidate’s Name:

Requirement: Must have at least three (3) years of experience in designing/architecting mobile user interfaces.

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

Page 25 of 33

(Experience and Qualifications)

Position #3 – Mobile Test Lead

Candidate’s Name:

Requirement: Must have twenty-four (24) months of experience as a mobile test lead on at least two (2) projects of similar size and scope.

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android- Indicate Yes or No Platform iOS- Indicate Yes or No

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

Page 26 of 33

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

(Experience and Qualifications)

Position #3 – Mobile Test Lead

Candidate’s Name:

Requirement: Must have at least thirty-six (36) months of experience in testing mobile user interfaces.

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

Page 27 of 33

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

(Experience and Qualifications)

Position #4 – Mobile Application Lead

Candidate’s Name:

Requirement: Must have twenty-four (24) months of experience as a mobile application development lead on at least two (2) projects of similar size and scope.

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

Page 28 of 33

PERSONNEL PROFILE SUMMARY

CANDIDATE REQUIREMENTS

(Experience and Qualifications)

Position #4 – Mobile Application Lead

Candidate’s Name:

Requirement: Must have at least thirty-six (36) months of experience in designing/building native/hybrid web applications for smartphones and mini-tablet

Company Name: Contact Name: Primary or Alternate?

Contact Title:

Address: Contact Phone Number:

Email Address:

Work Name: Beginning Date of Experience

Month/Year

Ending Date of Experience

Month/Year

Similar size and scope information for the reference.

Number # of Screens: Number of on-line programs:

Platform Android Platform iOS

Candidate’s experience for the referenced project.

Description of technical experience, capacity performed, and role that is related to services to be provided for the Work:

Description of how client work size and complexity are similar to the Work:

*** COPY THIS FORM AS MANY TIMES AS NEEDED ***

Page 29 of 33

SUPPLEMENT ONEMatching hierarchy for OJI_INBOUND

Start at the top and STOP when a unique match is found. Otherwise continue to the end.

1 - Claimant ID 2 – SSN, Last Name, First Name 3 - SSN, Soundex(Last Name), First Name 4 - SSN, Soundex(First Name), Last Name 5 - SSN, Last Name, DOB 6 - SSN, First Name, DOB 7 - SSN, Soundex(Last Name), DOB 8 - SSN, Soundex(First Name), DOB

---From here down, this would be handled by the intowner.seekerin_match process 9 - Last Name, First Name, Gender, DOB, Zip, Phone, Address1, Address2 10 - Last Name, First Name, Gender, DOB, Zip, Phone, Address111 - Last Name, First Name, Gender, DOB, Zip, Phone12 - Last Name, First Name, Gender, DOB, Zip, Address1, Address213 - Last Name, First Name, Gender, DOB, Zip, Address114 - Last Name, First Name, Gender, DOB, Zip15 - Last Name, First Name, Gender, DOB, Phone, Address1, Address216 - Last Name, First Name, Gender, DOB, Phone, Address117 - Last Name, First Name, Gender, DOB, Phone18 - Last Name, First Name, Gender, DOB, Address1, Address219 - Last Name, First Name, Gender, DOB, Address120 - Last Name, First Name, Gender, DOB21 - Last Name, First Name, Gender, Zip, Phone22 - Last Name, First Name, Gender, Zip23 - Last Name, First Name, Gender, Phone24 - Last Name, First Name, Phone, Zip25 - Last Name, First Name, Zip26 - Last Name, First Name, Phone

If only one match is found, continue processing as normal.

If no match is found at all – Create a new SCOTI record and continue processing as normal.

If too many matches are found (meaning we could not find a unique match) OR if we end up with a seeker match but that seeker is already matched to a different claimant id – return an error to OJI and do not process that record.

Page 30 of 33

Page 31 of 33

Page 32 of 33

SUPPLEMENT TWO

Embedded File

CLICK ICON BELOW TO ACCESS CONTENTS

Page 33 of 33