state of maine rfp / proposal master score sheet · state of maine team consensus evaluation notes...

Rev. 3/5/2018

State of Maine RFP / Proposal Master Score Sheet

Instructions: Complete the Master Score Sheet below providing all of the requested information for each bidder that submitted a proposal in response to the RFP. This document is to be included in the Selection Package submitted to the Division of Procurement Services for review/approval.

SCORESHEET FOR RFP# 201804073: Maine Education Enterprise Data Solution PROPOSAL SUBMITTED BY: UDP, LLC Double Line, Inc ESP Solutions Group

COST: Cost: Cost: Cost:

EVALUATION ITEM POINTS AVAIL.

Stage 1: Eligibility N/A No* Yes Yes Stage 2: Organization Qualifications and Experience 30 18 27

Stage 3: Proposed Services 30 10* 24

Stage 4: Cost Proposal 30 30

Stage 4: Maine Business and Economic Impact Consideration 10 6

TOTAL 100 87

*Indicates bidder did not meet minimum eligibility requirements to pass to the next stage of evaluation

Award Justification Statement RFP# 201804073 – Maine Education Enterprise Data Solution

1

I. Summary The purpose of the RFP is to provide the integration of the Departments current data collection systems into a single managed service to serve a wide range of stakeholders in support of providing quality data and more transparency to Districts, Policymakers, and the public.

Three bidders responded to RFP # 201804073 - Urban Policy Development, LLC d/b/a UPD Consulting, ESP Solutions Group, Inc and Double Line, Inc.

II. Evaluation Process

The evaluation and scoring of proposals was conducted using four (4) tiered/staged approach.

• Stage 1 – Eligibility to Submit Bids • Stage 2 – Organization Qualifications and Experience • Stage 3 – Proposed Services • Stage 4 – Cost Proposal

Proposals were required to meet or exceed minimum scoring requirements of the stage in which they were being evaluated to move onto the next stage of evaluation. Proposals not meeting the stated minimum scoring requirements of a stage, were not eligible for award consideration and removed from the evaluation process.

Stage 1 required vendors to have fully implemented at least one (1) Ed-Fi solution. All vendors were evaluated using the following definition of implementation:

Implementation is the carrying out, execution, or practice of a plan, a method, or any design, idea, model, specification, standard or policy for doing something.

In an information technology (IT) context, software or hardware implementation encompasses all the processes involved in something operating properly in a production environment, including analyzing requirements, installation, configuration, customization, running, testing, systems integrations, user training, delivery and making necessary changes.

UPD Consulting did not indicate in their proposal that they had a fully implemented Ed-Fi solution in Production at the time of RFP submission. Direct quote from UPD Consulting Eligibility to Submit Bids - “All three components are on track to move to production”. Based on the definition above, Maine DOE was seeking evidence of full production implementation.

Stage 2 and Stage 3 the evaluation team applied the consensus method in scoring Organizations & Experience and Proposed Services. For both sections the vendor had to obtain a score of 18 out of 30 points.


2

ESP and Double Line both achieved the minimum score passing stage 2. However, Double Line failed to pass stage 3 and was disqualified from the remaining stages of the evaluation process.

The following table summarizes the results by stage of all vendors:

Bidder

Eligibility to Submit

Bids (Yes/No)

Qualifications & Experience

Min 18, Max 30 Points

Proposed Services

Min 18, Max 30 points

Cost Proposal Max 30 points

MBEIC Max 10 points

TOTAL

UPD No N/A N/A N/A N/A N/A Double Line Yes 18 10 N/A N/A N/A ESP Yes 27 24 30 6 87

III. Qualifications & Experience

Double Line

• 9 years of experience in Ed-fi and initial developer of Ed-fi standard • Size of the organization and the number of projects and staff (38 supporting 15 projects

across 11 clients) • The stability of the staff – a lot of turnover in their technical team • Primary Bidder has no previous work history with subcontractor Winxnet. • No responses to reference checks for Winxnet, leaving the team unable to confirm this

subcontractor’s ability to perform cloud migration and satisfaction of services provided. • Overall Staff experience <60 months for Education Data Warehouse projects • Majority of staff filling multiple positions

ESP

• 8+ years of ongoing Ed-fi implementations • 25 years in business of education applications with no restructuring, downsizing or

mergers • Ability to augment staff using subcontractor NTT Data to meet demands • Subcontractor NTT Data has 50 years of experience in systems integration and

managed services, and also has State of Maine Experience • Primary Bidder managing 3 subcontractors Learning Mate, NTT Data, and eScholar;

however, vendor does have work history with each • Average Staff experience having >50 months, in Educational Data Warehouses, many

100+ months


3

IV. Proposed Services Double Line: • Bidder didn’t really address the RFP. Only provided minimal information and lack of

overall substance. • Appears to have a valid understating and approach to implementing the Ed-FI standards

however didn’t seem to grasp the size and complexity of the other components such as the Financial Data Warehouse, EDFacts, and MAARS

• There was no clear definition in Project Methodology and approach to the project

ESP: • Bidder reiterated the vision and clearly addressed all the high points of the project by

providing a list of project assumptions, including high-level success factors with an emphasis on on-site presences

• Clear and defined approach to overall project management and methodology • Good high-level vision of milestones with discovery happening throughout the lifetime of

the project • Seem to really understand concept of the NEO Migration Scope and showed experience

in other phases of the project such as EDFacts and Financial Reporting. • Provided detailed description of training with trainers that have instructional experience

V. Cost Proposal

ESP was the only bidder to pass stage 3 and the only cost proposal that was opened.

Description Cost Proposal Initial Period of Performance (2018-2023) $19,295,908.47 3 Renewal Periods – Estimate (2023-2035) $6,948,888.14 Total Proposal $26,244,796.61

The evaluation team felt that there was need to ask the bidder some clarifying questions regarding the cost proposal. Questions were sent and responded to via email.

VI. Conclusion ESP Solutions Group, through the staged evaluation process, was the only Bidder determined to have all the qualifications and proposed a solution that meets the scope of work outlined in the RFP. The evaluation team feels that the best course of action is to make a conditional award to ESP Solutions Group. There are many items that need to be discussed with the bidder during contract negotiations. The bidder articulated many assumptions in their response, and


4

although clarifying questions were sent regarding the cost proposal, there are still additional questions.

Stage 1

STATE OF MAINE TEAM CONSENSUS EVALUATION NOTES

RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: UPD, LLC dba UPD Consulting DATE: 8/16/18

Rev. March 5, 2018 1

*************************************************************************************************************************** Instructions: The purpose of this form is to record all evaluation notes and scoring that is obtained through consensus discussions among the full evaluation team for this Request for Proposals (RFP) process. The RFP Coordinator or Lead Evaluator should complete this form and maintain the only copy. This form should reflect the full team’s consensus evaluations, and this form is not meant to take the place of individual evaluation notes, which are still required from each member of the evaluation team. A separate form is available for individual evaluation notes. Please submit a copy of this document to the Division of Procurement Services as part of your contract award selection documents. DEPARTMENT NAME: Department of Education NAME OF RFP COORDINATOR: Lance Gilman NAMES OF EVALUATORS: Lance Gilman & Charlotte Ellis

***************************************************************************************************************************

SUMMARY PAGE

Pass/Fail Criteria Pass: Fail:

Proposal Cover Page x Debarment, Performance and Non-Collusion Certification x Eligibility to Submit Bids x Appeal Deposit Refund Form x

Justification

The following bidder was determined to not meet Stage One criteria; as they did not indicate in their proposal as having a fully implemented Ed-Fi solution in Production at the time of RFP submission: UPD, LLC d/b/a UPD Consulting The “fully implemented/implementation” definition that was used to determine bidder Eligibility requirements for Stage One of the RFP evaluation process is below:

Implementation is the carrying out, execution, or practice of a plan, a method, or any

design, idea, model, specification, standard or policy for doing something.

In an information technology (IT) context, software or hardware implementation

encompasses all the processes involved in something operating properly in a

production environment, including analyzing requirements, installation,

configuration, customization, running, testing, systems integrations, user training,

delivery and making necessary changes.


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: Double Line, Inc. DATE: 8/16/18

*************************************************************************************************************************** Instructions: The purpose of this form is to record all evaluation notes and scoring that is obtained through consensus discussions among the full evaluation team for this Request for Proposals (RFP) process. The RFP Coordinator or Lead Evaluator should complete this form and maintain the only copy. This form should reflect the full team’s consensus evaluations, and this form is not meant to take the place of individual evaluation notes, which are still required from each member of the evaluation team. A separate form is available for individual evaluation notes. Please submit a copy of this document to the Division of Procurement Services as part of your contract award selection documents.

DEPARTMENT NAME: Department of Education NAME OF RFP COORDINATOR: Lance Gilman NAMES OF EVALUATORS: Lance Gilman & Charlotte Ellis

***************************************************************************************************************************

Stage 1 EVALUATION

Eligibility Requirements

Pass/Fail Criteria Pass: Fail: Proposal Cover Page x Debarment, Performance and Non-Collusion Certification x Eligibility to Submit Bids x Appeal Deposit Refund Form x



RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: ESP Solutions Group, Inc. DATE: 8/16/18

*************************************************************************************************************************** Instructions: The purpose of this form is to record all evaluation notes and scoring that is obtained through consensus discussions among the full evaluation team for this Request for Proposals (RFP) process. The RFP Coordinator or Lead Evaluator should complete this form and maintain the only copy. This form should reflect the full team’s consensus evaluations, and this form is not meant to take the place of individual evaluation notes, which are still required from each member of the evaluation team. A separate form is available for individual evaluation notes. Please submit a copy of this document to the Division of Procurement Services as part of your contract award selection documents.

DEPARTMENT NAME: Department of Education NAME OF RFP COORDINATOR: Lance Gilman NAMES OF EVALUATORS: Lance Gilman & Charlotte Ellis

***************************************************************************************************************************

Stage 1 EVALUATION

Eligibility Requirements

Pass/Fail Criteria Pass: Fail: Proposal Cover Page x Debarment, Performance and Non-Collusion Certification x Eligibility to Submit Bids x Appeal Deposit Refund Form x


Stage 2


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: Double Line, Inc. NAMES OF EVALUATORS: Ida Batista, Brian Keene, Charlotte Ellis, Cathy Harrison, Paula Roberts, Hazel Stevenson, Joseph Larrabee DATE: September 10, 2018


*************************************************************************************************************************** STAGE 2 EVALUATION

Organization Qualifications and Experience

Total Points Available: 30 Score: 18

*************************************************************************************************************************** Evaluation Team Comments:

Overview of the Organization: 1. Organization Overview

*9 Years of experience, 4 years collaborating with Dell Foundation *Number of employees seems low, and workload may be high with team of 38 supporting 15 projects across 11 clients *Hasn’t worked with subcontractor Winxnet before *Company status in flux as of 2015 *Possible issue with Winxnet merging with California company in June not being noted in the proposal

2-4. Experience in Enterprise Data Solutions, Ed-Fi and Cloud Migration/Managed Services *They have developed the Ed-Fi standard *Experience with integrating multiple systems *Designed certification process *Sub has hosting experience, SoM experience, and Migration to Cloud experience, but provides little detail or examples *No information about Winxnet experience outside SoM projects *Haven’t actually migrated a system? 5. Responses to Use Cases/Questions *Talk about how student data is stored in their systems, but don’t speak about redaction at all *Didn’t speak about problems with reporting data and student information *Going to use a rules engine for data quality *Seem to be aware of the process and what needs to be done *Didn’t answer the question as to how they are going minimize duplicate IDs *Good that they have merge and split tools, but needs to go beyond SIS package *See security as a big hurdle, but are well prepared and know how to handle it *Winxnet has a strong managed services background *Identified an issue with hours of support not being high enough for productivity *Acknowledged that they have not worked with Winxnet or SoM before *Both have long term relationships with AWS 6. Cloud Implementation *Didn’t indicate what major cloud provider they are using *Very light on details 7. Disaster Recovery *Same response as Cloud Implementation, didn’t provide much detail 8. Personnel *Some positions were left blank (technical writer, trainer)


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: Double Line, Inc. NAMES OF EVALUATORS: Ida Batista, Brian Keene, Charlotte Ellis, Cathy Harrison, Paula Roberts, Hazel Stevenson, Joseph Larrabee DATE: September 10, 2018


*Overall length of experience is less than 60 months on average for Ed-Fi and Data Warehouse projects *Some people doing multiple different jobs *Subcontractor has no experience on Ed-Fi and Data Warehouse projects *Concerning that project managers don’t have PMPs *On org chart, Winxnet is just listed by name: No people listed, no title, no roles or resumes *Several resources are on recently started reference projects *Really low onsite presence since the main company is in Texas *Several people show years of experience, but not how many years in the position they are being assigned to *Managed services listed under System Architect, not its own thing *Listed managed services for Winxnet separate from managed services for Double Line, no crossover *Some highly qualified people on staff 9. Primary or Subcontractor Contractor References *Two references only just started their projects this year *Only one of five refence projects is completed 10. Primary and Subcontractor Previous Working Relationship *Left blank, no previous work history 11. Subcontractor References *None Litigation: *No litigation Financial Viability: *1R3 Rating (Fair) *Moderately lower than average financial stress *552 credit score *Credit and payment risks are low *$5M sales organization, but this project is close to 40% of that annually Certificate of Insurance: *Provided one, but expired in June 10th, 2018 *Did not have student data insurance/cyber security insurance *Questioning the dates on the certificate References: *Nebraska mentioned having an issue limited resources to complete what they wanted *No responses from anyone about Winxnet


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: ESP Solutions Group, Inc. NAMES OF EVALUATORS: Ida Batista, Brian Keene, Charlotte Ellis, Cathy Harrison, Paula Roberts, Hazel Stevenson, Joseph Larrabee DATE: September 10, 2018


*************************************************************************************************************************** ***************************************************************************************************************************

STAGE 2 EVALUATION Organization Qualifications and Experience


***************************************************************************************************************************

Evaluation Team Comments:

Overview of the Organization: 1. Organization Overview

*Stable over 25 years w/no restructuring, downsizing or mergers *Acquired a number of former Double Line employees *Experience with state education, data standards, managing implementations *NTT has 50 years of system integration, large IT projects managed services and SoM experience *Some on team have concern with there being 3 subcontractors *Prime contractor on 8 or more major state implementations *ESP has worked with every state, including 2 projects in Maine

2-4. Experience in Enterprise Data Solutions, Ed-Fi and Cloud Migration/Managed Services *Ed-Fi projects and data warehouse projects show on time and still operational *Project for NTT Data shows experience with complex cloud migration 5. Responses to Use Cases/Questions *Really seem to get student privacy and disclosure/reporting *Presented a privacy paper to national organization *Department can control rules engine, but need to know XML *Rules engine data quality service is being considered by Ed-Fi Alliance as reference validation module *Understand the Ed-Fi mapping process *Have financial system modernization experience *Has experience working with CGI and have experience with interfaces like Advantage *Very robust solution for Unique IDs that is being used in 10 states *NTT has application outsourcing management matrix *Proactive, good assessment of challenges and solutions *Commented that communication is number 1 tool for success (Status updates and such) *Recognized that we have a small staff and that will be a challenge (availability of SMEs) *Meaningful metrics in term of implementation 6. Cloud Implementation *Said they enhanced Salesforce, not a new implementation 7. Disaster Recovery *Good description of DR criteria 8. Personnel *Most staff have 50+ months experience in Educational Data Warehouses, many 100+ months *NTT doing Data Warehouse, but none of their people have education data warehouse experience? *NTT project manager does not have references as a project manager


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: ESP Solutions Group, Inc. NAMES OF EVALUATORS: Ida Batista, Brian Keene, Charlotte Ellis, Cathy Harrison, Paula Roberts, Hazel Stevenson, Joseph Larrabee DATE: September 10, 2018


*Are the “to be assigned” positions from current staff or new staff? *Concern over single person being connected to executive advisory committee, instead of team being connected to executive advisory committee *Many inconsistencies in dates on some resumes *Trainer only has a couple of months experience as a trainer *PM and BA permanently on site *Project manager is listed on org chart as something different 9. Primary or Subcontractor Contractor References 10. Primary and Subcontractor Previous Working Relationship *ESP has experience working with Learning Mate and E-Scholar *No listed experience of NTT and ESP working together 11. Subcontractor References (From Proposal) *Only for E-Scholar in this section, the others were in the first section of the proposal Litigation: *None Financial Viability: *No corporate debt *Significant investment in R&D *Good payment history *Appears to be a low financial risk *Good credit score *Subcontractors are self-financing Certificate of Insurance: *Has one that is current, though ends soon *Aggregate limit is by location instead of for whole project *Cyber liability will be required if project is awarded References: *All subs seemed well reviewed, any areas of concern seem manageable

STATE OF MAINE INDIVIDUAL EVALUATION NOTES

RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Doubleline DATE: 9/6/2018 EVALUATOR NAME: Brian Keene EVALUATOR DEPARTMENT: Regional School Unit No. 56 – Technology Director ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of Organization P – Going on their 10th year – not a new business P – Subcontractor is local to us so even though the HQ is in Texas we would have local support through the sub. P – Significant Experience in Educational EDFI solutions M – DoubleLine hasn’t worked on a project with their subcontractor Winxnet before P – Developer of the EDFI data standard

2. Litigation P – None noted

3. Financial Viability P – Looked like nothing raising a red flag

4. Certificate of Insurance

P – Provided Cert of Insurance

5. Required Attachments Related to Organization Qualifications and Experience P – DoubleLine included resumes of project staff that have a lot of experience with EdFi, data warehouse applications and configurations M – The note at the end of the subcontractor references regarding WinXNet not having involvement as of yet with public K-12 in Maine. Something to think about. P – Infrastructure Architect looks to be very qualified in technology infrastructure management P – A lot of experience with database administration and management

6.

STATE OF MAINE

INDIVIDUAL EVALUATION NOTES

RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution

BIDDER NAME: Double Line, Inc. DATE: 9/7/2018 EVALUATOR NAME: Joseph W Larrabee EVALUATOR DEPARTMENT: Education

************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP.

************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization

Appendix H

1. Organization Overview:

M: Doubleline team is only 38 people which have the Ed-FI experience. This

wouldn’t be an issue if they currently have 15 projects across 11 clients. Might be

spread a little too thin

P: Original Implementer of Ed-Fi

P: Developer of Ed-Fi standard

2. Experience in Education Enterprise Data Solutions:

P: Implemented Student and Staff (page 6-Nebraska)

M: Have 2 new projects that kicked off in February and March 2018. Concerned

with volume of work. Would have liked to see more reference to completed work

3. Experience in Ed-Fi:

P: Double Line is the developer of the Ed-Fi data standard, Ed-Fi solution data

components, and the Ed-Fi Dashboards, as well as the Ed-Fi Data Warehouse

P: Described performance management

M: No mention of integration activities

4. Experience in Cloud Migration / Managed Services:

M: Could have provided more experience in Cloud Migration of an existing

product. Appears to have a reference to cloud implementation. Could have

provided more detail of project (ie statistics, tables ect). At this point not sure if the

BI tool migration is anywhere as complicated as NEO is going to be.

5. Responses to use cases/questions:

STATE OF MAINE




5.1 Student Privacy

P: Mentions Privacy of data at infrastructure level keeping different

environments for DEMO and not keeping student data local

M: Didn’t mention any details on redaction of data.

M:

5.2 Data Quality and Monitoring

M: Could have taken more opportunity to explain how managed data quality in

past projects and how will fit into DOE project. Didn’t use much of their

allocated space of 1 page to get into detail

P: Have rules engine that uses “natural language”

5.3 Financial Data

P: Detailed with respect to how the process will work to extend the ED-FI

(API, ODS Ect) and working with district vendors to use API

M: No mention of working to incorporate the NCES Standards

5.4. Unique IDs:

M: Would like to have seen more detail. Didn’t use much of their allocated

space of 1 page to get into detail

M: Could have indicated more experience and how it was done in the past

M: Doesn’t get into detail on what elements are used to match on

M: No mention of using the Ed-FI API

M: Seemed more like a white paper description of Identity management than

experience

5.5. Managed Services (limit 2 pages):

General Concern: Think that Managed services might just be just NEO when it

really its NEO and the entire DW

5.5.1

M: Would like to have seen more detail around customer support

Q: Strongest In – not sure if they are referring to they are strongest in

Operational Maturity. Couldn’t really tie their answer to one of the key

elements.

P: ITIL structure – but didn’t mention ITIL certified

5.5.2

P: Recognize security is a large concern

Q: Not sure the information is a response to working with State Agencies.

Would have expected more possible around working with State government

and the concerns around that.

5.5.3

STATE OF MAINE




M: Kind of a “canned answer” common understanding of objectives is a

concern for anything. Would like to have more information around the

interaction between the DOE and vendor with it comes to the SLA

5.5.4

M: First partnership with Winxnet for managed services. No predefined roles

between Prime and Sub

5.5.5

P: Provide a ticketing management system

M: Didn’t really answer the question with respect to upgrades, downtime.

5.5.6

P: Use tickets to track trends and provide common issues

M: Said they can provide reports, but would have like to have seen mention of

Realtime dashboards that the Department can monitor

6 Cloud Implementation

M: Not much description about the project. Would have like to see more technical

details or another reference that had more information

P: Have worked with the State so should understand the entire OIT Deployment

Cert process

7 Disaster Recovery

M: Not much description about the project. Would have like to see more technical

details or another reference that had more information

M: Didn’t mention any capabilities but said will work with the state to provide one

P: Have worked with the State so should understand the entire OIT Deployment

Cert process

2. Litigation

P: No litigation in past 5 year

3. Financial Viability

Read – No feedback


M: All Polices had expired in June 2018 and the report was done on 7/16/2018

5. Required Attachments Related to Organization Qualifications and Experience

8 Personnel

8.1 Summary Table

M: There are no Winxnet employees listed.

M: There are no trainers identified

STATE OF MAINE




M: There are no technical writers identified

M: Average EDW Experience is <36 months (consider>60 a Plus)

M: Average Edfi Experience is <40 months (consider>60 a Plus)

8.2 Org chart

M: There are no Winxnet employees show

M: Training doesn’t show up in Org Char

Q: Not sure what Double Line Managed Services is vs Winxnet. Why

Wouldn’t Winxnet report to them instead of an Architect. Would have liked

to see an Support manager and managed services associated to them. Same

with NEO team. There should be one managed services team that supports

the DW and NEO

M: Have a lot of resources assigned to multiple roles. Assume that the

project will have different deliverables at difference phases. That there

might be a conflict between someone that is (ie) a BA and QA for different

deliverables. Or Data Analyst and QA

8.3 Resumes

Q: Was difficult understanding how the roles tied to the org chart because

of individuals assigned to multiple roles. I think it would have been more

clear if they did a resume for each of the roles instead of trying to combine

them.

M: David Doyle – Senior Developer – does not appear to have 7 years of

Dw experience.

M: Eugene Bird – Architect – does not appear to have 6 years of required

experience in required categories

M: There are no Winxnet employees resume

M: BA’s don’t have min 3 years of experience

M: QA’s don’t have min 7 years’ experience

M: There are no trainers resumes

M: There are no technical writers resumes

M: Hard to verify years of education and ed-fi experience

M: References for a lot of resumes don’t tie to the project position. For

example (Lille Anderson) is DA/QA but the references are for developer.

In fact for this person, there is no QA experience

9 Prime / Subcontractor References

Prime:

P: Appears to have significant experience implementing EdFi standard

P: Appears to have experience working with State and Local Ed’s

STATE OF MAINE




M: didn’t see any experience in building a DW from ground up. Experience

appears to be on existing structures and not completely new ones.

Sub:

M: Resources are new and projects are not completed. Would have liked to

have seen at least one complete project to follow up on

P: Experience working for the state

10 Prime and Subcontractor previous working relationships

M: This is a new relation. There no previous working relationships

11 Subcontractor K-12 references

M: Subcontract has no K-12 experiences


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/6/18 EVALUATOR NAME: Hazel Stevenson EVALUATOR DEPARTMENT: Admin and Finance




sub contractor Winxnet 9 years’ experience; 4 years acquired by Dell Foundation, 4 years’ vendor and collaborator with Dell Foundation Team of 38 supporting 15 projects/ 11 clients Sub is twice size than prime in employee count P Reference projects provided – successful Ed-fi, DWH

P Developed Ed-Fi standard, have “longest track record” P Experience with integrating, multiple systems and designed the certification process

working with 13 States meets requirements for similar experience P sub has hosting experience, migration to cloud, SOM experience

Privacy procedures “beyond compliance”, anonymized test data P- rules engine for data quality P- developed Finance domain in ed-fi standard P – sub has security practice, strong managed service background, but no ed-fi P- both have long AWS partnerships M- no experience working together; acknowledge it will be a challenge Staffing- M PMs do not have PMP M several resources are on recently started reference projects M low onsite presence; Majority of team in TX M Did not provide sub’s roles, names, resumes M Overall length of experience is less than 60 months on average in Education DWH; several positions less than 2 years. I Infrastructure architect/ DB admin = same person M 4 of the reference projects are ongoing; two started in 2018 sub reports to infrastructure Architect

2. Litigation

none




Provided D&B for prime 5M sales; credit and payment risks are low; moderate risk of severe financial stress D&B was unable to calculate business ratios Unable to ascertain financial viability of prime for a project this size.


Provided; cyber liability will be required if awarded contract


Provided where applicable (no #7)


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/7/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: DAFS/OIT ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization– ok • Double Line – 9 yr history (38 EEs) • Developed the Ed-Fi standard - P • 1 sub: Winexnet – no previous working relationship with Double Line • Total # EEs (both vendors) 130 EEs - M

2. Litigation– P

None

3. Financial Viability – ok • D&B rating 1R3 • “Moderately lower than average … Financial Stress” pg 26 – M • 2017 Sales: $5 Mil • Credit Score: 552 Percentile: 81

4. Certificate of Insurance– P

Good – High Claim Limits

5. Required Attachments Related to Organization Qualifications and Experience • Winexnet – never a prime; always a sub; - M • Winexnet – has not migrated to a major cloud provider? – Q • Winexnet – has not completed the SOM “Hosted Kofax BI Platform” project? – Q • How can Double Line have 15 projects across 11 clients with only 38 EEs? (pg 5) - Q • What is Winexnet’s relationship with AWS? Why were they recommended? - Q • 5 Client References as PRIME – P • Where is the “DHHS environment” located? Pg 73 - Q


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/7/18 EVALUATOR NAME: Ida Batista EVALUATOR DEPARTMENT: DOE ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization (P) Experience with ed data & Ed-Fi technology; Experience with other SEAs currently still working with many. Strong on student data privacy. (I)Architects on existing Finance domain Ed-Fi standard. This would be bleeding edge development for them. Not clear if proposal answers questions posed in RFP on student identifiers. Wanted to hear more on trainings and transition of ownership/stewardship of products and Data Quality and Monitoring (Q)New close relationship with Winxnet. More information on Winxnet.

2. Litigation (P)Double Line had no litigations.

3. Financial Viability Double Line appears financially viable. However, it does look like there was major company ownership transition in 2015.

4. Certificate of Insurance (P,Q) Double Line has one. Is the value of the insurance sufficient for this project? Are the expiration dates of 06/2018 problematic?

5. Required Attachments Related to Organization Qualifications and Experience (P,Q)Newer members to staff. P39 organization chart how do rotational and multiple manned roles work? Though Winxnet has experience with state government will that translate to work as done in the DOE. No experience with Ed-Fi nor education data warehouse projects. What type of man-power resources would be facilitated by Winxnet. Many employee several years of experience with Ed-fi and data warehouse projects (Q) Double Line and Winxnet have not worked together.


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 08/31/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization 1. √ 2. √ 3. √ P – Developed Ed-Fi standard – very experienced 4. √ I – Subcontractor 5. √ P – Experience with anonymized databases for training and demonstration purposes

M – Didn’t mention redaction or challenges of privacy in reporting 5.3 √ P – Seems to understand the challenges and has a good grasp of the process 5.4 √ M – Didn’t really discuss how the system will minimize duplicates

P – Merge and Split tools 5.5 .1 √ P – Has analyzed for weak points and is addressing

.2 √ M – Not sure that I agree that security will be the biggest issue, especially if they have robust security like they discussed in 5.1 .3 √ .4 √ P – Realizes that roles and responsibilities will be a challenge .5 √ .6 √

6. √ M – “Work With” doesn’t sound like self-support 7. √ M – Same description as above, no disaster recovery description 8. √ M Didn’t really include sub contractors in organization chart

8.1 √ M – Some people are doing double duty – multiple roles M – Some positions were left blank 8.2 √ M – Didn’t identify staff at WinXNet 8.3 √ I Becky McKnight (BA/QA) – How does one go from a Benefits Administrator to a Career Counselor to a Data Architect in 2 years? She will be a Business Analyst on this project and does have a graduate degree in this and 2 years of experience. Both references are from the same project Melissa Ford (BA) – Only 4 years background shown. Not sure of amount of actual experience as a BA Lillie Anderson (Data Analyst/QA) – Only 6 years background shown Morinsola Akinduro (Data Analyst) – P – Seems very qualified Dennis Piatt (Infrastructure Architect/Database Administrator) - P – Seems very qualified Stacy-Ann Salter (Database Administrator) ? – Not sure list of tasks actually equates to a DBA Matt Warden (Solution Architect) – ? Only dates worked are those at DoubleLine I – published – M – Does really show experience as an architect


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 08/31/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education

Ben Meyers (Solution Architect/Software Engineer) - ? 3 years as an architect or software engineer? Positions overlap Eugene Bird (AWS Infrastructure Architect) – P- Seems very qualified Kyung Huh (Senior Project Manager) ? How long as this person been working – only time given is at Double Line (5yrs), P – published in education publications as far back as 2005 Jackie Williams (Associate Project Manager) ? Work experience starts at 2015 – graduate degree in Psychology 2014 David Doyle (Senior Software Engineer) √ Michael Nichols (Junior Software Developer) √ Adam Miller (Junior Software Developer) √

9. √ I – NDE project description nearly word for word from NDE website 10. √ M – No previous work experience together 11. √ M – No education experience

2. Litigation

√ P – No litigation

3. Financial Viability √ M – Financial stress score of 3 seems concerning, but according to the D&B report, this is lower than average risk P – Good credit – pays bills on time

4. Certificate of Insurance √ M – All of the policies expired 06/10/2018 ? Certificate of insurance doesn’t seem to include data breach insurance, that would need to be added if they don’t already have it

5. Required Attachments Related to Organization Qualifications and Experience Attachment 1 √ Attachment 2 √ Attachment 3 √ Attachment 4 √ Attachment 5 √ Attachment 6 √ Attachment 7 √ Attachment 8 √


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/6/2018 EVALUATOR NAME: Paula Roberts EVALUATOR DEPARTMENT: District reprehensive



1. Overview of the OrganizationLot of resources, changes and acquisitions

2. Litigationnone

3. Financial ViabilityModerate risk of financial stress On time payments Comfortable with the financial Viability


Provided

5. Required Attachments Related to Organization Qualifications and ExperienceOrganization chart looks appropriate for the project. The staff has a substantial work experinece Concerned that Winxnet does not have any K-12 experience


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group DATE: 9/7/2018 EVALUATOR NAME: Brian Keene EVALUATOR DEPARTMENT: Regional School Unit No. 56 – Technology Director

************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of Organization P – 25 years of experience. Hooked in with very highly qualified team members. NTT Data is out of Boston. P – Experience with using EdFi solutions to build data warehouse projects and work with school’s data for EdFi dashboards P or M – Depending on how you look at this it could be a plus or minus in the fact that there are multiple sub-contractors. Could be a positive thing as more hands on desk means things get done and also the more expertise involved is always a good thing, but on the other hand can there be too many hands in the fire so to speak. P – Worked with connectors and APIs for EDFI solutions P – Data Quality and Monitoring – Setup with validation rules

2. Litigation P - None

3. Financial Viability P – Looks like cash flow etc looks good no major issues


P- Included Certificate

5. Required Attachments Related to Organization Qualifications and Experience M – NTTData subcontractors listed do not have experience with data warehouse and or EdFi projects P – UniqID project members involved in this list also P – Experience in Educational data warehouses in other states by many of the employees


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 09/04/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization 1. √ M – number of ESP employees is missing as is the ratio of employees to clients

P – ESP, Learning Mate, and E-Scholar have lots of experience in the education space I – ESP seems to have acquired a number of former Double Line employees

2. √ M - Reference #5 for NTT Data is not an education related project reference 3. √ I – ESP directed Double Line in the Delaware project 4. √ P – NTT has cloud experience 5. √

1. √ P – Really gets the student data privacy aspect and understands disclosure avoidance techniques

P – Has experience with de-identified research databases 2. √ P – Department can control the rules engine – although knowledge of XML is needed 3. √ P – Experience with the state of Arizona

P – Understands the Ed-Fi mapping process 4. √ P – 10 states

P – Robust solution – even works with nicknames P – Merge and separate tools

5. √ 1. √ I – Didn’t indicate strengths, but seems to have a good model for managed services

I – Also didn’t identify any weaknesses, but realizes that communication is vital 2. √ P – Understands about pro-action instead of reaction 3. √ P – Understands that communication and setting priorities is important, while

understanding the small staff component 4. √ P – Strong processes and shared expectations

M – Didn’t describe processes 5. √ P – Plan seems strong 6. √ P – Plan seems strong

6. √ M – “enhanced” a cloud based product – didn’t implement a new system in the cloud 7. √ P – USDOE experience for disaster recovery 8. √

1. √ - All roles accounted for 2. √ - ? Org chart states that some will be assigned later, but all roles were accounted for in

regular chart 3. √

Uday Singh (Project Manager) √ - M - References are for solution architect, not for project manager. ? Architect experience is for Oracle – Ed-Fi and most education edw are in MS technologies Joshua Goodman (Project Manager) √– P - Lots of project management experience in education technology projects


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 09/04/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education

I – Executive Advisory Team – Glyn (ESP) and Mark (LearningMate) have a lot of education technology experience – But NTT has none Ben Granade (Business Analyst) √ - James Rife (Project Director/Information Architect) √-M-Responsible for all product development and service hosting engagements at ESP – since they didn’t quantify their current engagements, we do not know if he has capacity for this project Stan Jordan (Senior Software Developer) √ - P Seems qualified Rebecca Fanestiel (Database Administrator) √ - P – Microsoft Certified Chandana Chelamkuri (Quality Analyst) √ - P – Ed-Fi experience – M – Both references are from the same project Ed Jung (Infrastructure Architect) √ - P – Very qualified, lots of education experience Deborah Sterling (Business Analyst) √ - P – lots of education experience Amit Raval (Senior Software Developer) √ - P – Seems qualified Richard Helm (Data Analyst) √ - P – Ed-Fi experience, former Double Line employee Kathleen Browning (Infrastructure Architect) √ - P – Ed-Fi experience, former Double Line employee – I – References from same organization, but not same project Juan Guerrero (Infrastructure Architect) √ - P- seems very qualified Jessica Swafford (Technical Writer/Editor) √ Jeremy Thigpen (Trainer) √ - M – Only a few months experience as a trainer

9. √ 10. √ P – ESP and LearningMate have experience working together

P – ESP and eScholar have experience working together M – No reference as to whether or not ESP has worked together with NTT Data

11. √ Two references for eScholar M – no references for LearningMate M – no references for NTT Data

2. Litigation √

P - No litigation ? – UCC filings

3. Financial Viability √ P – Low risk – Good credit – pays bills on time ? – UCC filings

4. Certificate of Insurance√ I - some of the coverages end in the next couple of months ? Certificate of insurance doesn’t seem to include data breach insurance, that would need to be added if they don’t already have it

5. Required Attachments Related to Organization Qualifications and Experience Attachment 1 √ Attachment 2 √ Attachment 3 √ Attachment 4 √ Attachment 5 √ Attachment 6 √ Attachment 7 √



Attachment 8 √

STATE OF MAINE



BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/7/2018 EVALUATOR NAME: Joseph W Larrabee EVALUATOR DEPARTMENT: Education




Appendix H

1. Organization Overview:

P: ESP appears to have significant amount of Education experience

M: I think having 3 subcontracts is a lot to manage.

M: NTT Data – refers to as being responsible for NEO Cloud migration, but didn’t

find any references to projects or customers where they have performed a

migration as complex as NEO

Q: Not sure why need to partner with learning Mate if ESP has “DEEP” Ed-Fi

experience. If its for extending the financial piece, the process for extending any

Edfi api is the same.

M: NTT Data is going to provide on-site coordination but doesn’t have any EDW,

Edfi or education experience at all. I am guessing that they are also providing the

onsite work for the DW component as well. Hard to tell at this point without the

proposed solution

2. Experience in Education Enterprise Data Solutions:

Q: I am not sure they have ever implemented a major Education data warehouse.

It appears that they have taking existing systems and enhanced or built on top of

them.

Q: ESP: I am not sure if they have ever architected and implemented a warehouse

from the ground up. It appears that they might have created a significant number

dashboards and reporting tools. If they have, not sure why NTT data is performing

this work

M: Also, most of the references provided are older projects. While it might

indicate a positive relationship, long term contracts can be negative well. Could

imply that they are slow at completing the work

M: Reference 3: Says it’s the prime contractor but think it was Learning Mate that

did the implementation

STATE OF MAINE




M: Reference 5 : not a k-12 reference. Also says prime when it’s a reference for

NTT

M: There are no other refences for e-Scholar

3. Experience in Ed-Fi:

Q: They indicate experience but not extensive experience. For someone that states

they have DEEP ed-fi experience in 2.1.1 it’s not visible here. I am not sure if

leveraging ed-fi as indicated for Arizona is really a full implementation of ed-fi

M: There was no extensive demonstration of ODS other than being the integration

contractor for Doubleline and OTIS. Curious why they didn’t partner with OTIS

again.

P: Seem to have a fair amount of dashboard experience

M: Should have maximized the space allowed and provide more detail around

ODS and integration activities

4. Experience in Cloud Migration / Managed Services:

M: Category says prime, but description states it’s a subcontractor

M: Explains mostly what they did from a project approach. Would like to have

seen more details about the technical aspects of the app that they moved. Similar

to the specs that we provided about NEO ect.

M: While we asked for 1 reference, it would have been helpful if there where more

refences to determine if this was a onetime project or have had many projects of

moving to the clouse

5. Responses to use cases/questions:

5.1 Student Privacy – seems like a very general response

P: Provided 4 methods for managing privacy

P: Research paper appeared to detail out privacy. Not sure how this trickles

down to implementation

5.2 Data Quality and Monitoring

M: should have used the entire space allowed to provide as much detail as

possible

P: Appears that they understand the need and have a mechanism (xml) in place

for the department to manage rules. Also mentions the Level 1 and Level 2

validation process.

5.3 Financial Data

STATE OF MAINE




M: Offered 6 steps to implementing – but basically repeated steps in RFP scope

(23)

M: Didn’t speak to data migration from existing Financial WH

M: Spoke to more about GASB than NCES

Q: Not sure they understand what the Financial WH contains for information.

Our warehouse is about budget data.

5.4. Unique IDs:

P: Offered a short-term solution and long-term integration solution (Page 20) It

appears that the short term solution is a plug in to get the districts using the tool

and a long term integration to the DW

M: While there was mention of merging IDS in the tool, there was no mention

on how merging of IDS would occur in the DW

5.5. Managed Services (limit 2 pages):

5.5.1

P: Listed some proactive steps how and key elements

M: Didn’t identify which ones they were strong in

M: Stated they don’t have any weakness

5.5.2

P/M: Mentioned about having a framework but didn’t provide any details what

that frame really was about. Not sure if this is something that have

implemented else where

5.5.3

Q: Appears they were responding to the project challenges vs the challenges of

managed service. References project and project schedule

5.5.4

M: Didn’t really identify a problem. Used the word “may”

5.5.5

Q: Appears that they are referring to the project level details. Not Managed

Services type communication

M: No mention of help desk type of communications

5.5.6

Q: Not sure I understand the information presented. They offer some data but

not sure that it addresses behavior and performance of the system

6 Cloud Implementation

M: I don’t think a 5 month project is large for a new project and the

description says it’s an enhancement

STATE OF MAINE




M: States that it was the Prime Contract that did the implementation, but refers

to NTT (sub) in the project description

7 Disaster Recovery

M: Not sure if the project best describes their experience in Disaster Recovery

2. Litigation

Doesn’t appear that any of the Contractors (prime or subs) have any litigation against

them


From the untrained eye, looking at the ratings provided, they seem to be financially

stable and capable of supporting a project of this size.


Policy is active but due to expire in September/October 2018


8 Personnel

8.1 Summary Table

P: A name was provided for each of the Services as well as Executive Level

Positions

M: Wish the resumes went in order of the chart.

M: Project Manager with 0 Experience - – concerned if he is going to be the

onsite representative for the DW component

M: Business Analyst with 0 Experience – concerned if he is going to be the

onsite representative DW component

M: Couldn’t really verify years of Edfi/EDW experience with information

that was provided in the resumes.

P: Average EDW Experience > 72 months (consider>60 a Plus)

P: Average Edfi Experience is <40 months (consider>60 a Plus)

8.2 Org chart

P: Have identified resources in respective Teams/Functions

M: Project Manager – Josh Goodman shows up as Prime in summary table

but shows up as Business Manager on Org Chart. They should only

identify a single PM for the entire project.

Q: Not sure if the unassigned positions will be coming from current staff

and resumes or are they planning on adding new team members.

STATE OF MAINE




M: No indication of managed services and who that reports too. Shows

phase 4 – but is that for support? Would have liked to see an Support

manager. Right now, the support is to the PM

8.3 Resumes

M: Uday Sing – according to the Attachment 4 has extensive experience in

warehouse implementation. Would have like to see at least 1 reference to

leading a warehouse project.

M: Kathleen Browning – inconsistency in dates. Texas ED says 2000-2015

but in detail says 2000-2010 (p 78.79)

M: Jessica Swafford - inconsistency in dates. ESP says start in 2010 but

detail says 2009 (p 82)

M: Jeremy Thigpen – inconsistency in dates. POEEd says 2014-2018, but

in detail Says 2010-2018 (84)

M: Some reference went over the 2 page limit

P: Staff appear to have strong technical backgrounds, still not sure about

overall cloug migration experience.

9 Prime / Subcontractor References

Comments provided as part of Experience

10 Prime and Subcontractor previous working relationships

M: NTT Data – No previous working relationship

M: e-Scholar and Learning Mate – only provide 1 example each and learning

mate is over 5 years ago.

11 Subcontractor K-12 references

M: Provide 2 references for escholar but the references are 6 and 13 years ago.

No recent references

M: No K-12 reference for NTT Data

M: No K-12 reference for Learning Mate


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/6/18 EVALUATOR NAME: Hazel Stevenson EVALUATOR DEPARTMENT: Admin and Finance




Subs NTT data, LearningMate, eScholar 25 years’ experience with state education, data standards (ed fi), managing implementations Deep understanding of data systems ESP and LM licensees of Ed-Fi, involved in working groups NTT 50 years of system integration, large IT projects and managed services, significant SOM experience, very large company, LM, large number of resources, industry standards, business intelligence ESP previous partner with eScholar, prime contractor 8+ major statewide implementations, leading experts in data and technology incl EDFacts

P Reference projects – Ed-Fi and DWHs, phased and ongoing successes, on time and still operational P Complex cloud migration and managed services

P Presented privacy paper 2017 P exp in confidential data in all sector; collective experience with similar mandates P-rules engine/ data quality service being considered by ed-fi alliance as reference validation module for ed-fi AZ financial system modernization experience; use Maine’s financial warehouse Escholar solution – join 10 states /37m unique ids P- NTT application management outsourcing framework; top 10 global IT service provider P- good assessment of challenges; solutions suggested Staffing: P- most staff have experience 50 months in Ed DWH, several100+months, well qualified P – experienced PM staff, some with PMP P- org chart has Project management depth and an executive advisory team, PM and BA on-site, support team onsite

2. Litigation

none



RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/6/18 EVALUATOR NAME: Hazel Stevenson EVALUATOR DEPARTMENT: Admin and Finance

Subs self-finance Privately held but initiated D&B for proposal as prime Appears low financial risk


Provided; cyber liability will be required if contract awarded


provided


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/7/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: DAFS/OIT ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization - P • ESP – 25 yr history • ED-FI certified & Extensive ED-FI experience – prime in 8 states • 3 reputable subs – NTT Data, Learning Mate, eScholar – total of 47,000 EEs+, well defined roles on MEEDS

project • Both ESP & Learning Mate are ED-FI certified since 2014 • ESP has worked with 25 states • NTT Data has moved applications to a major cloud vendor • Learning Mate has experience working with CGI (Advantage) 2. Litigation- P No litigation participation (in any fashion) in ESP’s 25 year history 3. Financial Viability- P • No corporate debt • Significant investment in R&D • Good payment history – all cash x-acts, no outstanding debt as of 7/2018 • Very low risk • D&B rating ER5 • Credit Score: 590 Percentile: 94 4. Certificate of Insurance- P Good – General Aggregation Limit applies per location 5. Required Attachments Related to Organization Qualifications and Experience • When did they work with Maine? – Q see page 8 • What cloud provider was used for LexCall Phase2 – Q see page 23 • Learning Mate implemented same FERPA/HIPAA mandates that Maine has to use • Using Sub eScholar for unique IDs – P see page 8 • Organized – will schedule SMEs so no surprises P see page 21 • NTT Data has successfully moved an application to Azure (major cloud provider) • ESP 3 client references: pge 86, 87 • NTT Data 3 client references: pgs 14 & 16 • Learning Mate 3 client references: pgs 12,13, 87 • eScholar 3 client references: pgs 87, 89


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/6/18 EVALUATOR NAME: Ida Batista EVALUATOR DEPARTMENT: DOE ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization (P,Q) Organization has solid history working with Ed-FI and EDS. However, it is unclear to me how these multiple entities are organized together. When were the variety of entities brought on? How they work together? I also found the information on eScholar limited.

(I) LM has experience with SEA finance with Az. (I) eScholar unique identifiers appears comprehensive.

NTT Data experience with cloud migration and managed services. (Q) Data Quality and Monitoring rules editing – are there modes to monitor redundant rules or track historic rules and present utility. (Q) Vague on Performance metrics in Managed Services

2. Litigation (P,Q) ESP had no litigations, which is good. Does that also mean that the companies in the group also have had no litigations related to the type of work they could be proving for us?

3. Financial Viability (Q) IF ESP works on a cash basis and is the single point of contact for the DOE, what does it mean or translate in implementation that “each subcontractor will self-finance their own operations and any retainage on their deliverable”? In and of itself ESP appears financially safe however there is no information about NTT Data, LM, and eScholar?

4. Certificate of Insurance (Q) Does EPS certificate of Insurance blanket all companies under it? Is the value of the insurance sufficient for this project? Are the expiration dates of 09/2018 or 10/2018 problematic?

5. Required Attachments Related to Organization Qualifications and Experience (P)The personnel presented have a lot of experience with Ed-fi, Ed Facts, data warehousing, ETL, architecture, project management, training, technical writing, business & data analytics. (Q) Seems like several personnel on multiple projects (how will time be split among projects?) NTT Data experience little to no experience with Ed-Fi or education data warehouse. (Q) P42 Organizational chart, It is not quite clear to me how the role of Project executive and executive advisory team works with the other branches of the chart and if this meets our needs. Also, question where the NEO Helpdesk fits into this chart and if some of the onboarding of new members is realistic?


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/16/2018 EVALUATOR NAME: Paula Roberts EVALUATOR DEPARTMENT: District Representative ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Overview of the Organization Concerned about the number of different companies, but each company looks stable and experienced.

2. Litigation None

3. Financial Viability Financial Condition fair

4. Certificate of Insurance Supplied

5. Required Attachments Related to Organization Qualifications and Experience Project manager has 0 months of experience in Data Warehouse nor edfi

Stage 3


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: Double Line NAMES OF EVALUATORS: Ida Batista, Brian Keene, Charlotte Ellis, Cathy Harrison, Paula Roberts, Hazel Stevenson, Joseph Larrabee DATE: September 24, 2018



Proposed Services



Appendix I:

1. Project Understanding Recapped understanding Seemed to understand Ed-Fi and Student portions of the project

2. Overall Project Organization and Approach 2.1 - Overview Minimal information, did not address RFP Lacked substance in the proposal Stated they would leverage experience, no examples given 2.2 Project Management Plan Methodology based on Agile, Scrum, Kanban and Pmbok (quite a mix, no specifics on how they interact) Named a lot of different tools, could be hard to manage that many Operational readiness is not specific (baked in) All work tested in local environment (how does this work with AWS?) No real details about communications 2.3 Strategic Roadmap and Timeline Provided roadmap: training plan and deliverables are at end of project which is a conflict/concern Doing Identity Management after Student and Staff data seems backwards No real detail to timeline beyond basic requirements from RFP to show how it would be implemented 2.4 Technical Architecture Show all data going through ODS, may not work for all data sets No specific detail about how requirements will be met Only minimally addressed technical requirements Some security details, don’t specify which SOC2 report type 2.5 Overall Staffing Plan Confusing Lots of overlaps No trainer or technical writer Unclear if they are only committing to keep initial NEO team for 1 year or 1 year post-implementation of NEO Managed services Didn’t have time or FTE requirements for State staff No continuity between phase 2 and phase 3 for staffing plan Good onboarding and cross-training of new staff




3. NEO Cloud Migration 3.1 Approach Doesn’t demonstrate an understanding of the scope of the project Assuming some existing systems are data warehouses, when they are much more complex and the interfaces require more than a data warehouse would 3.2 Transition Roadmap and Workplan Don’t appear to be bringing in Winxnet until implementation 3.3 Roles and Responsibilities Lacking detail in roles and responsibilities on both vendor and State side

4. Proposed Enterprise Solution 4.1 Product Roadmap and Workplan No key milestone dates 4.2 Solution Discovery for entire project is only 2 months Using a lot of AWS features and scalability for peak demand Don’t seem to be aware of the size and complexity of the Ed-Facts reporting (reported they expected to do up to 10 reports while Ed-Facts has over 200 reports that need to be maintained) Rules engine is in front of ODS and Data Warehouse Seems to be conflict of ID Management System diagrams, one showing data going back to ID Management System, one shows no data going back, all one-way flow MAARS and financial all seemed to be lumped together with minimal information given AWS Cloud Formation template is a good tool for use until everything is migrated 90-day warranty is unacceptable 4.3 Data Migration Only discuss migration of staff and student data, no mention of financial and organizational, nor any mention of non-Ed-Fi data moving No clear indication of where all data was migrating to Lists that department “may” review data

5. Operation Support Model In the diagram, nobody seems to be responsible for NEO Winxnet only has one task Two prong support: who decides who is responsible for the support? Footnote 13 has no reference Details are going to be established later, nothing solid now Routine maintenance will be done Saturday and Sunday which is good Chart shows only one option from Help Desk, which is “Issue Resolved” but what if it isn’t resolved? Data Management team is not clearly defined (workflow/role) 5.1 NEO DoubleLine and Winxnet support roles not clear Very vague




5.2 Enterprise Solution Mention technical deployment, no details about transition to production No mention of knowledge transfer

6. Training Mention technical architecture and maintenance training. Unclear what level of responsibility this transfers back to the State to take over these duties to maintain and support it Mention training for a single SOM DBA Training section response leads to lack of information on ongoing responsibility for maintenance and support Recapped RFP requirements without much additional detail Assumptions No assumptions given


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER: ESP Solutions NAMES OF EVALUATORS: Ida Batista, Brian Keene, Charlotte Ellis, Cathy Harrison, Paula Roberts, Hazel Stevenson, Joseph Larrabee DATE: September 24, 2018



Proposed Services



Appendix I:

1. Project Understanding Reiterate the vision, assumptions show understanding of the project and need for clarity in order to be successful Some items will need to be negotiated Showed experience in phases (Ed-Fi and Financial Management) of project

2. Overall Project Organization and Approach 2.1 - Overview Concise but clearly hit the high points Include high level success factors Really emphasize on-site presence (very good thing) 2.2 Project Management Plan Using NTT project management methods, have a governance model, and baseline the plan Didn’t really identify a specific methodology they are following, appears to be Waterfall Test operational readiness before deployment 2.3 Strategic Roadmap and Timeline Appropriate training is outlined Good high-level vision of milestones Want to implement latest and greatest versions at each step Too many concurrent tasks going on, may not leave enough time to respond Recognize school calendar as being important ID Management and Data Integrity taken care of up front 2.4 Technical Architecture Single portal/landing page for all user roles Azure Platform will only include US Cloud Regions, only stateside which shows they understand requirements Utilizing Microsoft Azure helps leverage Microsoft Stack used by Ed-Fi and NEO Security availability Integrity and confidentiality all handled with SOC2 Type 2, referenced in Appendix J 2-Factor Authentication Seem to really understand concept of the NEO Migration Scope No subcontractor involvement in Technical Architecture




2.5 Overall Staffing Plan Good contingency plan with 100,000 professionals worldwide Depth of experience with team, and ability to bring them in Show flexibility in staffing being able to dynamically adapt as needed Strategic Staffing Division at NTT is a benefit Don’t reference any other State technical resources that might be necessary, only SME and PM

3. NEO Cloud Migration 3.1 Approach Made assumption that application will work in new environment with configuration only Identified that they need clarity in the magnitude of the NEO migration Going to figure out details in discovery phase 3.2 Transition Roadmap and Workplan Identified that they don’t know the full level of complexity Use current NEO developers in the onsite support team Not much detail 3.3 Roles and Responsibilities No mention of SOM’s roles and responsibilities Advisory has authority to directly direct resources, there are positives and negatives to this

4. Proposed Enterprise Solution 4.1 Product Roadmap and Workplan 30 months to accomplish Phase 3 is aggressive 4.2 Solution Recognize risk of ODS API version 3.1 being announced byEd-Fi Alliance, may affect the timeline Discovery happens at varying points depending on the timing of the component Written requirements will come out of every discovery Initial Ed-Fi implementation will have data mappings created by hand Seem to have robust ID Management system that even deals with twins Mature rules engine for data integrity Rules engine can be used through XML Question of how accurate assumptions are for financial information system Single place to look for all validation errors Understand the need to work with other vendors Interactive workshop approach is good Public reports are available in 4 most popular Maine languages ID Solution is part of the community of 10 states Question about IDs being given to parents as well as Students and Staff Understand need to document data warehouse rules Reference to section 508 guidelines shows understanding of Maine’s technical requirements Tools are scalable and sustainable Understand complexities of assessment and accountability data Seem to expect department to provide antivirus software All data is encrypted at rest and in transit as required in Appendix J Recommended a pilot which is good




4.3 Data Migration Reference to offshore project manager and offshore data migration, could be conflict to data residency No details on how data that doesn’t fit Ed-Fi model will be migrated to data warehouse No information on how data from other systems like financial or MAARS will be migrated Good identification of SOM resources and roles

5. Operation Support ModelNo mentions of outages, planned maintenance, no mention of workflow No reference to Appendix M in this section

5.1 NEO Tailor life cycle methodology and use life cycle process Said team will use existing State DOE ticketing system, but were asked to provide one for this project Assumption seems to be that if someone is working on site they will use State equipment Don’t give number of people who will be on site

5.2 Enterprise Solution 90-day warranty is unacceptableReference to deployment certification shows they understand State processes See one, do one, teach one is a good approach with job shadowing Training analysis is good Unclear what level of responsibility will be transferred over to the State staff including technical staff

6. TrainingStated that trainers will have instructional experience Will record webinars for subsequent delivery Have a training plan as a deliverable Developing material as it is being phased in Including field training 120 end users seems too low

Assumptions Each assumption will have to be negotiated. Examples include but are not limited to: Assumption 3, 3 days for turnaround is problematic Assumption 4 is unclear as to what “certain responsibilities” are Assumption 6 (facilities, software, etc.) is unclear as far as what is being provided and how many of each, and who it is for Assumption 8, it is unclear if the times listed are per time zone, and if this is for help desk Assumption 11, how will State be expected to load its own UAT database? Assumption 13, will state be responsible for all legacy systems, including NEO? Assumption 15, what would be the impact of changing assumptions on cost? Assumption 16 does mention SLAs and Appendix M Assumption 19, some concern about state providing licenses vs vendor providing licenses


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 09/19/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Project Understanding M – Double Line seems to understand the Ed-Fi and student portion of this project, but doesn’t seem to understand the scope of the other portions

2. Overall Project Organization and Approach 2.1 Overview – M – No real substance as to requirements, challenges, outcomes, or conditions 2.2 Project Management Plan -

• Governance – P/M Lots of tools – this can be a plus because there is lots of ways to query and find information, it is also a minus – how does one know which one to use?

• Scope Management √ • Schedule/Time Management P – JIRA to determine Estimated Time to Completion and

Actual Time to Completion • Communication – M – no details • Risk √ • Document Management √ • Operational Readiness √

2.3 Strategic Roadmap and Timeline – ? How can they do Identity Management after student data and staff data – the IDs are needed for those data collections. ? Why are EdFacts and Generate so far apart on the schedule? M – No training until 2022 2.4 Technical Architecture – M-they have all data going through the Ed-Fi ODS into the data warehouse. That may not work for all datasets, especially historical data 2.5 Overall Staffing Plan

• Staff Contingency Plan – P Cross-project discussions, onboarding documents • Staff Management –

o ? Is it industry standard to have PM be responsible for Human Resource and Issue Management – seems like a lot while also trying to manage the project itself

o M - Chart has some people doubled, testing and training positions, other positions (IT Support) were not identified in File 2

• Use of State Staff – M – no detail as to time or FTE requirements for State staff 3. NEO Cloud Migration

3.1 Approach – M -May be underestimating the scope of the project 3.2 Transition Roadmap and Workplan M – Appear to be bringing Winxnet in at Implementation and not involving them in Discovery and Interfacing 3.3 Roles and Responsibilities √

4. Proposed Enterprise Solution 4.1 Product Roadmap and Workplan √ 4.2 Solution


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 09/19/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education

• M – didn’t follow the Discovery, System Design, System Construction, Integration and Testing, Documentation, Hosting, Application and Warranty Services flow for each component –Some items are discussed in each section, others are mentioned as their own pieces. Difficult to follow. Only mention discovery for the overall project – they do mention that it will be iterative.

• M – Do not seem to be aware of the complexity and size of EDFacts reporting (this is public information)

• P – I like the rules engine in front of the ODS and another one in front of the Data Warehouse • ? - Weird sample • P – I like their testing strategy • M – Warranty of 90 days is not appropriate or acceptable. Some functionality only gets used

once a year 4.3 – Data Migration M – only detail staff and student data migration, assessment data migration was mentioned in 3.4.2, but financial and organization data is not mentioned at all, nor how data warehouse elements that are not ED-Fi related will be moved into the EDW

5. Operations Support Model M – in their diagram no one seems to really be responsible for NEO M – Concerned about two pronged support – who decides what the issue is (environment/application) – how is that decision made – don’t want to get caught in a circular mess ? Footnote 13 has no reference (page 95) 5.1 NEO √ 5.2 Enterprise Solution – M Description is for deployment methods, not a process for transitioning M – No details

6. Training √

STATE OF MAINE



BIDDER NAME: Double Line, Inc. DATE: 09/21/2018 EVALUATOR NAME: Joseph W. Larrabee EVALUATOR DEPARTMENT: Department of Education



Overall – I thought the RFP didn’t read well and was very hard to follow. Since this was the proposed solution section, I expected more detail around the solution and project methodology. There was a lot of vagueness in the use of Winxnet that I thought would be resolved in the solution, but it didn’t provide much additional detail on how they were going to be utilized. The proposal seems to be more about what the vendor will do vs what the vendor already has for functionality and can provide. I was hoping that the vendor had more functionality already in place from other states but it seems like there is a lot of ground up customization. Still concerned about the vendors limited staff and ability to complete the work in a timely manner.

Appendix I – Proposed Services

1.0 Project Understanding

- Have a basic understanding of what is expected from the Ed-FI/DW functionality but nomention of the NEO component as part of the overall project scope.

- Spent half of their allotted space (last 2 paragraphs) talking about their qualifications and notabout what they understand the project to be

2.0 Overall Project Organization and Approach

2.1 Overview

- No mention of any potential challenges or any significant requirements that might have to beovercome

- Again, spent more time explaining who they are then what they are going to do

2.2 Project Management Plan

Methodology

- mention of using “Adaptive Practices” such as agile and scrum – appears that they willbe using a hybrid methodology. Uses a mix of terminology through response – refersto Kanban

- essentially it appears that they are using an interative development process with somescrum activities

Governance

- Didn’t really go into detail on a governance strategy- Does indicate some components of a plan such as ownership, expectations,

interventions. Would like to see more detail around these components

Scope Management

STATE OF MAINE




- Combined with changed management

Schedule/Time Management

- typically, with Agile projects, time is based on estimating the work and teams velocity and ability to complete the work.

Communications

- did mention the daily standup/scrum meetings. Would have also liked to see other agile means of communications. IE Sprint Planning, Reviews, retros ect

Risk

- again, if this is an agile effort, risk management is not a structured as with a waterfall project. Risk management is typically handled in daily standups and sprint reviews. Having iterations is the way to manage the risk

Change Management

- Refer to a change control process to evaluate items. If they are truly using a Agile methodology, change control doesn’t truly exist.

- Change management in Agile is typically handled in prioritization of the backlog. The only place that change management might have to be implemented if there additional functionality that was not identified in the RFP.

Document Management

- describes the physical process and location of storing documents. I would have liked to see more information on creating and updating system type documents. There was no technical writer provided as part of the staff in Qualifications. I am not sure who is actually going to be creating documents

- Graphic in document management section didn’t seem to fit with section. Not sure what it was intended for

Operational Readiness

- stated its “baked in” but no mention of what it is. Could have elaborated more -

2.3 Strategic Roadmap and Timeline

General Comments

- Don’t think they provided a roadmap. Looks more like a modified gantchart with its associated graph.

- Timeline has dates for phase and deliverables but doesn’t really indicated any major milestones with the specific phases

- Based on the limited staff – I am concerned with the amount of concurrent activities they have and not knowing how the staff will be resourced

- There is no indication of UAT/Deployment certification in their timeline

- Not sure if their end dates for the deliverables is into production. There are no migration/implementation activities

- Plan doesn’t indicate any dependencies between deliverables

- Overall Plan seems a little aggressive. Leaves me to think they might not understand the complexity. All the work is done 14 months before the end of the first period

- Does appear that they have Organization and Student training at the end of major deliverables, not sure why the other modules don’t have training identified

- Deliverables – only deliverables that were shown on timeline was what was in the RFP. There was no additional thought put into other deliverables that might be required.

Roadmap Specific Content

STATE OF MAINE




- Not sure why the Warehouse component is broke out as its own

- Ed-fi Dashboards are indicated twice – under a.i and a.ii Not sure if a.ii was supposed to be data visualization since its not shown on the plan

- If generate is out of the box why does it take 12 months to implement..I am guessing there are other dependencies but doesn’t really

- Organizations and Student data have training identified in the roadmap but the training plan is delivered until the last month of the development work

- Reporting Dashboards – there was no mention of the reporting dashboards but show up on timeline. 6 Months to develop 8 dashboards seems aggressive

Timeline

- Timeline is really just date ranges with the deliverables by funationlity, that we had in the RFP.

- Would have like to see more about their milestones and what they were proposing along the way than repeating back the RFP

2.4 Technical Architecture

- looks like the details provided in the RFP are only for the Ed-FI and not NEO as well. Not sure where NEO stands in the architecture. Is it sitting in the same space as EDW.

- While there is a diagram of the architecture, which is very similar in nature to the one provided in the RFP, doesn’t specify if its SQL server based

- Is it PAAS or IAAS - Mentions amazon as a partner but what about Winxnet?

2.5 Overall Staffing Plan

- Staff contingency – speaks to forecasting sprints, but doesn’t really talk to estimating the backlog. Also refers to adding resources to sprints as needed. Typically a scrum team is consistent and doesn’t change in size based on a sprint. It’s hard to get a team velocity and project the future if the team is constantly changing. I would have expected them possible say they could spin up another scrum team but to meet the need. Also, they don’t mention where these resources will be coming from. DO they augment with contracts to meet the demand?

- Staff management – doesn’t describe policies that they have, just speaks to following them. Again, staff augmentation through contractors?

State Staff

- Mentions the need of state staff for requirements, but no mention of testing? - Reference a SOM DBA – not sure what they think the DBA is for. What data

structures are they expecting to understand? These are all new to the department. Are they referring to NEO because the footnote mentions data structures for ED-FI and new modules.

- I think there should have been mention of the state staff coordinating with existing vendors. I don’t think they realize that most of the existing systems are coming from outside vendor apps

There is a table in the appendix with roles and responsibilities

- IT support not on original staff list provided with Organization Experience - Solution Architect not on original staff list provided with Organization Experience

STATE OF MAINE




- Have total of 23 people dedicated to the project for the entire duration. They only have

a total of 38 people in their organization

? Is there a scrum master? Who oversees sprints? Is it a PM?

3.0 NEO Cloud Migration

3.1 Approach

- There was no mention of any assumptions that might impact the overall timeline - Uses AWS Application Discovery Service and AWS Migration HUB - refers to their local AWS team – thinking this might be Winxnet. Would have been a good

opportunity to go into more detail on their use of them - Seems like a lot of emphasis is being put on AWS and their toolset, concerned that this

migration might not be a straight forward effort and I don’t think they are prepared for it. - Testing – There was no mention under roles of testing expectations of the SOM. There were

technical expectations but no business. - will the Department allow these AWS services to be installed on servers and VMS to be able to

collect the data? - Do they have to go through some kind of deployment certification process first?

3.2 Transition Roadmap and Workplan

Overview - there are no State Resources mentioned in the roadmap/workplan. Also, would have

liked to have seen more detail.

Roadmap - Not sure what is mean by execute support services. And how the that differs from

Managed services. - Also not really sure when the cutover period is, if that’s the SLA agreement milestone

Managed Services Task - Since DOE is providing level 1 support, I would have expected to see the Helpdesk as

a resource. Also not sure why the PM is part of the managed services task. I would have expected to see like an Operations Manager at this point.

3.3 Roles and Responsibilities

- refers to 4 individuals but in the description of the individuals calls for a development team. Is a team of 4 skill sets (PM, AIL, Dev and BA).

- No mention of testing – who is responsible for testing? I would have expected to see something to the effect of UAT will be completed by each Module/Business Owner.

- NEO Team – is their plan to consume the NEO team before migration to the cloud. It appears that they think this team are SOM but dedicated to the migration. This team will still need to support existing functionality while the migration is complete

- There is no mention of what is expected from state resources outside of the NEO team.

4.0 Proposed Solution

Overview - based on experience with other states, I feel that the vendor does understand the

Departments desire to implement the Edfi standard.

STATE OF MAINE




- I don’t feel that the RFP fully understood or addressed the other areas that are notrelated to Edfi such as MAARS and NEO cloud migration.

4.1 Product Roadmap and Workplan

- States that they will complete the four phases of MEEDS by 6/30/23 but there are really only 3phases. Phase 4 is managed services and long term support. In fact RFP only mentionsphases 1-3

4.2 Solution

Overview - Components didn’t go in the order of the RFP. Hard to follow

Discovery

- If they are doing an iterative approach as indicated in their schematic, discovery shouldn’tbe a series of meetings. Discovery should be the creation of a backlog with constantgrooming and refinement. Creating backlog items as you go.

- Appears that their plan is to come and get all the requirements up front - will only haveaddition meetings if necessary. Should be conducting a story mapping/requirementssession at the start of each deliverable.

Ed-FI

- 20 data collection systems? Where did they get this number? Are they considering NEOas part of them?

Data Visualization Tool

- The reporting and dashboard tools – are these custom or 3rd party such as Tableau?- Looks like they don’t even have a tool already but instead are going to be building one

from the ground up. Not sure why they aren’t using 3rd party tools.

CIID Generate Tool

- Assumption is that it will work out of the box except to connect to ODS

Education Organizations Management Tool

- Has implemented similar functionality for Michigan- Does not mention any data migration

Student Data

- Lumps Student and Staff into same section- Not much of a proposal rather give us back the information in the RFP in paragraph format

ID Management Tool

- Implementing the ED-FI ID API- Uses AWS Elastic Search as its matching tool for matching- Website says its encrypted data, not sure about meeting requirements for PII- Not sure how new IDS get created and what is the numbering system for it- Migration of existing data?- Utilizing of existing IDs?- What is the process of Merging IDS – is this a custom function- No mention of working with the SIS vendors other than the diagram

STATE OF MAINE




- Diagram on Page 89 shows data going back to SIS, Diagram on 78 shows ID

management as a different Path to ODS - Does this process tie into the Rules engine – not sure if DOE can manage the matching

rules and the feedback reports from the AWS search tool - Not sure if this tool is for just getting an ID or creating and storing the student profile as

well. Maybe Profiles are stored in ODS and this is just and ID generator?

Data Integrity Solution

- Tier 1 validations – stated that contains many out of the box validations – would like to have seen an example of what one might be

- Reports - typically on building 2 reports for data validation. Reference 2 - Mentions training DOE to manage data rules - Speaks to running rules against DW – is this considered a tier 3 rules enginer. Rules

against DW might be different than ODS - States failure reports can be sent to emails. I would think that this might an issue with PII

data and not a secured email server. - I didn’t seen any mention of Dashboards to review the reports. RFP calls for a DOE and

Data owner dashboards

Financial Accounting for Local and State School Systems - Lumps MAARS and Financial into same section - References extending API for NCES - Working with account vendors to be certified - Extending rules engine - Data conversion and migration from legacy dw

Maine Assessment and Accountability Reporting System (MAARS)

- Lumps MAARS and Financial into same section - References MAARS rfp and updating WH for data - Migration of existing data from the MAARS system is different than the RFP. - Does not address future assessment updates from assessment vendors, does show on

graphic on page 78 but there is no mention in text - No mention in text about creating APIs for future updates - Not mention of Updating Rules engine

EDFacts Reporting

- Only going to maintain 10 Edfacts reports outside of what is in Generate - Not sure why it only references phase 3 for EdFacts. EdFacts will have to be supported

until all reports are in Generate

Staff Data - Lumps Student and Staff into same section - Doesn’t mention about extending the ID tool for Staff data. Whats the process of making

sure that there are multiple Staff person profiles. - Will update Rules engine - Will work with district HR vendors for certification? - Would have liked to see more around process they go through for Certify any API with

external vendors (staff, student, financial) - No migration of legacy data

STATE OF MAINE




- No mention of modifying NEO to connect to ODS for districts without HR systems

Reporting Dashboards

- Almost missed this section. Section header was by itself on previous page- Not much details about what the dashboard looks like or functions. Appears that the

vendor doesn’t have a public reports dashboard in toolset.

Testing

- RFP specifically mentions use of automated and repeatable testing tools, there strategylooks manual based and doesn’t speak any regression type testing.

- Doesn’t speak to roles and responsibilities of testing. Appears they do it all, but myassumption is that UAT is done by DOE

- Uses AWS tools to test and validate environments/infrastructure

4.3 Data Migration

- First paragraph only mentions migration from student and staff. Seems to only focus on thatthroughout the section.

- Will data go through Tier 1 and 2 validations before it’s into the warehouse- Does everything go into the ODS from the lacy systems? What about data going directly into

the warehouse? If items go into the ODS first, do special ETL’s have to be created then movedata into the WH. Thinking more in the line of financial or other summarized data

- What are the SOM responsibilities with respect to data migration?

5.0 Operation Support Model

- Managed services are split into 2 categories – Application and Environment – concern who ismanaging the inbox for the helpdesk. Everything should go through Dbl line and then routed

- Application – Double Line- Environment – Winxnet- No mention of a ticketing system – diagram refers to logging an issue but not where- No mention of Staffing plan- Has no existing support model states WILL create one- No mention of knowledge transfer- Operational fixes – do they follow the same development/testing methodology as the initial

product does. Analysis, development, testing, promoting through environments?- How are routine/upgrades ect fixes implemented? Talks about downtime, but what about roles

and responsibilities.- I am assuming that the digraph and process flow apply to both NEO and EDS- Footnote on Process flow calls for a different flow for “smaller” pieces of work and idffernt for

larger. Not sure what constitutes small vs large. Is it tier 2 or tier 3?- Is the workflow the same for the Warranty items?

5.1 NEO

- Refences existing NEO team to answer questions. Is this the expanded NEO support or the 5developers?

- Are the 5 NEO developers part of the on-site Double line team?

- Doesn’t mention any SOM needs/roles?

5.2 Enterprise Solution

STATE OF MAINE




Transitioning from dev to Prod

- Use automated deployment across all environments.- No mention of UAT/Deployment Certification

Warranty

- Refers to warranty of 90 days but doesn’t indicate how issues will be handled. Do they gothrough the same process (graphic) shown for levels 1 to 3

Knowledge Transfer

- Refers to training approach, but training is different than knowledge transfer.

6.0 Training

- There was no mention of training districts- Limited information to documentation/training materials- Not sure of who is going to be doing most of the training. We had asked for a formal Trainor

position in RFP but there wasn’t a resource provided





Proposed Services:

1. Project understanding iterated AWS cloud platform

2. Project Organization M Overview -minimal information, did not address RFP, repeats org quals Plan: Methodology based on PMBoK, agile and scrum, mention KanBan, discuss source control tools, deployment tools JIRA for tasks, SP, Confluence M -operational readiness is not specific

Roadmap and timeline- Provided roadmap M Training plan and deliverables is at end of project – either after deployment or nothing is deployed? 1b migration overlaps Phase 2 ed-fi API and ODS. Phase 3 and Phase2 overlap – reasonable? M – work plan does not have level of detail beyond requirement stated in RFP to indicate how bidder came up with timeline.

Architecture AWS architects involved with Winxnet to meet hosting and availability requirements - some security details, SOC2 (type not indicated) - not much specific detail about how requirements will be met Did not address App J requirements

Staffing plan List state roles/responsibilities at high level M Only commit to keeping current team for 1 year which doesn’t get through 18 month NEO migration

3. NEO Cloud migration Team of 4 M- planning “AWS hybrid cloud” for data warehouses – left on-prem M -Responsibility of SOM to provide resource to assist with utilization, performance and data transfer Appears to be inconsistency about contracting NEO team for 1 year Does not address interfaces

4. Enterprise Solution P Uses AWS features, scaleability for peak demand



P uses Ed-FI on AWS cloudformation template developed by DL P Data Warehouse solution customizes existing solution, leverages components from other states M-CIID assumptions need validation P- rules engine for data validation will also validate DW Data quality validation in discovery phase

Operation Support model M- split managed service model – details will be established M- relies on SOM HelpDesk as intermediary for communication of issues Neo – DL/WinX support roles not clear Enterprise – focus on technical deployment, doesn’t address operational

Training TTT, provide high level info, recaps requirements in RFP without much additional detail


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/22/2018 EVALUATOR NAME: Brian Keene EVALUATOR DEPARTMENT: Technology Director – Regional School Unit No. 56



3.1.1 – Project Understanding P – Ed-FI Experience includes state implementations for Nebraska, Arkansas, DE, Michigan, Penn, New Mexico, Nevada, Texas, Indiana, and Minnesota

3.1.2 – Overall Project Organization & Approach M – Out of 9 projects still have 5 of them in for more than two years? Problems or Still in progress? P – Using Git as a code repository P – Code will be tested in local environments before being pushed to testing environments Q – Governance – How items are approved? Would this be by PM or Double Line, or SOM? Technical Arch: P – System will be Hosted on AWS. M – Doesn’t say what WinXNet will do other than support Staff Management – ? SOM staff contracted by Double line? Is this a normal setup? M – Staffing Plan doesn’t talk about winxnet at all 3.1.4 Proposed Enterprise Solution: P – MEEDS hosted on AWS. P – Utilizing the EdFi cloud template P – Data exchanges encrypted using SSL ? – With SOM permission enhancements to data warehouse? ? - System Admins will have access for maintaining accounts as well as extended access. Does this mean SOM helpdesk as system admins? P – Use AWS scalability features to address peak demand times P – AWS Elasticsearch service can provide near matches for ID’s also ? – Data Integrity – Select from an existing set of rules for data integrity – Does this mean SOM is going to be doing this portion of the work or Double line? ? – Use dedicated time to empower clients to build and maintain rules - Why would SOM do this when integrity should be done by the project contractor? P – 90-day warranty to all services and elements ? Does the 90-day warranty start when the entire project is complete or 90-days when each piece is complete ? – SOM “may” review comparison results before approving? 3.1.5 – Operation Support Module P – Divide support into Application and environment I – WinxNet is doing hardware side and Double Line is on the application side of support and management


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/22/2018 EVALUATOR NAME: Brian Keene EVALUATOR DEPARTMENT: Technology Director – Regional School Unit No. 56

3.1.6 Training - P – Train the trainer setup P – End user trainings


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/21/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: SOM/OIT



P M I Q

Winxnet – providing security, DR, backup (and more) in AWS. Have they done this before (in AWS)? Q

Pg 79 – SOM needs 8 staff in toto from Discovery thru Phase 1c? Q

Pg 81 – Sentence that begins, “The finalized solution will incorporate the migration of all current on-premise servers and locally hosted virtual machines to the AWS cloud …” THIS IS INCORRECT, our current on-prem servers will NOT be moving to the cloud. AND The diagram on pg 81 at the bottom shows VMs that I do not believe exist. We need to understand HOW Doubleline plans to get our windows server application into the AWS cloud. M

Pg 81, There are 5 systems that NEO must interface with (AND none are data warehouses) M

Pg 83 Discovery is shorter than needed – only 2 months M

Pg 87 Estimate of 10 EDFacts reports will be maintained or built new if they are not contained in Generate Q

Pg 89 Step #3 of Data Quality Rules Creation Steps … is this configurable [such that] MDOE can say if they want an error or warning thrown? Q

Pg 90 Testing Strategy looks good! P

Pg 93 Waranty of 90 days .. does this begin at GO LIVE of entire system? Q

Pg 94 Flow seems to not show Winxnet doing any remediation? Q

Pg 95 What is first paragraph saying? Q

Pg 96 under Data Stewardship .. it states that SOM DBAs are needed. This is not what we are anticipating – Doubleline and Winxnet are supposed to do this type of maintenance. M

Pg 97 more technical training of SOM staff… not our desire M


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/21/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: SOM/OIT

PG 99 NEO Managed Services has an end date of 9/30/2020 (which implies that NEO will be fully absorbed into the new solution by this end date) however, there are still references to changes happening to the NEO modules on the subsequent pages 100, 101, 102.


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/20/18 EVALUATOR NAME: Ida Batista EVALUATOR DEPARTMENT: DOE



1. Project Understanding:Yes, understand and have implemented similar projects for 11 other states.

2. Overall Project Organization & Approach2.1. Project Overview Partner with Winxnet. Still not much on Winxnet as a company. Leverage experience. 2.2. Project Management Speaks heavily to coding development. Plans on developing relationship with SOM. Items in & out of scope to be well defined. What is the difference between local development environment and testing environment (p75). Operational readiness written statement seemed “canned” did not expand on what they can offer.

2.3. Strategic Roadmap & Timeline Data Visualization and Training times seem short.

2.4. Technical Architecture Will work with AWS. Plans on using a series of security measures

2.5. Staffing The project manager/s have a lot to do: staff, project & technical elements just to name a few. No Trainer nor technical writer in staff plan. This plan seems different than the one from phase 2. Wonder how multiple people in multiple roles from phase 2 translate into this staffing plan and the amount of FTE per role.

3. NEO Cloud Migration3.1. Cloud Migration Approach Linear plan heavy on jargon. Heavy reliance on communication with SOM.

3.2 Transitions Roadmap & Workplan Mention Winxnet but no details offered.

3.3. Roles & Responsibilities Project manager heavy load. Reliance on SOM NEO group.

4.Proposed Enterprise Solution4.1 Product Roadmap & Workplan Why does this timeline only go to 2022?


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: Double Line, Inc. DATE: 9/20/18 EVALUATOR NAME: Ida Batista EVALUATOR DEPARTMENT: DOE

4.2 Solution Contract with AWS for private cloud. Relationship with AWS leveraged to improve efficiencies and reduce costs. Experience with other states with warehouses, dashboards, edfacts, visualization tools, EdFi ODS interactive. AWS scalability seems dynamic. ID manager does not mention issues or corrections with duplicates. Experience with Ed-Fi alignment. Plugin rules engine into warehouse for additional checks for longitudinal data. Robust rules and willing to teach client to write (code) own rules to remove dependency. Financial plan not specific to ME. Need to do what they are discussing anyway to gain expertise with Financial standards. What happens if financial legacy and edfi needs differ? What does 90 day warranty mean? 4.3. Data Migration Perform gap analysis. Discuss with SOM. Comparison test to confirm validity and correctness of data migration.

5. Operation Support Model5.1 NEO Close communication with SOM; Leverage NEO team. Winxnet mentioned vaguely.

5.2 Enterprise Solution Automated deployment. 90 day warranty?

5.3 Training Train the trainer model. Time seems short. Development of multiple formats for training – online, videos, written material, in person trainings and special area training. Also multiple audience types.


************************************************************************************************************************ Individual Evaluator Comments:Paula Roberts

Since the establishment of the Ed-Fi Alliance in 2012, Double Line has been its technical arm responsible for most roadmap development, enhancements, maintenance, and technical support

11 state implementations

Winxnet expert understanding of SOM

Long timetable 2022 is this a staffing issue or be design Is it really going to begin in October? modern AngularJS data filtering control Good to see

utilize AWS scalability features to address peak

we assume Generate will work “out-ofthe-box” with no customization What if it does not?

ID management is establishment- Michigan and Indiana

It’s important to note, the Ed-Fi Identity API defines the interface by which the tool’s actions can occur, but the Ed-Fi API does not include an implementation of the matching mechanism. This matching mechanism in our solution is AWS Elasticsearch Service10

Routine maintenance (including planned outages, application upgrades, and scheduled environment fixes) will occur on non-business days, either Saturday or Sunday,

Train-the-trainer

Student data before ID management?


STATE OF MAINE



BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/21/18 EVALUATOR NAME: Hazel Stevenson EVALUATOR DEPARTMENT: Admin and Finance



1. Project understandingRe-iterates visionP Includes assumptions showing understanding of complexity of project and need for clarity to besuccessful. Some will need to be negotiated if awarded.

2. Project OrganizationInclude high level success factors

High level approach of concurrent discovery and analysis for Phase 2 P - Plan - use NTT project Management, governance model, baselined plan P - Operational readiness test

Roadmap and timeline- Roadmap lays out all phases and includes professional services 1a/b and Phase 2 overlap – reasonable? P- good high level description of strategic milestones with implementation strategies

Architecture P- MS Azure – leverages MS stack used by NEO and Ed-Fi, PAASLists tools; additional tools will be identified in the to-be analysisP SOC2 Type2 to address security, availability, integrity, confidentiality – some references to AppendixJ requirementsLeverages Azure featuresHigh level description addresses architecture for both NEO and MEEDS

Staffing plan P Depth of experience from team, with additional resources if needed and capability to bring in P- has strategic staffing division at NTTProvides State SME’s (50%) and PM; project plan and schedule for planning

3. NEO Cloud migrationP Describe 3 step technical approach, determine details in discovery, automated tools – short butlooks reasonableP will make inline improvements if beneficial to overall MEEDSUse current NEO developers in onsite support teamRecap staffing R&R

STATE OF MAINE



BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/21/18 EVALUATOR NAME: Hazel Stevenson EVALUATOR DEPARTMENT: Admin and Finance

4. Enterprise SolutionP Explains components implemented in Phase 2 and 3 and explains data design processLists tools – DataSpecs for metadataP Mature rules engine for data integrityRecommends pilotGood summary of solution approachP Interactive workshop approach, recommendations for meeting requirements, good details inresponse with samples show understanding of the requirements while meeting proposal guidelinesP tools are scalable and sustainableM offshore PM / offshore resources in data migration – conflict with data residency requirements?

5. Operation Support modelFixed capacity model using rate card and change managementNEO – tailor life cycle methodology, use knowledge Acquisition ProcessEnterprise- SIT to UAT, with deployment cert – understand state processesSodoto and job shadowing for knowledge transfer

6. TrainingP Describe training model at a high level and provide training implementation plan as deliverable


************************************************************************************************************************ Individual Evaluator Comments:Paula Roberts

on site support- Issues managed by onsite Support team

We want to partner with Maine to build and implement the most cutting edge and comprehensive education enterprise data solution in the US.

Edfi & initial already in production +ID 12 year experience + experience working with State

Staffing- Depth to ad additinal resources if needed

Concern-This presents a schedule risk to the project, since the release date of ODS/API v3.1 is not yet announced by the Alliance.

I like this approach:developing one report for district administrator level access to all data, with drill through capability

Identified importance of Synchronize data with SIS vendors

10 states using ID Solution

There is a single place to look to find all data validation errors (both Tier 1 and Tier 2).

Like this approach, standard operating procedure is to begin with the question to be answered by the data, design the visualization/report to inform that question, determine the data/metric/statistics requirements to deliver the report—then to define and collect the appropriate data.

focus groups to obtain design and functionality feedback.

Our proposed solutions are scalable both vertically and horizontally and are sustainable.

90 days warranty sounds short

120 participants trains sounds low

Will it really start October 1, 2018?


STATE OF MAINE



BIDDER NAME: ESP Solutions Group, Inc. DATE: 09/25/2018 EVALUATOR NAME: Joseph W. Larrabee EVALUATOR DEPARTMENT: Department of Education



Overall –

I think the vendor has an overall understanding of what the work is and has a viable solution. I feel that they might not totally understand the complexity of the WH needs and the NEO migration. While there was an expectation of concurrent tasks and activities, I feel that it might be too much for the department to manage and have a 3 day limit to respond.

Appendix I – Proposed Services

1.0 Project Understanding

- It appears that they have an understanding of the scope of work. Mention other states andhow much more effort Maine is going to be than others

- Second half refers more to their Organization qualifications than a project understanding.- Provided appendix of assumptions

2.0 Overall Project Organization and Approach

2.1 Overview

- Concurrent activities between NEO and EDW Discovery- Onsite from Phase2 through end of initial period of performance

2.2 Project Management Plan

Methodology

- Methodology looks very “old-school”. Didn’t really identify a methodology that theywould be using. Based on the graphics appears to be a waterfall approach. RFP didmention an Iterative project.

Governance / Scope Management / Schedule/Time Management ect

- Appear to be very structured and old methodology.

Document Management

- No mention

Operational Readiness

- consider operational readiness and getting the system ready and “activities”. Wouldhave liked to see more description of what those activities might be. IECommunications, training, training, business process.

2.3 Strategic Roadmap and Timeline

Timeline

- Concerned with so much overlap in phase 3 that the department has the capacity tosupport vendor and turnaround time

- Releases are planned to summer to try minimizing impacts to districts and vendor

STATE OF MAINE




- Strategic Milestones o Cloud environment ready for development o NEO migrated to cloud o Initial Infrastructure and pilots o Edfi releases planned for summer o Discovery multiple checkpoints o Efficiency in development of phase 3 components o Indicates a reduction in NEO support after 6/30/22 depending on remaining

functionality left in NEO

2.4 Technical Architecture

- Visual Studio, Power BI, SSRS - AsIsToBe Mindset – current and future state - Migration Plan - NEO - Azure Web PAAS - MEEDS – IAAS and PAAS

2.5 Overall Staffing Plan

- Staff contingency – mentions that they have experience and depth to additional resources if needed. Not sure how they will acquire them. Not sure if it’s using NTT data and augmentation?

- Staff management – right skills and create motivated and “excited team”. Does mention here that they will use NTT data to provide specific skillsets.

State Staff

- SMEs and State PM. - Keep SME time to <50% on project

3.0 NEO Cloud Migration

3.1 Approach

- 3 Step process Requirements, Readiness and rollout - Rollout as Hour by Hours (HxH) – not sure what this means but appears to be tasks a time

boxed guide for managing expectations and tasks - Using automated tools where possible - Detailed Migration and Roadmap with HXH plan - NEO 5-member team will work with cloud infrastructure team - Integration and UAT testing - Assumption – not sure if the NEO developers will have interface points working in target or

they will be using the knowledge of NEO to help get them working - Says they will use existing ticketing system (126) when the RFP states they will supply one

3.2 Transition Roadmap and Workplan

- Will determine complexity and build a tactical roadmap

STATE OF MAINE




- Steps o Transition o Service and SLA Stabilization o Service Optimization o Service Transformation

3.3 Roles and Responsibilities

- Advisory Team led by President and CEO – showing executive oversight and provide authority to direct resources as needed

- NTT on site Executive - Daily Operations by PM - No mention of roles and responsibilities between ESP and State

4.0 Proposed Solution

- Conceptual Architecture by end of Phase 2 - Uses DataSpecs and ISSIght tools .

o DataSpecs appears to be a ESP custom application for standardizing data elements, naming conventions ect

o Insight – 3rd party tool for documenting data center transformations - 30 months to Complete phase 3 is very aggressive

4.1 Product Roadmap and Workplan

- Same concern as above – too many concurrent activities for the Department to Manage and support as well as districts.

4.2 Solution

- Phases 2 and 3 structured approach - Discovery - happens at very points depending on the timing of the component - System Design – expects SOM to approve in 10 days - Construction - mostly done offsite with onsite team refining requirements - Integration testing – done by onsite team. - Documentation – detail documentation for each component - Hosting – Azure - Warranty – done by onsite team

Ed-FI

- v3.0 API/ODS with the Ed-Fi v2.1 dashboard - Risk - Ed-Fi dashboards for v3.0 ODS/API are not intended to be operational until

ODS/API release v3.1 o Need to implement over 12 months pilot districts with varying SIS packages

- DW based on Power BI - Uses implementation approach to Plan/Define/Evolve/Map - Asses NEO for Data and use cases

Data Visualization Tool

- Edfi dashboards and Power BI - SSIS for ETLs - SSRS / Power BI for reporting

STATE OF MAINE




- Data Marts for each reporting area

CIID Generate Tool - Stand up around midpoint of phase 2 - CIID has integration strategy which ESP will use - Generate currently has 16 - Will work with CIID too monitor

Education Organizations Management Tool

- Need to identify who’s responsible for each resource (LEA) and how they relate - Create initial definitions by hand to be able to determine Org attributes - Have worked with SIS vendors in the past coordinating -

Student Data

- Gap analysis of NEO and MEEDS to determine gaps in data elements - Develop mapping document to support legacy data - Modify NEO once MEEDS functionality is built

ID Management Tool

- Will use e-Scholar solution - Allows for NON-sis districts to submit for SSID - Allows for batch searches or batch loading - Match logic is First, middle, last, alt last, birthdate and gender - Provides a match score 0-100 - Will work with SIS vendors to be Certified

- Migration of historical IDs

Data Integrity Solution

- Error reports in Web UI - Same rules engine as in Tier 1 is applied to tier2 for ODS and WH - Rules either SQL or C# - Rules are stored in configuration files and can be edited in XML -

Financial Accounting for Local and State School Systems - Updated Tier 1 and 2 rules - Have 20 tables that will be migrated from legacy??? Not sure that this is an assumption or

a fact - Not sure if they understand that the Financial DW is for accounting and budgeting

purposes rather than financial analysis - Will work with the financial vendors to implement – attending certification session blocks

Maine Assessment and Accountability Reporting System (MAARS) - Evaluation of current MAARs Data - Build MAARS framework using BI and quick reports - Will create training - Doesn’t really mention following the old RFP but rather a complete evaluation and new build

EDFacts Reporting

STATE OF MAINE




- Currently supporting or have support 9 states for EdFacts - Moved implementation to 1/1/21 to leverages more files in Generate - Doesn’t mention supporting non-generate edfacts reports in the interim

Staff Data - Updates to ID Management - Updates to Tier1 and Tier2 rules - Extend APIS to get staff data from district HR and Financial Systems - Modify NEO to remove functionality - Will follow same process as financial for Certifying district vendors

Reporting Dashboards

- Work with Focus groups to obtain design and functionality - Multiple views for district, school, state, student - 5-7 new dashboards per over 3 years - Update data marts - Performance issues with dashboards with large data volumes -

Training

- System Administration Training; End-User/Stakeholder Training; and MEEDS Training. - System Admin

o 2 weeks instructor led o Operational and Updating of Major Components o ETLS Process

- End User o 4 weeks instructor led o Visualization tools o Supported by live web classes

- Organization Tool Training - Student Data Training - ID management Training

4.3 Data Migration

- Merging of legacy and new data in ODS before pushing to DW - 10 critical Modules (Student Data, Financial, Maine Schools, Staff, School and Program

Approvals, Core, Essential Payment Services, Special Ed, Graduation, and Dashboard) - State Responsibilities (PM, SME, Tech Analysts DBA) - MEEDS Responsibilities (PM, Offshore PM, Architect, BI Analyst, Tech Lead, QA, DBA)

5.0 Operation Support Model

- Recommend a fix Capacity model - Fixed monthly invoice

o data corrections, simple code corrections, backup and restore, DR support testing, and other tasks which fall under the capacity of the team

- Formal RFC for larger SOW - No mention of Maintenance or Planned outages - No mention of what support workflow is. DOE helpdesk role

STATE OF MAINE




5.1 NEO

- Leverage past transitions and technical experience - Define governance and SLA - Work with SMEs to prepare for on-going support - Knowledge Acquisition Process (KAP) - Existing NEO team will serve as support team - Use code walkthrough and shadowing to gain more insight - On-site similar to how NTT does it now - State to provide a work space with standard tools, laptop, access to state network, 24x7 or

agreed upon access to State offices, remote access to State network, and tools as needed. - State ticketing systems in place to coordinate with the helpdesk

5.2 Enterprise Solution

- UAT to staging then prod - 90 days warranty - See One, Do One, Teach One” (SODOTO) - ILT, webinars, documentation, job aids,

6.0 Training

- Training Implementation Plan - Admin - 60 Participants to be trained, 2 days, 4 offerings - End User – 120 participants, 2 days, 8 offerings - Curriculum – changes in workflow, enhancements, interoperability - Format - Training Implementation Plan; Instructor Manuals; Participant Manuals; PowerPoint

Slide Decks; Instructor-Led Training Classes; Live Web Classes; Recorded Web Classes; and On-line Help

Appendix Assumptions of Concern (numbers correspond to bullets)

#2 Reviews – not sure what is meant by single review cycle (specifically: submit, review, cure, and accept). Review process will be addressed and agreed upon as part of the contract language

#3 Reponses to information – 3 days might not be enough depending on who is required to respond. Getting information from the districts or OIT might take more than 3. Working days are M-F and non-holidays

#4 “Certain responsibilities” – kind of nebulous and doesn’t define what they are

#6 facilities, software, hardware and other resources – kind of generic. What specifically are they referring too

#8 Provide of hour access to buildings. Also, not sure if 8:30-5 applies to Support or not. Asked for different times in Appendix M

#11 UAT – page 45 of the RFP states the bidder will develop a testing plan. Is the DW. On page 96 for NEO it says they will create UAT testing strategy.

#15 Estimates can be adjusted based on discovery. Cost estimates or time estimates?

#19 State licensing structure

#21 Number of tables to be migrated

#22 50 Reports and Dashboards (is this assuming to include MAARs)


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/23/18 EVALUATOR NAME: Brian R. Keene EVALUATOR DEPARTMENT: Technology Director – Regional School Unit No. 56 ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

3.1 – Project Understanding P – Mentions different strengths and things that each sub contractor will be doing Overview: P – hosted and operational NEO in 9 months 3.1.2.4 – Technical Architecture P Asure hosted P – Secure and seamless access to the system with a single login portal then redirected upon access level P – Two Factor authentication MEEDS Implementation P – No subcontractor involved done by a single company 3.1.2.5 – Overall Staffing Plan P – NTT backup with 10,000 employees to draw from ? State provide the PM at the state level to work with it 3.1.3 – NEO Cloud Migration I - Migration to Cloud NEO will be lift and shift P – Onsite PM Eric Hills from NTT P – Onsite Senior PM Singh Data Visualization Tool - Capabilities on one report for the district administrator

ID Assignment P – Match, No Match, Near Match P – Mentioned being able to handle Twins in ID generations Data Integrity: P – Rules engine working in XML for management of the rules Training – P – End-User Training Four weeks of instructor led P – Data Tool Training will be offered to all end users ? – Are the training dates listed too late in the plan? 3.1.5.2 – Solution - P – State UAT testing P – 90 Day Warranty ? – Maximum of 120 people trained?


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/24/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: DAFS/OIT ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

3.1.1 Project Understanding Clearly states high level credentials of all 4 vendors 3.1.2 1. Overview Concise, but hit the high points, clear

2. Project Mgt Plan Proposed Go Live activities not shown 3. Strategic Roadmap and Timeline 4. Technical Architecture States that Azure platform will only include the US cloud regions States that Azure has achieved SOC II Type II certification 2 factor authentication Although incorrectly referred to as a “Lift and Shift”, their understanding is sound States “There will be no subcontractor involvement” + 5. Overall Staffing Plan GREAT Staff Contingency Plan! + Goal is to keep all SMEs < 50% committed + 3.1.3 NEO Cloud Migration GREAT understanding of what this involves. + 3.1.4 Proposed Enterprise Solution Not clear if “on or near January 1, 2020 – the initial MEEDS system will be operational with Ed-Fi Core elements” means this is a true Go Live into production? 2. Solution REALLY appreciate their attention to the SDLC! 10 Days for SOM to review the System Design docs + There is a Discovery Phase for each Project Phase! + Good breakdown of Phase II!


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/24/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: DAFS/OIT

GLAD they chose the Microsoft suite for BI/DW! GOOD explanation of DW approach: (star-schema and snowflake structure) Demonstrate clear understanding of the CIID Generate Tool Phase 3 a.i Educational Organization Management Tool Demonstrate clear understanding of the EOM Tool! Phase 3 a.ii Student Data Sounds like ESP understands that we don’t want to keep NEO around forever, but rather migrate the functionality to MEEDS! (PG 110) Phase 3 a.iii ID Management Tool The eScholar Uniq-ID solution sounds awesome! + Phase 3 a.iv Data Integrity Solution REALLY GOOD explanation of this part of the solution; like the ‘child before parent’ example of a data integrity violation Phase 3 a.v Financial … Good flow of development activities Phase 3 a.vi MAARS (Maine assessments…) Phase 3 a.vii EDFacts Reporting Significant experience in enhancing EDFacts reporting: completed in 4 states, onging in 5 additional states! Good strategy for the CIID Generate tool component of the project! (pg 119) Phase 3 a.ix Reporting Dashboards Seems pretty comprehensive with references to ESSA, OSEP and CNA/SAU! Encryption in transit and at rest!! Phase 3 a.x Training Mentions MEEDS Training “related to enhanced functionality of MEEDS and development of new tools” WE DON’T WANT THIS, DO WE? WE AREN’T GOING TO DO TECHNICAL MAINTENANCE …- 2. Data Migration (pg 124) Well thought out! 3.1.5 Operations Support Model NEO support sounds correct (they understand the reqs) KT for the “Enterprise Solution” is more nebulous … AND “SODOTO will be the primary training methodology used to prepare technical staff to assume the responsibilities initially carried out by the ESP MEEDS team!! NOT GOOD .. WE DON’T WANT OIT TO TAKE ON SUPPORT OF THIS SYSTEM.


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/24/18 EVALUATOR NAME: Cathy Harrison EVALUATOR DEPARTMENT: DAFS/OIT

3.1.6 Training Sounds fine Attachment 1 Strategic Milestone F YEHAW! ESP does plan to do ID mgt, data integrity and EOM functions to production readiness first (before the remaining Phase 3 components) … production ready by 1/1/2020! … but the Gantt shows data integrity not completed until a year later?!? Appendix 1 General and Technical Assumptions Like the review cycle BUT the project related decisions made ‘within 3 working days’ isn’t very accommodating … Pg 136 … “User acceptance testing” bullet: Why does the state need to provide resources to assist with system configuration”? Pg 137 … Can ESP leverage our State’s “hybrid Microsoft licensing structure to prices the OS and SQL Server licenses for the target Azure environment?


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/21/18 EVALUATOR NAME: Ida Batista EVALUATOR DEPARTMENT: DOE ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Project Understanding: They appear to understand the project complexity and scope, referring to it as an “information ecosystem”. Some large assumptions made on pages 135-137. Question some of the responsibilities of parties and roles.

2. Overall Project Organization & Approach 2.1. Project Overview Linear approach outlined with awareness of overlapping tasks. Partial answer to the prompt. No challenges or obstacles mentioned. 2.2. Project Management Leverage experience; iterative process; use of change management process. Not much detailed methodology discussed on topic areas. 2.3. Strategic Roadmap & Timeline Heavy focus on Ed Fi. Plans to pilot components on actual SAUs with different vendors Ed Fi release in the summer 2.4. Technical Architecture Tools mostly /all in the Microsoft suite; US cloud only; segmentation of information by user and role; Focus on security; Little information on Version control. 2.5. Staffing Staffing plan? They have the ability to add more staff depending on project need Staff management did not discuss management tools or type. Mentions using SME but not the retention of contracted SOM NEO staff

3. NEO Cloud Migration 3.1. Cloud Migration Approach Methodology process discussed (R,R,R & HxH). Discussed role of subcontractor. Indicated inline improvements. I’m not sure if assumption is valid. 3.2 Transitions Roadmap & Workplan Looks sound and within a 9 mo. Timeframe. 3.3. Roles & Responsibilities Advisory team has authority to direct resources as needed. Not so sure of this as one of the roles of an advisory team. NEO support personnel mentioned and PM Singh. Briefly rehashed prompt. 4.Proposed Enterprise Solution


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 9/21/18 EVALUATOR NAME: Ida Batista EVALUATOR DEPARTMENT: DOE

Confused by flow pattern in Figure 3. Use DataSpecs to inventory & ISInsight to develop MEEDs ecosystem. No discussion of non-conforming data. 4.1 Product Roadmap & Workplan Organized seems to have a logic in keeping with SOM wants/needs. 4.2 Solution Assumptions made as to SOM role. Application warranty to be discussed as to what is warrantable. Mention of a risk with Ed-Fi V3.0 API/ODS and release of ODS.API v3.1 Conduct interactive workshops to understand key strategic objectives and focus areas; Describe an iterative process of development. Visualization tools and roles for access to data discussed. (p104) Will there be access to data not through a visualization tool. Data integrity and quality enforced. Describe the data mart setup. Set up of an authoritative agent for the ed org management tool. Discussion of data synchronization challenges with student data and feel capable and experienced in handling them. Id manager is comprehensive for new id. Little discussion on historic ids besides saying they will move them in. Exp with the rules engine solution proposed. Number of financial reports low and examples for outside end user not internal SOM user. Want to make sure Me’s financial data needs met even if they do not conform with Ed-Fi Understand the complexity of assessment data; want to make it available in other languages and understand privacy and security needs of such data. Various forms and formats of reporting assessment data. Discuss the need to define assessment data from various both state and LEA levels. Want to address assessment analytics. A bit confused by the link between district HR and financial accounting process and staff data. Dashboard focus groups. Approx. number of dashboards able to be created within a year and have designs to develop ESSA, OSEP and CAN/SAU reports. Security encryption; Antivirus; firewalls and user roles. (p122) 4.3. Data Migration 10 critical modules Three module at a time approach. Willingness to shift order if necessary. Data normalization. Discusses roles and responsibilities on both the state and vendor side. 5. Operation Support Model: Fixed capacity model & change management processes. Fixed monthly invoice 5.1 NEO Leveraging prior experience and expertise. Collaboration. Plan for onboarding. Want clear definitions of success criteria and exit criteria. 5.2 Enterprise Solution What is with the 90 day warranty? UAT responsibility? Knowledge transfer plan adaptable though they caveat sharing out of scope. SODOTO. 5.3 Training Plan two-day offerings. Webinars. Use of material developed in earlier phases. Instructor lead trainings. Are the caps on specific audience types reasonable?


RFP #: 201804073 RFP TITLE: Maine Education Enterprise Data Solution BIDDER NAME: ESP Solutions Group, Inc. DATE: 09/21/2018 EVALUATOR NAME: Charlotte M. Ellis EVALUATOR DEPARTMENT: Maine Department of Education ************************************************************************************************************************ Instructions: The purpose of this form is to record proposal review notes written by individual evaluators for this Request for Proposals (RFP) process. It is required that each individual evaluator make notes for each proposal that he or she reviews. No numerical scoring should take place on these notes, as that is performed only during team consensus evaluation meetings. A separate form is available for team consensus evaluation notes and scoring. Once complete, please submit a copy of this document to your Department’s RFP Coordinator or Lead Evaluator for this RFP. ************************************************************************************************************************ Individual Evaluator Comments:

1. Project Understanding √ 2. Overall Project Organization and Approach √

2.1 Overview √ P – I like the onsite presence and iterative approach

2.2 Project Management Plan -√ - P Nice graphic in Appendix 2 that lays out the plan – seems to be NTT’s model

• Governance √ • Scope Management √ • Schedule/Time Management √ • Communication √ • Risk √ • Document Management √ • Operational Readiness √ 2.3 Strategic Roadmap and Timeline √ - ? – in diagram point E – they are proposing Power BI, but aren’t really sure if it is compatible with Ed-Fi dashboards 2.4 Technical Architecture √

P – proposing Microsoft platform and tools – current team is familiar with these tools 2.5 Overall Staffing Plan √

• Staff Contingency Plan √ - P Subs have lots of staff to augment • Staff Management √ • Use of State Staff √ - M – referenced SME and PM, but made no reference to technical

resources that might be necessary 3. NEO Cloud Migration√

3.1 Approach √ - P – Plan seems solid M – Their assumption about lift and shift and the system working with just configuration is not valid 3.2 Transition Roadmap and Workplan √

3.3 Roles and Responsibilities √ P – Executive oversight and involvement M – No mention of Maine DOE roles and responsibilities

4. Proposed Enterprise Solution√ 4.1 Product Roadmap and Workplan √ 4.2 Solution√ I – Ed-Fi dashboards will not be available until Ed-Fi Alliance releases them. M – Education Organization Management – Initial Ed-Fi implementation will have orgs created “by hand” Unclear what this looks like and its effect on the project as a whole



M Unclear how Generate will work when “historical assessment data store” is gone and assessment data is in the Data Warehouse – is it expected that assessment data is loaded into the ODS M – How do you rerun Ed-Facts datasets if Generate is operating out of the Ed-Fi ODS and the ODS is only a single year database? P – Robust ID Management tool – even deals with twins P – Like rules engine in front of ODS and in front of data warehouse P – Data integrity solution offers advice on how to resolve issues and errors M – Question how accurate their assumptions are regarding financial tables for migration, data integrity and rules P – Understand the need to work with outside vendors (SIS, HR, Financial) P – Public reports are available translated into Maine’s top 4 languages P – I like their approach to creating visualizations P – Understand the need to document data warehouse business rules as to how metrics are calculated P – Understand the complexities of assessment and accountability data P – Understand the need for external stakeholder involvement in ESSA report card design M – Expect Department to supply anti-virus software? 4.3 Data Migration √ M – No details on how data that does not fit the Ed-Fi data model will be migrated M – Assumption that financial data is in NEO is incorrect

5. Operations Support Model √ 5.1 NEO√ P – Strong plan for knowledge transfer M – Plan to keep existing ticketing system in place –unclear as to how that will work 5.2 Enterprise √ M – Details were about deployment, not transition M – 90 day warranty is not adequate P – I like the idea of a training needs analysis M – Unclear as to what level of responsibility will be turned over to the Maine DOE

6. Training √ P – Trainers will have instructional experience M – I don’t understand the part about live webinars in place of instructor led training Assumptions Assumption 6 – M – Confused about how they expect Maine to obtain software licenses Assumption 11 – M – State is responsible for populating UAT database Assumption 13 -M – For legacy systems, do they mean other than NEO? Assumption 15 – M – What would be the impact of them adjusting their assumptions and estimates? Assumption 19 - ? Is it even possible for them to leverage our licensing?

state of maine rfp / proposal master score sheet · state of maine team consensus evaluation notes...

Documents