for... · web viewapplication penetration testing ... design - word, excel, visio4. unit testing...
TRANSCRIPT
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
1
40 Process
As mentioned in the RFP "Offeror's must fully describe how they will be fully operational by July 1, 2015."
We assume that the vendor will be provided a 3 months’ time period to complete the transition phase. (i.e. starting from award start date of 03/31/2015 to 01/07/2015). Please confirm.
The vendor will be provided the period of time from contract award start date through June 30, 2015 to complete the transition phase. It is anticipated that this will be three months.
2
40 Scope
As mentioned in the RFP " How it will confirm it has transitioned all assets and materials needed to continue systems operations."
Considering the above Statement, does the State expects the vendor to perform any mandatory activity like: Operational readiness testing (ORT) to ensure Operation Readiness & Business Continuity?
The State will perform a readiness assessment with the vendor near June 1, 2015 in order to validate Operational preparedness.
3
General Other
We assume that the vendor will be provided with the existing test artifacts (Manual/automation test scripts, test results, etc.) which will help to bring in reusability during the M&O and enhancement testing phase. Please confirm.
The vendor will have available all existing test artifacts.
1
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
4
58 Administrative
As part of the Staffing Experience mentioned in the RFP, the Quality Assurance Testing Analyst shall be responsible for “Development of automated test scripts utilizing Commercial-Off-the-Shelf (COTS) tools such as HP Quick Test Pro; and Development of automated load testing scripts utilizing COTS tools such as HP Load Runner"
Will the State agree for multiple resume's for the QA position (more than 2 resources) 1. With manual testing Exp. & With QTP experience (Automation) 2. With HP load runner experience (Performance Tester). As availability of resources having both manual/automation and performance/load testing experience is scarce. Hence we may have to look for different profiles for each of the above asks. Please confirm.
The State will agree to multiple resumes for the QA position.
5
38 Cost
We assume that 2 tester mentioned as part of open positions for the Quality Assurance Testing Analysts are only for functional testing requirement (Manual & Automation). As this RFP requires other testing like Performance testing, Security testing - We assume that we will require additional QA staffing. Please confirm
The vendor should propose resources to meet the RFP requirements.
6General Cost
Is the State willing to accept any open source (free of cost) tool like selenium for automation testing?
The vendor should be prepared to use existing tools however HSD is open to recommendations of changes or additions to these tools.
2
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
7
130 Other
As mentioned in the RFP "The Contractor shall create and maintain manual test scripts when the automated tool cannot be utilized allowing common areas, such as interfaces that receive files from trading partners to be regression tested."
Do we have any analysis for the coverage of automation v/s manual scripts for the existing regression test suite, if so, please provide details?
There are approximately 81 QTP scripts written for ASPEN functionality that are used in automated regression testing for each release. The State would like to increase the number and usage of these in the future. There are hundreds of manual scripts that can be used based on the functionality being tested. We do not have a current analysis of the coverage of automation v/s manual.
8
36 Scope
The scope of testing mentioned in the RFP describes the vendor to perform the following testing types: Testing will include:a. Unit/Integration Testingb. System Testing (i.e. Quality Assurance Testing)c. Regression Testingd. Performance and Load Testinge. Security Testingf. Disaster Recovery/Business Continuity Testing
Will the vendor be responsible to perform any penetration or network based security testing or is the scope of security testing limited to role-based security validation. Please confirm.
HSD conducts in-house as well as independent security scanning and testing on a regular basis. The vendor will be responsible for correcting any items found from these scans related to the application or their supported infrastructure. The vendor is not responsible for conducting penetration or network based security testing.
9General Other
Please provide the average number of incidents/break fixes that the State is currently handling for each month as part of the ongoing Operations and management phase.
Current monthly builds on average include 40 work requests that are related to incidents/break fix needs.
3
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
10
General Scope
Please provide the scope for the following Application Security testing:SAST - Static Application Security Testing (Source Code Level review of the code)DAST - Dynamic Application Security Testing (UI level testing for identifying vulnerabilities at the runtime)Application Penetration Testing
SAST/DAST – Static application Security Testing must meet all applicable controls from NIST 800-53.
Application Penetration Testing – Will be done as part of the annual HSD security assessment and not part of ASPEN M&O
11General Other
Do we have application component Categorization (simple/medium/complex) of the given applications that are in scope for security testing?
Yes. CMS has categorized this system as moderate.
12 General Scope Do we need to include re-scan and remediation support efforts as part of scope?
Yes.
13 General Other Will State provide the security scanning tools to perform the security assessment?
Yes. State has a continuous monitoring program and annual third party assessment.
14
32 Scope
"HSD introduced an Interactive Voice Response (IVR) service in February 2014. It allows clients to call a toll-free phone number and receive predetermined information about their cases. The information comes from the YES-NM database"We assume that IVR performance testing is part of scope. Please confirm.
IVR performance testing is included in scope as it relates to the database performance. The performance of the external call tree and telephonic operations is outside the scope of this procurement.
15
32 Scope
"With ASPEN implementation, HSD has introduced a web portal with a client-facing system allowing clients “self-service” capabilities"Do we have other interfaces (handheld/mobile) to access the ASPEN application other than web interface? We assume that only the web interface is part of performance testing. Please confirm.
YES NM is currently the only web portal access into ASPEN that includes handheld/mobile access. There is a web service supporting Department of Work Force Solutions as a real time scan from their systems - not for use by clients directly and not used via a mobile device.
4
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
16
32 Scope
"Section 504 and 508 accessibility requirements, and Section 1561 recommendations from the Department of Health and Human Services (DHHS)."We assume that accessibility testing is part of the scope. Please confirm.
The external facing YES NM portal requires accessibility testing and compliance - this is outside the scope of this RFP as that portal is supported by HSD/ITD staff. ASPEN is not required to meet Section 504 and 508 accessibility requirements.
17
33 Scope
"ASPEN is used by approximately 1,300 end users"
How many of these are concurrent users?
We assume that end-user volume would be 1300 only as part of the performance testing scope. Please confirm
is the number of concurrent users.Do we have data for end users volume growth for next 2-5 years? If yes, please provide the details.
Concurrent users can be as high as 900 on a regular basis. We do not have data for end user volume growth for next 2-5 years at this time.
18
33 Scope
As stated, "ASPEN to retrieve case information or reports from ASPEN"How many different type of reports are getting generated/available? Are we using COGNOS for generating reports? Please confirm.
Over 300 different types of reports are being generated from ASPEN. Some of these reports are built and run as canned reports within the application, some are done as Ad-Hoc 'one off' reports, some are built by another HSD/ITD team out of a data warehouse using extracted ASPEN data, and some are done via an Oracle tool 'APEX' by other HSD/ITD staff using ASPEN data. COGNOS is not being used to generate any reports.
19
33 Scope
As stated, "The majority of HSD staff are housed in 36 field offices located throughout New Mexico as well as in the Central Administration building located in Santa Fe" We assume that the Performance testing from different geo-location and from cloud LG is out of scope? Please confirm.
Performance testing from different geographical locations has been conducted in the past and could be in scope depending on the size of a future functionality change or enhancement.
5
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
20
33 Scope
As stated "Currently HSD provides at least one public assistance benefit to more than 800,000 low-income New Mexicans"We assume ASPEN would support overall member volume of 800,000 plus. Are we expecting overall member volume %growth in next 1-5 years? If yes, Please provide details of the expected % growth
We only have projections for FY15 and FY16 and at this time we don’t see an overall projected growth, however some programs may see growth, mainly Medicaid.
21
33 Scope
As stated "In addition, field staff members complete eligibility determinations on new applications, a significant portion of which will not be eligible for benefits"How many applications are getting ineligible for benefits in year/month/day period and what is the %growth? Please provide the details
Approximately 76,587 applications were denied during the last State fiscal year, with an average monthly decrease of 16%. Detailed information to be provided upon contract award.
22
33 Scope
As stated and provided the count under "Active Case and Recipient Counts by Program as of JULY 2014"Are these active cases, Recipient clients processed through web interface only? Do we have any other interface entry into the system for processing the active cases, recipient clients? If yes, Please provide details
These are not processed ONLY through interface.
Interface entry in to ASPEN can come from field office Lobby kiosk entry, Federally facilitated Marketplace and Yes-NM.
23
33 Scope
As stated and provided the count under "Active Case and Recipient Counts by Program as of JULY 2014"Are expecting any %growth in the active case and recipient count in next 1-5 years? If so, Please provide the details
We only have projections for FY15 and FY16 and at this time we don’t see an overall projected growth, however some programs may see growth, mainly Medicaid.
24
33 Scope
As stated" Use of the term “ASPEN” in this RFP shall include the ASPEN application, YES-NM web service support with ASPEN, Electronic Document Management (EDM), all Interfaces, and xml support to the IVR system"We assume that performance testing on YES-NM web service and EDM and other interfaces alone is out-of-scope. Please confirm
Performance testing for the YES NM web services within ASPEN, all EDM functionality, and all ASPEN interfaces are within scope.
6
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
25
33 Other
"The support and enhancements to be provided for ASPEN include but are not limited to: Point 4"We assume that State will provide performance test tool, profiling tool, monitoring tool for the performance testing. Please confirm.
The State will provide all tools required.
26
35 Process
As stated, Purchase of hardware, software, hosting, license fees or other commodities is not within the scope of the RFPWe assume that State will provide all necessary software, tool. Please confirm.
The State will provide all necessary software tools.
27
38 Scope
As Stated," 2. Testing will include: Point f. Disaster Recovery/Business Continuity Testing"Do we need testing to support "Disaster Recovery “exercise and Business Continuity Testing?
The State conducts Disaster Recover testing at a minimum annually and more often as needed based on new functionality. The vendor must support this testing as an integrated partner within the scope of this RFP.
28
121 - Point # 6 Process
Below point represents SLA."The Contractor shall respond to and comply with HSD’s direction and timeline for remediation of problems and incidents. ASPEN will continue to change in reaction to business needs, and federal and state legislative mandates and new enhancements will be added through the application maintenance process."Please share current SLAs for various priority incidents
There is no current SLA in place for the Maintenance and Operation of ASPEN. The State will establish an SLA with the vendor following contract award.
29 122 - Point # 14 Process
Are there any tools available to check & monitor system availability? Please provide details
Yes, HSD has industry standard monitoring tools in place for monitoring availability. Specifics will be shared with vendor after contract awarded.
30122 - Point #
15 Process
Are there any tools available to check & monitor application environments, logs, etc.? Please provide details
Yes, HSD has industry standard monitoring tools in place for monitoring application environments and log consolidation. Specifics will be shared with vendor after contract awarded.
7
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
31
126 - Point # 47 Process
What are all the tools used for (Other than Clearquest & JIRA) 1. Requirement gathering2. Data Models3. Design4. Unit Testing5. System & Integration Testing6. Source code management
The RFP procurement library includes a list of all tools. The following are used currently for items listed: 1. Requirements gathering - Word requirements documents, Excel, ClearQuest and JIRA2. Data Models - Word, Excel, ErWin, Visio3. Design - Word, Excel, Visio4. Unit Testing - Clover (YES NM) 5. System & Integration Testing - QTP, ClearCase, and Atlassian tools6. Source code management - ClearCase, Subversion
32 133 - Pint # 106 Scope
Please share past 12 months ticket count by priority, request type (incidents/Service Requests/etc.), function, etc.
A slide deck for November 2014 information has been added to the Procurement Library.
33
135 - Point # 127 Technical
What is current release management process and release planning?How many releases are happening in a month for1? Enhancement2. Incidents3. Break fixes
Current release management process includes weekly meetings with vendor, IT and business to determine what Work Requests (Incidents and Break Fixes) will go into next major and immediate releases if needed. This is planned for 2-3 releases in the future. Each month has one major release and 1-3 immediate releases if needed. Enhancements are determined with monthly Steering Committee meeting of all Divisions and are scheduled into releases based on priority and hours required to complete the change request. These are determined 2-3 releases out as well. In additional daily data fixes are submitted by vendor for approval and run as required.
34 137 - Point # 140 Technical What tools are used for batch scheduling and batch
monitoring?OpCon tools are used for batch scheduling and batch monitoring currently.
35138 - Point #
142 Technical
Please provide details of OpCon environment Please refer to OpCon Integration documentation in the Procurement Library in the Technical Architecture folder (Deliverable A4_Technical Architecture Plan-Appendix L Opcon Integration)
8
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
36
142 - Point # 157 Scope
How will the contractor be notified of any Ad hoc report requests?Provide some examples/cases for Ad hoc reports and Recurring report requests.
Ad Hoc report requests are submitted via the Help Desk ticket tracking tool - Cherwell. They are submitted with requirements and requested due dates. They are managed via the ISD Business owner and the vendor as to priority and due dates. See new items posted in the procurement library Ad Hoc Request Summary Spreadsheet and Sample Ad-Hoc Request.
37
General Other
Do the applications owners have specific expectations by moving towards Managed Service model
We understand your question as asking what are the Department’s expectations of a vendor maintained system versus a State staff maintained system. The ISD eligibility system has always been primarily vendor maintained so this is not a new concept for the application owners. There are state staff supported various components alongside the vendor.
38 General Other Are there any regulatory/compliance and audit requirements for these apps?
Yes. CMS Mars-e requirements as well NIST 800-53, SSA, FNS, and HSD security directives
39
General Other
What is the current level of documentation application-wise? Do you have support documentation available?
The application documentation is extensive and includes storyboards, decision tables, requirement and design documents, Operations documentation and a Disaster Recovery plan that includes extensive support documentation. In addition the procurement library has a section detailing the training that is available for the actual application usage documentation.
9
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
40
General Other
What are the definitions of L1/L2/L3/L4 for the applications in scope
Level 1 support is the ASPEN Help Desk and/or ASPEN Customer Service Center depending on whether question is coming from HSD internal staff or from HSD customers using the YES NM Portal. Level 2 support is via ASPEN Help Desk Support Supervisor and Manager, Level 3 support is via the current support contract vendor staff and Level 4 support is via the current support contract vendor manager. The scope of this RFP includes levels 3 and 4 although resolutions are directed back through the ASPEN Help Desk staff and not directly between vendor and requester.
41
General Other
What are the definitions of P1/P2/P3/P4 incidents? Are these definitions common across different applications?
Priorities are assigned via the help desk tool (Cherwell) by either ASPEN Help Desk staff or by HSD staff if submitted via the self-service portal. Priorities are determined based on a matrix that uses impact and priority as follows:
Dept. Wide Office Bus
Unit Single User
Now 1 1 2 3ASAP 1 2 3 4Soon 2 3 4 4Later 3 4 4 4
In addition there are priorities related to if it impacts benefit issuance, has a work around, is coming from a constituent complaint, etc. The definitions are common across the applications within scope of this RFP.
42
General Other
What is the definition of Minor/Major enhancements? - What is the effort consideration to classify enhancements into Minor and Major enhancements?
There is no strict definition between Minor and Major enhancements currently. Each enhancement is given to the vendor with high level requirements for a level of effort estimate. Based on this estimate it is determined when it can fit into a release or if additional resources need to be pulled from break fix work to assist.
10
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
43
General Other
What are the key KPIs/Metrics and SLAs for the applications in scope
What are the key SLAs that need to be monitored and reported across applications?
Current M&O year contract does not contain specific SLAs. Please review the Monthly Report sample for KPIs/Metrics that are expected to be tracked and reported. The State is interested in developing additional metrics in the future including methods to track rework.
44General Other
What is the SLA trend and KPI/metrics trend over the last 6 months for the applications in scope
A slide deck for November 2014 information has been added to the Procurement Library in response to this question.
45
Page 32/System
BackgroundScope
Are there any other major enhancements currently going on which may affect the project scope?
There are three known major enhancements that could affect maintenance scope. One is replacement of the current Correspondence module to use HP ExStream tools instead of current OPUS toolset. Two is the planned move of the current Master Client Index (MCI) into a separate service architecture that can be utilized by other applications instead of its current ASPEN embedded architecture. Three is adding real time eligibility to the interface between ASPEN and YES NM for Medicaid.
11
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
46
Page 40/Require
mentsOther
Current contractor M & O staff count is 26. Can we get the title (skill set) of these resources.
The current M&O staff count required by the contract is 26. The current vendor has additional staff currently supporting the system (as needed) that are not accounted for. The contracted staff by track are:3 - Technical Support3 - Benefit Management1 - Correspondence6 - EDBC3 - Front Office/Self Service2 - Interfaces1 - Project Manager2 - Reports1 Production Support2 - Testing1 - Admin Support1 - Application Development ManagerThe skill sets include track leads, business analysis and java development spread among each team.
47 Page 39/Require
mentsScope
What is the current size of the "Aspen Enhancement" team? The new 28 resources will be added in addition to the current count or will they be replacing the current contractor.
Current size of the vendor ASPEN Enhancement team is 24. These will be replaced by RFP awarded vendor. There is additional State ASPEN Enhancement staff that support YES NM and will not be replaced with this RFP.
48General Contract
Will State be prepared to negotiate a contract based on industry-standard terms applicable to top tier providers
Please refer to Section II, C, parts 15 and 16 under “Sample Contract Terms and Conditions” and “Offeror Terms and Conditions.”
49
General Cost
Please clarify if proposed costs should include New Mexico sales tax (Gross Receipts Tax). Please clarify the I.T services subject to the tax and confirm the statutory rate.
The proposed costs should include the NM Gross Receipts Tax. The rate may vary based on an Offeror’s status under NM tax code, location of services, etc. Due to various aspects as to how the NM Tax and Revenue Department (TRD) establishes tax rates, Offerors should confirm their tax status and liability with the TRD.
12
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
50
Page 28 Cost
It is our understanding that a Contractor may lease the space at the location and cost indicated on page 28 but is however not required to do so. Please confirm that the should the successful Contractor elect to lease this space pursuant to this RFP, the Contractor would not be required to enter into a lease arrangement for a period prior to the start of delivery or during a period of any delay caused solely by the Department
A signed contract would be required to allow the successful contractor to both enter into a lease agreement as well as collocate.
51 189 /Table B.1. Key
Contractor Technical Tasks and Activities
Technical
Please validate the tools name /provide the tools used to manage the ASPEN environment? ITSM - Cherwell Monitoring - Nimsoft & Oracle Ent MangerBackup /recovery - ?Batch jobs processing /automation - ?Others - ?
ITSM - Cherwell with interface to ClearQuestMonitoring - Nimsoft & Oracle Enterprise Manager as well as custom scripts.Backup/Recovery - Oracle tools as well as NetBackup and DataDomainBatch Jobs Processing/Automation - OpCon
52 191 / Table B.1 - System
patch and upgrade
Technical
How many OS upgrade is expected per month? There can be many devices that can have an Operating System upgrade - i.e. desktops, Vblock, Exadata, etc. - on average we are doing Operating System upgrades 3-4 per year across all devices.
5334 / Other
general information
Technical
What is the failure rate of batch jobs? Are these batch jobs running off business hours?
Batch jobs are rarely failing unless an interface file is not received - then they are cancelled. We have less than one batch job failure per month currently. Batch jobs are primarily run off business hours - exceptions are those batch jobs supporting EDM and YES NM.
54
Page 120 Administrative
Within Appendix 2-B Functional Specifications, the table of contents provided on page 121 indicates that the appendix contains A.8 Staffing and Functional areas; however the appendix doesn't contain any requirements for that category. Can you please confirm there are no requirements under thissubheading of A.8?
The Reference to A-8 was a typographical error and should have been deleted.
13
ASPEN M&O RFPQuestions and Answers
# Page Question Type Question Response
55
Page 28 Administrative
Could you please clarify that if a Contractor selects the collocation option, would HSD consider providing 15 additional spots at the Siler location? What is the associated cost for additional spots?
Per Section IV, E., page 47, the Siler location only has space for 45 contractor staff. The state would not be able to host an additional 15 spots at the Siler location.
56 Page 65 Administrative
Could you please clarify that there is not any preference points associated to firms that have a Local Business Certification? If so, can you please provide details of this preference as it relates to additional points within the evaluation criteria.
There are no preferential points associated with this procurement. Procurements which include federal funding do not qualify for Local Business or Veteran Owned Business preferences.
14