data collection questionnaire : agile-scrum€¦ · web viewdata collection questionnaire :...

Data Collection Questionnaire

Application Development & Enhancement

Agile - Scrum

Version : 1.0 Final Release Date : 7 January 2019

© ISBSG 2019

`Data Collection Questionnaire : Agile-Scrum

Table of Contents

Introduction.............................................................................................................................................................4

The International Software Benchmarking Standards Group...............................................................................4

Submitter..................................................................................................................................................................6

General Information.................................................................................................................................................6

Organizational Information......................................................................................................................................7

Application Information...........................................................................................................................................8

Application...............................................................................................................................................................9

Application type...................................................................................................................................................9

Application Type Details.......................................................................................................................................9

Application Environment......................................................................................................................................9

Users....................................................................................................................................................................9

Application Complexity..........................................................................................................................................10

Organization.......................................................................................................................................................10

Application.........................................................................................................................................................10

Technology.............................................................................................................................................................11

Package (Commercial of the shelve)...................................................................................................................11

Programming Language......................................................................................................................................11

Database............................................................................................................................................................11

Architecture...........................................................................................................................................................11

Environment...........................................................................................................................................................12

Middleware........................................................................................................................................................12

Instances............................................................................................................................................................12

Team......................................................................................................................................................................13

Scrum Attributes....................................................................................................................................................15

Agile Practices........................................................................................................................................................16

Tools.......................................................................................................................................................................17

Quality Metrics.......................................................................................................................................................18

Scrum Metrics........................................................................................................................................................19

Test Automation Metrics........................................................................................................................................20

Continuous Integration, Delivery and Deployment................................................................................................21

© ISBSG Page: 2


Estimation..............................................................................................................................................................22

Actual Effort and Cost.............................................................................................................................................23

Size Estimate Other than Story Points....................................................................................................................24

Team and User Satisfaction Survey........................................................................................................................25

Additional Information...........................................................................................................................................26

© ISBSG Page: 3


Introduction

The International Software Benchmarking Standards Group

The International Software Benchmarking Standards Group, (ISBSG), is a not-for-profit organization.

The ISBSG Software Project Repository exists: to provide software development practitioners with industry output standards against which they may

compare their aggregated or individual projects, and; to provide real data of international software development that can be analyzed to help improve the

management of IT resources by both business and government.

Benefits of Your InvolvementContributing to the ISBSG Project Repository provides you with an opportunity to see how your projects compare with the industry. You will be able to answer the question:

"Where do we fit in and why?" In addition, you are contributing to a body of software engineering knowledge that is open, available

and used for the betterment of the ICT profession.

AnonymityThe anonymity of contributors to the ISBSG repository is assured. The ISBSG has a process in place that ensures that the identity of contributors remains confidential.

Submitting informationPlease use the e-mail address [email protected] (administrator) for all communication with ISBSG and submitting your information.

Further steps1. The administrator assigns a submission ID to this submission.2. The administrator removes all data from the sheet that references to the submitting organization.3. The administrator forwards the anonymized submission to the repository manager.4. The repository manager analyzes the data and may have some questions.5. The repository manager sends the questions to the administrator.6. The administrator contacts the submitter in order to have the questions resolved.7. The repository manager processes the submission.8. The repository manager creates a benchmark report for the submission.

Thank you for submitting data! You are helping the industry advance step by step!In case of questions, the front office will contact you on the provided e-mail address:

© ISBSG Page: 4

mailto:[email protected]


Glossary of TermsA Glossary of Terms is available as a separate document that you can download from http://www.isbsg.org

© ISBSG Page: 5

http://www.isbsg.org/


SubmitterContact Information Answer

Contact person E-mail Role Organization

General InformationQuestion Answer Notes

Amendment to earlier submission <Previous ISBSG ID, if any> Please provide ISBSG IDDevelopment Type New Devlopment Please give details in case of

‘N/A’

What is the Project ID you wish to use for this project?

<Project Identification> This is for your own reference ISBSG will assign an ISBSG ID as well

Start date of the data/benchmark period DD-MM-YYYYEnd date of the data/benchmark period DD-MM-YYYYWhat was the most important objective for this release?

<Select Answer>

What was the second most important objective for this release?

<Select Answer>

What was the third most important objective for this release?

<Select Answer>

Release Definition of Done (DoD) <DoD> Please list the most important DoD criteria In case DoD criteria s not defined please mention ‘N/A

Was Release Definition of Done (DoD) completed?

<Select Answer> Did the release comply to all DoD criteria? Specify details in case the Release DoD was not completed.

Release Elapsed duration in months <Number> Mention the duration planned for the release in months.

For e.g.., 3 months, 1.5 months, 0,5 months

Release Delivered Date DD-MM-YYYY Mention the date the release is delivered

© ISBSG Page: 6


Organizational InformationQuestion Answer Notes

In which industry(s) do the software’s end users work?

If other or N/A, please specify

<Select Answer> End users are the people who physically interact with the software.

<Industry Details> Please give details in case of Others or N/A’

What is the type of the organization the software’s end users work?

<Organization Type> For instance, Banking or Insurance

End user’s organization location(s)/countries <Location(s) / Country Name(s) >

Conversion rate of hours per full time equivalent (FTE) per year

<Number> The number of hours an FTE works in the country where the software was developed

© ISBSG Page: 7


Application InformationQuestion Answer Notes

Is the application developed/maintained / supported internal (in house) or external?

Internal (in house) Please give details in case of ‘N/A’

Is the application developed/maintained/ supported on an onshore, nearshore or offshore location?

☐ Onshore

☐ Nearshore

☐ Offshore

Please select multiple options, as applicable

Onshore Country <Country Name> If Work was carried out in an onshore country, which country was this?

Nearshore Country <Country Name> If Work was carried out in a nearshore country, which country was this?

Offshore Country <Country Name> If Work was carried out in an offshore country, which country was this?

Is the application bespoke or Commercial off the Shelf (CotS) software?

<Select Answer> Example of packages: SAP, Peoplesoft, Oracle

If Commercial of the shelf, please specify the supplier

<Supplier Name>

Application go live date DD-MM-YYYY

Remarks

<Additional remarks, if any>

© ISBSG Page: 8


ApplicationQuestion Answer

Notes

Application typeWhat type of application, within the selected application group, has the project produced or enhanced?

Real-time Application Please provide details if answer is ‘N/A’

In case of a Business Application, select from the following list

<Select Answer> Please provide details if answer is ‘N/A’

In case of a Realtime application, select from the following list


In case of a Mathematically intensive application, select from the following list


If other, please specify <Specify>

Application Type DetailsIf you have selected an application within more than one application group, please estimate amount of functionality provided in each application group.

Business application <nn.nn>%Real-time application <nn.nn>%Mathematically-intensive application <nn.nn>%

Application EnvironmentImplementation platform <Select Answer> Please provide details if

answer is ‘N/A’If other, please specify <Specify>If mobile or device embedded, please specify <Select Answer> Please provide details if

answer is ‘N/A’If other, please specify <Specify>

UsersQuestion Answer Notes

Number of active (distinct) end users <Number>Number of key users <Number>Average number of concurrent users <Number>Maximum number of concurrent users <Number>Number of distinct installations <Number>

© ISBSG Page: 9


Application ComplexityQuestion Answer

Notes

OrganizationNumber of different countries servicedNumber of different user locationsNumber of different user profiles

ApplicationApplication importance for the organization

☐ Process not important

☐ Process important

☐ Process critical

☐ Business not important

☐ Business important

☐ Business critical

☐ N/A

Please specify details if answer is ‘N/A’

Total number of interfaces <Number>Number of batch interfaces <Number>Number of real time-online interfaces <Number>

Operating System (OS) on which the application is deployed

<Specify>

Hardware (Brand, type) on which the application is deployed

<Specify>

Age of the application <Years>Expected date of retirement/replacement DD-MM-YYYYNumber of releases in this benchmark period <Number>Percentage code changed (in benchmark/data period)

<nn.nn>%

Percentage functionality changed (in benchmark / data period)

<nn.nn>%

Percentage of changes caused by rework (in benchmark / data period)

<nn.nn>%

© ISBSG Page: 10


TechnologyQuestion Answer

Notes

Package (Commercial of the shelve)Degree of customization (%) <nn.nn>%

Programming LanguageMain programming language <Specify>2nd programming language <Specify>3rd programming language <Specify>…. programming language <Specify>

DatabaseMain database <Specify>2nd database <Specify>3rd database <Specify>…. database <Specify>

ArchitectureQuestion Answer Notes

Architecture <Specify> Architecture of the application: stand alone, multi-tier, Client-server, multi-tier with web public interface

Client / Server <Select Answer> Indicator of whether the application or product requires more than one computer to operate different components or parts of it

Client roles <Specify> The roles performed by the computers that provide interface to the software’s external users.

Server roles <Specify> The services provided by the host/server computer(s) to the software application or product.

Type of server <Specify> A description of the server to the software application or

© ISBSG Page: 11


Question Answer Notes

product.Client server description <Specify> A description of the

architecture of the client/server software application or product.

Web development <Specify> A derived indicator of whether the project data includes any comment that it is a web-development.

EnvironmentQuestion Answer Notes

MiddlewareWeb server <Specify>Message server <Specify>Another platform <Specify>

InstancesNumber of production instances <Number>Number of acceptation instances <Number>Number of test instances <Number>Number of development instances <Number>Number of other instances <Number>

Remarks


© ISBSG Page: 12


TeamQuestion Answer Notes

Scrum Training Received by team Details of Scrum training received by the team. Also mention if any role/work specific training undergone by the teamSpecify details of training conducted onshore, offshore and near shore

Number of Agile teams working on this release <Number>Number of Distributed teams <Number> Mention the number of

distributed teams that are part of the release. Teams are considered as ‘co-located’ if all team members are located in same office or city. Otherwise team is ‘distributed’ team

Number of Co-located Teams <Number> Mention the number of co-located teams that are part of the release. Teams are considered as co-located if team members are located in same office or city. Otherwise team is distributed team

Team Cross functional <Select Answer>

Average No. of people in a team <Number> Average Team Size.Average per day Productive Hours considered <Number> Specify the average per day

productive hours considered for a sprint.

For e.g. 6.5 Hours/ day or 7 Hours/Day or 8 Hours/Day

Number of Scrum masters with more than 5 years of experience

<Number>

Number of Scrum masters with experience between 3 years to 5 years

<Number>

Number of Scrum masters with less than 3 years of experience

<Number>

Number of times Scrum Master replaced during the release

<Number> Number of times scrum master changed during the project/release.

© ISBSG Page: 13


Number of Product Owners with more than 5 years of experience

<Number>

Number of Product Owners with experience between 3 years to 5 years

<Number>

Number of Product Owners with less than 3 years of experience

<Number>

Number of times Product Owners replaced during the release

<Number> Number of times product owner changed during the project/release.

Number of Scrum team members with more than 5 years of experience

<Number>

Number of Scrum team members with experience between 3 years to 5 years

<Number>

Number of Scrum team members with less than 3 years of experience

<Number>

Average per day Productive Hours considered <Number>Total Number of Sprints in this Release <Number>Sprint duration (in weeks) <Number>

Remarks


© ISBSG Page: 14


Scrum AttributesQuestion Answer Notes

Total Number of Sprints in a Release <Number> Mention the number of sprints executed in the release. Note: Provide the sum of sprints executed by all teams part of the releaseE.g., Release duration is 3 months, Sprint duration is 2 weeks. So, 6 sprints by a team in a releaseNumber of teams part of release: 3Total number of sprints in a release : 18

Sprints Duration (in weeks) <Number> Ideally sprint duration should be within 2-4 weeks

Sprint Definition of Done <SelectAnswer>

Acceptance Criteria Defined for each user story <SelectAnswer>

Backlog Refinement Session <SelectAnswer>

Sprint Review <SelectAnswer>

Customer Demo performed at regular intervals during the project

<SelectAnswer>

Sprint Retrospection <SelectAnswer>

Product Backlog <SelectAnswer>

Mention if the teams are using the product backlog to maintain user stories, bugs and other work items

Sprint Backlog <SelectAnswer>

Mention if the teams are using the sprint backlog to maintain the committed user stories, bugs and other work items for the sprint.

Impediment List <SelectAnswer>

Mention if the team is using the impediment list artifact to maintain impediments encountered during sprint

Architecture/ Design/ Functional specification <SelectAnswer>

Mention, if any specification document is used. Please provide details in case of ‘N/A’

Remarks


© ISBSG Page: 15


Agile PracticesQuestion Answer Notes

Test Driven Development <Select Answer>Pair Programming <Select Answer>Business Driven Development <Select Answer>Feature driven development <Select Answer>Other development practices <specify>

Remarks


© ISBSG Page: 17


ToolsQuestion Answer Tool Name Notes

Agile Project Management Tool used?

<SelectAnswer>

<Specify tool name, in case answer is yes>

Code Quality Tools used? <SelectAnswer>


Test Automation Tools used? <SelectAnswer>


Build / CI Tools used? <SelectAnswer>


Test Management tools used? <SelectAnswer>


Performance Testing Tools used?

<SelectAnswer>


Vulnerability Testing Tools used?

<SelectAnswer>


Release Automation / Continuous Delivery Tool used?

<SelectAnswer>


Any other tools used <SelectAnswer>

<specify any other tools used that are not listed elsewhere>

Remarks


© ISBSG Page: 18


Quality MetricsQuestion Answer Notes

Customer Reported Minor Defects <Number> Minor defects reported within three months of release Go Live

Customer Reported Major Defects <Number> Major defects reported within three months of release Go Live

Customer Reported Extreme Defects <Number> Extreme (Blocker and Critical) defects reported within three months of release Go Live

Total Defects Reported by Customer <Number> Total number of defects reported within three months of Go Live This column shows the total of defects reported (Minor, Major and Extreme).or Where no breakdown is available, the single value is specified, if known.

Defects Density <Number> Defects per story points calculated as: Total Defects Reported by Customer / Total Release Story Points delivered

Measures the quality of software in terms of defects delivered against total story points delivered for the release

Code Coverage Percentage 1st Language <%> Code Coverage(%) is calculated as: Lines of Code Covered by Automated Testing / Total Lines of Code

Code Coverage Percentage 2nd Language <%> Code Coverage(%) is calculated as: Lines of Code Covered by Automated Testing / Total Lines of Code

Code Coverage Percentage 3rd Language <%> Code Coverage(%) is calculated as: Lines of Code Covered by Automated Testing / Total Lines of Code

Remarks


© ISBSG Page: 19


Scrum MetricsQuestion Answer Notes

Committed Velocity (in Story Points) <Number> Sum of story points for the committed Features/User Stories for the release

Release Velocity (in Story Points) <Number> Sum of story points for completed Features/User Stories delivered in the release

On Time Delivery (%) <%> Mention the percentage of functionality/requirements delivered within the agreed timelines for the release

Is all Committed Functionality Released within committed time?

<SelectAnswer>

Mention Y/N if the agreed scope / functionality for the release are delivered within the committed time

‘Y’ indicates the agreed scope are delivered within time‘N’ indicates the agreed scope were not delivered within time

Remarks


© ISBSG Page: 20


Test Automation MetricsQuestion Answer Notes

Automated Unit testing – 1st Language <%> Mention the percentage of unit test automation done for the 1st language

Automated Unit testing – 2nd Language <%> Mention the percentage of unit test automation done for the 2nd language

Automated Unit testing – 3rd Language <%> Mention the percentage of unit test automation done for the 3rd language

Automated API testing <%> Mention the percentage of API Testing automation

Automated Functional testing <%> Mention the percentage of automation done for Functional Testing

Automated Regression testing <%> Mention the percentage of automation done for Regression Testing

Automated Performance testing <%> Mention the percentage of automation done for Performance Testing

Automated Vulnerability testing <%> Mention the percentage of automation done for Vulnerability Testing

Any other test automation <%> Mention if any other testing performed and percentage of automation done for that testing type

Overall Automation Testing <%> Mention the overall percentage of automation for a Product/Release

Remarks


© ISBSG Page: 21


Continuous Integration, Delivery and DeploymentQuestion Answer Notes

Build Frequency <Specify> Mention the frequency as on demand, multiple times in a day, daily, weekly, bi-weekly, monthly etc.

Build Trigger <Select Answer> <

Mention the mode of build triggered Manual/Automated

Is automated unit testing executed with build <SelectAnswer>

Common Artifact Repository <Specify> Mention if versions are maintained in common repository for every build that is successful

Deployment frequency to Production <Specify> Mention the frequency as Hourly, daily, weekly, bi-weekly, monthly etc.,

How many times code was deployed in production?

<Number>

Number of sites to be deployed <Number>

Remarks


© ISBSG Page: 22


EstimationQuestion Answer Notes

Size estimate method <Specify> Method used to estimate the story size e.g. planning poker

Release size estimate in Story Points <Number> Estimated story points for the release

Release Capacity – (FTE Days) <Number> Mention the available or planned capacity for the release. Capacity is calculated based on the total team size for the release, release duration and removing the vacations, training, organizational activities during the release duration

Release Delivery Date Committed <DD-MM-YYYY>

Delivery date estimate: Estimate of the delivery date for the release made during the planning activity.

Release Cost Estimate <Number> Estimated cost of Release made during the planning activity.

Cost Estimate Currency <Specify> USD, GBP, Euro etc.Currency Exchange rate to the USD at the time of estimate

<Number> Mention the currency conversion rate to USD

Cost estimate method <Specify>Method used to estimate the cost. <Specify>Estimation comments <Specify> Additional comments related to estimation

Remarks


© ISBSG Page: 23


Actual Effort and CostQuestion Answer Notes

Total Team Effort Estimate <Number> Estimate of the total effort of the release of the teams.

Average hour rate of all hours spent in the Project/Release

<Number> The sum of all effort hours spent divided by all cost related to these hours.

Actual Total Team Effort The sum of all effort hours spent on the project/release. Mention the effort only for scrum team including scrum master

Team Cost <Number> The sum of the cost of all effort hours spent on the project/release. Mention the total cost only for scrum team including scrum master

%Effort spent on Requirements <nn.nn>% Indicate the percentage of the effort hours spent on requirements gathering and analysis by the team, not the product owner.

%Effort spent on Design <nn.nn>% Indicate the percentage of the effort hours spent on User story definition and possible other design documentation

%Effort spent on Coding <nn.nn>% Indicate the percentage of the effort hours spent on Coding, configuring or parametrization and bug fixing after testing

%Effort spent on Test <nn.nn>% Indicate the percentage of the effort hours spent on all Test activities, e.g. test case design, development and execution

%Effort spent on Implementation <nn.nn>% Indicate the percentage of the effort hours spent on brining the software in a live production environment

Total Release Cost Estimate <Number> Estimate of the total cost of the release. This includes Human Resource, Software, Hardware, Infrastructure, Facilities etc.,

Currency <Specify> USD, GBP, Euro etc.Currency Exchange rate to the USD at the time of estimate

<Number> Mention the currency conversion rate to USD

Actual cost comments <Specify> Additional comments related to actual cost

© ISBSG Page: 24


Remarks


Size Estimate Other than Story PointsQuestion Answer Notes

Function Points (Project Size) <Number> The number of function points measured for the Product/Release

Functional Size Measurement Standard <Specify> Specify the method used to measure the functional size: IFPUG, Nesma, COSMIC, FISMA, Mark II, CISQ

Version of the Standard <Specify> For instance, Nesma 2.3 or IFPUG 4.11Method for calculating Project Size <Select

Answer>Did you measure the Project Size using the following formula?Project Size = FP Added + FP modified + FP deleted?

Nesma FPA enhancement <SelectAnswer>

Did you the Nesma FPA for Enhancement method? This method can also be applied to IFPUG. It considers impact factors for changed functionality, based on complexity of the change, and deleted functionality.

Is the FPA analyst certified? <SelectAnswer>

Please specify if the FPA analyst holds a valid certification for the method used to measure.

Lines of Code <Number> The number of the source lines of code (SLOC) produced for the Product/Release.

Tools used to capture LoC <Specify> Specify the tools used to calculate LoCOther Size Units <Specify> Specify any other Sizing methods used e.g.

Function pointsOther Size <Number>

Remarks


© ISBSG Page: 25


Team and User Satisfaction SurveyQuestion Answer Notes

Team Happiness Rating <Select Answer>

1 is Low and 5 is High

User Satisfaction Rating <Select Answer>

1 is Low and 5 is High

User Satisfaction method <Specify> Specify the method/tool used to capture user satisfaction ratinge.g.., Questionnaire, Net Promoter Score etc.,

Remarks


© ISBSG Page: 26


Additional InformationQuestion Answer Notes

Is the team involved in implementing Process Improvement Program?

<Select Answer> A process improvement program comprises a planned series of actions to improve how the software development team does its work

Does your organization have standard metrics program?

<Select Answer> Specifies whether the organisation has implemented a metrics program.

Metrics program duration (Years) <Number> Duration of the organisation's metrics program (in years) in case the organisation has implemented a metrics program.

Remarks


© ISBSG Page: 27

data collection questionnaire : agile-scrum€¦ · web viewdata collection questionnaire :...

Documents