#442 towards a needs assessment process … · towards a needs assessment process model for...

TOWARDS A NEEDS ASSESSMENT PROCESS MODEL FOR

SECURITY, EDUCATION, TRAINING AND AWARENESS

PROGRAMS: AN ACTION DESIGN RESEARCH STUDY

Lebek, Benedikt, Leibniz Universität Hannover, Königsworther Platz 1, 30167, Hannover,

Germany, [email protected]

Uffen, Jörg, Leibniz Universität Hannover, Königsworther Platz 1, 30167, Hannover,


Neumann, Markus, bhn Dienstleistungs GmbH & Co. KG, Hans-Lenze-Str. 1, 31855 Aerzen,


Hohler, Bernd, bhn Dienstleistungs GmbH & Co. KG, Hans-Lenze-Str. 1, 31855 Aerzen,


Abstract

Employees are considered to be the weakest link in information systems (IS) security. Many companies

and organizations started to implement security education, training and awareness (SETA) programs.

These provide their employees awareness of information security risks and the necessary skills to

protect a companies’ or organizations’ information assets. To ensure that SETA programs are

efficiently aligned to an organization’s objectives, it is essential to identify the most important areas

on which to concentrate. In research, there is a lack of generic process models for conducting SETA

needs assessments. In this study, we aim to close this gap by suggesting a systematic approach to

capturing, evaluating, and depicting the current state of employees’ security awareness and behavior.

Actual behavior is evaluated by determining the target values and measuring actual values with

respect to security metrics. In order to contribute to both, practical and academic knowledge, we used

an action design research (ADR) approach to draw general design principles from organizational

intervention within an international engineering company.

Keywords: SETA program, needs assessment, security behavior, security metrics, action design

research, process model.

Proceedings of the 21st European Conference on Information Systems

1

1 Introduction

The proliferation of a wide variety of complex and multinational information security risks leads to

major challenges for information security management (ISM). An important issue of IT managers is to

determine how to create efficient and sustainable organizational information security. Since

researchers refer to employees as the weakest link in information security (e.g. Bulgurcu et al., 2010;

Spears & Barki, 2010) security education, training, and awareness (SETA) programs have garnered

increasing attention. To maximize the number of prevented and deterred security breaches by

explaining and predicting employees’ security-related behavior, researchers have begun to incorporate

multidisciplinary theories, including theories from psychology, sociology, and pedagogy, into

integrated information security success outcome models (Karjaleinen & Siponen, 2011). But a

generally accepted approach that focuses on basic organizational requirements does not exist.

Practitioners face the problem of how the theoretical constructs that were found to be determining

employees’ behavior can be adopted. A gap between theoretically founded explanation of employees’

security behavior and the need of practitioners to know which interventions to apply can be identified

(Workman et al. 2008). As a result, and due to the complex nature of the information security domain,

organizations often face difficulties in managing an efficient and sustainable SETA approach when

considering personnel security, user access control, and network security (Eloff & Eloff, 2005).

In the development process and to ensure that SETA programs are efficiently aligned with

organizational objectives, some areas need to receive more attention and in turn should receive more

resources (Kruger & Kearney, 2006). To assist organizations in determining a risk and priority

measurement, the purpose of this paper is to provide a systematic and individualized approach to

capturing, evaluating, and depicting the state of employee security awareness and behavior. To specify

organizational needs in enhancing security awareness and behavior, a process model that theorizes the

needs assessment in an organizational context was developed and tested in an international

engineering company. To build a bridge between research and practice, we adapted a research

approach that is relatively new in IS research, namely action design research (ADR) by Sein et al.

(2011). With the use of different cycles, ADR allows continuous interaction between researchers and

practitioners in early stages. We explore the following research question:

What are the design principles for developing and implementing a needs assessment

process for SETA programs that considers an organization’s individual context?

The remainder of this paper is structured as follows: because of the relevance of ADR in this research

study, an overview and theoretical basis is presented first, followed by a description of the evaluated

process model. Subsequently, the process model is tested within the organizational setting: Actual

behavior is evaluated by determining the target values and measuring actual values with respect to

security metrics. We conclude with a discussion of general design principles that outline theoretical

and practical implications, as well as limitations, and then give an outlook for future research.

2 Research Design

The objective of this research study is to derive a needs assessment for a SETA program that can be

applied within multiple organizations. Therefore, we chose the action design research approach by

Sein et al. (2011) as the underlying research methodology. The term ‘action design research’ was first

mentioned by Iivari (2007) to describe the combination of action research (AR) and design research

(DR). Motivated by an increasing debate about the gap between organizational relevance and

methodological rigor (Lindgren, 2004; Iivari, 2007), Sein et al. (2011) introduced the ADR approach

in order to close this gap by presenting an integrative research approach of AR and DR. In their ADR

approach, the authors incorporated two challenges: First, by addressing a problem in a specific

organizational setting, ADR takes the influence of practitioners and the ongoing interaction with

researchers within the specific organizational context into account. Second, to meet the requirement of

academic contributions, ADR designs and evaluates generalized IS artifacts that address a class of


2

problems through formalized learning from organizational intervention. Although Sein et al. (2011)

primarily see technical products as the outcome of DR, we argue that the ADR approach is also

applicable to an extended definition of the term artifact that includes organizational and social aspects

of IS (Hrastinski et al., 2008), as well as concepts (Järvinen 2007), models, methods and instantiations

(March & Smith, 1995; Hevner et al., 2004). We adopted the four stages of ADR as proposed by Sein

et al. (2011): (1) problem formulation, (2) building, intervention and evaluation (BIE), (3) reflection

and learning, and (4) formalization of learning (Figure 1).

Researchers

Practitioners

(IT managers)

Employees

(End users)

Process

model

Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5

Applied

methods

Stage 3: Reflection and learning Stage 4: Formalization of learning

Stage 1: Problem formulation

ADR Team

Sta

ge

2:

Bu

ild

ing

, in

terv

en

tio

n a

nd

ev

alu

ati

on

(BIE

)

Literature

analysis

Semi-structured interviews

Goal question metric (GQM)

Online questionnaires

Analytical hierarchy process (AHP)

Contributions

Design

principles

Process

model for

needs

assessments

Utility for

needs

assessment

Alpha

version

Beta

version

Figure 1. Research design based on ADR approach by Sein et al. 2011

The first stage was triggered by a problem perceived in the practical setting. In order to conduct a

needs assessment for a SETA program, the target organization faced the problem of how to capture the

actual level of employees’ security awareness and behavior. Based on a review of academic and

practical literature, the specific practical problem was formulated as an instance of a broader class of

problems. An ADR team was formed, made up of researchers from a German university and members

of the SETA project team within the target organization, including the company’s CIO and the

security project manager. The shared competencies facilitated the problem definition and formulation.

The adopted problem framing in stage one provides the baseline for the following stages. The BIE

stage is consisting of five iterative cycles carried out in a real-world environment in order to build and

continuously evaluate a process model to conduct a SETA needs assessment. An initial process model

design (‘alpha version’) was developed throughout cycle one and introduced to practitioners for the

purpose of evaluation in cycle two. The first practical iteration did not shape the third level

(‘employees’) because of the needed expertise of designing the artifact. Based on feedback from the

practitioners, the initial process model design was specified in cycle three (‘beta version’). The

applicability of the proposed needs assessment approach was tested within IT department of the target

company. Feedback of the participating employees was used to refine the model again in cycle five

until the final version was reached and adopted by the participating organization. In order to evaluate

the process model, stage three (reflection and learning) was carried out simultaneously to the previous

stage. On the basis of feedback from cycles one to four, this stage allowed to transfer experiences from


3

the specific problem solution within the target organization into knowledge that addresses the broad

class of problems. It also helped to gain a clear understanding of the problem due to early evaluation.

The fourth stage aims to provide a general solution for the broad class of problems as it outlines the

results of this study as design principles.

3 Process Model Development

Problem Formulation

The present study emerged from a project for developing and implementing a SETA-program within

an engineering company. The company operates in 60 countries from its headquarter in Germany with

a total of 3,200 employees. The SETA-project is based on the NIST SP-800-50 standard and consists

of four phases: (1) program preparation, (2) program development and implementation, (3) program

execution, and (4) program evaluation. Part of the first phase of the SETA-project is the execution of a

needs assessment (cf. NIST SP-800-50) to determine the extent of the lack of security awareness and

potential need for action by using training- and awareness measures. Although company management

is generally aware that the security behavior of employees plays an important role in any information

security concept, employees’ security behavior was considered inadequate. To enhance security

awareness, the company used general information security presentations on a regular basis. However,

the state of employees’ actual security behavior was not monitored. As Abdulrazeg (2012) pointed out,

security behavior cannot be improved if it cannot be measured, so we saw the need of a structured

approach for capturing, evaluating and depicting the state of employees’ security awareness and

behavior.

A comprehensive literature review (for details see Lebek et al. 2013) based on the structured approach

by Webster and Watson (2002) was conducted to access the current state of information security

awareness research. We searched through ten databases: AISeL, ScienceDirect, IEEEXplore, JSTOR,

SpringerLink, ACM, Wiley, Emerald, InformsOnline, Palgrave Macmillan. A list of search terms was

pre-defined, including ‘security awareness’, ‘awareness training’, ‘awareness program’, ‘awareness

campaign’, ‘security education’, ‘security motivation’, ‘security behavior’ and ‘personnel security’. In

total 113 articles were identified to be relevant. Results indicate that in the past decade of security

awareness research, researchers mainly focused on the application of behavioral-cognitive models

(Lebek et al., 2013). These models explain behavioral factors that raise employees’ security

awareness. Researchers have begun to incorporate multidisciplinary theories, including theories from

psychology, sociology, and pedagogy into integrated information security success outcome models

(Karjalainen & Siponen, 2011). We determined a lack of general accepted meta- or process models

that theorize the needs assessment in an organizational context. For the theoretical foundation of the

proposed method presented in this paper, we made use of both theoretical and practical models and

guidelines. On the theoretical side, we used prior work, for example from Kruger and Kearney (2006).

The authors developed a prototype to measure security awareness levels of employees based on six

focus areas. However, the needs assessment procedure was underrepresented. On the practical side, we

adapted NIST SP-800-50, which provides guidelines for SETA needs assessments in organizations.

Initial Process Model Design

For the design of an initial process model we used primarily two data sources: (1) The results of the

comprehensive literature review and (2) the results of semi-structured interviews which were

conducted with 6 IT managers of the partner company. The interviews aimed at initially collecting

requirements regarding the process “needs assessment” in the context of a SETA program. To analyze

the results of both data sources coding methods were used according to Strauss and Corbin (1990). We

applied open, axial, and selective coding to get categories, sub-categories, attributes and relationships

out of the raw material. These constructs were finally used to design an initial rudimentary process

model (‘alpha version’) for identifying information security training and awareness needs (Figure 2).


4

Comparison

and

evaluation

Identifying

roles and

focus areas

Weighting

importance

and risk

Definition

of target

values

Identifying

measurement

goals

Measuring

actual

values

Develo-

ping

metrics

Determining target values Measuring actual values

Sets up

Figure 2. Process model for evaluating information security training and awareness needs

The ADR team agreed that evaluating employees’ security behavior as a whole was inapplicable to

determining information security training and awareness needs. For this reason, we decided to

implement several perspectives on employees’ security behavior. First, we assumed that employees in

different roles or positions demonstrate different security-related behavior, resulting in a role-based

view. Secondly, we adopted the concept of focus areas from Kruger and Kearney (2006). For the

purpose of this research study, focus areas are defined as critical risk areas in which the behavior of

the employee is evaluated (e.g. ‘use of mobile devices’). Because we assumed that each focus area

contains a different risk potential, the focus areas need to be weighted amongst each other. Further, the

ADR team supposed that each focus area is of differing importance for the different roles within the

organization. For example, the focus area ‘use of mobile devices’ is obviously less important for roles

that do not use mobile devices in their work environment, such as application developers. On the other

hand, the focus area is more important for roles with extensive use of mobile devices such as

management. After the roles and focus area definition process, the measurement goals have to be

defined. Applicable security metrics have to be identified based on the measurement goals. In

information security research, the use of self-reporting data to determine employees’ information

security behavior is predominant (e.g. Ifinedo 2011). However, the use of self-reported data to

measure security-related behavior is prone to the problems of common method variance, consistency

motif, and social desirability, and results may be biased (Workman et al., 2009). Therefore, the

integration of empirical data that determines actual behavior (e.g. system monitoring data, incident

records) into the measurement process is preferable. With the purpose of defining desired behavior,

the importance and risk weightings have to be transformed into specific target values. In order to

evaluate the gap between actual and desired behavior, a normalization process is needed to ensure that

target and actual values are comparable. The general requirements for a needs assessment defined in

the problem formulation stage were refined as shown in Table 1.

Table 1. General requirements for a needs assessment process

4 Target Value Determination

Definition of Focus Areas and Roles

Following the requirements set up by the initial design of the process model, it was necessary to define

the observation levels (i.e. roles, focus areas) in the first instance. The employees’ roles were

Determination of desired behavior

• To determine desired behavior, different observation levels (i.e. roles, focus areas) must be considered.

• Each focus area must be weighted by its inherent risk potential.

• The importance of each focus area must be weighted for each role.

Measuring employees’ actual behavior

• Applicable metrics must be developed based on the measurement goals.

• Reliable data sources must be included (e.g. system monitoring data, incident reports).

Evaluation of the gap between actual and desired behavior

• Target values and actual values must be normalized in order to establish comparability (e.g. by using a

points-based system).

• Training and awareness needs per role and focus area should be presented in a short table form which

is intuitive to IT managers (named ‘awareness map’).


5

predetermined by the organization’s business processes. In order to get a valid theoretical foundation

for defining the focus areas, we utilized an approach similar to the initial process model development

as we used the perspectives of prior academic work from e.g. Drevin et al. (2007) and Kruger and

Kearney (2006) and semi-structured interviews with IT experts within the company.

In their research study, Drevin et al. (2007) derived a value-focused information security awareness

approach whose fundamental objectives included a network of key areas that must be taken into

account in security decisions. The authors identified thirteen mean objectives, e.g. maximize logical

access control, minimize virus infection, and responsible use of e-mail and internet. Within these

assisting objects, one limitation was that there was no generally accepted information security object

with coherent areas or labels that addressed the assessment of information security behavior

(Kritzinger & Smith, 2008; May & Dhillon, 2010; Torres et al., 2006). With the purpose of gaining a

more practical view on relevant focus areas, we also considered several information security reports

from recent years (e.g. Verizon – 2011 Data Breach Investigation Report, KPMG - The e-Crime

Report 2011, CERT 2011 Cyber Security Watch Survey). Based on the literature analysis, a general

list of focus areas was prepared. Due to their generic scope, each focus area had to be validated within

the context of the target organization. For this purpose, an additional team was formed, consisting of

six members who are well versed regarding the underlying topics (named ‘expert team’). The expert

team includes three IT managers, one information security manager, one governance, risk and

compliance manager, and one IT security expert. We used semi-structured single interviews to present

the focus areas to the expert team members. Tape recording supported the authors in collecting and

analyzing data accurately. First, all interviewees were asked to select the focus areas that are relevant

for the target organization and whether they considered any addition or changes of the focus areas to

be necessary. A list of nine critical areas of information security awareness was resulting: access

control, client workplace, storage media, mobile devices, software, internet, e-mail, handling of

critical information, and physical safeguarding of the workplace. Second, each interviewee was asked

to determine factors that accounted for each focus area in the project organization from his or her point

of view. For example, for the focus area ’mobile devices’, the interviewees named ’damage to

devices’, ’network access’, ’apps’, and ’securing of mobile devices’.

Focus Area Weighting and Target Value Definition

In order to determine the inherent risk potential (RP) for each focus area and the importance (I) of

each focus area per role, we made use of the analytic hierarchy process (AHP) as proposed by Saaty

(1980). This method was developed to solve complex, multi-criteria decision problems. Four major

arguments influenced our decision to use AHP: AHP provides explicit specifications in analysis,

intuitiveness, validated measurement scales, and has robust built-in consistency assessments.

Following the AHP approach, a specified number of questions were developed for pairwise

comparison of the focus area measures. The weights were obtained from the members of the expert

team and the company’s CIO by using an online questionnaire. The results of the pairwise comparison

were aggregated in a (n x n) comparison matrix. Normalized eigenvectors with a sum to one indicated

the relative importance/inherent risk for the different focus area measures. For each individual

judgment matrix, this procedure was used to derive the average risk and priority matrix for each focus

area. Overall weights were built by calculating the average value of each expert’s individual

weightings of importance and inherent risk, resulting in one matrix for importance (I) for each focus

area per role and one matrix for inherent risk potential (RP) for each focus area. The impact value (IV)

of each focus area per role IV = I x RP was calculated (Table 2). Subsequently, the calculated impacts

were used to determine target values on a scale ranging from 0 to 100 by using a spreadsheet

application. In order to explain the awareness level, the following target corridors were derived in

accordance with the expert team: 100 - 75 = good; 74.9 - 50 = average; 49.9 - 25 poor; 24.9 and less =

unacceptable. The lower limit of the section ‘good’ (=75) was multiplied by (1+IV) for each focus

area and role. In order to avoid having target corridors that were too small, a minimum size of the

corridor ‘good’ was set to 10 points. All other lower limits were raised by the same amount. The


6

resulting target corridors for two example focus areas, ‘client workplace’ and ’mobile devices,’ are

shown in Table 3.

Focus areas Roles

Onsite staff Management

Server

administration

Application

development

Client workplace 0.18 0.11 0.11 0.14

Mobile devices 0.28 0.30 0.27 0.22

Table 2. Example of impact values

Focus Areas Roles

Onsite staff Management

Server

administration

Application

development

G* A* P* G A P G A P G A P

Client workplace 88 63 38 83 58 33 83 58 33 85 60 35

Mobile devices 90 65 40 90 65 40 90 65 40 90 65 40 *Lower limits of corridors: G = Good, A = Average, P = Poor

Table 3. Example of target corridors

5 Actual Value Measurement

Metrics Development

Actual behavior was measured with security metrics. To select security metrics, we used the goal-

question-metric (GQM) approach introduced by Basili and Weis (1984). This validated approach

facilitated the selection and implementation of useful metrics and aligned them to the identified focus

areas. The GQM method was originally used to develop software metrics, but was also applied in

literature in the context of security metrics (e.g. Hayden 2010; Abdulrazeg et al., 2012). In general, the

GQM approach consisted of three steps (Ebert et al., 2005). First of all, a clear formulation of concrete

goals for improving security behavior was required. The aim of the proposed needs assessment

process is to measure employees’ behavior within organization specific focus areas. Consequently

nine goals were derived directly from areas defined above. In the second step, questions were

developed from the defined goals. For this purpose, we used the factors which were named during the

expert team interviews to define the focus areas. The formulated questions related to the essential

aspects of goal achievement. In the third step, the corresponding metrics were defined by the ADR

team. Figure 3 shows an excerpt from the GQM approach used within the focus area ‘mobile devices’.

Appropriate use of mobile devices

Which apps are used by employees? How do employees secure the mobile devices?

Number of

installations of

unauthorized apps

Number of devices with

installed unauthorized

apps

Frequency of use

of data

encryption

Frequency

of use of

PINs

PIN

Complexity

Frequency of

leaving devices

unattended

Goal

Questions

Metrics

Figure 3. Excerpt from the GQM approach for the focus area ’mobile devices’

Following this process, a total of one hundred metrics were developed for the nine defined focus areas.

Subsequently, the results were discussed with the project company’s information security manager and

IT security expert. During this discussion it became apparent that some of the defined metrics were

unnecessary. For example, since the use and complexity of PINs for mobile devices is inevitable due


7

to technical restrictions, the corresponding metrics were dropped. Other metrics were withdrawn since

no explicit regulations had been defined within the company’s security policies.

Metrics collection

Reliable data sources were determined to collect these metrics. Not every required metric could be

obtained from either system monitoring data or incident management records (e.g. frequency of

writing down passwords). Due to the sensitive context, additional methods for collecting the required

data became necessary. We had to resort to employee self-reports. The use of questionnaires for this

purpose provides several advantages (Malhotra 1999): first of all, structured questionnaires are easy to

administer and provide reliable and comparable data since the respondents are limited to a

predetermined set of answers. Moreover, online surveys can be distributed to all employees via the

project company’s communication systems and data is collected as soon as an employee finishes the

questionnaire, providing a time advantage. However, questionnaires incur another difficulty, the

phenomenon of ‘social desirability’ (Oppenheim 1992; Fowler 1995; Malhotra 1999; Pauls & Crost,

2004). However, keeping the difficulties of obtaining security-related data in mind, (Katoulic et al.,

2004) online questionnaires are least susceptible to social desirability and are therefore suitable for

obtaining sensitive data (Malhotra 1999). Since questionnaires that control for social desirability have

been proven to be inapplicable (Pauls & Crost, 2004), we opted not to implement these controls, but

instead took additional steps to mitigate the social desirability effect and facilitate the employees’

motivation in participating. We assured participants that anonymity and confidentiality measures were

in place and communicated the necessity of response accuracy (Fowler 1995). Furthermore, we

followed the rules for questionnaire design proposed by Oppenheim (1992).

As mentioned above, the proposed needs assessment approach was tested within the company’s IT

department. Consequently the survey was sent to all 50 IT employees, 29 of which returned a

completed questionnaire. At the beginning of the online questionnaire, each participant had to select

his or her role within the organization (e.g. onsite-staff, management, server administration, or

application development). Based upon role specification, the online survey tool provided a specific set

of questions for each participant. For example, the roles ‘application development’ and ‘server

administration’ were not asked about mobile devices, since they do not use mobile devices during their

work. The questionnaire was divided into two sections. In the first section, the employees were asked

about security behavior in the focus areas that were relevant for their role according to the expert

group weighting. In the second section, the employees were asked about their attitudes towards

information security in the respective focus areas.

Behavior Evaluation

Subsequently, the collected data was normalized for comparison with the target corridors (cf.

section 4) and evaluation of the gap. For this purpose, we used a scale ranging from 0 to 100. A score

for both the behavior and the attitude measurement section was determined per role and focus area.

During the process of collecting system monitoring data and incident records, we had to deal with

insufficiently detailed data in some focus areas, i.e. metrics were not drilled down to organizational

unit level or even role level. In order to achieve comparability, we resorted to the experience of the

expert team members. We found that the experts were able to break down the global metrics to

required details. Furthermore, the experts evaluated the metrics using a five-point Likert scale.

Points Role specific corridors

Score behavior: 89.1 (●●)

Good Average Poor

Score attitude: 82.7 (●)

Score monitoring: 60.0 (●)

Overall score (Ø): 77.3 (●) 85 60 35

Difference to corridor 'good': -7.73 ●● = Good, ● = Average, ○ = Poor, ◊ = Inacceptable

Table 4. Scores for role ’application development’


8

Focus areas Roles

Onsite staff Management Server

administration

Application

development

Client workplace ● (-16.43) ●● (+1.9) ● (-4.92) ● (-7.73)

Mobile devices ● (-23.75) ●(-14.97) n/a n/a ●● = Good, ● = Average, ○ = Poor, ◊ = Inacceptable; (+/- X) difference from overall score to the lower limit of the corridor ‘good’

Table 5. Excerpt from the awareness map of the IT support process

After that, the transformation process described above was carried out and a score was that averaged

each evaluated metric was determined. The overall score was determined by averaging the three single

scores (Table 4). In the last step, the overall scores were compared to the determined corridors for

each role. The degrees of goal achievement were transferred to the awareness map (Table 5). The

difference between each role’s overall score and the lower limit of the respective corridor ‘good’ was

calculated.

6 Formalization of Learning and Discussion of Results

Following the ADR approach, we reflected on each step during the problem formulation and BIE

stages to learn from the practical intervention. Through formalization, the learning was transformed

into general design principles with the purpose of contributing academic knowledge to the respective

research field. The final results are presented in Table 6.

Table 6. Set of Design Principles

At the beginning of the project, the consolidation of stakeholders emerged as a necessary condition to

successfully implement the needs assessment process for several reasons. First, the support of the

company’s top management was needed to emphasize the importance of a needs assessment process.

Second, the expert team forms the connector to the human factor. By developing and weighting of

focus areas, the expert team fits the needs assessment process to the individual requirements of the

organization. Due to their practical experience, the experts were able to compensate for insufficient

data from system monitoring. Through the early inclusion of key users, an understanding for the

purpose of the project could be accomplished among the employees, which has been proven to be

beneficial in the sensitive context of employees’ information security behavior.

Design principle Description

Stakeholder

integration

It is necessary to consider relevant stakeholders (i.e. management, experts, key-users) to

reduce barriers within the organization and understand the purpose. Experts and key-users

provide valuable experiences that complement measured data.

Perspectives

Different observation levels should be integrated to enable a selective analysis of the

current state of employees’ security behavior. The selection and combination of

observation levels depends on the organizational context.

Weighted focus

areas

Focus areas are critical risk areas of employees’ security behavior. To determine adequate

target values, the risk potential and importance of each focus area has to be evaluated.

Applicable

metrics

A standardized process for developing metrics that correspond to organization-specific

focus areas is a basic condition to ensure the validity and reliability of measuring

employees’ security behavior.

Reliable data

sources

Instead of relying completely on employees’ self reports, the use of reliable data sources

such as system monitoring should be aspired to. However, the integration of system

monitoring data requires the establishment of a mature and detailed monitoring process.

Normalization To make metrics comparable, normalization of data is needed.

Awareness map

By depicting results from the evaluation process in an awareness map, needs for training

and awareness measures can easily be identified. However, proper documentation of the

measurement process is necessary to develop concrete measures.


9

With the purpose of providing a basis for determining and developing appropriate training and

awareness measures, we emphasize the necessity of integrating different perspectives into the needs

assessment process. Those perspectives can be roles and focus areas, as in our case, but also business

units, departments, or business processes. The combination of several perspectives facilitates a

variable consideration of employees’ security behavior within an organization.

Focus areas constitute critical risk areas in which employees’ security behavior is evaluated. Although

several propositions for focus areas exist in literature, organization-specific customizing is necessary

(cf. Chapter 3.2). This requires a standardized selection process (e.g. expert interviews, focus group

discussion). Based on the assumption that focus areas provide different inherent risk potentials and are

of deferring importance for each role, a weighting process is needed. The adoption of the AHP

approach turned out to be an applicable method of developing weights in this context. However, the

use of online questionnaires to conduct pairwise comparisons entailed unanticipated difficulties. Even

though a definition of each focus area was sent to the participants, the expert team members struggled

to understand the focus areas. We solved the problem by individually explaining the focus areas to

each expert team member as the problem occurred. The online questionnaire consisted of 180 pair

wise comparisons, which meant a high workload for each expert team member. This led to a high

number of questionnaires being incomplete. To avoid this problem, we recommend using a method

that a priori allows interactions between researchers and participants (e.g. focus group discussions) to

perform the AHP process.

To measure employees’ actual behavior within the defined focus areas, applicable metrics had to be

defined. The GQM approach provides a simple and easy way of developing metrics from the goals set

up by the defined focus areas. Data from system monitoring or incident records is considered more

reliable data than results from self-reported data. However, the use of questionnaires is necessary in

order to gain full coverage of employees’ behavior and security related attitudes. Additionally,

questionnaires are better for subsequent analysis, because results can be compared by using

homogeneous scales. A major challenge emerged in regard to the inclusion of system monitoring data.

Although we anticipated that adjustments would be necessary to make the data comparable, we

discovered that the available data was not sufficiently detailed, e.g. metrics for unauthorized software

installations were not drilled down to organizational unit level or even role level. A mature system

monitoring process is a necessary precondition for successfully integrating system monitoring data

into a SETA needs assessment process. By normalizing collected metrics, the measurements were

made comparable. The use of a scale from 0 to 100 was proven to be applicable. The depiction of the

degree of goal achievement in an awareness map enables managers to gain a fast initial overview of

the current state of employees’ security behavior and to identify areas that need security training and

awareness measures. Furthermore, through step-by-step documentation of the measurement process, a

more detailed view of the identified needs was gained, thus providing a basis for developing training

and awareness measures.

7 Limitations and Outlook

This study is subject to the following limitations: First, in order to solve a specific organizational

problem and derive solutions for a class of problems, an ADR approach was used. Even if this study

has proven that ADR is suitable for drawing design principles for SETA needs assessment processes

from a specific organizational context, only one organization participated in the research process. It

can be argued that this fact challenges the generalizability of the study’s findings, but Lee and

Baskerville (2003) showed that a greater sample size within qualitative studies is not an indicator of

greater generalizability. However, artifact quality might benefit from further evaluation and

refinement by including several companies into a field study. In addition, cross-organizational

differences may affect the needs assessment for SETA programs with regard to external variables.

Future studies could investigate differences in branch or company size. The suggested needs

assessment process was applied to one business process within the target company and measured

employees’ security behavior in two out of nine focus areas. Since the suggested approach is repetitive


10

for each business process and focus area, we do not expect substantial changes to the general design

principles when more processes and focus areas are included. However, the design principles can be

refined through experience from practitioners and through employee feedback during an organization

wide roll-out of the needs assessment process. The focus of this paper was to develop and validate an

approach for needs assessments which represents the first step in the overall process of implementing

a SETA program. It would be interesting for future research to investigate the long term experiences

of the application of the proposed needs assessment approach. Particularly in the context of

developing concrete information security awareness and training measures, the suggested approach

has to prove its utility, which is part of an ongoing research process as mentioned in the problem

formulation stage. In the course of this study, an organization specific list of security metrics was

developed. It would be valuable if future research provides a generic list of security metrics in order to

complement the proposed process model.

8 Conclusion

This research study is a first step to provide a needs assessment process for SETA programs. Based on

an ADR process, the gap between organizational objectives and current awareness was explored. For

this purpose, we built an ADR team that consisted of researchers and IT managers from an

international engineering company. We emphasized the target value definition and development of a

reliable and valid measurement process as the two major challenges to conducting a SETA needs

assessment within the target company. On this basis, initial requirements for a process model were

developed and refined during several cycles of theoretical and organizational intervention until general

design principles were set up. After considering the limitations, the suggested process model and

particularly the proposed design principles contribute to practical and theoretical knowledge. This

study is focusing on the gap between theoretically founded explanation of employees’ security

behavior in academic literature and the need of practitioners to know which interventions to apply.

From a practical perspective, the developed model assists organizations in implementing a needs

assessment for SETA programs and builds directly on the NIST-SP-800-50 standard. It supports IT

managers in identifying and evaluating the undesired security behavior of employees and provides a

basis for developing adequate training and awareness measures. On the theoretical side, this study

contributes to scientific literature as it focuses on reducing the lack of generic process models in the

context of employees’ security behavior. Whereas previous research is mainly focused on the adoption

of different cognitive factors to explain and predict the security related behavior of employees’ (Lebek

et al., 2013), this study facilitates the development of concrete training and awareness measures to

improve employees’ behavior. The suggested needs assessment approach enables dynamic depiction

of the current state of employees’ security behavior within organizations and its changes over time.

This provides the basis for future research to test and evaluate the efficiency of different SETA

measures in the organizational context.

References

Abdulrazeg, A.A.; Norwawi, N. & Basir, N. (2012): Security Measurement Based On GQM to

Improve Application Security During Requirements Stage, International Journal of Cyber-Security

and Digital Forensics 1(3), pp. 211-220.

Basili, V. & Weiss, D. (1984), A Methodology for Collecting Valid Software Engineering Data,

Software Engineering 10 (6), pp.728-738.

Bulgurcu, B.; Cavusoglu, H. & Benbasat, I. (2010): Information security policy compliance: An

empirical study of rationality-based beliefs and information security awareness, MIS Quarterly

34(3), pp. 523-548.

Drevin, L.; Kruger. H.A. & Steyn, T. (2007): Value-focussed assessment of ICT security awareness in

an academic environment, Computer & Security 26(1), pp. 36-43.

Ebert, C.; Dumke, R.; Bundschuh, M. & Schmietendorf, A. (2005); Best Practices in Software

Measurement - How to use metrics to improve project and process performance, Springer, Berlin.


11

Eloff, J.H.P. & Eloff, M.M. (2005): Information Security Architecture. Computer Fraud & Security

11(1), pp. 10-16.

Fowler, A. & Floyd J.Jr. (1995): Improving Survey Questions: Design and Evaluation, Applied Social

Research Methods Series 38, SAGE Publications Inc., Thousand Oaks (CA).

Hayden, L. (2012): IT Security Metrics - A Practical Framework for Measuring Security & Protecting

Data , McGraw-Hill Publ. Comp.

Hevner, A.; March, S.; Park, J. & Ram, S. (2004): Design Science in Information Systems Research,

MIS Quarterly 28 (1), pp. 75-105.

Hrastinski, S.; Carlsson, S.; Henningsson, S. & Keller, C. (2008): On How to Develop Design

Theories for IS Use and Management, ECIS 2008 Proceedings, Paper 138.

Ifinedo, P. (2011): Understanding information systems security policy compliance: An integration of

the theory of planned behavior and the protection motivation theory, Computer & Security 31(1),

pp. 83-95.

Iivari, J. (2007): A paradigmatic analysis of information systems as a design science, Scandinavian

Journal of Information Systems 19 (2), pp. 39-63.

Järvinen, P. (2007): Action Research is Similar to Design Science, Quality and Quantity 41 (1), pp.37-

54.

Karjalainen, M. & Siponen, M. (2011): Toward a New Meta-Theory for Designing Information

Systems (IS) Security Training Approaches, Journal of the Association for Information Systems

12(8), Paper 3.

Kruger, H.A. & Kearney, W.D. (2006): A prototype for assessing information security awareness,

Computers & Security 25 (4), Pages 289-296

Lebek, B.; Uffen, J.; Neumann, M.; Hohler, B. & Breitner, M.H. (2013): Employees’ Information

Security Awareness and Behavior: A Literature Review, Proceedings of the HICSS 2013.

Lee, A.S. & Baskerville, R.L. (2003): Generalizing Generalizability in Information Systems Research,

Information Systems Research 14 (3), pp. 221–243.

Lindgren, R.; Henfridsson, O. & Schultze, U. (2004): Design Principles for Competence Management

Systems: A Synthesis of an Action Research Study, MIS Quarterly 28 (3), pp. 435-472.

Malhotra, N.K. (1999): Marketing Research: An Applied Orientation, third edition, Prentice-Hall

International Inc.

March, S. & Smith, G. (1995): Design and Natural Science Research on Information Technology,

Decision Support Systems 15, pp.251 - 266.

May, J. & Dhillon, G. (2010): A holistic approach for enriching information security analysis and

security policy formation, ECIS 2010 Proceedings, Paper 146.

Oppenheim, A. N. (1992): Questionnaire Design, Interviewing and Attitude Measurement,

Continuum.

Pauls, C.A. & Crost, N.W. (2004): Effects of faking on self-deception and impression management

scales, Personality and Individual Differences 37, pp. 1137-1151.

Saaty, T.L. (1980): Multicriteria Decision Making: The Analytic Hierarchy Process; McGraw-Hill.

Sein, M.K.; Henfridsson, O.; Purao, S.; Rossi, M. & Lindgren, R. (2011): Action Design Research,

MIS Quarterly 35 (1), pp.37-56.

Spears J.L. & Barki, H., (2010): User Participation in Information Systems Security Risk

Management, MIS Quarterly, 34 (3), pp. 503-522.

Strauss, A. & Corbin, J. (1990): Basics of Qualitative Research: Grounded Theory Procedures and

Techniques, Sage Publications.

Torres, J.M.; Sarriegi, J.M.; Santos, J. & Serrano, N. (2006): Managing information systems security:

Critical success factors and indicators to measure effectiveness. ICIS 2006 Proceedings, pp. 530-

545.

Webster, J &Watson, R.T. (2002): Analyzing the Past to Prepare for the Future: Writing a Literature

Review, MIS Quarterly 26, pp. xiii-xxiii.

Workman, M.; Bommer, W.H. & Straub, D. (2008): Security lapses and the omission of information

security measures: A threat control model and empirical test, Computers in Human Behavior 24,

pp. 2799–2816.


12

#442 towards a needs assessment process … · towards a needs assessment process model for...

Documents