societal impact assessment manual and...

Criteria for Assessing and Mainstreaming

Societal Impacts of EU Security Research Activities.

Coordination and Support Action.

Societal Impact Assessment Manual and

Toolkit

Deliverable 3.1

Trilateral Research & Consulting

April 2014

ASSERT is co-funded by the European Commission under the 7th Framework Programme, theme „security“, call FP7-

SEC-2012-2, work programme topic 6.3.2. „Criteria for assessing and mainstreaming societal impacts of EU security

research activities – Coordination and support action“.

2 D3.1 Societal Impact Assessment Manual ASSERT

Imprint

Responsible project partner:

Trilateral Research & Consulting (TRI)

Author(s):

David Barnard-Wills, Kush Wadhwa, David Wright

Contact:

David Barnard-Wills, [email protected]

ASSERT website

www.assert-project.eu

Version history

Version Date Change/Remark Responsible (person,

beneficiary/function)

0.1 26 March 2014 David Barnard-

Wills/TRI/Author

0.2 23 April 2014

Following review, ex-

panded introduction,

added section four

David Barnard-

Wills/TRI/Author

0.3 23 April 2014 Final quality review Kush

Wadhwa/TRI/Author


EXECUTIVE SUMMARY

The focus of WP3 is to operationalise the assessment of societal impacts of security

research through a structured approach, drawing upon the outcomes of WP1. Task 3.1.

and 3.2 are interlinked, resulting in a highly usable and effective assessment tool de-

ployed via the web. The toolkit itself is an interactive, visual representation of the SIA

manual that is developed in task 3.1. This deliverable sets out the following:

The first section of the report sets out a state-of-the art-methodology for societal impact

assessment for security research. This is a structured methodology for conducting a

societal impact assessment (SIA) of security research and security measure implemen-

tation. The section first provides an overview of the need for and role of societal impact

assessment, then presents an account of the existing impact assessment methodologies

that have influenced this guide. Section two describes the core methodology based up-

on an interactive approach to six key sectors of impact, then provides analytical ques-

tions for use in this process, before setting out a step-by-step process guideline. This

guideline includes guidance on identifying stakeholders and incorporating best practice

in impact assessment. The third section provides guidance on the content of a societal

impact assessment report. The section concludes with recommendations as how to best

embed such a methodology within the broader security research process.

The second section of the report presents the findings of our review of existing decision

support tools and resources (decision tools, learning management systems, online

guides and handbooks, and EU decision support projects). This review was conducted

to support the development of the ASSERT toolkit. It is critical to understand existing

potential solutions, including platforms that might be adapted or customised, in order

to make the appropriate tool selection and to avoid wasted effort. Our review consid-

ered the inherent usability of the tool in relation to societal impact assessment, as well

as fit with the objectives of the ASSERT project. The analysis of existing tools for deci-

sion support in this section has supported the decision to develop the toolkit on the

basis of a combination of online guide or handbook combined with a learning manage-

ment system.

The third section presents a summary of the ASSERT online societal impact assessment

for security research toolkit, including its contents and structure as well as the key de-

sign decisions underpinning the toolkit


Table of Contents

1 Introduction 6

2 A State-of-the-art methodology for Societal Im-pact Assessment for security research

7

2.1 Introduction 7

2.1.1 Societal Impacts of security research 7

2.1.2 Influences of societal impact assessment 10

2.2 Step-by-step guide for SIA in security research 13

2.2.1 Methodology 14

2.2.2 Process 17

2.2.3 The societal impact assessment report 31

2.3 Conclusions and recommendations 36

3 Review of existing decision support tools 38

3.1 Decision support tools 38

3.1.1 D-SIGHT 38

3.1.2 VISA 39

3.1.3 Web-HIPRE 40

3.1.4 1000minds 41

3.1.5 Transparent Choice 43

3.1.6 SMART 44

3.2 Learning support environments/learning management systems

46

3.2.1 Moodle 46

3.2.2 WordPress + plugins 47

3.2.3 WordPress + multiple plug ins 49

3.2.4 Big Blue Button 49

3.2.5 OpenMOOC 49

3.2.6 Blackboard 50

3.2.6 EdX 50

3.3 Handbooks and guides 51

3.3.1 IDEO HCD toolkit 51

3.3.2 Research toolkit 52

3.3.3 Crisis Guide: Iran 54

3.4 EU Decision support projects 55

3.4.1 SIAM 55

3.4.2 EURO summer school 57


3.4.3 MCDA-RES 57

3.4.4 DESSI 58

3.4.5 DREAM 59

3.4.6 FIRST 59

3.4.7 DISASTER 60

3.4.8 ESS 61

3.5 Analysis and conclusions 61

4 The ASSERT online toolkit 64

5 References 71


1. Introduction

The focus of WP3 is to operationalise the assessment of societal impacts of security

research through a structured approach, drawing upon the outcomes of WP1. Task 3.1.

and 3.2 are interlinked, resulting in a highly usable and effective assessment tool de-

ployed via the web. The Toolkit itself is an interactive, visual representation of the SIA

manual that is developed in task 3.1. This deliverable sets out the design process for the

ASSERT toolkit. This process starts with a methodology for societal impact assessment,

and then through a review of existing tools, develops a set of design requirements for

the ASSERT toolkit, which leads to the selection, development and testing of the final-

ised ASSERT Toolkit.

Section 2 features a state of the art methodology for societal impact assess-

ment for security research. This methodology is based upon the review of existing

methods of social impact assessment produced in D1.2, the best practice guidelines for

societal impact assessment in D1.3, and existing impact assessment methodologies. The

syncretised method set out in this section is intended to guide a user through planning

and conducting a social impact assessment exercise for a security research project. It

also contains guidance on the best way to record and report the process.

In support of developing the ASSERT Toolkit, section 3 documents a review of exist-

ing tools for supporting decision making and impact assessment, encom-

passing FP7 decision support projects, online and offline handbooks, decision support

tools, and learning support tools. This review provided the consortium with a grounded

understanding of the existing field, and potential tools that could 1) be used as the basis

for the ASSERT Toolkit, in terms of features and appropriate functionality, and 2) other

tools that might be included in or referenced by the toolkit. The conclusions of the tool

review provided the direction for the best way to manifest the societal impact assess-

ment methodology in the form of an online tool. This review guided the decision to

construct an online handbook to guide users through the impact assessment methodol-

ogy, whilst providing additional support and guidance material in an accessible form,

and which could also provide integrated support to the other outward-facing elements

of the ASSERT project – the Masterclass concept and the expert database.

The third stage of the design process, the creation of the ASSERT online toolkit is

documented in section 4. This section presents the logical structure and content of the

finished toolkit design choices, and structure of the ASSERT Toolkit. This section in-

cludes the key design decisions that were made on the basis of the preceding sections as

well as the requirements that emerged from the ASSERT project more broadly. It also

explains how the selected software packages (WordPress and Moodle) were adapted for

the needs of the ASSERT toolkit. The ASSERT online toolkit is a combination of an

online societal impact assessment “handbook”, with multimedia content, structured in

an accessible manner, with the additional features provided by a Leaning Management

System that offers additional tools to registered users. This combination allows the

consortium to support the Masterclass experience with online content, to make infor-

mation on societal impact assessment in security research accessible for a wider range

of users, and also to host the ASSERT expert database.


2. A State-of the-art methodology for Societal Impact As-

sessment for security research

This section sets out a structured methodology for conducting a societal impact as-

sessment (SIA) of security research and security measure implementation. The section

first provides an overview of the need for and role of societal impact assessment, then

presents an account of the existing impact assessment methodologies that have influ-

enced this guide. Section two describes the core methodology based upon an interactive

approach to six key sectors of impact, then provides analytical questions for use in this

process, before setting out a step-by-step process guideline. This guideline includes

guidance on identifying stakeholders and incorporating best practice in impact assess-

ment. The third section provides guidance on the content of a societal impact assess-

ment report. The paper concludes with recommendations as how to best embed such a

methodology within the broader security research process. The methodology has par-

ticular relevance for security research within the European Union.

2.1 Introduction

This section sets out a structured methodology for conducting a societal impact as-

sessment (SIA) of security research and security measure implementation. The paper

first provides an overview of the need for and role of societal impact assessment, then

presents an account of the existing impact assessment methodologies that have influ-

enced this guide. Section two describes the core methodology based upon an interactive

approach to six key sectors of impact, provides analytical questions for use in this pro-

cess, before setting out a step-by-step process guideline, including guidance on identi-

fying stakeholders and incorporating best practice in impact assessment. The third sec-

tion provides guidance on the content of a societal impact assessment report. The paper

concludes with recommendations as to how best embed such a methodology within the

broader security research process. The methodology has particular relevance for securi-

ty research within the European Union, however as the approach is based upon princi-

ples and methodology rather than legal compliance, it therefore has a wider potential

applicability.

2.1.1 Societal impacts of security research

Security research and innovative application of security technologies is a complex and

heterogeneous field that pulls together various academic disciplines and different types

of organisation. Several features characterise the field of security research. These fea-


tures include ethical and social issues that have been identified by different research

fields.

Security itself is socially and institutionally valued, attracting significant funding

streams and market attention. The security industry is regarded as an important eco-

nomic sector for Europe.1 Security has a particular institutionalisation within policy-

making communities.2 Security research programmes, such as the EU’s Secure Socie-

ties3 effort, therefore take place within these social, economic and political contexts.

Identifying some feature of the world as a security issue is to grant it a particular privi-

leged political status and to start to frame the issue in a specific way, deserving political

and social responses and identifying it as the responsibility of certain political actors.4

This framing of what counts as a security risk is known as securitization and has been

the subject of considerable research within international relations security studies.5

Similarly, security practices, including research and innovation, can contribute to the

normalisation of security6, with security becoming an organising principle across many

areas of social life, sometimes to the detriment of other values or principles, such as, for

example, privacy, transparency, freedom of speech or the democratic process. Security

impacts can be powerful, and frequently distributed unevenly across society (for exam-

ple, border security measures may be intended to increase national security, but can

increase insecurity for asylum seekers and immigrants). Some groups are particularly

vulnerable to the negative impacts of security research and implementation and often

excluded from decision-making processes. These groups can suffer from cumulative

disadvantage, which in turn can have negative implications for social cohesion.7 Securi-

ty processes often have the potential to negatively impact citizen’s fundamental rights.

Security (and security research) is part of the political process; however, societies often

experience periods of delay between the societal impacts of security policy and inter-

vention, social awareness of these impacts, their examination in the democratic process

1 McCarthy, Sabhbh, Report of the Societal Impact Expert Working Group: EC DG ENTR Re-port, February 2012. 2 Chilton, P., Security Metaphors: Cold War Discourse from Containment to Common House, Peter Lang, New York, 1996, p. 23. 3 European Commission, “Secure Societies – protecting freedom and security of Europe and its citizens”. http://ec.europa.eu/programmes/horizon2020/en/h2020-section/secure-societies-%E2%80%93-protecting-freedom-and-security-europe-and-its-citizens 4 Buzan, Barry, Ole Weaver and Jaap de Wilde, Security: A New Framework for Analysis, Lynne Rienner, London, 1998, pp. 25-6; Deibert, Ron, “Circuits of Power: Security in the Internet En-vironment” in J.P. Singh and James N. Rosenau (eds.), Information Technologies and Global Politics: the Changing Scope of Power and Governance, Suny Press, New York, 2002, pp. 115-142 [p. 115]; Neocleous, Mark, “Security, Liberty and the Myth of Balance: Towards a critique of security politics”, Contemporary Political Theory, Vol. 6, Issue 2, May 2007, pp. 131-149 [p. 144]. 5 Buzan, Barry, and Lene Hansen, The Evolution of International Security Studies, Cambridge University Press, Cambridge, UK, 2009, p. 142. 6 Nissenbaum, Helen, Privacy in Context: Technology, Policy, and the Integrity of Social Life, Stanford Law Books, Stanford, 2009, p. 161. 7 Lianos, Michaelis, “Dangerization and the End of Deviance: The Institutional Environment”, British Journal of Criminology, Vol. 40, No. 2, Spring 2000, pp . 261-278. http://bjc.oxfordjournals.org/content/40/2.toc


and responses to them from law and regulation. During this lag, security practices and

technologies can become entrenched in areas of social life making it difficult to dislodge

them. A related problem is function creep, where security technologies installed for one

purpose are used for secondary, initially unintended or unstated uses. Misalignment

and disjunction between agencies and actors with responsibility for promoting security

technology, and those with responsibility for assessing, controlling and in some cases

limiting and preventing negative impacts, can also exacerbate these situations.8

Security research, especially in the fields of crime prevention and national security,

often has less transparency than in areas where national interests are not seen as af-

fected in such an immediate way, and governmental bodies are less centrally involved.

There is limited public access to knowledge about the research. This often serves to

exclude stakeholders, whilst at the same time limiting the acceptability of security

measures, causing them to be viewed with suspicion.

It is important for people and institutions involved in both security research and the

innovative application of security technologies to consider the wide range of possible

societal impacts of these activities, as a response to the above political and social issues,

and as part of maximising the positive social benefits of security research whilst mini-

mising the negative effects. Societal impact assessment therefore has a critical role in

the security research and implementation process.

SIA is the process of understanding, managing and responding to the societal impacts

that arise from security research and the application of innovative security measures.

The use of the term societal (rather than social) connotes the inclusion of anything af-

fecting human, natural or artefactual systems, rather than just those effects that impact

upon humans and their interactions. It also allows us to distinguish the process from

social impact assessment, as discussed below.

From a positive perspective, societal impact assessments can also provide a better un-

derstanding of the productive and socially desirable impacts that arise from security

research, including how best to maximise these contributions. Conducting impact as-

sessment exercises as part of security research and innovation contributes towards the

evidence base for these activities. SIA can contribute towards understanding the socie-

tal impact of larger scale research funding frameworks and policies, including their

contribution toward policy objectives (for example, the partial goal of European Union

security research funding in boosting the competitiveness of the EU security industry).

Security research and innovation are, by definition, intended to produce, encourage or

support security as societal impact. An inclusive definition of security includes those

practices and technologies aimed at strengthening social bonds and social resilience

using social policy tools as well as just preventive measures against particular threats.

This understanding of security aligns with the concept of human security.9 Rather than

8 Rip, Arie, and Johan Schot, “The Past and Future of Constructive Technology Assessment”, Technological Forecasting and Social Change, Vol. 54, Issues 2–3, Feb-Mar 1997, pp. 251-268. http://www.sciencedirect.com/science/journal/00401625/54/2-3


conceptualising security as an outcome of using security technologies, security is con-

ceptualised here as one of the societal impacts of security measures. Although security

research and the implementation of security measures have internal differences, this

paper discusses them together, as they are both amenable to the assessment approach

set out here.

2.1.2 Influences on societal impact assessment

Societal impact assessment for security research has not emerged in a vacuum. SIA

approaches draw upon a range of previously established methodologies and practices,

both in security research and related areas. These influences include constructive tech-

nology assessment (CTA), privacy impact assessment (PIA) and recent developments in

surveillance impact assessment (SuIA), social impact assessment and the impact as-

sessment activities of the European Commission. There is some limited influence from

the responsible research and innovation debate and from research ethics, although this

is primarily concerned with the way that research is conducted, rather than concerns

about the goals and subjects of the research.10

Constructive technology assessment is defined by Rip and Schot as an approach that

"shifts the focus away from assessing impacts of new technologies to broadening de-

sign, development, and implementation processes",11 the aim of which is to contribute

“better technology to a better society”. By anticipating impacts, involving users and

stakeholder communities in the design process in an interactive manner and by har-

nessing social learning, it is possible to avoid the human costs associated with trial and

error social responses to new technologies. Social aspects of technologies are explicitly

included as design criteria.12 CTA claims no inherent normative agenda, rather focusing

upon expanding the reflexivity of the design process, however it serves to close conflicts

and increase acceptance. Tools and procedures of CTA include understanding the in-

novation journey, identifying points of intervention at different phases of a project,

anticipating the outcome of a research process, reflexivity exercises, stakeholder work-

shops, technology forcing13 and strategic niche alignment.14

Privacy impact assessment is a methodology for assessing the impacts on privacy of a

project, policy, programme, service, product or other initiative that involves the pro-

cessing of personal information and in consultation with stakeholders for taking reme-

10 McCarthy, 2012, p. 10 11 Rip and Schot, 1997. p. 251 12 Ibid, p. 251 13 Genus, Audley, “Rethinking constructive technology assessment as democratic, reflective dis-course”, Technological Forecasting and Social Change, No. 73, No. 1, 2006, pp. 13-26 [p. 19]. http://www.sciencedirect.com/science/journal/00401625/73/1 14 Schot, Johan and Frank W. Geels, “Strategic niche management and sustainable innovation journeys: Theory, findings, research agenda and policy”, Technology Analysis & Strategic Man-agement, Vol. 20, No. 5, 2008. pp. 537-554.


dial action as necessary in order to avoid or minimise negative impacts.15 Surveillance

Impact Assessment is an expansion of privacy impact assessment that takes into ac-

count a wider range of issues, impacts and stakeholders across the surveillance sys-

tem.16 The aims of PIA and SuIA are better privacy protection, increased transparency

of personal information processing technologies and increased accountability (and for

SuIA mitigation of the risks of surveillance systems). It is also intended to engage

stakeholders and affect the direction of a project from its earliest stages. PIA has been

adopted across several countries. 17

Potentially the broadest of the influential approaches, social impact assessment as un-

derstood by Vanclay is a participatory process for the analysis, monitoring and man-

agement of the intended and unintended, positive and negative consequences of

planned interventions.18 The goal is a more sustainable and equitable physical and hu-

man environment, and key methods include increasing awareness of potential unin-

tended consequences, through the exploration of the likely impacts of a research pro-

ject on people’s lives, cultures, communities, political systems, environment, health and

wellbeing, personal and property rights, fears and aspirations.

As part of the legislative process, the European Commission undertakes impact as-

sessments of policies, legislation, trade agreements and other measures. This includes

publishing roadmaps and economic, social and environmental impact assessments of

planned initiatives prior to EU action. Post-intervention, this is followed by evaluation

of the performance of initiatives and by subjecting them to regulatory fitness and per-

formance (REFIT) assessment. The Commission seeks public consultation throughout

the process.1920

The three theoretical approaches share reflexivity as core principles, and all four ap-

proaches have a strong focus on participation. Similarly, the approaches all share the

perspective that impact assessments should be initiated at the earliest possible stages of

a project or intervention, and conducted as an ongoing activity. Social impact assess-

ment contributes the need for an awareness of societal dimensions and how they im-

pact the security research and development process. It highlights stakeholder dissatis-

faction with the reports of existing social impact processes, especially post-impact stud-

15 Wright, David and Paul de Hert, “Introduction to Privacy Impact Assessment”, in David Wright and Paul de Hert (eds.), Privacy Impact Assessment, Springer, Dordrecht, 2012, pp. 3-33 (p. 5). 16 Wright, David and Charles Raab, “Constructing a surveillance impact assessment”, Computer Law & Security Review, Vol. 28, No. 6, Dec 2012, pp. 613-626. 17 Wright, David, Raphaël Gellert, Rocco Bellanova, Serge Gutwirth, Marc Langhenrich, Michael Freidewald, Dara Hallinan, Silvia Venier and Emilio Mordini, Privacy Impact Assessment and Smart Surveillance: A State of the Art Report, Deliverable 3.1, SAPIENT Project, May 2013. 18 Vanclay, Frank, “Social Impact Assessment: International Principles”, IAIA, Special publica-tion series No.2, May 2003, p. 2. http://www.iaia.org/publicdocuments/special-publications/sp2.pdf 19 European Commission, “Smart Regulation”, 13 January 2014. http://ec.europa.eu/smart-regulation/index_en.htm 20 For an early review of this process, see: Lee, Norman and Colin Kirkpatrick, “Evidence-based policy making in Europe: an evaluation of European Commission integrated impact assess-ments”, Impact Assessment and Project Appraisal, Vol. 24, No. 1, 2006, pp. 23-33.


ies, and the importance of a social impact assessment as a tool and investment in risk

management.21 CTA recommends researchers exercise critical reflexivity and empha-

sises the prominent role of societal issues in research consortia.22 The European Com-

mission’s impact assessment process – which currently is applicable only to policies

developed by EU institutions - demonstrates the value of publishing impact assess-

ments and building a common methodology within a sector. Surveillance impact as-

sessments already include privacy and ethical impact assessments and are therefore a

suitable model for rigorous impact assessment. The following table shows the influ-

ences from existing approaches that have contributed to the Societal Impact Assess-

ment approach set out in this paper.

Table 1: combination of elements across impact assessment (source: authors)

Elements of Societal Im-pact Assess-ment

Influences from existing approaches

Social Impact Assessment

Privacy Impact assessment

Constructive technology Assessment

European Impact assessment

Including all human, natu-ral and arte-factual sys-tems

Human interac-tions/social di-mensions

Inclusion of social science in technology assessment

Critical reflex-ivity of as-sessment

Reflexivity to-wards societal dimensions

Critical reflex-ivity

Stakeholder involvement

Stakeholder in-volvement


Stakeholder involvement


Publishing Publishing

Common methodology

Common methodol-ogy

Societal impact assessment as a broad category of practice, therefore includes the other

approaches, as shown in the following diagram. The existing approaches are all forms

of societal impact. In this paper we draw upon these existing best practices to we pre-

sent a composite approach that is applicable to security research.

21 Vanclay, Frank, and Ana M. Esteves (eds.), New Directions in Social Impact Assessment: Conceptual and Methodological Assumptions, Edward Elgar, Cheltenham, 2011 , p. 11 22 Rip, Arie, and Harro van Lente,“Bridging the gap between innovation and ELSA: The TA pro-gram in the Dutch Nano-R&D Program Nano Ned”,Nanoethics, Vol. 7, Issue 1, April 2013, pp. 7-16. http://link.springer.com/journal/11569/7/1/page/1


Figure 1: Societal Impact Assessment methodologies (source authors)

2.2. Step-by-step guide for SIA in security research

The objective of the SIA process is to increase the reflexivity of the process and of deci-

sion-making, not to pre-script or pre-define the results of an assessment process. Re-

flexivity in this context is the ability of researchers to take stock of their role in the re-

search process, and subject their research activity to the same level of critical scrutiny

as the rest of their “data”.23 Reflexivity is a process of critical reflection both on the

kind of knowledge generated from research and on how that knowledge is generated.24

Reflexivity and an openness to alternatives is crucial to act upon the “irritation” that

SIA is capable of creating by ensuring that those results that challenge core assump-

tions of the planned project, technology or policy can also have an impact on the plan-

ning process. One cannot assume a simplified binary process in which security technol-

ogies are invented, and then go on to have implications and impacts.25 Innovation stud-

ies have established the observation that innovation frequently does not occur in a line-

ar process but in a complex and uncertain process of trial, error and unintended devel-

opments.26 As with PIA,27 one-size does not fit all with regard to SIA, and this method-

ology should be adaptable to the specific needs and contexts of a given research project

or innovation. Using a shared framework and accepted common approach increases the

transferability across domains of the assessment, and is likely to increase external con-

fidence in the process. The offered approach is rigorous and detailed. This is because a

23 Mason, Jennifer, Qualitative Researching, Sage, London, 1996. 24 Guillemin, Marilys, and Lynn Gillam, “Ethics, Reflexivity and “ethically Important” moments in research”, Qualitative Inquiry, Vol. 10, No. 2, 2004, pp. 261-280. 25 Rip, Arie, and Johan Schot, “Identifying Loci for Influencing the Dynamics of Technology Development”, in Knut H. Sorensen and Robin Williams (eds.), Shaping Technology, Guiding Policy: Concepts, Spaces and Tools, Edward Elgar, Cheltenham, 2002, pp. 155-172. 26 Braun-Thürmann, Holger, Innovation, Transcript, Bielefield, 2005. 27 Wright et al., 2013.


security research project is often operating in relatively unknown or conjectured ter-

rain. The additional information produced for such a project by a detailed impact as-

sessment supports better understanding of the security intervention being researched.

The following section first presents the core elements of the assessment methodology,

before situating this methodology as part of a structured societal impact assessment

process.

2.2. 1 Methodology.

The core of the methodology proposed is based upon analysis driven by interaction

with the stakeholders and posing a series of questions that enable the discovery of vary-

ing views on impacts. This enquiry is divided into six different aspects of societal im-

pacts as shown in the figure below, and the enquiry is completed in an approach com-

monly used in curriculum design28 (moving from basic questions to increasingly more

complex ones as more is learned, cycling or spiralling back through the same topics

multiple times).

Figure 2: Methodology (source: authors)

28 “Spiral Curriculum” was originally suggested by Jerome Bruner. See Bruner, Jerome, The Process of Education. The President and Fellows of Harvard College, Cambridge, MA, 1960.


The various aspects of societal impact that are evaluated include those suggested by

Vanclay (attributed in part to ideas developed by Armour)29, which have been com-

bined here into six main groups (Vanclay separates culture from community as well as

way of life from fears and aspirations):

• Way of life, fears and aspirations (how people live and interact

with each other on a daily basis, their perceptions about their safety and

that of their communities, and their aspirations for future, including the

future of their children);

• Culture and community (people’s shared beliefs, customs, values

and languages, the cohesion, stability and character of their communi-

ties);

• Political systems (participation in the decisions and processes that af-

fect people’s lives, the nature and functioning of democratic processes,

and the resources available to support people’s involvement in these);

• Environment (access to and quality of air, water, and other natural re-

sources, the level of exposure to pollutants and harmful substances, ad-

equacy of sanitation);

• Health & well-being (physical and mental wellbeing, not just an ab-

sence of infirmity);

• Personal and property rights (economic effects, civil rights and lib-

erties, personal disadvantage).30

The assessor should start the SIA evaluation process by looking at any one of the above

aspects in three different dimensions:

1) First examine whether the security research project meets the needs of soci-

ety;

2) Iterating a second time through the same six aspects, review the potential

externalities or costs to society, enumerating risks and identifying ways to

mitigate them;

3) Finally, pass through the six societal impact aspects a third time to identify

potential benefits to society.

With each re-evaluation of the six aspects, the assessor and stakeholders have the op-

portunity to rethink answers to previous questions based upon answers across each of

the earlier assessments, providing an opportunity to go back and rethink impacts where

new information has been brought to light or where new ideas have emerged.

The following set of questions provides a basis for examining the social needs, potential

costs and risks, and potential benefits of security research. The questions are based

upon the societal impact checklist for R&D developed by the Societal Impact Expert

29 Vanclay, Frank, “Conceptual and methodological advances in social impact assessment” in Henk A. Becker and Frank Vanclay (eds.), The International Handbook of Social Impact As-sessment: Conceptual and Methodological Advances, Edward Elgar, Cheltenham, 2003, p. 7. 30 Vanclay, F., “International Principles for Social Impact Assessment”, Impact Assessment and Project Appraisal, Vol. 21, No. 1, 2002, pp. 5-11.


Working Group in their report to the European Commission’s Directorate-General for

Enterprise and Industry.31

Table 2: Assessment questions (source: authors)

Me

ets

ne

ed

s o

f so

cie

ty?

1. Which documented societal security need(s) does the proposed research

address? (e.g., life, liberty, health, employment, property, environment,

values).

2. How will the research output meet these needs? How will this be demon-

strated? How will the level of societal acceptance be assessed?

3. Is the research project aware of challenges to these needs?

4. Does addressing the documented societal needs through the proposed

research require any trade-offs with other documented societal needs?

How is this trade-off decided? Is this trade-off still valid if the research is

less effective than anticipated?

5. What threats to society does the research address? (e.g., crime, terror-

ism, pandemic, natural and man-made disasters).

6. How is the proposed research appropriate to address these threats?

7. What other measures could be adopted to address these threats?

31 McCarthy 2012, pp. 17-18


En

su

rin

g s

ec

ur

ity

re

se

ar

ch

do

es

no

t h

av

e n

eg

ati

ve

im

pa

cts

on

so

cie

ty

8. How could the research have a negative impact on human dignity?

9. … on the right to life?

10. … on equality before the law?

11. … on freedom of thought?

12. … on freedom of opinion and information?

13. … on privacy?

14. … on protection of the family?

15. … on freedom of movement?

16. … on rights of ownership?

17. … on freedom of assembly?

18. … on freedom to choose an occupation?

19. … on working conditions?

20. … on collective social rights?

21. … on social welfare?

22. … on rights to an education?

23. … on the principle of democracy?

24. … on rights of access to information?

25. … on rights of access to the courts?

26. … on access to public space?

27. If implemented, how could the research have a negative impact on this

aspect (culture and community, way of life, etc.)?

28. How could the research impact disproportionately upon specific groups

or unduly discriminate against them? How could the research increase

discrimination?

Could the research have impacts upon vulnerable groups (including, but

not limited to: women, the elderly, disabled people, children and young

adults, homeless people, economically disadvantaged people and people

in precarious situations, immigrants or non-citizens, and lesbian, gay,

bisexual, transgender or queer (LGBTQ+) identifying people.

En

su

rin

g s

ec

ur

ity

re

se

ar

ch

be

ne

fits

so

cie

ty

29. What segment(s) of society will benefit from increased security as a re-

sult of the proposed research?

30. How will they benefit?

31. Are additional measures required to achieve this benefit?

32. Are additional measures possible to extend these benefits to other seg-

ments of society?

33. In what contexts might this benefit be lacking or not be delivered by the

research project?

34. How will society as a whole benefit from the proposed research?

35. Are there other European societal values that are enhanced by the pro-

posed research, e.g., public accountability and transparency; strength-

ened community engagement, human dignity; good governance; social

and territorial cohesion; sustainable development.

2.2.1 Process

To effectively employ this evaluation methodology, it should be put in the context of the

following 15-step process:


Table 3: Steps in the SIA process (source: authors)

Preparation

and plan-

ning

1. Identify the SIA team and set the team’s terms of reference, re-

sources and time frame.

2. Prepare the SIA plan.

3. Determine the budget for the SIA.

4. Describe the project to be assessed.

5. Identify stakeholders.

Consultation

and analysis

6. Conduct the spiral assessment of the six core areas of societal im-

pact to identify impacts associated with needs, externalities/costs, and

benefits.

7. Consult with stakeholders.

8. Determine whether the project complies with legislation; assess

whether the security research has the potential to generate results that

require new legislation to address potential gaps.

9. Identify risks and possible solutions.

10. Formulate recommendations.

Reporting

and re-

sponding

11. Prepare and publish the report, e.g., on the organisation’s website

and/or in a suitable repository.

12. Implement the recommendations.

13. Ensure a third-party review and/or audit of the SIA.

14. Update the SIA if there are changes in the project.

15. Refer to the SIA in any post-project evaluation.

This process is based upon PIA methods, and shares common features with the EU im-

pact assessment procedure.32 As the project progresses, there is a shift from foresight,

to management, to evaluation. Guidance for each individual step in the process is pro-

vided below.

32 European Commission, “Key procedural steps for the Commission/smart-regulation/impact”, 20 Dec 2013. http://ec.europa.eu/smart-regulation/impact/ia_key/ia_key_en.htm


Preparation and planning

This approach acknowledges that for many types of security research, there is a need to

conduct societal impact assessments during the research planning or proposal stage. At

this stage, there may be minimal resources available to support extensive societal im-

pact efforts, including in-depth consultation. Structured consideration of societal im-

pact at this stage can improve the quality of the research design being proposed or con-

sidered, and is crucial to ensure that negative societal impacts are not locked into the

research design.

Identify the SIA team and set the team’s terms of reference, resources and time frame

The research project manager should be responsible for the conduct of a SIA, but she

may need some additional expertise, perhaps from outside her organisation. Depending

on the estimated scale of the SIA, the project manager or the designated societal impact

assessor may need to form a team to undertake the SIA. The team could bring together

expertise from information security experts, lawyers, operations managers, ethicists,

public relations experts, etc. As the SIA progresses, the assessor may find that she

needs still other expertise. The benefits of a dedicated manager and interdisciplinary

team, which ideally includes some social science expertise, is supported by existing

methodologies of social impact assessment.33 Eurobarometer research suggests that the

European public is most receptive to accounts of research that come from researchers

themselves.34

The project manager and/or the organisation’s senior management should decide on

the terms of reference for the SIA team, its budget and its time frame. The assessor may

come under considerable pressure to complete the SIA quickly so as not to delay the

project, but she may need to resist compromising the integrity and adequacy of her SIA

mission and may need to ensure she has the full support of the organisation’s CEO

and/or its management board. The terms of reference should make clear that the socie-

tal impact assessment is a process, and that the process will need to continue beyond

preparation of the SIA report. If the assessor’s work or that of an external consultant

comes to an end with publication of the report, the project manager and/or the organi-

sation’s CEO and/or management board should decide how implementation of recom-

mendations will be monitored and who will be responsible for the monitoring and what

factors will determine whether the SIA report needs to be updated.

For research projects, it may be appropriate to dedicate a work package or stream of

activity to the societal impact role. If this is done, it is important to ensure that this

work package is integrated with the other elements of the project, perhaps through

33 Kemp, Deanna, “Understanding the Organizational Context”, in Frank Vanclay and Ana Maria Esteves, (eds.), New Directions in Social Impact Assessment: Conceptual and Methodological Advances, Edward Elgar, Cheltenham, 2011, p. 30. 34 TNS Opinion & Social, Special Eurobarometer 401: Responsible Research and Innovation (RRI) Science and Technology, European Commission, Directorate General for Communication, November 2013, p. 5.


shared personnel or making later elements of the work dependent upon stages of the

SIA report.35

Prepare the SIA plan

The assessor should prepare a plan for conducting the SIA. She can prepare the SIA

plan using this SIA process document, but may need to tailor it to the exigencies of the

project to be assessed. The plan should spell out the objectives of the SIA, what is to be

done to complete the SIA, who on the SIA team will do what, the SIA schedule and,

especially, how the consultation will be carried out. An important part of the plan

should address consultation. It should specify why it is important to consult across each

of the six societal impact areas, who will be consulted and how they will be consulted

(e.g., via public opinion survey, workshops, focus groups, public hearings, online expe-

rience, specialist consultation tools).

The SIA should also include if and how societal impact will be included in any post-

research evaluation activity.

Determine the budget for the SIA

Once the project manager and/or assessor have prepared an SIA plan, they can esti-

mate the costs of undertaking the SIA and seek the budgetary and human resources

necessary from the organisation’s senior management. Unfortunately, the assessor may

be constrained in what she can do in the SIA by the budget allocated by the organisa-

tion. If the assessor is unable to do an adequate SIA, she should note this in her SIA

report. The assessor may need to revise her SIA plan based on the budget available.

In general, the budget for performing an effective SIA will depend upon a number of

factors, including 1) the stage at which an SIA is being performed, and 2) the scale,

scope, and overall complexity of the project. For example, if an SIA is being performed

at the inception of the project, while there remains adequate opportunity to effect

change, it may be more cost-effective than if the SIA is performed after a great deal of

research, design, and development has been completed.

As a matter of practical estimation, it can be expected that the SIA will require budget

for the salary of a senior level consultant and a junior level consultant for approximate-

ly one person-month, presuming that the consultants are experienced in performing

SIAs and already come to the task armed with procedures and templates to perform it.

If so equipped, they will use some of their time to understand the project and subse-

quently survey or interview key stakeholders before completing their analysis. If this is

the first time an organisation is doing an SIA, they will need to take an extra two or

three months to develop their process before beginning the work of the SIA itself. Ex-

perienced consulting companies will have already completed this work, and of course,

35 In the course of the SIA, it may become apparent to the assessor that the organisation needs to spend more time on raising the awareness of employees (including researchers) about societal impact issues. The background context section of the report can be used to state what the organ-isation does now to raise employee awareness of societal impacts, and where it could improve.


there will likely be additional costs of addressing the outcomes of the SIA (potentially

greater if the SIA is performed later in the research and development cycle).

Describe the project to be assessed

The assessor should describe the project or technology or service to be assessed. As the

development of the project or technology or service may still be at an early stage, there

may not yet be that much known about the project. The assessor can update the de-

scription as more becomes known. The description can be used in at least two ways – it

can be included in the SIA report and it can be used as a briefing paper for consulting

stakeholders. The description of the project should provide some contextual infor-

mation (why is the project being undertaken, how is it funded, who are the project

members and participants, who are the intended audiences for the findings of the re-

search, how does it relate to other ongoing security research activity conducted by the

project members). The project description should state who is responsible for the pro-

ject. It should indicate important milestones and, especially, when decisions are to be

taken that could affect the project’s design.

Identify stakeholders

A critical component of societal impact assessment is the participative inclusion of

stakeholders in the assessment of the security research project or security measure ap-

plication. The assessor should identify stakeholders, i.e., those who are or might be in-

terested in or affected by the project, technology, service or other initiative. The stake-

holders could include people who are internal as well as external to the organisation.

Involving a variety of stakeholders provides an opportunity for any potential risks to be

highlighted and eventually managed. Given the potential breadth of societal impact

across different categories of impact, the way that “stakeholders” is understood should

be broad and inclusive.

Kemp provides a list of parties who could potentially be affected by a planned project or

policy and should thus be engaged within the context of SIA.36 Building upon this list,

and customising it for the security research process provides the following summary:

• personnel or managers with carriage of the social agenda within project propo-

nent organisations;

• researchers, designers, engineers, developers, potential suppliers, security ex-

perts and others who will carry out the research activity;

• assessors who are commissioned to undertake or facilitate the societal impact

assessment process, either internally or from outside of the organizational

structure of the project proponent;

• project-affected peoples, up to and including representatives of the general pub-

lic

• regulators;

• civil society organisations, including civil rights advocates;

36 Kemp, 2011, pp. 21-2


• the media;

• academics;

• businesses.

The assessor should identify these different categories and then identify specific indi-

viduals from within each category, preferably as representative as possible. The stake-

holders of a project, and particular people who might be affected by the project, will be

dependent upon the context of the project and the way in which it is conducted. This

means that the above list cannot be inclusive and the project assessor must make ef-

forts to identify other stakeholders as appropriate. Social relations are complex, and

stakeholders, especially those affected by unintended consequences, may not be appar-

ent to the SIA team. The SIA process should therefore include opportunities for indi-

viduals, groups and organisations to self-identify as stakeholders and request participa-

tion in the assessment activity. Some stakeholders may only become apparent as the

SIA progresses. If necessary or useful, they too should be brought into the consultation

process. The range and number of stakeholders to be consulted should be a function of

the likely societal impact as identified in early stages of the spiral methodology, includ-

ing the number of people who could be affected. Thus, the number of stakeholders to be

consulted could be relatively limited if the project or service is also expected to be

small, e.g., the project or service might involve only employees of a small or medium-

size enterprise. The proper involvement of stakeholders may require additions to the

SIA team, either to encourage participation by stakeholders, or to bring key stakehold-

ers, who may be most affected by the project into the SIA team directly.

Consultation and analysis

The four stages in this section involve the analysis of societal impacts using an iterative

process based upon the methodology, and in concert with identified stakeholders, that

looks at each of six aspects of societal impacts as described above.

Consult with stakeholders

The project manager and/or societal impact assessor should enter a dialogue with as

many stakeholders as appropriate or meaningfully possible (taking into account the

available budget). There are many reasons for doing so, not least of which is that they

may identify some societal risks not considered by the project manager or assessor. By

consulting stakeholders, the project manager may forestall or avoid criticism that they

were not consulted. If something does go wrong downstream – when the project or

technology or service is deployed – an adequate consultation at an early stage may help

the organisation avoid or minimise liability. Furthermore, consulting stakeholders may

provide a sort of “beta test” of the project or service or technology. Consulted stake-

holders are less likely to criticise a project than those who were not consulted. The ear-

lier a consultation process is entered into, the more benefits an organisation can expect

to draw from it as any learning produced can be integrated into the project more rapid-

ly, and additional information may help to avoid unanticipated problems in the project.

There are several different ways of consulting stakeholders and the assessor should

consider which will be most appropriate in the circumstances. The assessor or other

members of the SIA team could interview stakeholders directly. They could convene


workshops of experts or stakeholder representatives. They could hold focus groups of

ordinary consumer-citizens. They could conduct surveys by telephone or e-mail or face

to face. They could post the project description on the organisation’s website and invite

comments. They could hold public hearings where they describe the project and invite

comments from the audience or from experts and then invite comments after the ex-

perts have spoken. They could prepare stories or adverts in the media and invite com-

ments from readers. They could conduct a Delphi survey of experts, to query them on

potential societal risks now and in the future.37 There are current EU-funded research

projects which are working to produce structured methods for consultation in security

research.38

Social impact assessment has attracted a range of cautionary comments in relation to

community participation. That participation can become a tokenistic exercise, can be

inconsistent, often does not involve enough participation in actual decision-making,

being reduced to a form of consultation.39 Moreover, community participation is some-

times used as a seeming quick fix for problems, without addressing the root of the issue

(which often lies in unequal power distributions that are deeply engrained in polities,

and sometimes in the very institutions of the community whose participation is

sought). As a result, community participation sometimes takes the form of a tokenistic

exercise; once “affected communities” – or their leaders – have been heard, the respec-

tive box can be ticked off, and afterwards the rule of previous power relations re-

sumes.40 To avoid this, the participation of stakeholders in an SIA exercise should be

dialogic and a partnership, and go beyond simply writing down what stakeholders have

to say. Stakeholders should have access to the researchers and ideally be able to exer-

cise some influence over the direction of the project.

Compliance with legislation

A societal impact assessment for security research is more than a compliance check; nevertheless, the assessor or her legal experts should ensure that the project complies with any legislative or regulatory requirements. These may be high level laws relevant across contexts, such as (at the European level) the European Convention on human rights41 and the EU Charter of fundamental rights42 as well as more specific laws, regu-

37 For a longer list of possible techniques, see OECD, Stakeholder Involvement Techniques, ISBN 92-64-02087-X, Paris, 2004, pp. 30-32 [Box 2. Commonly cited techniques for informing deliberation through stakeholder involvement] 38 see http://securitydecisions.org/about-dessi/, and http://www.siam-project.eu/ 39 Peterman, William, “Advocacy vs. collaboration: comparing inclusionary community planning models”, Community Development Journal, Vol. 39, No. 3, 2004, pp. 266-276; O’Faircheallaigh, Ciaran, “Public participation and environmental impact assessment: Purposes, implications and lessons for public policy making”, Environmental Impact Assessment Review, Vol. 30, Issue 1, January 2010, pp. 19-27 [p. 19]; Müth, Matthias, Verkherspolitik in Metropolen Südostasiens (the politics of traffic in South East Asian Metropolis areas), Abera, Hamburg, 2000; Bishop, Patrick, and Glyn Davis, “Mapping public participation in policy choices”, Aus-tralian Journal of Public Administration, Vol. 61, No. 1, 2002, pp. 14-29. 40 Peterman, 2004. 41 http://www.echr.coe.int/Documents/Convention_ENG.pdf


lations, codes and guidelines applicable to the specific context and aims of the project being assessed. Research projects attracting institutional support will also have to com-ply with relevant standards and criteria selected by funding or sponsoring institutions. Individual institutions will likely have their own guidance on this, which should be identified here. The exercise of producing the SIA report will likely assist in compliance with these requirements.

The following table presents some legislation at the EU level that may be applicable to security research projects.

Table 4: potentially applicable legislation at the EU level (source: authors)

Way of life, fears

and aspirations

Charter of Fundamental Rights of the European Union; Europe-

an Convention on human rights; Council Directive 2000/78/EC

of 27 November 2000 establishing a general framework for

equal treatment in employment and occupation; Directive

2004/38/EC on the right to move and freely reside; Gender

recast Directive 2006/54/EC; Employment equality Directive

2000/78/EC/

Culture and com-

munity

Charter of Fundamental Rights of the European Union; Council

of the European Union, Directive 2000/43/EC of 29 June 2000

implementing the principle of equal treatment between persons

irrespective of the racial or ethnic origin; Racial equality Di-

rective 2000/43/EC

Political systems Charter of Fundamental Rights of the European Union; Europe-

an Convention on human rights.

Environment Directive 2008/1/EC of the European Parliament and the Coun-

cil of 15 January 2008 concerning integrated pollution preven-

tion and control: Directive 2011/92/EU of the European Par-

liament and the Council of 13 December 2011 on the assessment

of the effects of certain public and private projects on the envi-

ronment.

Health and well-

being

National Legislation for health; Directive 2011/24/EU of the

European Parliament and of the Council of 9 March 2011 on the

application of patients’ rights in cross-border healthcare; Coun-

cil Directive of 12 June 1989 on the introduction of measures to

encourage improvements in the safety and health of workers at

work (89/391/EEC).

42 European Parliament, Council and Commission, Charter of Fundamental Rights of the Euro-pean Union, 2010/C 83/02, OJ, Brussels, 30.3.2010. http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2010:083:0389:0403:en:PDF


Personal and prop-

erty rights

Charter of Fundamental Rights of the European Union; Di-

rective 95/46/EC of the European Parliament and the Council

of 24 October 1995 on the protection of individuals with regard

to the processing of personal data and on the free movement of

such data; Directive 2002/58/EC of the European Parliament

and the Council of 12 July 2002 concerning the processing of

personal data and the protection of privacy in the electronic

communications sector; Employment Equality Directive

2000/78/EC; Gender goods and services Directive

2004/38/EC; Framework Decision 2008/977/JHA of

27 November 2008 on the protection of personal data processed

in the framework of police and judicial cooperation in criminal

matters.

Identify societal impacts and possible solutions

The assessor and her SIA team, preferably through stakeholder consultation, should identify all possible negative societal impacts, who these will impact and their likeli-hood (frequency) and consequence (magnitude of impact) as well as the numbers of people who could be affected. Often, the best way to identify these impacts is to consid-er principles associated with each type of societal impact and/or a set of questions which can help identify negative societal impacts, as provided in section 2.1. The asses-sor will benefit from engaging stakeholder representatives and experts to have their views. The assessor, other members of the SIA team and stakeholders consulted should raise other questions that can help to identify the societal impacts of the proposed pro-ject.

The following tables demonstrate how an assessor should use the spiral methodology

and approach the six relevant sectors in three phases.


Table 5: Assessment phases (source:authors)

Assessment Round 1:

Ensuring security re-

search meets the needs

of society

Wa

y o

f li

fe,

fea

rs a

nd

asp

ira

tio

ns

Cu

ltu

re a

nd

co

mm

un

ity

Po

liti

cal

syst

em

s

E

nv

iro

nm

en

t

He

alt

h a

nd

we

ll-b

ein

g

Pe

rso

na

l a

nd

pro

per

ty r

igh

ts

Which documented societal secu-

rity need(s) does the proposed

research address? (e.g. life, liberty,

health, employment, property,

environment, values).

How will the research output meet

these needs? How will this be

demonstrated? How will the level

of societal acceptance be assessed?

Is the research project aware of

challenges to these needs?

Does addressing the documented

societal needs through the pro-

posed research require any trade-

offs with other documented socie-

tal needs? How is this trade-off

decided? Is this trade-off still valid

if the research is less effective than

anticipated?

What threats to society (e.g.,

crime, terrorism, pandemic, natu-

ral and man-made disasters) does

the research address?

How is the proposed research ap-

propriate to address these threats?

What other measures could be

adopted to address these threats?


Assessment Round 2:


search does not have

negative impacts on

society

Wa

y o

f li

fe,

fea

rs a

nd

asp

ira

tio

ns

Cu

ltu

re a

nd

co

mm

un

ity

Po

liti

cal

syst

em

s

E

nv

iro

nm

en

t

He

alt

h a

nd

we

ll-b

ein

g

Pe

rso

na

l a

nd

pro

per

ty r

igh

ts

How could the research have a

negative impact on freedom of

association?


negative impact on freedom of

expression?


negative impact on protection of

personal dignity?


negative impact on privacy and

data protection?


negative impact on property

rights?


negative impact on access to pub-

lic space?

If implemented, how could the

research have a negative impact

on this aspect (culture and com-

munity, way of life, etc.)?

How could the research impact

disproportionately upon specific

groups or unduly discriminate

against them?

Could the research have impacts

upon vulnerable groups (includ-

ing, but not limited to: the elderly,

the disabled, children and young

adults, homeless people, economi-

cally disadvantaged people and


people in precarious situations,

immigrants or non-citizens, and

lesbian, gay, bisexual, transgender

or queer (LGBTQ+) identifying

people).

How could the research increase

discrimination?

Assessment Round 3:


search benefits society

Wa

y o

f li

fe,

fea

rs a

nd

asp

ira

tio

ns

Cu

ltu

re a

nd

co

mm

un

ity

Po

liti

cal

syst

em

s

E

nv

iro

nm

en

t

He

alt

h a

nd

we

ll-b

ein

g

Pe

rso

na

l a

nd

pro

per

ty r

igh

ts

What segment(s) of society will

benefit from increased security as

a result of the proposed research?

How will they benefit?

Are additional measures required

to achieve this benefit?

Are additional measures possible

to extend these benefits to other

segments of society?

In what contexts might this bene-

fit be lacking or not be delivered

by the research project?

How will society as a whole benefit

from the proposed research?

Are there other European societal

values that are enhanced by the

proposed research, e.g., public

accountability and transparency;

strengthened community engage-

ment, human dignity; good gov-

ernance; social and territorial co-

hesion; sustainable development.

It should be noted that “impacts” are not solely negative, and part of this approach in-

cludes the assessment of potential benefits from security research (including, for exam-

ple, impacts on welfare, growth and competitiveness). This is particularly the focus of


assessment round 1: ensuring security research meets the needs of society. This is not

just a threshold check, but an opportunity for better understanding the pathways to

desirable beneficial outcomes.

Deciding how to mitigate or eliminate or avoid or transfer negative societal impacts is

also a somewhat political decision as is the decision regarding which benefits to pursue.

The assessor or project manager or organisation may decide that the benefits of the

project or technology outweigh the perceived negative impacts arising from its devel-

opment and deployment. Societal impact assessment should be regarded as part of the

organisation’s risk management, although this should be balanced against the need to

actually learn through the process. In order to facilitate socially robust innovation, it

could be argued, SIA needs to provide room for genuine learning on the side of all ac-

tors involved, and the possibility that core policies and plans will be aborted or refor-

mulated needs to remain a possibility.

The organisation should maintain an impact register, wherein the assessor (and/or

other organisation employees) identifies the impacts, their seriousness, what the organ-

isation has decided (if anything) to do about them, who is the person who is responsible

for managing it. The impact register should be regularly updated (e.g., every six months

or at appropriate milestones in the project), depending upon the length of the research

project. It is important to include all identified impacts in this register even if they are

accepted at later stages in the process.

Formulate recommendations

Based on her analysis of the societal impacts, the assessor should prepare a set of rec-

ommendations, which will form part of the SIA report. The assessor should be clear to

whom her recommendations are directed – some could be directed towards different

units within the organisation, some to the project manager, some to the CEO, some to

employees (including researchers) or employee representatives (e.g., trade unions), to

regulatory authorities, etc. The assessor should provide the rationale for each of her

recommendations. The recommendations could include procedural and more general

organisational matters, e.g., relating to training and raising awareness and accountabil-

ity, as well as those relating specifically to societal impact.

Potential venues and options for the publication of research findings (for example,

open-access publication) can be identified at this stage as they will contribute to the

positive societal impacts of the security research.

Reporting and responding

Prepare and publish the report

The assessor should prepare her SIA report, and the organisation should publish it on

its website and/or submit it to an appropriate repository.

An outline and recommended contents of a SIA report are provided in section 3 of this

paper.

Research funding bodies may have specific reporting requirements for societal impact

assessment exercises.


Some organisations may be reluctant to publish their SIAs because they fear negative

publicity or they have concerns about competitors learning something they don’t want

them to. Such concerns seem overdone. Publication offers many benefits and opportu-

nities to the organisation. It demonstrates that the organisation treats societal issues

seriously, and consequently its customers or citizens. Customers and citizens are more

likely to invest their trust in an organisation that treats their wellbeing, environment,

individual rights and other concerns with respect. It offers an opportunity to gather

additional feedback from stakeholders. It offers the organisation an opportunity to dis-

tinguish itself from its competitors. For organisations concerned about publishing

commercially sensitive information or security sensitive information, there are solu-

tions. The organisation can simply redact the sensitive bits or put them into a confiden-

tial annex or just publish a summary of the project or, if necessary, provide a copy to

the regulator.

Implement the recommendations

The project manager and/or the organisation does not need to accept all these recom-

mendations, but they should say which recommendations they have implemented al-

ready or intend to implement and which they do not intend to implement and the rea-

sons why they do not intend to do so. The organisation’s response to the assessor’s rec-

ommendations should be posted on the organisation’s website. This transparency will

show that the organisation treats the SIA recommendations seriously, which in turn

should show consumers and citizens that the organisation merits their trust. The or-

ganisation should put in place a mechanism or system for updating the SIA report as

necessary and, especially, for monitoring the implementation of the recommendations.

Research funding and support institutions may also wish to be informed of how a re-

search institution is implementing the recommendations.

Recommendations from the SIA may have implications for the research methods and

research design used in a security research project.

Ensure a third-party review and or/audit of the SIA

The value of independent third-party review or audit has been established for privacy

impact assessments, in term of guaranteeing quality and rigour.43 This is likely to hold

true for societal impact assessments. For research projects this review will need to be

planned for in advance, with appropriate third parties identified. Existing review bod-

ies, for example research funding agencies, will have their own evaluation and report-

ing requirements, which may support the external review of the SIA, but these agencies

may not yet have the capacity to fully audit the SIA process.

Update the SIA if there are any changes in the project

Many projects undergo changes before completion. Research on technological devel-

opment may go in several different directions before achieving its goal. Research with a

social dimension may also uncover previously unidentified societal impacts. Whenever

43 Stoddart, Jennifer, “Auditing Privacy Impact Assessments: The Canadian Experience”, in David Wright and Paul De Hert (eds.), Privacy Impact Assessment, Springer, Dordrecht, 2013.


material changes occur, the project manager and/or assessor should revisit the societal

impact assessment to see whether it needs to be amended, which will almost certainly

be the case where new societal impacts become apparent that were not previously con-

sidered. The value of the spiral methodology is that it highlights the importance of re-

visiting key questions through the lifetime of the project, and holding initial findings

contingent. Depending on the magnitude of the changes, the assessor may need to re-

visit the SIA as if it were a new initiative, including a new consultation with stakehold-

ers.

Refer to the SIA in any post-project evaluation

The SIA process does not end with the publication of the report, but should be contin-

ued into any evaluation work related to the security research project. Depending upon

the scope and scale of the project, additional resources and methods may be available

to evaluate the efficacy of the security research or applied security measure, these activ-

ities should include consideration of societal impact.

2.2.3: The societal impact assessment report

A societal impact assessment will have several key outcomes. For example, a key out-

come will be the identification and overcoming of any negative societal impacts. Anoth-

er outcome will be the benefits of stakeholder interaction and engagement. Yet another

outcome will be the discovery of new knowledge and learning (by the organisation as

well as others). Still another key outcome of this methodology will be a societal impact

assessment report for the security research project. This section of the paper provides

guidance on the contents and purpose of the societal impact report. The report docu-

ments the assessment process, and contains the resulting findings. It acts as a reference

document during the conduct of the project and as part of evaluation work afterwards.

The report can serve both as evidence and a record of the societal impact assessment

process. It can serve as a touchstone during the project for project staff and other par-

ticipants, and as a way to demonstrate commitment to understanding and managing

societal impacts of the project.

This reporting guidance draws upon the structure of privacy and surveillance impact

assessments44, as combined with the influences of CTA and social impact assessments,

and configured for application to societal impacts of security research. This section

describes the structure of the final, completed report; however, several sections of the

report, particularly those related to background, planning and project description,

should be initiated in the planning and preparation stage detailed above.

The report should contain the following sections, details of which are provided below.

44 Wright, David, and Kush Wadhwa, “A step-by-step guide to privacy impact assessment”, presentation paper for the second PIAF Workshop, Sopot, Poland, 24 April 2012. http://www.piafproject.eu/ref/A_step-by-step_guide_to_privacy_impact_assessment-19Apr2012.pdf


1. Background and identifying details

2. Introduction and overview of the SIA process

3. Project description

4. Societal Impacts

5. Options and alternatives

6. Design features to manage societal impacts

7. Compliance with laws, regulations, codes and guidelines

8. Stakeholder analysis and result of consultation(s)

9. Recommendations

1. Background and identifying details

The SIA report should state on its cover page at least the following elements:

• Societal impact assessment on [name of the project]

• Name and address of the organisation sponsoring the SIA

• Contact person (the assessor), title and e-mail address

• Date of the SIA report.

The length of the SIA report may justify an executive summary, which should state why the SIA was undertaken, who initiated the SIA and who conducted it. The executive summary should provide a brief description of the security research project or technol-ogy application that was the subject of the SIA. It should say which stakeholders (or stakeholder groups) were consulted. It should identify the principal societal impacts, across the six categories and the alternatives for minimising or avoiding negative im-pacts. The summary should contain the principal recommendations of the SIA report.

The SIA report should include a section that describes how senior management is in-volved in decision-making related to societal impacts of security research. Does the senior management board regularly discuss the impacts of security research or applied security measures? Are there specific office holders whose responsibility includes socie-tal impacts?

This background section should identify any organisational issues that are directly or indirectly implicated by the development of the project. For example, it may become apparent that the development of the project requires putting in place an organisation-al mechanism for ensuring accountability, i.e., that the CEO or her designated senior manager is responsible for ensuring that the development of the project does not nega-tively affect the organisation or stakeholders, and that beneficial societal impacts are maximised. The organisation should have procedures in place whereby funding for the proposed project is tied to completion of a satisfactory SIA beforehand.

2. Introduction and overview of the SIA process

The Introduction should outline the scope of the SIA, when, why and for whom it was performed and by whom. It should provide initial information about the project or se-curity measure assessed. It should introduce the methodology employed in the SIA, which can be drawn from section 2.1 of this paper, or adapted as required. Adaptations


should, however, be documented. It should also set out the terms of reference for the assessment.

This section should describe the SIA process undertaken (similarly, this can be drawn from section 2.3) and what was the outcome of each stage of the process. It should de-scribe the scale of the SIA undertaken and why the organisation developing the project and (presumably) sponsoring the SIA felt the scale of the SIA undertaken was appro-priate. It should refer to any stakeholder consultations undertaken and the approach adopted to support these.

This section of the report should also describe any measures that have been undertaken to attempt to increase the role of stakeholders in decision-making processes relating to the research project or security measure implementation.

3. Project description

This section should provide a detailed description of the project including its objectives and justification for the project. This should include initial answers to the societal needs questions from section 2.1 and can assist with project planning.

Details of the project can be added as the SIA progresses, as the “spiral” expands and greater knowledge regarding societal impacts is produced.

This section should include information on the documented societal needs to which the security research or application is addressed, and how the project will address these needs. It should contextualise the project against its theoretical background and core assumptions.

This section should also state the main aims of the project or technology? Why is this research being conducted (or proposed) and the system or technology being estab-lished? What are the principle features of the security measure proposed?

The project description should state who is undertaking the development of the project, when it is expected to be conducted or deployed. It should state the subject of the pro-ject, potential beneficiaries, and stakeholders who might be interested in or affected by the project.

The project description should provide some contextual information about how the project fits in with the organisation’s other services or activities. It should state whether aspects of the project are or will become proprietary. It should indicate the intended outcome of the project (for example, fundamental scientific advances, a new technolo-gy, a demonstrator or a commercial product or service).

4. Societal impacts (risks)

This section should list and describe the societal impacts posed by the project. Thus, the organisation or project manager or assessor should consider the impacts of the proposed project on all six categories of societal impact identified above. This section can set out the responses to the questions from section 2.1. These reflections in initial drafts of the reports can be expanded upon in later and final versions as more infor-mation is produced by investigation and consultation. The assessor should state how she and/or the organisation believe the impacts will affect the project objectives.


5. Options and alternatives

The SIA report should include a section that identifies the options and alternatives available to the organisation in order to mitigate, avoid, transfer, eliminate or accept the negative societal impacts identified by the SIA, as well as those that could optimise the realisation of societal benefits. The report should say why particular options or al-ternatives were rejected or discounted and why a particular course of action has been recommended. If the organisation has decided to proceed with the research project or technology implementation despite the SIA raising the risk of negative societal impact, the assessor should say (if she knows) how the organisation justifies these. Opinions and alternatives should be understood broadly to also include alternative policy direc-tions.

6. Design features to manage societal impacts

This section should describe the design features adopted within the project or technol-ogy application to reduce or avoid negative societal impacts and to maximise positive societal impacts, and state what are the implications of these design features (e.g., how they affect the viability of the project).

The report could include a table like the one below:

Table 6: Societal impact response table (source: authors)

Societal

Impact

Category Descrip-

tion

Mitigation /

benefit maxi-

misation

measures

Implications

for the project

Way of life, fears and aspi-

rations

Culture and community

Political systems

Environment

Health and well-being

Personal and property

rights

7. Compliance with laws, regulations, codes and guidelines


The SIA report should identify the laws, regulations, codes of conduct and guidelines with which the project complies or should comply.45 This section should also provide an assessment of whether the security research has the potential to generate results that require new legislation to address potential gaps. For example, if a new technology were to drastically improve the surveillance capacity of an existing technology, perhaps by removing any possibility for anonymity in public space, would this require revision of data protection legislation? In addition, this section should state how the organisa-tion monitors compliance with the laws, regulations, codes of conduct and guidelines it has identified.

8. Stakeholder analysis and results of the consultation(s)

The report should identify who are the principal stakeholders interested in or affected by the project, and how the assessor or the organisation arrived at this list. The report should specify what efforts the organisation has made to consult with stakeholders, including the public, to gather their views and ideas about potential societal impacts, how they might be affected by the project (positively and/or negatively) and how nega-tive impacts could be mitigated, avoided, minimised, eliminated, transferred or accept-ed. The assessor should specify which consultation techniques were employed (surveys, interviews, focus groups, workshops, conferences, Delphi, surveys, etc.), when they were undertaken, the results of each consultation exercise and whether differences in opinions were discovered when different techniques were used. If any particular stake-holder consultation tools were used in the process, this should also be noted.

The assessor should state who was consulted and what information materials the or-ganisation provided to stakeholders, including the public. Such materials might be in-cluded as an annex to the report. The assessor should state whether the consultations yielded any new findings and what efforts the organisation has made to take into ac-count stakeholder views and ideas in the design of the project. Did any fundamental changes result from stakeholder consultation? The assessor should state how she views public acceptance of the project (does the public accept the project? Or not? Or is opin-ion divided? Or does anyone care?).

9. Recommendations

The assessor should set out her recommendations, which could be a few or many, de-pending on the case. They could be detailed and specific or high level. The recommen-dations are primarily aimed at reducing or removing negative societal impacts and in-creasing positive societal impacts. Some negative impacts may, on balance, be worth accepting and, if so, the assessor should explain why. The assessor should be clear who

45 Privacy Impact Assessment in the European Union is conducted with reference to the Data Protection Directive 95/46/EC, the E-Privacy Directive 2002/58/EC and the Data Retention Directive 2006/24/EC. Because societal impact assessment takes into ac-count a broader range of potential impacts, it must also take into account a correspond-ingly wider range of legislation.


will bear the negative and positive impacts (i.e., will it be society, specific social groups, the organisation conducting the project, stakeholders, suppliers or others?).

The assessor should also set out what further work is necessary or desirable with regard to the societal impact assessment. For example, the assessor should make recommen-dations with regard to the need for independent third-party monitoring of implementa-tion of the recommendations.

The assessor should also make recommendations re whether the SIA report should be made public. The default mode should be to make the SIA report public (e.g., to post it on the organisation’s website), however, there may be circumstances where it might not be appropriate to make the SIA or parts of it public – e.g., there may be security, com-mercial-in-confidence or other competitive reasons. Often the report can be redacted here or there and then made public or sensitive bits can be placed in a confidential an-nex.

2.3: Conclusions and recommendations

A societal impact assessment should be completed for all security research, and lessons

learnt from PIAs, SuIAs, CTA and social impact assessment to ensure high standards

are met in terms of real assessment, and to prevent against SIA becoming a box-ticking

exercise. A coherent methodological approach and structured processes, integrated

with existing research practice, combined with clear reporting should contribute to

minimising negative societal impacts and increasing the positive societal benefits of

security research. The structured methodology presented here provides guidance

through such a process.

Whilst benefits can be achieved by security researchers and security research organisa-

tions conducting SIAs, additional benefits can be achieved by mainstreaming such as-

sessments within wider security research contexts. Options for achieving this include

integrating SIA into research frameworks, gaining the support of research support in-

stitutions and policy-makers, sharing expertise and providing training in societal im-

pact assessment, and collating together evidence of the benefits of societal impact as-

sessment activities.

Societal impact assessment methodologies and guidance could be trialled in the Euro-

pean Commission’s Horizon 2020 Secure Societies work programme (which lists vari-

ous calls for security proposals), mirroring the deployment of the Societal Impact

Working Group checklist. However, as the SIA approach builds upon that checklist, and

adds a structured methodology for assessing and reporting upon societal impact, the

SIWG work could stand a precursor, allowing a more rapid adoption of SIA. Many re-

search actors will have some measure of experience with the preceding approach, upon

which this builds. Deployment of societal impact assessment with large scale research

frameworks will help to achieve policy objectives of secure societies without infringing

upon fundamental values. Such an effort would require the support of the organisations

responsible for such frameworks such as the EU’s Research Executive Agency (REA).

The Social Impact Measurement Working Group identified the following phases, into

which societal impact assessment might be implemented: work programmes and annu-


al calls (setting the security research funding framework), proposals (the exercise of

putting together a research proposal and responding to the funding framework), nego-

tiation (between funders and researchers, and involving the alteration of the research

proposal to fit the framework), project execution, and the implementation of a com-

pleted product, system or technique in different contexts.46

Training in this methodology could be supported by institutional funding bodies, re-

search prioritisation agencies, research institutions and other appropriate actors. This

would increase the capability, skill level and experience of security researchers, across a

broad range of organisations, in the conduct of societal impact assessment. This train-

ing could be supported by developing a network of practitioners with expertise in socie-

tal impact assessment. This network would act as a resource and store of experience for

conducting societal impact assessments. It would also facilitate organisations seeking

advice or looking to include societal impact assessment in their research activity, either

on a contractual basis, or through partnerships. The network of practice could also col-

late societal impact assessment reports and examples of good practice in societal im-

pact assessments. This would contribute towards building an evidence base for the abil-

ity of SIA to increase the positive societal impacts of security research and decrease the

negative impacts, as well as providing future research projects with models of how to

achieve this. It would also facilitate analysis of long term security research funding

frameworks as a whole, including the ability to reflect upon the societal impact of agen-

da setting.

46 McCarthy 2012, p. 11


3. Review of existing decision support tools

In this section of the report we present the findings of our review of existing decision

support tools and resources. This review was conducted to support the development of

the ASSERT SERIA toolkit. It is critical to understand existing potential solutions, in-

cluding platforms that might be adapted or customised, in order to make the appropri-

ate tool selection and to avoid wasted effort. Our review considered the inherent usabil-

ity of the tool in relation to societal impact assessment, as well as fit with the objectives

of the ASSERT project.

3.1 Decision support tools

Decision support tools generally assist the user in making a structured decision from

amongst a set of available options, to set values to options, and provide choice rules.

These tools are modular and generic, and therefore useable, but do not currently have

inherent support for societal impact assessment in security research. Potential uses for

ASSERT include: 1) as part of, or basis, for SERIA 2) interactive toolkit integrated with

Masterclass, 3) inspiration for own tool specifically designed for security impacts. The

following decision support tools are based around a range of decision support method-

ologies. Is not within the scope of this deliverable to delve too deeply into these meth-

odologies other than to indicate that they are primarily concerned with different ways

to compare a set of relatively comparable options, through shared criteria. A common

example with such software is the decision of which new to buy. In this case common

criteria would include the cost, the fuel economy, the maximum speed, number of pas-

sengers, safety record etc. Each of these criteria could be attributed a value (100mph

for speed), and the criteria could be weighted against each other (for example, I may

express a preference for speed over fuel economy).

3.1.1 D-SIGHT

D-SIGHT in an online multi-criteria decision analysis tool. The software itself supports

a range of methods for the comparison of alternatives. These include pairwise

weighting, ADD, MCDA, and SMART. D-SIGHT has a strong visual presentation, and

the analysis of each alternative is well established. It is Internet based, with a free trial

available, followed by a paid version.

The user specifies criteria for the decision, and can then compare options side-by-side

in interactive graphs and visualisations. The tool has a predefined workflow which is

intended to increase the efficiency of decision making. The tool also provides support

for exporting, recording and communicating the decision made, as well as some collab-

oration tools for collaborative decisions.

Figure 3: D-SIGHT screenshots


3.1.2 VISA

Visual Interactive Sensitivity Analysis (VISA) is another Multi-Criteria Decision Analy-

sis software, which provides multiple methodologies for valuing and weighting alterna-

tives in a decision, including SMART, MCDA and ADD. VISA has particular support for

integrating the weightings and valuing of several participants into a shared decision. It

also allows for the use of decision tree methods. VISA is a relatively straightforward

system, with an intuitive interface which is possible for first-time users to understand

with some support. A web based version is also available.


Figure 4: VISA screenshots

3.1.3 Web-HIPRE

Web-HIPRE47 is a web-based decision support software offers a wide range of decision-

support methodologies, including value-trees, swing weighting, pairwise comparison,

non-linear value functions, sensitivity analysis, MCDA, ADD, SAT, and SMART. It is

another tool for the evaluation of alternatives. In terms of impact assessment this sug-

gests a requirement for the impact to be already known, at least to some extent. The

methodologies underpinning Web-HIPRE are robust and supported by current best

practices in decision support research in academia, however the tool is less refined that

47 hipre.aalto.fi


some of the other options here, and has a basic (if functional) interface, based on Java.

Web-HIPRE also offers support for group decision support.

Figure 5: web-HIPRE screenshots

3.1.4 1000minds

1000minds48 is a web-based decision support software. It is based upon a proprietary

weighting method called PAPRIKA (Potentially All Pairwise RanKings of all possible

Alternatives) which is intended to be a more intuitive and natural pairwise weighting

method. The tool has a strong visual presentation, and the analysis of each alternative

is well established. 1000minds won a Consensus Software Award sponsored by IBM

and Microsoft.

Figure 6: 1000minds screenshots

48 www.1000minds.com


3.1.5 Transparent Choice

Transparent Choice49 is a collaborative online decision making software intended to

support multi-stakeholder decisions, with features that allow the division of tasks

among stakeholders. It is a decision support tool, intended for the evaluation of alter-

natives with known (or estimated) impacts, and known (or estimated) values for key

criteria. It has an intuitive interface and its collaborative functions work in a manner

similar to dropbox, and stakeholders can be invited to collaborate via email.

Figure 7: Transparent Choice screenshots

49 www.transparentchoice.com


3.1.6 SMART

SMART50 is another online decision support software, and is one of the simplest tools

we evaluated in this section. Clear instructions and guidance are given for each step.

Figure 8: SMART screenshots

50 www.smart-decisions.net


3.2 Learning Support Environments / Learning Management Sys-

tems

This section reviews a set of tools that support learning activities, including the frame-

works for online courses. These tools often allow for multi-media presentation and in-

clude communication tools for collaborative learning. Potential uses for ASSERT in-

clude the creation of a learning environment or online course for assessing social im-

pact of security research, or integration with Masterclass

Many of these products are really very similar in terms of functionality. The all offer

ways of setting up a “course” in a web based online format. Generally, the courses are

composed of a set of content generate or curated by the course provider as you might

find in a traditional offline course or module (often divided up into “Lessons” or “clas-

ses” of appropriately sized learning units), supported with tools to facilitate communi-

cation between teachers and students, and between students, and to facilitate the man-

agement of the class (for example enrolment, setting quizzes or other assessment exer-

cises. These products generally allow the course provider to monitor some activity on

the course (for example, which students have accesses a particular course or assign-

ment) and to restrict access to enrolled students.

3.2.1 Moodle

Moodle51 is a software package for producing Internet-based courses and websites.

Moodle is open source software under the GNU General Public License. Moodle is in-

stalled on a server and can then offer a fully functional learning management system.

Moodle is a mature product, with a large number of users and developed documenta-

tion and support base. It is relatively rapid to set up in a basic configuration, and pro-

vides a large number of education tools (including chat, discussion forum, lessons, da-

tabases, connections to external tools, assignment setting and collection functions).

Moodle has a large number of option features which can be activated to customise the

installation for particular purposes including unusual class formats.

Moodle offers the following features52

• Collaborative tools and activities (forums, chat, wikis, glossaries, activities data-

bases, collaborative publishing)

• Support for various pedagogic approaches (instructor-led, self-paced, blended

learning or online online).

• Calendar

51 https://moodle.org/

52 http://docs.moodle.org/26/en/Features


• File management

• Multimedia integration

• Text editors

• Profile management and user access management

• Notifications

• Progress tracking

• Customisable site design

• Support for open standards

• Management of user roles and permissions

• A wide range of plug-ins and plug in management systems

• Reporting and logs

The lesson function seems particularly appropriate for some of the ASSERT features,

for example, a step-by-step impact societal impact assessment:

The lesson activity module enables a teacher to deliver content and/or practice activities in inter-esting and flexible ways. A teacher can use the lesson to create a linear set of content pages or instructional activities that offer a variety of paths or options for the learner. In either case, teachers can choose to increase engagement and ensure understanding by including a variety of questions, such as multiple choice, matching and short answer. Depending on the student's choice of answer and how the teacher develops the lesson, students may progress to the next page, be taken back to a previous page or redirected down a different path entirely.

A lesson may be graded, with the grade recorded in the gradebook.

Lessons may be used

• For self-directed learning of a new topic • For scenarios or simulations/decision-making exercises • For differentiated revision, with different sets of revision questions depending upon an-

swers given to initial questions

3.2.2 WordPress + plugins

WordPress is one of the most popular and commonly used website production and

blogging platforms, which makes it remarkably easy to create a set of web pages, which

can either be hosted by WordPress and be supported by adverts, or can be uploaded

onto a server and hosted by the consortium. WordPress’s functionality can be expanded

through a range of plug ins that provide the educational tools needed. There are several

of these available, as discussed below. There is a large amount of supporting material

available for WordPress, ranging from custom templates up to bespoke website design.

Moodle (integrated into WordPress)

Further to the discussion of Moodle above, Moodle can also be integrated into Word-

Press. The advantage of doing this would be to benefit from the ease of use of the


WordPress front-end. The WordPress appearance and templates are more attractive

than those available through Moodle, and using WordPress in this manner would allow

the consortium to have an accessible website with much of the SERIA material that

we’d wish to make public also available on it (without having to log in, create an ac-

count, learn how to navigate the LMS), but also to have all the LMS functionality avail-

able to registered users.

Namaste LMS - http://namaste-lms.org/

Namaste! Also runs as part of a WordPress installation. It offers the following func-

tions:

• Manage unlimited courses and enroll students or let them enroll themselves

• Auto-approve or manually review students who want to enroll in a course.

• Publish the course descriptions on your site to encourage subscribing.

• Charge to enroll in courses - Paypal and Stripe integrations available

• Manage unlimited number of lessons in each course.

• Lessons support rich text, media and plugin contents just like any other Word-Press post or page.

• Define various criteria for lesson completeness - such as completing assign-ment, manual approval, completing exams.

• Define which lessons should be completed to complete a course. Some lessons can be optional.

• Create assignments and approve or reject student's solutions.

• Add notes to student's solutions to help them resolve the problems.

• Manage students in each course and see their to-do lists for every lesson

• Create certificates that will be assigned to students for completing course(s)

• Allow different user roles to administrate and use the learning material

• Connect to exams created with plugins like Watu or WatuPRO

• Run powerful reports using the Namaste! Reports plugin.

• Optionally allow discussion to assignments - powered by the additional Na-maste! Connect plugin

• Receive and send automated email notices about various LMS actions like en-rollments, comments, notes, submitted solutions, etc - via Namaste! Connect.

• Enable a dashboard widget or a page with activity stream showing you latest X days of activity in the system - via Namaste! Connect

LearnDash

LearnDash53 is a plug-in that turns WordPress into a learning management system by

the addition of functions for tools for online lectures, quizzes and checking how stu-

dents are progressing through the material. It provides a simple way of creating an

adaptable custom learning platform. LearnDash introduces an authentication systems

which allows designers to charge for access to particular courses. A demo version is

available, but the full version charges a fee. This fee includes access advice and guid-

ance to individuals setting up learning management programs. LearnDash has good

53 www.learndash.com


quality design, but a limited range of functions in comparison to some other LMS

plugins. LearnDash has been used by a number of for-profit education providers.

3.2.3 WordPress + multiple plug-ins

It is possible to create a functional online learning environment through the use of a

WordPress website, with the addition of a larger range of individual plug-ins, which

would replicate the functionality of one of the all-inclusive LMS packages. This would

allow some flexibility and options. However, it would take more effort and resources to

identify and select the appropriate plug-ins and to set them up. There is a greater po-

tential for having something missing that would otherwise be useful, or for conflicts

between the various plugin applications. Given that single plug-ins could provide the

required functionality, this avenue is not explored further.

3.2.4 Big Blue Button

Big Blue Button54 is a plug in for other websites, or hosting on dedicated bigbluebutton

servers. User downloads a desktop client, and it features open source licenses. This has

a tool for setting up education focused online meetings and web conferences, with sup-

port for video and audio communication and presentations.

Perhaps most appropriate for active teaching (video/audio and text chat in real time).

Big Blue Button provides several tools for moderating audio/conversation in real time,

looking at a presentation in real time with students etc. The ASSERT project does not

currently envisage (or has not yet discussed) this form of real-time teaching activity as

part of the SERIA toolkit, and its sustainability is not covered by the ASSERT project

agreement. If the Masterclass concept was to be expanded at a later point, this might a

useful tool for doing that. Big Blue button does not appear to offer all of the functionali-

ty required for ASSERT

3.2.5 OpenMOOC

OpenMOOC55 is an open source massive open online course platform, intended to pro-

vide a highly customisable and free-to-use system for hosting online courses. A MOOC

(massive open online course) is an online course aimed at large-scale interactive partic-

ipation and open access via the web. In addition to traditional course materials such as

videos, readings, and problem sets, MOOCs provide interactive user forums that are

54 http://bigbluebutton.org/

55 http://openmooc.org/


intended to help build a community for the students, professors, and teaching assis-

tants (TAs). MOOCs are a recent development in distance education. OpenMOOC is an

open source platform (Apache license 2.0) that implements a fully open MOOC solu-

tion. It is relatively lightweight – for example, rather than using any internal media

hosting, its course videos are hosted on YouTube. Unfortunately, whilst the project was

started in 2012, it is not yet completed, and critically, documents for installation have

not yet been made available. Therefore use of OpenMOOC for ASSERT would require

substantial additional coding effort.

3.2.5 Blackboard

Blackboard56 is an industry standard learning management systems that is used by a

large number of educational institutions. Blackboard’s current offering is a set of five

different platforms with specialist applications for business, education and industry.

This includes the ability to set up a fully functioning custom learning supporting envi-

ronment. Recent additions to the learning platform have included social functionality,

akin to Facebook and Twitter in the form of creating and inviting events, and allowing

for group chat. Blackboard is a very established provider, however it not open source,

nor free to use, but a range of educational and institutional licenses are available.

Key features include a central hub/portal with key information on the course displayed

in a simple place, online assessment functions, rich content editors, a user-centric in-

terface, surveys and course evaluation and tools for collaboration between students.

3.2.6 EdX

EdX57 is a massive open online course (MOOC) platform founded by the Massachusetts

Institute of Technology and Harvard University in May 2012 to host online university-

level courses in a wide range of disciplines to a worldwide audience at no charge and to

conduct research into learning. Potential students can choose from a wide range of

courses, with details of their requirements in terms of commitment and time. Once

registered on the system and on a course, the student has access to the courses. The

courses include automated feedback to quizzes and assessments, and participants can

gain a certificate for completing a course. The set-up of EdX courses, and of similar

MOOCs such as Coursera can provide inspiration for the structure and content of the

SERIA toolkit, but given the complex licensing requirements for courses to be submit-

ted and approved by EdX, currently only extended to a select set of partners, it does not

appear currently possible to host an ASSERT course on the EdX platform. The source

code is available under open source licensing agreements.

56 www.blackboard.com/Sites/International/EMEA/index.html 57 https://www.edx.org


3.3 Handbooks and Guides

These tools take the form of a physical or web-based handbook, giving a user access to

best practices and techniques in a particular area. Traditional handbooks are primarily

textual, with some visual material. Online guides provide the opportunity to add mul-

timedia content such as audio, video and animation. Potential use for ASSERT as a

model for what a non-interactive or offline assessment tool might look like – e.g. a

“Handbook for assessing the social impact of security research”.

3.3.1 IDEO HCD toolkit

IDEO58 is a global design and innovation consultancy, which has adopted a research-

informed design methodology to approach a range of problems for its clients. Their

approach, which they term “design thinking” brings together what is possible from a

design point of view with what is desirable from a human perspective. The intent is to

allow people who have not been trained as designers to use the same creative tools to

approach problems.

The Human Centered Design (HCD) Toolkit is a freely downloadable and open source

toolkit designed to help international staff and volunteers with social enterprises and

NGOs understand a community’s needs in new ways, find innovative solutions to meet

those needs, and deliver solutions with financial sustainability in mind.59 The kit

“walks users through the human-centered design process and supports them in activi-

ties such as building listening skills, running workshops, and implementing ideas.”60

IDEO also hosts HCD Connect, an online platform for people taking a human centered

design approach to connect and engage with each other. The IDEO website also con-

tains case studies of successful application of the approach including a project on safe

access to drinking water, and another on children’s eye care in developing countries.

HCD also refers to the suggested approach – Hear, Create, Deliver.

Hear (6 Steps in total) During the hear phase, Design Team will collect stories and

inspiration from people. This includes preparing for and conducting field research.

Create (7 Steps in total) In the Create phase, participants work together in a work-

shop format to translate which is heard from people into frameworks, opportunities,

solutions, and prototypes. During this phase you will move together from concrete to

more abstract thinking in identifying themes and opportunities, and then back to the

concrete with solutions and prototypes.

58 http://www.ideo.com/about/ 59 http://www.ideo.com/work/human-centered-design-toolkit/ 60 Ibid.


Deliver (6 Steps in total) The Deliver phase will begin to realize your solutions

through rapid revenue and cost modeling, capability assessment, and implementation

planning61

Figure 9: Human Centered Design Toolkit62

3.3.2 Research Toolkit

The Research Toolkit63 is a website intended to provide tools and guidance for all as-

pects of a clinical research project. The website is organized by phases of a research

project, with resources that have been vetted and curated by the project team.

The Research Toolkit project was funded through two consecutive administrative sup-

plements to the University of Washington's Institute for Translational Health Sciences

Clinical and Translational Sciences Award (CTSA) UL1 RR025014 from the NIH Na-

tional Center for Research Resources.

The toolkit is divided into six sections, each with sub sections.

1. Building Collaborations: A solid foundation is essential to any successful project. The resources and tools found in this section cover fundamental aspects of partnerships, from finding partners to ensur-ing sustained engagement.

2. Developing Proposals: This section provides links to funding sources, budget development aids, and tips on good grant-writing.

61 IDEO, HCD Toolkit. London & New York, IDEO, 2009 62 http://www.ideo.com/work/human-centered-design-toolkit/ 63 http://www.researchtoolkit.org/


3. Starting Up a Study: Able to find resources for including contractual agreements, ethical/regulatory approvals, and training study staff, as well as a guide for developing participant-centered study mate-rials.

4. Conducting and Managing Projects: Helpful array of resources to aid the study management process, including procedures for informed consent, operations manuals for clinical research, and ways to keep both research staff and participants engaged over time

5. Disseminating and Closing Research: Sharing research results is both an obligation and an opportuni-ty.

6. Research w/Hispanic and Indigenous Populations: As this is a fundamental and cross-cutting aspect of research, we have compiled a few dozen resources that cover a range of issues from engaging with communities and ensuring research is culturally appropriate, to IRB and regulatory processes and study management.64

The Research Toolkit provides clear, structured research guidance, directed towards

practitioners, and this would have clear transferability to ASSERT in the form of pre-

senting a structured approach to a specific element of a research project. It is primarily

an information resource with no interactive elements.

Figure 10: Research toolkit screenshot

64 http://www.researchtoolkit.org/


3.3.3 Crisis Guide: Iran

The US Council on Foreign relations produced an interactive online guide to Iran.65 The

presentation traced Iran’s history, its evolution as an Islamic republic, and its nuclear

programme. It also presented a set of policy options relation to Iran. The guide includes

an animated timeline, details on the regime, nuclear programme and the broader re-

gion, as well as photography, maps and video. This is an example of an attractive and

accessible packaging of information, which makes good use of a combination of ap-

proaches to address a complex subject. Top level information in presented in the intro-

ductory video, whilst exploring the pages provides more detail. This type of presenta-

tion is increasingly common in online information sources, and is build upon common

modular approaches to web development. The key requirements are appropriate con-

tent (in the form of text, maps, video, images and audio material) and determining an

appropriate structure for this material that maximises flow through the site and acces-

sibility of the information. This form of presentation may be most effective upon first

viewing, and its valuable as a reference source or guide to practice may be limited.

However as an introduction to a topic, this approach can be powerful.

Figure 10: Crisis Guide: Iran

65 http://www.cfr.org/interactives/CG_Iran/index.html#/overview/


3.4 EU Decision support projects

Several current EU-funded projects include an element of decision support, and the

construction of tools. Some of these also have a specific security dimension. The AS-

SERT project may be able to build upon, expand, customise or incorporate the tools

produced by these projects.

3.4.1 SIAM66

The SIAM project is described by its consortium in the following manner:

“SIAM is an EU-funded research project that provides support to end-users in the as-

sessment process of security measures and technologies (SMT). The overall objective is

to create an Assessment Support Toolkit that takes the complexity of technologies, eco-

nomic aspects, cultural differences and societal dimensions into account. SIAM worked

to help stakeholders to cope with the increasing complexity of assessments by provid-

ing a systematic approach for assessing the potential impact of SMTs.”67

The purpose of the SIAM Assessment Support System is to guide users in their assess-

ment of security measures and technologies. This is realised by means of a set of struc-

tured questions about issues of security, trust, efficiency, and freedom infringement.

These questions have been develop by experts in their respective fields and take into

account social and legal issues. The SIAM system function as a database and stores

store these questions and their associated assessment paths and make them available to

different kinds of users.

The SIAM68 toolkit is best described as a tool for conducting an analysis of the potential

future impacts of security measures and technologies based upon the structured collec-

tion of relevant perspectives, through answers to expert curated questions. It is in a

beta stage with some functions still to be completed, but is functional. It attempts to

help develop a picture of the potential likely impacts of a security intervention by draw-

ing upon the collective knowledge embedded in its multiple participants. Unlike some

of the other tools we have assessed through ASSERT, the SIAM tool is inherently based

upon multiple participants.

The tool is aimed at organisational decision makers seeking to evaluate a potential se-

curity technology intervention (for example, if they should install an abandoned bag-

gage detection system in an airport). The tool identifies a number of roles that should

66 Members of the ASSERT Consortium were invited to participate in the SIAM User Forum II in Berlin 31st October – 1st November, to experience the tool, participate in its testing and to pro-vide feedback on the tool to the SIAM consortium. The following material is based upon that meeting as well as a broader examination of the tool. 67http://www.tu-berlin.de/ztg/menue/forschung/projekte_-_laufend/security_impact_assessment_measures_siam/ 68 http://siam-project.eu/


be involved in the assessment process (for example, lawyers, stakeholders, investors,

etc). These actors are invited to participate in the use of the tool. The tool is based

around a large pool of 300 or so questions. These questions have been developed by the

SIAM consortium on the basis of research and literature in areas of law, sociology and

technology, and are intended to get at the key issues in security technology assessment.

actors are presented with a sub-set of the question pool relevant to their role, and with

conditional logic (for example, an initial question might ask if there is any anticipated

risk to human life from a technology, and if the answer is yes, then subsequent ques-

tions will be displayed which probe into this risk in greater detail). The answers to these

questions are collated together from the various participants, and can be used to pro-

duce a final report. There is also a reflexivity score which gives a quantitative evaluation

of how close to inclusive the process was (for example, if it should have involved eight

different roles, but only included four, and of these four some only answered half of the

questions, it would receive a lower reflexivity score). Therefore the SIAM tool can also

be understood as a consultation management tool.

One strength of the tool is the collation of different perspectives. The danger is that

they are currently just reported. The tool needs to help this process reveal diversity, and

potentially develop new approaches/solutions. The tool currently has a static model of

the security intervention being assessed whereas the final intervention might be

changed through this process. The tool needs to identify ways in which the technology

being assessed can be improved (for example by implementing it technologically in a

particular way, by changing elements so that it avoids infringing fundamental rights).

Participants need to be able to suggest such changes or approaches. Explanations and

evidence for decisions are important in this. This also might be part of the value-added

case of why users would make use of this tool.

The tool currently examines a single technology measure at a time, when often the need

will be to comparatively examine a range of options. There will therefore be some re-

dundancy in the tool, with participants being invited several times and having to an-

swer the same question multiple times. Could imagine ways in which parallel cases

could be linked in some ways. Each gets an individual report/case, but could the way

the questions are served reduce the workload of this?

The outputs of this process could with the appropriate legal/institutional/reputational

support be valuable in themselves. A tool like this could potentially be part of (self) cer-

tification approach, or as evidence that best practice in consultation/security impact

assessment has been followed. We discussed at the session the possibility of crypto-

graphically proving that a report generated through the process had not be edited, for

example.

It was suggested of the tool required user training. A requirement for training to answer

questions, introduces an overhead, and a bias towards “professional stakeholders”. The

questions should be answerable by a professional in the relevant field, but without ad-

ditional training.


3.4.2 EURO Summer school

The EURO Summer School69, also known as the International Summer School on "Mul-

ti-Criteria Decision Aid": Methods, applications and software. The regular event was

formed by a group of well-known North American and European scientists with the

purpose of promoting the diffusion of the potentialities of multi-criteria models as val-

uable supporting instruments in decision-making. The aim was to provide a continuing

environment for face-to-face networking. Starting in 1983, the Summer School on

MCDA/M is a joint event of the International Society on Multiple Criteria Decision

Making70 and the EURO Working Group on Multi-criteria Decision Aiding.71 The aim

of the school is to give to doctoral students/young researchers a state-of-the-art presen-

tation of multiple criteria methods, applications and software. The school does this

through inviting a combination of researchers intending to develop a thesis in MCDA

alongside industry and government professionals who use such approaches in their

work.

3.4.3 MCDA-RES

The aim of the MCDA-RES project72 is to develop a software decision tool that will ena-

ble multi-criteria decision analysis (MCDA) of renewable energy source investments

and apply it to three case studies. The project concluded in 2004 and had several work

packages were devoted to tool development. The tool was intended to increase coopera-

tion between experts and analysts in the process of making a decision.

Figure 11: MCDA-RES methodology

69 http://www2.hsu-hh.de/logistik/summer-school-2013/history.html 70 International Society on Multiple Criteria Decision Making 71 www.cs.put.poznan.pl/ewgmcda 72 http://www.aegean.gr/environment/energy/mcda/mid.html


3.4.4 DESSI

The DESSI project intended to create both a decision making process and a decision

support system for use by end users of security investments. The system is intended to

give insights into the pros and cons of specific security investments. The aim was to

support a transparent and participatory decision making, which is able to take into ac-

count specific contexts and social issues. The targeted audiences were public authori-

ties, developers of security solutions, commercial enterprises and social organisations.

The DESSI tool could be used by decision makers to structure their own assessment

exercises.

The DESSI tool is a planning and decision tool, hosted online with both a security focus

and a reliance upon a participatory methodology. A decision maker could follow the

structured DESSI approach, and in doing so, would take into account a range of key

contextual variables as well as be supported in the conduct of workshops and other

forms of stakeholder engagement. The stakeholder engagement elements are integrated

into the process and form a key part of the process. The tool would keep track of the

process and assist the decision maker. The process is intended to incorporate a set of

dimensions such as security gain/loss, impact on fundamental rights and ethical as-


pects, legal framework, social implications, acceptability, political significance and

economy into a security decision making process.

Figure 12: DESSI tool screenshot

3.4.5 DREAM

The DREAM project (Decision support in Real-Time for Efficient Agile Manufactur-

ing)73 aimed to increase the competitiveness of the European manufacturing sector by

developing methods to support real time decision making in manufacturing. The ap-

proach was based upon developing simulation software, particularly for industrial re-

quirements in manufacturing. The project has only started recently and at the time of

writing, no deliverables were available.

3.4.6 FIRST

The FIRST74 project seeks to develop a large scale information extraction and integra-

tion infrastructure for supporting financial decision making. The project had a focus

73 http://dream-simulation.eu/index.php/project-overview 74 http://project-first.eu/


upon information gathering and real-time decision support. It intends to create a pro-

cess for integrating information from a knowledge base with real time information

feeds to create models for the detection of financial events, as well as visualisation

tools. The type of knowledge engaged with in this project is highly quantitative and the

approach is heavily mathematical, based upon the challenge of extremely large, highly

dynamics and heterogeneous sources of information. The aim is to increase transpar-

ency by providing a fast, real-time, automatic and more comprehensive information

base can help preventing false decisions. FIRST also seeks to identify insider trading

and other forms of abuse by using large volumes of unstructured financial data incor-

porated into a reputation engine.

Figure 13: FIRST market surveillance

3.4.7 DISASTER

DISASTER (Data Interoperability Solutions at STakeholders Emergencies Reaction)75

aims to increase mutual understanding between international EMS organisations and

so reduce response time by helping create an interoperable solution to solve these is-

sues. Many emergency responders use Emergency Management Systems to facilitate

communication and emergency responses. However there are also problems related to

cultural, legal and linguistic differences. The aim of the project is to increase under-

standing between different participants in a network of emergency actors and thereby

speed up decision making processes in operation environments. This is to be achieved

through the development of a collective ontology, as well as developing techniques for

interoperability between technology systems. The project target outcome is an integra-

tive and modular ontology for establishing a common knowledge structure between all

the first responders involved in an emergency, but being compliant with legacy interna-

75 http://disaster-fp7.eu/


tional data formats exchanged in the European Union as long as being seamlessly inte-

grated within current SOA-based Emergency Management Systems. Literally, this is to

make a ‘black box’ that converts the each countries ‘code’ for emergency situation to

other countries one, thus able to handle international emergency situation. The project

user case studies and proof of concept deliverables are available.

3.4.8 ESS

ESS – Emergency Support System76 aims to develop a revolutionary crisis communica-

tion system that will reliably transmit filtered and pre-organized information streams

to the crisis command system, which will provide the relevant information that is actu-

ally needed to make critical decisions. The project is focused on the provision of action-

able information to crisis managers during abnormal events. This information is to

come from the collection and fusion of real time data from deployed sensors. The aim is

for a very large scale process that involves multiple sources of information. Whilst secu-

rity focused, the decisions in these situations are very short term and focused upon

emergency response. This is a very different environment from that of security research

processes, and the information to be processed (number of casualties, locations of peo-

ple and resources) of a markedly different character.

3.5 Analysis and conclusions

Based upon the review of existing tools, we can make the following conclusions

Online decision support tools are potentially useful for making security decisions

(as they may be useful in support of any particular decision) and would require relative-

ly little development time from the ASSERT consortium. However, these tools do not

support the investigation of societal impact assessment if this is currently unknown

(which in many cases it will be). Rather they support the comparative evaluation of

multiple options. Societal impact (or various composite parts of societal impact) can be

attributed a value in these evaluations, and then involved in a weighted comparison

against other considerations (for example, costs, resource requirements, etc). Alterna-

tives do not emerge during the process, but must be identified in advance. Further-

more, using these decision support tools in a societal impact assessment is likely to re-

quire additional information or a paradigm shift to include social impacts well. Indi-

vidual tools might be included as potential tools in handbook or online course, but it is

not suggested that these form the core of a SERIA toolkit.

Many of existing EU projects are either 1) incomplete, 2) quite specific in their focus,

or 3) beyond the scope of ASSERT to replicate. There are, however some lessons that

can be extracted from their experience. The SIAM tool could be a useful way of con-

76 http://www.ess-project.eu/


ducting a societal impact assessment for a particular context. It is strongly orientated

towards decision makers looking to implement a particular technology. This may be

appropriate for some forms of security research that are more focused upon application

and demonstrations of particular security technologies. In this case the SIAM tool could

be used to model this assessment process. It would require participants, although in an

appropriately interdisciplinary project these may be the consortium participants. The

danger would be of missing the perspective of external stakeholders at the research

planning stage. In contexts where the intended security technology intervention is not

yet realised (for example if the aim of the project is to develop something, but its fea-

tures are currently unknown), then the SIAM tool may be less useful. It would be worth

linking to the SIAM tool through the ASSERT toolkit when it becomes available. The

ASSERT toolkit should probably provide a paragraph akin to the one above (if not more

detailed) outlining the best way to make use of the SIAM tool, and the contexts in which

its use is appropriate.

Handbooks/Online guides on societal impact assessment would be relatively

straightforward to produce. This could be an enhanced version of the societal impact

assessment methodology presented in section two of this report. The Research Toolkit

provides clear, structured research guidance, directed towards practitioners, and this

would have clear transferability to ASSERT in the form of presenting a structured ap-

proach to a specific element of a research project – in this case societal impact assess-

ment. Similarly, The IDEO toolkit aims support practitioners in investigating a complex

social problem through a range of research methods, an area which has clear parallels

with societal impact assessment. A similar approach for societal impact assessment

would use a structured approach and suitable illustrative content, presented in an ac-

cessible manner to walk people, potentially with little experience of the process,

through a societal impact assessment exercise as set out in section 2. Like the societal

impact assessment methodology, the HCD toolkit promotes high stakeholder involve-

ment, and provides a number of tools for considering the potential social impact of de-

signs or other interventions. The toolkit is non-interactive, although HCD Connect does

provide interaction and communication tools.

Finally, Online learning environments and learning management systems

can provide many of the features of a handbook or online SIA guide, but also potential-

ly link up with supports, and extend the life of the master class concept. Material from

the deliverables and from the master class can be re-purposed to provide content for

such an environment. In addition to providing support to practitioners in security re-

search by presenting structured information on societal impact assessment, such a

learning environment could potentially foster the development of a learning communi-

ty centred on societal impact assessment in security research. Most of these LMS re-

quire customisation for a particular learning goal/objective and context of use. The

services have lots of options that can be activated or deactivated as appropriate. Course

content has to be created and uploaded, and the lessons and tools have to be selected in

a manner that will support the intent of the course. So beyond the choice of tool, the

most important choices will be what material content to use, and how to structure this

within the learning environment. Some tools are available under Open Source software

licenses and can be rapidly adopted.


The analysis of existing tools for decision support in this section has supported the de-

cision to develop the SERIA toolkit on the basis of a combination of online guide or

handbook combined with a learning management system.


4. The ASSERT online Toolkit

The ASSERT online toolkit is a combination of an online societal impact assessment

“handbook”, with content derived from ASSERT project deliverables (particularly from

WP1), with new material created for WP3 (see section 2), with multimedia content,

structured in an accessible manner, with the additional features provided by a Leaning

Management System that offers additional tools (communication, resource hosting,

user authentication and log-in etc) to registered users. This combination allows the

consortium to support the Masterclass experience with online content, to make infor-

mation on societal impact assessment in security research accessible for a wider range

of users, and also to host the ASSERT expert database created in WP2.

The ASSERT online toolkit is accessible at http://assert.masiondx.com

The Toolkit is intended to meet the following design requirements:

• Accessibility – the toolkit must be straightforward to use, and not present an

unnecessary complication to the user. Information on the toolkit should be pre-

sented in a logical and structured manner. The toolkit should not present barri-

ers to access from a variety of users with a variety of needs. Part of this require-

ment is the need to leverage appropriate social media and web 2.0 tools.

• Appealing – the toolkit should present useful information (relevant to its audi-

ence and purpose) in a visually appealing manner. Part of this requirement is

for interactivity and multimedia (photos, graphics, video and audio) content.

This provides one component of the added value over and above the societal

impact assessment manual.

• Audience – the toolkit should be addressed to researchers who may be interest-

ed in conducting a societal impact assessment as part of a security research pro-

ject, but who may not currently have significant expertise in this area. The sec-

ondary audience for the toolkit should be the evaluators and funders of security

research projects.

• Flexible construction – the tool should be adaptable and changeable over time.

It should be straightforward to add new content, and multiple administrators

should be able to update the tool in order to add a measure of sustainability.

Part of this requirement is the need to build upon existing tools in order to in-

crease the speed of the design process, and ensure familiarity and convenience

for users and content producers.

• User-Authentication – the tool should be able to differentiate between guests

and registered users, and be able to provide different information to these two

groups.

• Support to the Masterclass – the toolkit should be able to support the learning

and networking activities of the ASSERT Masterclasses.

• Host the Expert database – the toolkit should be able to host the ASSERT Ex-

pert database, and provide tools and functions for communicating to and be-

tween this group of users.

• Security and privacy protection – the toolkit should protect the personal infor-

mation of registered users and respect their preferences in terms of communica-

tions.


Structurally, the resulting toolkit is a combination of two software packages hosted on

the same server. WordPress is an industry-leading content management and web-

publishing tool. It is free to use and highly customisable, as well as being straightfor-

ward to add and edit content. Similarly, Moodle is a highly-regarded and well-known

learning management system. It is also free to use, and highly customisable. The adapt-

ability of both tools has allowed the ASSERT consortium to create a tool which is ad-

dressed to the specific audience for the tool (security researchers and evaluators), and

that provides the required functionality for the tool. The use of two tools allows for an

accessible (and easily updatable) set of front-end, public facing pages that can be ac-

cessed by anybody with an internet connection, combined with a password-protected

environment accessible to registered users only. The structural arrangement of the two

components is depicted in the following figure. In addition to this core arrangement,

the toolkit also makes use of video content hosted on www.vimeo.com and embedded

in the appropriate WordPress. The server hosting this content is owned by an ASSERT

consortium member which brings the content fully under the control of the consortium.

Figure 14: Structure of toolkit components

The general visitor to www.assert.maisondx.com will encounter the WordPress pages,

which they can navigate in the conventional fashion. On the “online course log-in” page

there is an inline frame which loads content from the Moodle LMS. This allows LMS

users to remain within the general environment of the toolkit, and provides a cohesive

appearance. The graphical appearance of the LMS was customised to be harmonious

with the general appearance of the toolkit.


Users can also access the LMS directly at http://www.assert.maisondx.com/lms which

may be preferable for users with smaller displays.

Users of the LMS are registered for one or more “Classes”, which are primarily a way of

structuring access to different types of content. The ASSERT toolkit features three clas-

ses: one dedicated class for each of the Masterclass events and one class which works as

the ASSERT expert database.

The database class is currently the largest group with around 170 registered partici-

pants. The main utility of this class is provided within Moodle by two forums. One of

these is a news forum for posts by the ASSERT consortium under the terms of use pro-

vided to participants. This forum has been configured to operate like a mailing list. The

second forum is an internal forum to which users can subscribe and share messages

between themselves. Email from this forum has been disabled by default in order to

prevent spamming, but users are able to reactive receiving email for their own account

if they wish. Users are able to modify their profile in the LMS and view the profiles of

other participants. By default only basic information is provided.

The Masterclass groups are more fully featured and provide a library of ASSERT doc-

umentation (all the published deliverables, group exercises, additional briefing papers

and reading material for the masterclass, as well as administrative and organisational

documents such as the agenda), chatrooms and forums, as well as user profiles. Users

can send direct messages to other participants. After the Masterclass events, these clas-

ses can persist and host content generated at the Masterclass, including photos, notes,

etc. Users of these classes are divided into two types “teachers” and “students” (the

terminology reflects Moodle’s general use in educational settings). In addition to the

access available to students, teachers are able to edit the content of the class and send

messages to all participants.

Moodle offers a very large range of features and tools for educational purposes, such as

quizzes, assessment and feedback tools, and even gamification “badges” for particular

achievements. Many of these were unnecessary for the purposes of the toolkit and have

been deactivated. Additionally, the guiding principles of the set up of the toolkit are to

provide users with access to important and useful information, but not to overburden

them with unnecessary content, overwhelm them with spam emails, or put their per-

sonal data at risk.

The following figure shows the structure of the public-facing pages created with Word-

Press. This structure is intended to cover the range of topics and material created and

curated by the ASSERT project whilst still being easy to navigate. The website is only

ever one two pages “deep” making it easier to navigate back to the main page.

Figure 15: Structure of public-facing content pages


The section “What is societal impact?” is historical and theoretical, providing a sum-

mary of societal impact in past and current research frameworks. This section is in-

tended to provide background. “Conducting a societal impact assessment” is the largest

section and the core of the public facing toolkit. This section draws upon the ASSERT

deliverables, to provide a structured approach to practically incorporating a societal

impact assessment exercise in a security research project. “Case studies” provides con-

crete examples of how a societal impact assessment might work in practice. Many users

will need this section in order to conceptualise how and SIA might work in their own

areas of work. This section is intended to expand over time as more case studies be-

come available. The “Resources” section collates external resources, including tools

created by other FP7 research projects, which can contribute to the conduct of an SIA.

Finally the “ASSERT Masterclass” section contains information about previous and

future ASSERT masterclasses, as well as a web-form to express interest in participating

in these. This section also contains access to the LMS and the registered-user-only con-

tent hosted there.

The key intention with the content in these sections has been to re-structure the infor-

mation produced by the consortium in manner more accessible than the traditional

document-based project deliverables. The sections point towards the texts from which

they are drawn for interested readers, but aim to provide the key information in these.

Video content based upon interviews with security research experts at ASSERT project

workshops is in the process of being produced to add this element to the toolkit.

Figure 16: Screenshot of the front page of the toolkit.

What is Societal

Impact?

History of SIA

Societal impact

under FP7

Societal impact in

H2020

Conducting a

Societal Impact

Assessment

Why assess

societal impact?

Good practice

criteria

Methodology

Planning and

Preparation

Consultation and

analysis

Reporting and

responding

Case Studies

Public Transport

VIC-SIGMA

Resources

Articles and Books

Tools

Stakeholder

engagement and

consultation

ASSERT

Masterclass

Express interest

Online course log-

in

Stirling

Masterclass

Brussels

Masterclass


Figure 17: Screenshot of the learning management system


Figure 18: Screenshot of the Masterclass courses


Figure 19: Screenshot of user profile

The masterclass forum element of the toolkit was user-tested with the participants at

the ASSERT Masterclass event in Stirling in February 2014, and will be further tested

with the participants at the Brussels Masterclass in April 2014.


REFERENCES

Becker, Henk A., and Frank Vanclay, The International Handbook of Social Impact

Assessment: Conceptual and Methodological Advances, Cheltenham, Edward

Elgar, 2003.

Bishop, Patrick, and Glyn Davis, “Mapping public participation in policy choices”, Aus-

tralian Journal of Public Administration, Vol.61, No.1, 2002, pp. 14-29.

Braun-Thürmann, Holger, Innovation, Transcript, Bielefield, 2005.

Bruner, J., The Process of Education. The President and Fellows of Harvard College,

Cambridge, MA, 1960.

Buzan, Barry, and Lene Hansen, The Evolution of International Security Studies,

Cambridge University Press, Cambridge UK, 2009.

Buzan, Barry, Ole Weaver and Jaap de Wilde, Security: A New Framework for Analy-

sis, Lynne Rienner, London, 1998.

Chilton, P., Security Metaphors: Cold War Discourse from Containment to Common

House, Peter Lang, New York, 1996.

Deibert, Ron, “Circuits of Power: Security in the Internet Environment” in J.P. Singh

and James N. Rosenau (eds.), Information Technologies and Global Politics: the

Changing Scope of Power and Governance, Suny Press, New York, 2002, pp.

115-142.

Norman Lee and Colin Kirkpatrick, “Evidence-based policy making in Europe: an eval-

uation of European Commission integrated impact assessments”, Impact As-

sessment and Project Appraisal, Vol. 24, No. 1, 2006, pp. 23-33.

Lianos, Michaelis, “Dangerization and the End of Deviance: The Institutional Environ-

ment”, British Journal of Criminology, Vol. 40, No. 2, Spring 2000, pp . 261-278.

http://bjc.oxfordjournals.org/content/40/2.toc

European Commission, “Key procedural steps for the Commission/smart-

regulation/impact”, 20 December 2013. http://ec.europa.eu/smart-

regulation/impact/ia_key/ia_key_en.htm

European Commission, “Secure Societies – protecting freedom and security of Europe

and its citizens”. http://ec.europa.eu/programmes/horizon2020/en/h2020-

section/secure-societies-%E2%80%93-protecting-freedom-and-security-europe-

and-its-citizens

European Commission, “Smart Regulation”, 13 January 2014.

http://ec.europa.eu/smart-regulation/index_en.htm

European Security Research Advisory Board, Meeting the challenge: The European

Security Research Agenda, 2006.

http://ec.europa.eu/enterprise/policies/security/files/esrab_report_en.pdf

European Parliament, Council and Commission, Charter of Fundamental Rights of the

European Union, 2010/C 83/02, OJ, Brussels, 30.3.2010. http://eur-


lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2010:083:0389:0403:en:PD

F

Genus, Audley, “Rethinking constructive technology assessment as democratic, reflec-

tive discourse”, Technology Forecasting and Social Change, No. 73, No. 1, 2006,

pp. 13-26. http://www.sciencedirect.com/science/journal/00401625/73/1

Guillemin, Marilys, and Lynn Gillam, “Ethics, Reflexivity and “ethically Important”

moments in research”, Qualitative Inquiry, Vol. 10, No. 2, 2004, pp. 261-280.

Kemp, Deanna, “Understanding the Organizational Context”, in Frank Vanclay and Ana

Maria Esteves, (eds.), New Directions in Social Impact Assessment: Conceptual

and Methodological Advances, Edward Elgar, Cheltenham, 2011, p. 30.

Mason, Jennifer, Qualitative Researching, Sage, London, 1996.

McCarthy, Sabhbh, Report of the Societal Impact Expert Working Group: EC DG

ENTR Report, February 2012.

Müth, Matthias, Verkherspolitik in Metropolen Südostasiens (the politics of traffic in

South East Asian Metropolis areas), Abera, Hamburg, 2003.

Neocleous, Mark, “Security, Liberty and the Myth of Balance: Towards a critique of

security politics”, Contemporary Political Theory, Vol.6, Issue 2, May 2007, pp.

131-149. http://www.palgrave-journals.com/cpt/journal/v6/n2/index.html

Nissenbaum, Helen, Privacy in Context: Technology, Policy, and the Integrity of So-

cial Life, Stanford Law Books, Stanford, 2009.

Organisation for Economic Co-operation and Development (OECD), Stakeholder In-

volvement Techniques, Paris, 2004. http://www.oecd-

nea.org/rwm/reports/2004/nea5418-stakeholder.pdf

O’Faircheallaigh, Ciaran, “Public participation and environmental impact assessment:

Purposes, implications and lessons for public policy making”, Environmental

Impact Assessment Review, Vol. 30, Issue 1, January 2010, pp. 19-27.

Peterman, William, “Advocacy vs. collaboration: comparing inclusionary community

planning models”, Community Development Journal, Vol. 39, No. 3, 2004, pp.

266-276.

Rip, Arie, and Harro van Lente,“Bridging the gap between innovation and ELSA: The

TA program in the Dutch Nano-R&D Program Nano Ned”, Nanoethics, Vol. 7, Is-

sue 1, April 2013, pp. 7-16. http://link.springer.com/journal/11569/7/1/page/1

Rip, Arie, and Johan Schot, “The Past and Future of Constructive Technology Assess-

ment”, Technological Forecasting and Social Change, Vol. 54, Nos. 2-3, 1997, pp.

251-268.

Rip, Arie, and Johan Schot, “Identifying Loci for the influencing the dynamics of tech-

nology development”, in Knut H. Sorensen and Robin Williams (eds.), Shaping

Technology, Guiding Policy: Concepts, Spaces and Tools, Edward Elgar, Chel-

tenham, 2002, pp. 155-172.


Schot, Johan, and Frank W. Geels, “Strategic niche management and sustainable inno-

vation journeys: Theory, findings, research agenda and policy”, Technology

Analysis & Strategic Management, Vol. 20, No. 5, 2008. pp. 537-554.

Stoddart, Jennifer, “Auditing Privacy Impact Assessments: The Canadian Experience”,

in David Wright and Paul De Hert (eds.), Privacy Impact Assessment, Springer,

Dordrecht, 2013.

United Nations, Follow-up to General Assembly resolution 64/291 on human security -

Report of the Secretary-General, 5 April 2012.

https://docs.unocha.org/sites/dms/HSU/Publications%20and%20Products/Rep

orts%20of%20the%20Secretary%20General/A-66-763%20English.pdf

Vanclay, Frank, “International Principles for Social Impact Assessment”, Impact As-

sessment and Project Appraisal, Vol. 21, No. 1, 2003, pp. 5-12.

Vanclay, Frank,“International Principles for Social Impact Assessment: their evolu-

tion”, Impact Assessment and Project Appraisal , Vol. 21, No. 1, 2003a, pp. 3-4.

Vanclay, Frank, “Social Impact Assessment: International Principles”, IAIA, Special

publication series No.2, May 2003.

http://www.iaia.org/publicdocuments/special-publications/sp2.pdf

Vanclay, Frank, and Ana M. Esteves (eds.), New Directions in Social Impact Assess-

ment: Conceptual and Methodological Assumptions, Edward Elgar, Cheltenham,

2011.

Wright, David, “The state of the art in privacy impact assessment”, Computer Law &

Security Review, Vol. 28, No. 1, Feb. 2012, pp. 54-61.

http://www.sciencedirect.com/science/journal/02673649.

Wright, David and Paul de Hert, “Introduction to Privacy Impact Assessment”, in David

Wright and Paul de Hert (eds.), Privacy Impact Assessment, Springer, Dordrecht,

2012, pp. 3-33.

Wright, David, and Charles Raab, “Constructing a surveillance impact assessment”,

Computer Law & Security Review, Vol. 28, No. 6, Dec 2012, pp. 613-626.

Wright, David, and Kush Wadhwa, “A step-by-step guide to privacy impact assess-

ment”, presentation paper for the second PIAF Workshop, Sopot, Poland, 24

April 2012. http://www.piafproject.eu/ref/A_step-by-

step_guide_to_privacy_impact_assessment-19Apr2012.pdf

Wright, David, Raphael Gellert, Rocco Bellanova, Serge Gutwirth, Marc Langhenrich,

Michael Freidewald, Dara Hallinan, Silvia Venier and Emilio Mordini,”Privacy

Impact Assessment and Smart Surveillance: A State of the Art Report, Delivera-

ble 3.1, SAPIENT Project, May 2013.