societal impact assessment manual and...
TRANSCRIPT
Criteria for Assessing and Mainstreaming
Societal Impacts of EU Security Research Activities.
Coordination and Support Action.
Societal Impact Assessment Manual and
Toolkit
Deliverable 3.1
Trilateral Research & Consulting
April 2014
ASSERT is co-funded by the European Commission under the 7th Framework Programme, theme „security“, call FP7-
SEC-2012-2, work programme topic 6.3.2. „Criteria for assessing and mainstreaming societal impacts of EU security
research activities – Coordination and support action“.
2 D3.1 Societal Impact Assessment Manual ASSERT
Imprint
Responsible project partner:
Trilateral Research & Consulting (TRI)
Author(s):
David Barnard-Wills, Kush Wadhwa, David Wright
Contact:
David Barnard-Wills, [email protected]
ASSERT website
www.assert-project.eu
Version history
Version Date Change/Remark Responsible (person,
beneficiary/function)
0.1 26 March 2014 David Barnard-
Wills/TRI/Author
0.2 23 April 2014
Following review, ex-
panded introduction,
added section four
David Barnard-
Wills/TRI/Author
0.3 23 April 2014 Final quality review Kush
Wadhwa/TRI/Author
3 D3.1 Societal Impact Assessment Manual ASSERT
EXECUTIVE SUMMARY
The focus of WP3 is to operationalise the assessment of societal impacts of security
research through a structured approach, drawing upon the outcomes of WP1. Task 3.1.
and 3.2 are interlinked, resulting in a highly usable and effective assessment tool de-
ployed via the web. The toolkit itself is an interactive, visual representation of the SIA
manual that is developed in task 3.1. This deliverable sets out the following:
The first section of the report sets out a state-of-the art-methodology for societal impact
assessment for security research. This is a structured methodology for conducting a
societal impact assessment (SIA) of security research and security measure implemen-
tation. The section first provides an overview of the need for and role of societal impact
assessment, then presents an account of the existing impact assessment methodologies
that have influenced this guide. Section two describes the core methodology based up-
on an interactive approach to six key sectors of impact, then provides analytical ques-
tions for use in this process, before setting out a step-by-step process guideline. This
guideline includes guidance on identifying stakeholders and incorporating best practice
in impact assessment. The third section provides guidance on the content of a societal
impact assessment report. The section concludes with recommendations as how to best
embed such a methodology within the broader security research process.
The second section of the report presents the findings of our review of existing decision
support tools and resources (decision tools, learning management systems, online
guides and handbooks, and EU decision support projects). This review was conducted
to support the development of the ASSERT toolkit. It is critical to understand existing
potential solutions, including platforms that might be adapted or customised, in order
to make the appropriate tool selection and to avoid wasted effort. Our review consid-
ered the inherent usability of the tool in relation to societal impact assessment, as well
as fit with the objectives of the ASSERT project. The analysis of existing tools for deci-
sion support in this section has supported the decision to develop the toolkit on the
basis of a combination of online guide or handbook combined with a learning manage-
ment system.
The third section presents a summary of the ASSERT online societal impact assessment
for security research toolkit, including its contents and structure as well as the key de-
sign decisions underpinning the toolkit
4 D3.1 Societal Impact Assessment Manual ASSERT
Table of Contents
1 Introduction 6
2 A State-of-the-art methodology for Societal Im-pact Assessment for security research
7
2.1 Introduction 7
2.1.1 Societal Impacts of security research 7
2.1.2 Influences of societal impact assessment 10
2.2 Step-by-step guide for SIA in security research 13
2.2.1 Methodology 14
2.2.2 Process 17
2.2.3 The societal impact assessment report 31
2.3 Conclusions and recommendations 36
3 Review of existing decision support tools 38
3.1 Decision support tools 38
3.1.1 D-SIGHT 38
3.1.2 VISA 39
3.1.3 Web-HIPRE 40
3.1.4 1000minds 41
3.1.5 Transparent Choice 43
3.1.6 SMART 44
3.2 Learning support environments/learning management systems
46
3.2.1 Moodle 46
3.2.2 WordPress + plugins 47
3.2.3 WordPress + multiple plug ins 49
3.2.4 Big Blue Button 49
3.2.5 OpenMOOC 49
3.2.6 Blackboard 50
3.2.6 EdX 50
3.3 Handbooks and guides 51
3.3.1 IDEO HCD toolkit 51
3.3.2 Research toolkit 52
3.3.3 Crisis Guide: Iran 54
3.4 EU Decision support projects 55
3.4.1 SIAM 55
3.4.2 EURO summer school 57
5 D3.1 Societal Impact Assessment Manual ASSERT
3.4.3 MCDA-RES 57
3.4.4 DESSI 58
3.4.5 DREAM 59
3.4.6 FIRST 59
3.4.7 DISASTER 60
3.4.8 ESS 61
3.5 Analysis and conclusions 61
4 The ASSERT online toolkit 64
5 References 71
6 D3.1 Societal Impact Assessment Manual ASSERT
1. Introduction
The focus of WP3 is to operationalise the assessment of societal impacts of security
research through a structured approach, drawing upon the outcomes of WP1. Task 3.1.
and 3.2 are interlinked, resulting in a highly usable and effective assessment tool de-
ployed via the web. The Toolkit itself is an interactive, visual representation of the SIA
manual that is developed in task 3.1. This deliverable sets out the design process for the
ASSERT toolkit. This process starts with a methodology for societal impact assessment,
and then through a review of existing tools, develops a set of design requirements for
the ASSERT toolkit, which leads to the selection, development and testing of the final-
ised ASSERT Toolkit.
Section 2 features a state of the art methodology for societal impact assess-
ment for security research. This methodology is based upon the review of existing
methods of social impact assessment produced in D1.2, the best practice guidelines for
societal impact assessment in D1.3, and existing impact assessment methodologies. The
syncretised method set out in this section is intended to guide a user through planning
and conducting a social impact assessment exercise for a security research project. It
also contains guidance on the best way to record and report the process.
In support of developing the ASSERT Toolkit, section 3 documents a review of exist-
ing tools for supporting decision making and impact assessment, encom-
passing FP7 decision support projects, online and offline handbooks, decision support
tools, and learning support tools. This review provided the consortium with a grounded
understanding of the existing field, and potential tools that could 1) be used as the basis
for the ASSERT Toolkit, in terms of features and appropriate functionality, and 2) other
tools that might be included in or referenced by the toolkit. The conclusions of the tool
review provided the direction for the best way to manifest the societal impact assess-
ment methodology in the form of an online tool. This review guided the decision to
construct an online handbook to guide users through the impact assessment methodol-
ogy, whilst providing additional support and guidance material in an accessible form,
and which could also provide integrated support to the other outward-facing elements
of the ASSERT project – the Masterclass concept and the expert database.
The third stage of the design process, the creation of the ASSERT online toolkit is
documented in section 4. This section presents the logical structure and content of the
finished toolkit design choices, and structure of the ASSERT Toolkit. This section in-
cludes the key design decisions that were made on the basis of the preceding sections as
well as the requirements that emerged from the ASSERT project more broadly. It also
explains how the selected software packages (WordPress and Moodle) were adapted for
the needs of the ASSERT toolkit. The ASSERT online toolkit is a combination of an
online societal impact assessment “handbook”, with multimedia content, structured in
an accessible manner, with the additional features provided by a Leaning Management
System that offers additional tools to registered users. This combination allows the
consortium to support the Masterclass experience with online content, to make infor-
mation on societal impact assessment in security research accessible for a wider range
of users, and also to host the ASSERT expert database.
7 D3.1 Societal Impact Assessment Manual ASSERT
2. A State-of the-art methodology for Societal Impact As-
sessment for security research
This section sets out a structured methodology for conducting a societal impact as-
sessment (SIA) of security research and security measure implementation. The section
first provides an overview of the need for and role of societal impact assessment, then
presents an account of the existing impact assessment methodologies that have influ-
enced this guide. Section two describes the core methodology based upon an interactive
approach to six key sectors of impact, then provides analytical questions for use in this
process, before setting out a step-by-step process guideline. This guideline includes
guidance on identifying stakeholders and incorporating best practice in impact assess-
ment. The third section provides guidance on the content of a societal impact assess-
ment report. The paper concludes with recommendations as how to best embed such a
methodology within the broader security research process. The methodology has par-
ticular relevance for security research within the European Union.
2.1 Introduction
This section sets out a structured methodology for conducting a societal impact as-
sessment (SIA) of security research and security measure implementation. The paper
first provides an overview of the need for and role of societal impact assessment, then
presents an account of the existing impact assessment methodologies that have influ-
enced this guide. Section two describes the core methodology based upon an interactive
approach to six key sectors of impact, provides analytical questions for use in this pro-
cess, before setting out a step-by-step process guideline, including guidance on identi-
fying stakeholders and incorporating best practice in impact assessment. The third sec-
tion provides guidance on the content of a societal impact assessment report. The paper
concludes with recommendations as to how best embed such a methodology within the
broader security research process. The methodology has particular relevance for securi-
ty research within the European Union, however as the approach is based upon princi-
ples and methodology rather than legal compliance, it therefore has a wider potential
applicability.
2.1.1 Societal impacts of security research
Security research and innovative application of security technologies is a complex and
heterogeneous field that pulls together various academic disciplines and different types
of organisation. Several features characterise the field of security research. These fea-
8 D3.1 Societal Impact Assessment Manual ASSERT
tures include ethical and social issues that have been identified by different research
fields.
Security itself is socially and institutionally valued, attracting significant funding
streams and market attention. The security industry is regarded as an important eco-
nomic sector for Europe.1 Security has a particular institutionalisation within policy-
making communities.2 Security research programmes, such as the EU’s Secure Socie-
ties3 effort, therefore take place within these social, economic and political contexts.
Identifying some feature of the world as a security issue is to grant it a particular privi-
leged political status and to start to frame the issue in a specific way, deserving political
and social responses and identifying it as the responsibility of certain political actors.4
This framing of what counts as a security risk is known as securitization and has been
the subject of considerable research within international relations security studies.5
Similarly, security practices, including research and innovation, can contribute to the
normalisation of security6, with security becoming an organising principle across many
areas of social life, sometimes to the detriment of other values or principles, such as, for
example, privacy, transparency, freedom of speech or the democratic process. Security
impacts can be powerful, and frequently distributed unevenly across society (for exam-
ple, border security measures may be intended to increase national security, but can
increase insecurity for asylum seekers and immigrants). Some groups are particularly
vulnerable to the negative impacts of security research and implementation and often
excluded from decision-making processes. These groups can suffer from cumulative
disadvantage, which in turn can have negative implications for social cohesion.7 Securi-
ty processes often have the potential to negatively impact citizen’s fundamental rights.
Security (and security research) is part of the political process; however, societies often
experience periods of delay between the societal impacts of security policy and inter-
vention, social awareness of these impacts, their examination in the democratic process
1 McCarthy, Sabhbh, Report of the Societal Impact Expert Working Group: EC DG ENTR Re-port, February 2012. 2 Chilton, P., Security Metaphors: Cold War Discourse from Containment to Common House, Peter Lang, New York, 1996, p. 23. 3 European Commission, “Secure Societies – protecting freedom and security of Europe and its citizens”. http://ec.europa.eu/programmes/horizon2020/en/h2020-section/secure-societies-%E2%80%93-protecting-freedom-and-security-europe-and-its-citizens 4 Buzan, Barry, Ole Weaver and Jaap de Wilde, Security: A New Framework for Analysis, Lynne Rienner, London, 1998, pp. 25-6; Deibert, Ron, “Circuits of Power: Security in the Internet En-vironment” in J.P. Singh and James N. Rosenau (eds.), Information Technologies and Global Politics: the Changing Scope of Power and Governance, Suny Press, New York, 2002, pp. 115-142 [p. 115]; Neocleous, Mark, “Security, Liberty and the Myth of Balance: Towards a critique of security politics”, Contemporary Political Theory, Vol. 6, Issue 2, May 2007, pp. 131-149 [p. 144]. 5 Buzan, Barry, and Lene Hansen, The Evolution of International Security Studies, Cambridge University Press, Cambridge, UK, 2009, p. 142. 6 Nissenbaum, Helen, Privacy in Context: Technology, Policy, and the Integrity of Social Life, Stanford Law Books, Stanford, 2009, p. 161. 7 Lianos, Michaelis, “Dangerization and the End of Deviance: The Institutional Environment”, British Journal of Criminology, Vol. 40, No. 2, Spring 2000, pp . 261-278. http://bjc.oxfordjournals.org/content/40/2.toc
9 D3.1 Societal Impact Assessment Manual ASSERT
and responses to them from law and regulation. During this lag, security practices and
technologies can become entrenched in areas of social life making it difficult to dislodge
them. A related problem is function creep, where security technologies installed for one
purpose are used for secondary, initially unintended or unstated uses. Misalignment
and disjunction between agencies and actors with responsibility for promoting security
technology, and those with responsibility for assessing, controlling and in some cases
limiting and preventing negative impacts, can also exacerbate these situations.8
Security research, especially in the fields of crime prevention and national security,
often has less transparency than in areas where national interests are not seen as af-
fected in such an immediate way, and governmental bodies are less centrally involved.
There is limited public access to knowledge about the research. This often serves to
exclude stakeholders, whilst at the same time limiting the acceptability of security
measures, causing them to be viewed with suspicion.
It is important for people and institutions involved in both security research and the
innovative application of security technologies to consider the wide range of possible
societal impacts of these activities, as a response to the above political and social issues,
and as part of maximising the positive social benefits of security research whilst mini-
mising the negative effects. Societal impact assessment therefore has a critical role in
the security research and implementation process.
SIA is the process of understanding, managing and responding to the societal impacts
that arise from security research and the application of innovative security measures.
The use of the term societal (rather than social) connotes the inclusion of anything af-
fecting human, natural or artefactual systems, rather than just those effects that impact
upon humans and their interactions. It also allows us to distinguish the process from
social impact assessment, as discussed below.
From a positive perspective, societal impact assessments can also provide a better un-
derstanding of the productive and socially desirable impacts that arise from security
research, including how best to maximise these contributions. Conducting impact as-
sessment exercises as part of security research and innovation contributes towards the
evidence base for these activities. SIA can contribute towards understanding the socie-
tal impact of larger scale research funding frameworks and policies, including their
contribution toward policy objectives (for example, the partial goal of European Union
security research funding in boosting the competitiveness of the EU security industry).
Security research and innovation are, by definition, intended to produce, encourage or
support security as societal impact. An inclusive definition of security includes those
practices and technologies aimed at strengthening social bonds and social resilience
using social policy tools as well as just preventive measures against particular threats.
This understanding of security aligns with the concept of human security.9 Rather than
8 Rip, Arie, and Johan Schot, “The Past and Future of Constructive Technology Assessment”, Technological Forecasting and Social Change, Vol. 54, Issues 2–3, Feb-Mar 1997, pp. 251-268. http://www.sciencedirect.com/science/journal/00401625/54/2-3
10 D3.1 Societal Impact Assessment Manual ASSERT
conceptualising security as an outcome of using security technologies, security is con-
ceptualised here as one of the societal impacts of security measures. Although security
research and the implementation of security measures have internal differences, this
paper discusses them together, as they are both amenable to the assessment approach
set out here.
2.1.2 Influences on societal impact assessment
Societal impact assessment for security research has not emerged in a vacuum. SIA
approaches draw upon a range of previously established methodologies and practices,
both in security research and related areas. These influences include constructive tech-
nology assessment (CTA), privacy impact assessment (PIA) and recent developments in
surveillance impact assessment (SuIA), social impact assessment and the impact as-
sessment activities of the European Commission. There is some limited influence from
the responsible research and innovation debate and from research ethics, although this
is primarily concerned with the way that research is conducted, rather than concerns
about the goals and subjects of the research.10
Constructive technology assessment is defined by Rip and Schot as an approach that
"shifts the focus away from assessing impacts of new technologies to broadening de-
sign, development, and implementation processes",11 the aim of which is to contribute
“better technology to a better society”. By anticipating impacts, involving users and
stakeholder communities in the design process in an interactive manner and by har-
nessing social learning, it is possible to avoid the human costs associated with trial and
error social responses to new technologies. Social aspects of technologies are explicitly
included as design criteria.12 CTA claims no inherent normative agenda, rather focusing
upon expanding the reflexivity of the design process, however it serves to close conflicts
and increase acceptance. Tools and procedures of CTA include understanding the in-
novation journey, identifying points of intervention at different phases of a project,
anticipating the outcome of a research process, reflexivity exercises, stakeholder work-
shops, technology forcing13 and strategic niche alignment.14
Privacy impact assessment is a methodology for assessing the impacts on privacy of a
project, policy, programme, service, product or other initiative that involves the pro-
cessing of personal information and in consultation with stakeholders for taking reme-
10 McCarthy, 2012, p. 10 11 Rip and Schot, 1997. p. 251 12 Ibid, p. 251 13 Genus, Audley, “Rethinking constructive technology assessment as democratic, reflective dis-course”, Technological Forecasting and Social Change, No. 73, No. 1, 2006, pp. 13-26 [p. 19]. http://www.sciencedirect.com/science/journal/00401625/73/1 14 Schot, Johan and Frank W. Geels, “Strategic niche management and sustainable innovation journeys: Theory, findings, research agenda and policy”, Technology Analysis & Strategic Man-agement, Vol. 20, No. 5, 2008. pp. 537-554.
11 D3.1 Societal Impact Assessment Manual ASSERT
dial action as necessary in order to avoid or minimise negative impacts.15 Surveillance
Impact Assessment is an expansion of privacy impact assessment that takes into ac-
count a wider range of issues, impacts and stakeholders across the surveillance sys-
tem.16 The aims of PIA and SuIA are better privacy protection, increased transparency
of personal information processing technologies and increased accountability (and for
SuIA mitigation of the risks of surveillance systems). It is also intended to engage
stakeholders and affect the direction of a project from its earliest stages. PIA has been
adopted across several countries. 17
Potentially the broadest of the influential approaches, social impact assessment as un-
derstood by Vanclay is a participatory process for the analysis, monitoring and man-
agement of the intended and unintended, positive and negative consequences of
planned interventions.18 The goal is a more sustainable and equitable physical and hu-
man environment, and key methods include increasing awareness of potential unin-
tended consequences, through the exploration of the likely impacts of a research pro-
ject on people’s lives, cultures, communities, political systems, environment, health and
wellbeing, personal and property rights, fears and aspirations.
As part of the legislative process, the European Commission undertakes impact as-
sessments of policies, legislation, trade agreements and other measures. This includes
publishing roadmaps and economic, social and environmental impact assessments of
planned initiatives prior to EU action. Post-intervention, this is followed by evaluation
of the performance of initiatives and by subjecting them to regulatory fitness and per-
formance (REFIT) assessment. The Commission seeks public consultation throughout
the process.1920
The three theoretical approaches share reflexivity as core principles, and all four ap-
proaches have a strong focus on participation. Similarly, the approaches all share the
perspective that impact assessments should be initiated at the earliest possible stages of
a project or intervention, and conducted as an ongoing activity. Social impact assess-
ment contributes the need for an awareness of societal dimensions and how they im-
pact the security research and development process. It highlights stakeholder dissatis-
faction with the reports of existing social impact processes, especially post-impact stud-
15 Wright, David and Paul de Hert, “Introduction to Privacy Impact Assessment”, in David Wright and Paul de Hert (eds.), Privacy Impact Assessment, Springer, Dordrecht, 2012, pp. 3-33 (p. 5). 16 Wright, David and Charles Raab, “Constructing a surveillance impact assessment”, Computer Law & Security Review, Vol. 28, No. 6, Dec 2012, pp. 613-626. 17 Wright, David, Raphaël Gellert, Rocco Bellanova, Serge Gutwirth, Marc Langhenrich, Michael Freidewald, Dara Hallinan, Silvia Venier and Emilio Mordini, Privacy Impact Assessment and Smart Surveillance: A State of the Art Report, Deliverable 3.1, SAPIENT Project, May 2013. 18 Vanclay, Frank, “Social Impact Assessment: International Principles”, IAIA, Special publica-tion series No.2, May 2003, p. 2. http://www.iaia.org/publicdocuments/special-publications/sp2.pdf 19 European Commission, “Smart Regulation”, 13 January 2014. http://ec.europa.eu/smart-regulation/index_en.htm 20 For an early review of this process, see: Lee, Norman and Colin Kirkpatrick, “Evidence-based policy making in Europe: an evaluation of European Commission integrated impact assess-ments”, Impact Assessment and Project Appraisal, Vol. 24, No. 1, 2006, pp. 23-33.
12 D3.1 Societal Impact Assessment Manual ASSERT
ies, and the importance of a social impact assessment as a tool and investment in risk
management.21 CTA recommends researchers exercise critical reflexivity and empha-
sises the prominent role of societal issues in research consortia.22 The European Com-
mission’s impact assessment process – which currently is applicable only to policies
developed by EU institutions - demonstrates the value of publishing impact assess-
ments and building a common methodology within a sector. Surveillance impact as-
sessments already include privacy and ethical impact assessments and are therefore a
suitable model for rigorous impact assessment. The following table shows the influ-
ences from existing approaches that have contributed to the Societal Impact Assess-
ment approach set out in this paper.
Table 1: combination of elements across impact assessment (source: authors)
Elements of Societal Im-pact Assess-ment
Influences from existing approaches
Social Impact Assessment
Privacy Impact assessment
Constructive technology Assessment
European Impact assessment
Including all human, natu-ral and arte-factual sys-tems
Human interac-tions/social di-mensions
Inclusion of social science in technology assessment
Critical reflex-ivity of as-sessment
Reflexivity to-wards societal dimensions
Critical reflex-ivity
Stakeholder involvement
Stakeholder in-volvement
Stakeholder in-volvement
Stakeholder involvement
Stakeholder in-volvement
Publishing Publishing
Common methodology
Common methodol-ogy
Societal impact assessment as a broad category of practice, therefore includes the other
approaches, as shown in the following diagram. The existing approaches are all forms
of societal impact. In this paper we draw upon these existing best practices to we pre-
sent a composite approach that is applicable to security research.
21 Vanclay, Frank, and Ana M. Esteves (eds.), New Directions in Social Impact Assessment: Conceptual and Methodological Assumptions, Edward Elgar, Cheltenham, 2011 , p. 11 22 Rip, Arie, and Harro van Lente,“Bridging the gap between innovation and ELSA: The TA pro-gram in the Dutch Nano-R&D Program Nano Ned”,Nanoethics, Vol. 7, Issue 1, April 2013, pp. 7-16. http://link.springer.com/journal/11569/7/1/page/1
13 D3.1 Societal Impact Assessment Manual ASSERT
Figure 1: Societal Impact Assessment methodologies (source authors)
2.2. Step-by-step guide for SIA in security research
The objective of the SIA process is to increase the reflexivity of the process and of deci-
sion-making, not to pre-script or pre-define the results of an assessment process. Re-
flexivity in this context is the ability of researchers to take stock of their role in the re-
search process, and subject their research activity to the same level of critical scrutiny
as the rest of their “data”.23 Reflexivity is a process of critical reflection both on the
kind of knowledge generated from research and on how that knowledge is generated.24
Reflexivity and an openness to alternatives is crucial to act upon the “irritation” that
SIA is capable of creating by ensuring that those results that challenge core assump-
tions of the planned project, technology or policy can also have an impact on the plan-
ning process. One cannot assume a simplified binary process in which security technol-
ogies are invented, and then go on to have implications and impacts.25 Innovation stud-
ies have established the observation that innovation frequently does not occur in a line-
ar process but in a complex and uncertain process of trial, error and unintended devel-
opments.26 As with PIA,27 one-size does not fit all with regard to SIA, and this method-
ology should be adaptable to the specific needs and contexts of a given research project
or innovation. Using a shared framework and accepted common approach increases the
transferability across domains of the assessment, and is likely to increase external con-
fidence in the process. The offered approach is rigorous and detailed. This is because a
23 Mason, Jennifer, Qualitative Researching, Sage, London, 1996. 24 Guillemin, Marilys, and Lynn Gillam, “Ethics, Reflexivity and “ethically Important” moments in research”, Qualitative Inquiry, Vol. 10, No. 2, 2004, pp. 261-280. 25 Rip, Arie, and Johan Schot, “Identifying Loci for Influencing the Dynamics of Technology Development”, in Knut H. Sorensen and Robin Williams (eds.), Shaping Technology, Guiding Policy: Concepts, Spaces and Tools, Edward Elgar, Cheltenham, 2002, pp. 155-172. 26 Braun-Thürmann, Holger, Innovation, Transcript, Bielefield, 2005. 27 Wright et al., 2013.
14 D3.1 Societal Impact Assessment Manual ASSERT
security research project is often operating in relatively unknown or conjectured ter-
rain. The additional information produced for such a project by a detailed impact as-
sessment supports better understanding of the security intervention being researched.
The following section first presents the core elements of the assessment methodology,
before situating this methodology as part of a structured societal impact assessment
process.
2.2. 1 Methodology.
The core of the methodology proposed is based upon analysis driven by interaction
with the stakeholders and posing a series of questions that enable the discovery of vary-
ing views on impacts. This enquiry is divided into six different aspects of societal im-
pacts as shown in the figure below, and the enquiry is completed in an approach com-
monly used in curriculum design28 (moving from basic questions to increasingly more
complex ones as more is learned, cycling or spiralling back through the same topics
multiple times).
Figure 2: Methodology (source: authors)
28 “Spiral Curriculum” was originally suggested by Jerome Bruner. See Bruner, Jerome, The Process of Education. The President and Fellows of Harvard College, Cambridge, MA, 1960.
15 D3.1 Societal Impact Assessment Manual ASSERT
The various aspects of societal impact that are evaluated include those suggested by
Vanclay (attributed in part to ideas developed by Armour)29, which have been com-
bined here into six main groups (Vanclay separates culture from community as well as
way of life from fears and aspirations):
• Way of life, fears and aspirations (how people live and interact
with each other on a daily basis, their perceptions about their safety and
that of their communities, and their aspirations for future, including the
future of their children);
• Culture and community (people’s shared beliefs, customs, values
and languages, the cohesion, stability and character of their communi-
ties);
• Political systems (participation in the decisions and processes that af-
fect people’s lives, the nature and functioning of democratic processes,
and the resources available to support people’s involvement in these);
• Environment (access to and quality of air, water, and other natural re-
sources, the level of exposure to pollutants and harmful substances, ad-
equacy of sanitation);
• Health & well-being (physical and mental wellbeing, not just an ab-
sence of infirmity);
• Personal and property rights (economic effects, civil rights and lib-
erties, personal disadvantage).30
The assessor should start the SIA evaluation process by looking at any one of the above
aspects in three different dimensions:
1) First examine whether the security research project meets the needs of soci-
ety;
2) Iterating a second time through the same six aspects, review the potential
externalities or costs to society, enumerating risks and identifying ways to
mitigate them;
3) Finally, pass through the six societal impact aspects a third time to identify
potential benefits to society.
With each re-evaluation of the six aspects, the assessor and stakeholders have the op-
portunity to rethink answers to previous questions based upon answers across each of
the earlier assessments, providing an opportunity to go back and rethink impacts where
new information has been brought to light or where new ideas have emerged.
The following set of questions provides a basis for examining the social needs, potential
costs and risks, and potential benefits of security research. The questions are based
upon the societal impact checklist for R&D developed by the Societal Impact Expert
29 Vanclay, Frank, “Conceptual and methodological advances in social impact assessment” in Henk A. Becker and Frank Vanclay (eds.), The International Handbook of Social Impact As-sessment: Conceptual and Methodological Advances, Edward Elgar, Cheltenham, 2003, p. 7. 30 Vanclay, F., “International Principles for Social Impact Assessment”, Impact Assessment and Project Appraisal, Vol. 21, No. 1, 2002, pp. 5-11.
16 D3.1 Societal Impact Assessment Manual ASSERT
Working Group in their report to the European Commission’s Directorate-General for
Enterprise and Industry.31
Table 2: Assessment questions (source: authors)
Me
ets
ne
ed
s o
f so
cie
ty?
1. Which documented societal security need(s) does the proposed research
address? (e.g., life, liberty, health, employment, property, environment,
values).
2. How will the research output meet these needs? How will this be demon-
strated? How will the level of societal acceptance be assessed?
3. Is the research project aware of challenges to these needs?
4. Does addressing the documented societal needs through the proposed
research require any trade-offs with other documented societal needs?
How is this trade-off decided? Is this trade-off still valid if the research is
less effective than anticipated?
5. What threats to society does the research address? (e.g., crime, terror-
ism, pandemic, natural and man-made disasters).
6. How is the proposed research appropriate to address these threats?
7. What other measures could be adopted to address these threats?
31 McCarthy 2012, pp. 17-18
17 D3.1 Societal Impact Assessment Manual ASSERT
En
su
rin
g s
ec
ur
ity
re
se
ar
ch
do
es
no
t h
av
e n
eg
ati
ve
im
pa
cts
on
so
cie
ty
8. How could the research have a negative impact on human dignity?
9. … on the right to life?
10. … on equality before the law?
11. … on freedom of thought?
12. … on freedom of opinion and information?
13. … on privacy?
14. … on protection of the family?
15. … on freedom of movement?
16. … on rights of ownership?
17. … on freedom of assembly?
18. … on freedom to choose an occupation?
19. … on working conditions?
20. … on collective social rights?
21. … on social welfare?
22. … on rights to an education?
23. … on the principle of democracy?
24. … on rights of access to information?
25. … on rights of access to the courts?
26. … on access to public space?
27. If implemented, how could the research have a negative impact on this
aspect (culture and community, way of life, etc.)?
28. How could the research impact disproportionately upon specific groups
or unduly discriminate against them? How could the research increase
discrimination?
Could the research have impacts upon vulnerable groups (including, but
not limited to: women, the elderly, disabled people, children and young
adults, homeless people, economically disadvantaged people and people
in precarious situations, immigrants or non-citizens, and lesbian, gay,
bisexual, transgender or queer (LGBTQ+) identifying people.
En
su
rin
g s
ec
ur
ity
re
se
ar
ch
be
ne
fits
so
cie
ty
29. What segment(s) of society will benefit from increased security as a re-
sult of the proposed research?
30. How will they benefit?
31. Are additional measures required to achieve this benefit?
32. Are additional measures possible to extend these benefits to other seg-
ments of society?
33. In what contexts might this benefit be lacking or not be delivered by the
research project?
34. How will society as a whole benefit from the proposed research?
35. Are there other European societal values that are enhanced by the pro-
posed research, e.g., public accountability and transparency; strength-
ened community engagement, human dignity; good governance; social
and territorial cohesion; sustainable development.
2.2.1 Process
To effectively employ this evaluation methodology, it should be put in the context of the
following 15-step process:
18 D3.1 Societal Impact Assessment Manual ASSERT
Table 3: Steps in the SIA process (source: authors)
Preparation
and plan-
ning
1. Identify the SIA team and set the team’s terms of reference, re-
sources and time frame.
2. Prepare the SIA plan.
3. Determine the budget for the SIA.
4. Describe the project to be assessed.
5. Identify stakeholders.
Consultation
and analysis
6. Conduct the spiral assessment of the six core areas of societal im-
pact to identify impacts associated with needs, externalities/costs, and
benefits.
7. Consult with stakeholders.
8. Determine whether the project complies with legislation; assess
whether the security research has the potential to generate results that
require new legislation to address potential gaps.
9. Identify risks and possible solutions.
10. Formulate recommendations.
Reporting
and re-
sponding
11. Prepare and publish the report, e.g., on the organisation’s website
and/or in a suitable repository.
12. Implement the recommendations.
13. Ensure a third-party review and/or audit of the SIA.
14. Update the SIA if there are changes in the project.
15. Refer to the SIA in any post-project evaluation.
This process is based upon PIA methods, and shares common features with the EU im-
pact assessment procedure.32 As the project progresses, there is a shift from foresight,
to management, to evaluation. Guidance for each individual step in the process is pro-
vided below.
32 European Commission, “Key procedural steps for the Commission/smart-regulation/impact”, 20 Dec 2013. http://ec.europa.eu/smart-regulation/impact/ia_key/ia_key_en.htm
19 D3.1 Societal Impact Assessment Manual ASSERT
Preparation and planning
This approach acknowledges that for many types of security research, there is a need to
conduct societal impact assessments during the research planning or proposal stage. At
this stage, there may be minimal resources available to support extensive societal im-
pact efforts, including in-depth consultation. Structured consideration of societal im-
pact at this stage can improve the quality of the research design being proposed or con-
sidered, and is crucial to ensure that negative societal impacts are not locked into the
research design.
Identify the SIA team and set the team’s terms of reference, resources and time frame
The research project manager should be responsible for the conduct of a SIA, but she
may need some additional expertise, perhaps from outside her organisation. Depending
on the estimated scale of the SIA, the project manager or the designated societal impact
assessor may need to form a team to undertake the SIA. The team could bring together
expertise from information security experts, lawyers, operations managers, ethicists,
public relations experts, etc. As the SIA progresses, the assessor may find that she
needs still other expertise. The benefits of a dedicated manager and interdisciplinary
team, which ideally includes some social science expertise, is supported by existing
methodologies of social impact assessment.33 Eurobarometer research suggests that the
European public is most receptive to accounts of research that come from researchers
themselves.34
The project manager and/or the organisation’s senior management should decide on
the terms of reference for the SIA team, its budget and its time frame. The assessor may
come under considerable pressure to complete the SIA quickly so as not to delay the
project, but she may need to resist compromising the integrity and adequacy of her SIA
mission and may need to ensure she has the full support of the organisation’s CEO
and/or its management board. The terms of reference should make clear that the socie-
tal impact assessment is a process, and that the process will need to continue beyond
preparation of the SIA report. If the assessor’s work or that of an external consultant
comes to an end with publication of the report, the project manager and/or the organi-
sation’s CEO and/or management board should decide how implementation of recom-
mendations will be monitored and who will be responsible for the monitoring and what
factors will determine whether the SIA report needs to be updated.
For research projects, it may be appropriate to dedicate a work package or stream of
activity to the societal impact role. If this is done, it is important to ensure that this
work package is integrated with the other elements of the project, perhaps through
33 Kemp, Deanna, “Understanding the Organizational Context”, in Frank Vanclay and Ana Maria Esteves, (eds.), New Directions in Social Impact Assessment: Conceptual and Methodological Advances, Edward Elgar, Cheltenham, 2011, p. 30. 34 TNS Opinion & Social, Special Eurobarometer 401: Responsible Research and Innovation (RRI) Science and Technology, European Commission, Directorate General for Communication, November 2013, p. 5.
20 D3.1 Societal Impact Assessment Manual ASSERT
shared personnel or making later elements of the work dependent upon stages of the
SIA report.35
Prepare the SIA plan
The assessor should prepare a plan for conducting the SIA. She can prepare the SIA
plan using this SIA process document, but may need to tailor it to the exigencies of the
project to be assessed. The plan should spell out the objectives of the SIA, what is to be
done to complete the SIA, who on the SIA team will do what, the SIA schedule and,
especially, how the consultation will be carried out. An important part of the plan
should address consultation. It should specify why it is important to consult across each
of the six societal impact areas, who will be consulted and how they will be consulted
(e.g., via public opinion survey, workshops, focus groups, public hearings, online expe-
rience, specialist consultation tools).
The SIA should also include if and how societal impact will be included in any post-
research evaluation activity.
Determine the budget for the SIA
Once the project manager and/or assessor have prepared an SIA plan, they can esti-
mate the costs of undertaking the SIA and seek the budgetary and human resources
necessary from the organisation’s senior management. Unfortunately, the assessor may
be constrained in what she can do in the SIA by the budget allocated by the organisa-
tion. If the assessor is unable to do an adequate SIA, she should note this in her SIA
report. The assessor may need to revise her SIA plan based on the budget available.
In general, the budget for performing an effective SIA will depend upon a number of
factors, including 1) the stage at which an SIA is being performed, and 2) the scale,
scope, and overall complexity of the project. For example, if an SIA is being performed
at the inception of the project, while there remains adequate opportunity to effect
change, it may be more cost-effective than if the SIA is performed after a great deal of
research, design, and development has been completed.
As a matter of practical estimation, it can be expected that the SIA will require budget
for the salary of a senior level consultant and a junior level consultant for approximate-
ly one person-month, presuming that the consultants are experienced in performing
SIAs and already come to the task armed with procedures and templates to perform it.
If so equipped, they will use some of their time to understand the project and subse-
quently survey or interview key stakeholders before completing their analysis. If this is
the first time an organisation is doing an SIA, they will need to take an extra two or
three months to develop their process before beginning the work of the SIA itself. Ex-
perienced consulting companies will have already completed this work, and of course,
35 In the course of the SIA, it may become apparent to the assessor that the organisation needs to spend more time on raising the awareness of employees (including researchers) about societal impact issues. The background context section of the report can be used to state what the organ-isation does now to raise employee awareness of societal impacts, and where it could improve.
21 D3.1 Societal Impact Assessment Manual ASSERT
there will likely be additional costs of addressing the outcomes of the SIA (potentially
greater if the SIA is performed later in the research and development cycle).
Describe the project to be assessed
The assessor should describe the project or technology or service to be assessed. As the
development of the project or technology or service may still be at an early stage, there
may not yet be that much known about the project. The assessor can update the de-
scription as more becomes known. The description can be used in at least two ways – it
can be included in the SIA report and it can be used as a briefing paper for consulting
stakeholders. The description of the project should provide some contextual infor-
mation (why is the project being undertaken, how is it funded, who are the project
members and participants, who are the intended audiences for the findings of the re-
search, how does it relate to other ongoing security research activity conducted by the
project members). The project description should state who is responsible for the pro-
ject. It should indicate important milestones and, especially, when decisions are to be
taken that could affect the project’s design.
Identify stakeholders
A critical component of societal impact assessment is the participative inclusion of
stakeholders in the assessment of the security research project or security measure ap-
plication. The assessor should identify stakeholders, i.e., those who are or might be in-
terested in or affected by the project, technology, service or other initiative. The stake-
holders could include people who are internal as well as external to the organisation.
Involving a variety of stakeholders provides an opportunity for any potential risks to be
highlighted and eventually managed. Given the potential breadth of societal impact
across different categories of impact, the way that “stakeholders” is understood should
be broad and inclusive.
Kemp provides a list of parties who could potentially be affected by a planned project or
policy and should thus be engaged within the context of SIA.36 Building upon this list,
and customising it for the security research process provides the following summary:
• personnel or managers with carriage of the social agenda within project propo-
nent organisations;
• researchers, designers, engineers, developers, potential suppliers, security ex-
perts and others who will carry out the research activity;
• assessors who are commissioned to undertake or facilitate the societal impact
assessment process, either internally or from outside of the organizational
structure of the project proponent;
• project-affected peoples, up to and including representatives of the general pub-
lic
• regulators;
• civil society organisations, including civil rights advocates;
36 Kemp, 2011, pp. 21-2
22 D3.1 Societal Impact Assessment Manual ASSERT
• the media;
• academics;
• businesses.
The assessor should identify these different categories and then identify specific indi-
viduals from within each category, preferably as representative as possible. The stake-
holders of a project, and particular people who might be affected by the project, will be
dependent upon the context of the project and the way in which it is conducted. This
means that the above list cannot be inclusive and the project assessor must make ef-
forts to identify other stakeholders as appropriate. Social relations are complex, and
stakeholders, especially those affected by unintended consequences, may not be appar-
ent to the SIA team. The SIA process should therefore include opportunities for indi-
viduals, groups and organisations to self-identify as stakeholders and request participa-
tion in the assessment activity. Some stakeholders may only become apparent as the
SIA progresses. If necessary or useful, they too should be brought into the consultation
process. The range and number of stakeholders to be consulted should be a function of
the likely societal impact as identified in early stages of the spiral methodology, includ-
ing the number of people who could be affected. Thus, the number of stakeholders to be
consulted could be relatively limited if the project or service is also expected to be
small, e.g., the project or service might involve only employees of a small or medium-
size enterprise. The proper involvement of stakeholders may require additions to the
SIA team, either to encourage participation by stakeholders, or to bring key stakehold-
ers, who may be most affected by the project into the SIA team directly.
Consultation and analysis
The four stages in this section involve the analysis of societal impacts using an iterative
process based upon the methodology, and in concert with identified stakeholders, that
looks at each of six aspects of societal impacts as described above.
Consult with stakeholders
The project manager and/or societal impact assessor should enter a dialogue with as
many stakeholders as appropriate or meaningfully possible (taking into account the
available budget). There are many reasons for doing so, not least of which is that they
may identify some societal risks not considered by the project manager or assessor. By
consulting stakeholders, the project manager may forestall or avoid criticism that they
were not consulted. If something does go wrong downstream – when the project or
technology or service is deployed – an adequate consultation at an early stage may help
the organisation avoid or minimise liability. Furthermore, consulting stakeholders may
provide a sort of “beta test” of the project or service or technology. Consulted stake-
holders are less likely to criticise a project than those who were not consulted. The ear-
lier a consultation process is entered into, the more benefits an organisation can expect
to draw from it as any learning produced can be integrated into the project more rapid-
ly, and additional information may help to avoid unanticipated problems in the project.
There are several different ways of consulting stakeholders and the assessor should
consider which will be most appropriate in the circumstances. The assessor or other
members of the SIA team could interview stakeholders directly. They could convene
23 D3.1 Societal Impact Assessment Manual ASSERT
workshops of experts or stakeholder representatives. They could hold focus groups of
ordinary consumer-citizens. They could conduct surveys by telephone or e-mail or face
to face. They could post the project description on the organisation’s website and invite
comments. They could hold public hearings where they describe the project and invite
comments from the audience or from experts and then invite comments after the ex-
perts have spoken. They could prepare stories or adverts in the media and invite com-
ments from readers. They could conduct a Delphi survey of experts, to query them on
potential societal risks now and in the future.37 There are current EU-funded research
projects which are working to produce structured methods for consultation in security
research.38
Social impact assessment has attracted a range of cautionary comments in relation to
community participation. That participation can become a tokenistic exercise, can be
inconsistent, often does not involve enough participation in actual decision-making,
being reduced to a form of consultation.39 Moreover, community participation is some-
times used as a seeming quick fix for problems, without addressing the root of the issue
(which often lies in unequal power distributions that are deeply engrained in polities,
and sometimes in the very institutions of the community whose participation is
sought). As a result, community participation sometimes takes the form of a tokenistic
exercise; once “affected communities” – or their leaders – have been heard, the respec-
tive box can be ticked off, and afterwards the rule of previous power relations re-
sumes.40 To avoid this, the participation of stakeholders in an SIA exercise should be
dialogic and a partnership, and go beyond simply writing down what stakeholders have
to say. Stakeholders should have access to the researchers and ideally be able to exer-
cise some influence over the direction of the project.
Compliance with legislation
A societal impact assessment for security research is more than a compliance check; nevertheless, the assessor or her legal experts should ensure that the project complies with any legislative or regulatory requirements. These may be high level laws relevant across contexts, such as (at the European level) the European Convention on human rights41 and the EU Charter of fundamental rights42 as well as more specific laws, regu-
37 For a longer list of possible techniques, see OECD, Stakeholder Involvement Techniques, ISBN 92-64-02087-X, Paris, 2004, pp. 30-32 [Box 2. Commonly cited techniques for informing deliberation through stakeholder involvement] 38 see http://securitydecisions.org/about-dessi/, and http://www.siam-project.eu/ 39 Peterman, William, “Advocacy vs. collaboration: comparing inclusionary community planning models”, Community Development Journal, Vol. 39, No. 3, 2004, pp. 266-276; O’Faircheallaigh, Ciaran, “Public participation and environmental impact assessment: Purposes, implications and lessons for public policy making”, Environmental Impact Assessment Review, Vol. 30, Issue 1, January 2010, pp. 19-27 [p. 19]; Müth, Matthias, Verkherspolitik in Metropolen Südostasiens (the politics of traffic in South East Asian Metropolis areas), Abera, Hamburg, 2000; Bishop, Patrick, and Glyn Davis, “Mapping public participation in policy choices”, Aus-tralian Journal of Public Administration, Vol. 61, No. 1, 2002, pp. 14-29. 40 Peterman, 2004. 41 http://www.echr.coe.int/Documents/Convention_ENG.pdf
24 D3.1 Societal Impact Assessment Manual ASSERT
lations, codes and guidelines applicable to the specific context and aims of the project being assessed. Research projects attracting institutional support will also have to com-ply with relevant standards and criteria selected by funding or sponsoring institutions. Individual institutions will likely have their own guidance on this, which should be identified here. The exercise of producing the SIA report will likely assist in compliance with these requirements.
The following table presents some legislation at the EU level that may be applicable to security research projects.
Table 4: potentially applicable legislation at the EU level (source: authors)
Way of life, fears
and aspirations
Charter of Fundamental Rights of the European Union; Europe-
an Convention on human rights; Council Directive 2000/78/EC
of 27 November 2000 establishing a general framework for
equal treatment in employment and occupation; Directive
2004/38/EC on the right to move and freely reside; Gender
recast Directive 2006/54/EC; Employment equality Directive
2000/78/EC/
Culture and com-
munity
Charter of Fundamental Rights of the European Union; Council
of the European Union, Directive 2000/43/EC of 29 June 2000
implementing the principle of equal treatment between persons
irrespective of the racial or ethnic origin; Racial equality Di-
rective 2000/43/EC
Political systems Charter of Fundamental Rights of the European Union; Europe-
an Convention on human rights.
Environment Directive 2008/1/EC of the European Parliament and the Coun-
cil of 15 January 2008 concerning integrated pollution preven-
tion and control: Directive 2011/92/EU of the European Par-
liament and the Council of 13 December 2011 on the assessment
of the effects of certain public and private projects on the envi-
ronment.
Health and well-
being
National Legislation for health; Directive 2011/24/EU of the
European Parliament and of the Council of 9 March 2011 on the
application of patients’ rights in cross-border healthcare; Coun-
cil Directive of 12 June 1989 on the introduction of measures to
encourage improvements in the safety and health of workers at
work (89/391/EEC).
42 European Parliament, Council and Commission, Charter of Fundamental Rights of the Euro-pean Union, 2010/C 83/02, OJ, Brussels, 30.3.2010. http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2010:083:0389:0403:en:PDF
25 D3.1 Societal Impact Assessment Manual ASSERT
Personal and prop-
erty rights
Charter of Fundamental Rights of the European Union; Di-
rective 95/46/EC of the European Parliament and the Council
of 24 October 1995 on the protection of individuals with regard
to the processing of personal data and on the free movement of
such data; Directive 2002/58/EC of the European Parliament
and the Council of 12 July 2002 concerning the processing of
personal data and the protection of privacy in the electronic
communications sector; Employment Equality Directive
2000/78/EC; Gender goods and services Directive
2004/38/EC; Framework Decision 2008/977/JHA of
27 November 2008 on the protection of personal data processed
in the framework of police and judicial cooperation in criminal
matters.
Identify societal impacts and possible solutions
The assessor and her SIA team, preferably through stakeholder consultation, should identify all possible negative societal impacts, who these will impact and their likeli-hood (frequency) and consequence (magnitude of impact) as well as the numbers of people who could be affected. Often, the best way to identify these impacts is to consid-er principles associated with each type of societal impact and/or a set of questions which can help identify negative societal impacts, as provided in section 2.1. The asses-sor will benefit from engaging stakeholder representatives and experts to have their views. The assessor, other members of the SIA team and stakeholders consulted should raise other questions that can help to identify the societal impacts of the proposed pro-ject.
The following tables demonstrate how an assessor should use the spiral methodology
and approach the six relevant sectors in three phases.
26 D3.1 Societal Impact Assessment Manual ASSERT
Table 5: Assessment phases (source:authors)
Assessment Round 1:
Ensuring security re-
search meets the needs
of society
Wa
y o
f li
fe,
fea
rs a
nd
asp
ira
tio
ns
Cu
ltu
re a
nd
co
mm
un
ity
Po
liti
cal
syst
em
s
E
nv
iro
nm
en
t
He
alt
h a
nd
we
ll-b
ein
g
Pe
rso
na
l a
nd
pro
per
ty r
igh
ts
Which documented societal secu-
rity need(s) does the proposed
research address? (e.g. life, liberty,
health, employment, property,
environment, values).
How will the research output meet
these needs? How will this be
demonstrated? How will the level
of societal acceptance be assessed?
Is the research project aware of
challenges to these needs?
Does addressing the documented
societal needs through the pro-
posed research require any trade-
offs with other documented socie-
tal needs? How is this trade-off
decided? Is this trade-off still valid
if the research is less effective than
anticipated?
What threats to society (e.g.,
crime, terrorism, pandemic, natu-
ral and man-made disasters) does
the research address?
How is the proposed research ap-
propriate to address these threats?
What other measures could be
adopted to address these threats?
27 D3.1 Societal Impact Assessment Manual ASSERT
Assessment Round 2:
Ensuring security re-
search does not have
negative impacts on
society
Wa
y o
f li
fe,
fea
rs a
nd
asp
ira
tio
ns
Cu
ltu
re a
nd
co
mm
un
ity
Po
liti
cal
syst
em
s
E
nv
iro
nm
en
t
He
alt
h a
nd
we
ll-b
ein
g
Pe
rso
na
l a
nd
pro
per
ty r
igh
ts
How could the research have a
negative impact on freedom of
association?
How could the research have a
negative impact on freedom of
expression?
How could the research have a
negative impact on protection of
personal dignity?
How could the research have a
negative impact on privacy and
data protection?
How could the research have a
negative impact on property
rights?
How could the research have a
negative impact on access to pub-
lic space?
If implemented, how could the
research have a negative impact
on this aspect (culture and com-
munity, way of life, etc.)?
How could the research impact
disproportionately upon specific
groups or unduly discriminate
against them?
Could the research have impacts
upon vulnerable groups (includ-
ing, but not limited to: the elderly,
the disabled, children and young
adults, homeless people, economi-
cally disadvantaged people and
28 D3.1 Societal Impact Assessment Manual ASSERT
people in precarious situations,
immigrants or non-citizens, and
lesbian, gay, bisexual, transgender
or queer (LGBTQ+) identifying
people).
How could the research increase
discrimination?
Assessment Round 3:
Ensuring security re-
search benefits society
Wa
y o
f li
fe,
fea
rs a
nd
asp
ira
tio
ns
Cu
ltu
re a
nd
co
mm
un
ity
Po
liti
cal
syst
em
s
E
nv
iro
nm
en
t
He
alt
h a
nd
we
ll-b
ein
g
Pe
rso
na
l a
nd
pro
per
ty r
igh
ts
What segment(s) of society will
benefit from increased security as
a result of the proposed research?
How will they benefit?
Are additional measures required
to achieve this benefit?
Are additional measures possible
to extend these benefits to other
segments of society?
In what contexts might this bene-
fit be lacking or not be delivered
by the research project?
How will society as a whole benefit
from the proposed research?
Are there other European societal
values that are enhanced by the
proposed research, e.g., public
accountability and transparency;
strengthened community engage-
ment, human dignity; good gov-
ernance; social and territorial co-
hesion; sustainable development.
It should be noted that “impacts” are not solely negative, and part of this approach in-
cludes the assessment of potential benefits from security research (including, for exam-
ple, impacts on welfare, growth and competitiveness). This is particularly the focus of
29 D3.1 Societal Impact Assessment Manual ASSERT
assessment round 1: ensuring security research meets the needs of society. This is not
just a threshold check, but an opportunity for better understanding the pathways to
desirable beneficial outcomes.
Deciding how to mitigate or eliminate or avoid or transfer negative societal impacts is
also a somewhat political decision as is the decision regarding which benefits to pursue.
The assessor or project manager or organisation may decide that the benefits of the
project or technology outweigh the perceived negative impacts arising from its devel-
opment and deployment. Societal impact assessment should be regarded as part of the
organisation’s risk management, although this should be balanced against the need to
actually learn through the process. In order to facilitate socially robust innovation, it
could be argued, SIA needs to provide room for genuine learning on the side of all ac-
tors involved, and the possibility that core policies and plans will be aborted or refor-
mulated needs to remain a possibility.
The organisation should maintain an impact register, wherein the assessor (and/or
other organisation employees) identifies the impacts, their seriousness, what the organ-
isation has decided (if anything) to do about them, who is the person who is responsible
for managing it. The impact register should be regularly updated (e.g., every six months
or at appropriate milestones in the project), depending upon the length of the research
project. It is important to include all identified impacts in this register even if they are
accepted at later stages in the process.
Formulate recommendations
Based on her analysis of the societal impacts, the assessor should prepare a set of rec-
ommendations, which will form part of the SIA report. The assessor should be clear to
whom her recommendations are directed – some could be directed towards different
units within the organisation, some to the project manager, some to the CEO, some to
employees (including researchers) or employee representatives (e.g., trade unions), to
regulatory authorities, etc. The assessor should provide the rationale for each of her
recommendations. The recommendations could include procedural and more general
organisational matters, e.g., relating to training and raising awareness and accountabil-
ity, as well as those relating specifically to societal impact.
Potential venues and options for the publication of research findings (for example,
open-access publication) can be identified at this stage as they will contribute to the
positive societal impacts of the security research.
Reporting and responding
Prepare and publish the report
The assessor should prepare her SIA report, and the organisation should publish it on
its website and/or submit it to an appropriate repository.
An outline and recommended contents of a SIA report are provided in section 3 of this
paper.
Research funding bodies may have specific reporting requirements for societal impact
assessment exercises.
30 D3.1 Societal Impact Assessment Manual ASSERT
Some organisations may be reluctant to publish their SIAs because they fear negative
publicity or they have concerns about competitors learning something they don’t want
them to. Such concerns seem overdone. Publication offers many benefits and opportu-
nities to the organisation. It demonstrates that the organisation treats societal issues
seriously, and consequently its customers or citizens. Customers and citizens are more
likely to invest their trust in an organisation that treats their wellbeing, environment,
individual rights and other concerns with respect. It offers an opportunity to gather
additional feedback from stakeholders. It offers the organisation an opportunity to dis-
tinguish itself from its competitors. For organisations concerned about publishing
commercially sensitive information or security sensitive information, there are solu-
tions. The organisation can simply redact the sensitive bits or put them into a confiden-
tial annex or just publish a summary of the project or, if necessary, provide a copy to
the regulator.
Implement the recommendations
The project manager and/or the organisation does not need to accept all these recom-
mendations, but they should say which recommendations they have implemented al-
ready or intend to implement and which they do not intend to implement and the rea-
sons why they do not intend to do so. The organisation’s response to the assessor’s rec-
ommendations should be posted on the organisation’s website. This transparency will
show that the organisation treats the SIA recommendations seriously, which in turn
should show consumers and citizens that the organisation merits their trust. The or-
ganisation should put in place a mechanism or system for updating the SIA report as
necessary and, especially, for monitoring the implementation of the recommendations.
Research funding and support institutions may also wish to be informed of how a re-
search institution is implementing the recommendations.
Recommendations from the SIA may have implications for the research methods and
research design used in a security research project.
Ensure a third-party review and or/audit of the SIA
The value of independent third-party review or audit has been established for privacy
impact assessments, in term of guaranteeing quality and rigour.43 This is likely to hold
true for societal impact assessments. For research projects this review will need to be
planned for in advance, with appropriate third parties identified. Existing review bod-
ies, for example research funding agencies, will have their own evaluation and report-
ing requirements, which may support the external review of the SIA, but these agencies
may not yet have the capacity to fully audit the SIA process.
Update the SIA if there are any changes in the project
Many projects undergo changes before completion. Research on technological devel-
opment may go in several different directions before achieving its goal. Research with a
social dimension may also uncover previously unidentified societal impacts. Whenever
43 Stoddart, Jennifer, “Auditing Privacy Impact Assessments: The Canadian Experience”, in David Wright and Paul De Hert (eds.), Privacy Impact Assessment, Springer, Dordrecht, 2013.
31 D3.1 Societal Impact Assessment Manual ASSERT
material changes occur, the project manager and/or assessor should revisit the societal
impact assessment to see whether it needs to be amended, which will almost certainly
be the case where new societal impacts become apparent that were not previously con-
sidered. The value of the spiral methodology is that it highlights the importance of re-
visiting key questions through the lifetime of the project, and holding initial findings
contingent. Depending on the magnitude of the changes, the assessor may need to re-
visit the SIA as if it were a new initiative, including a new consultation with stakehold-
ers.
Refer to the SIA in any post-project evaluation
The SIA process does not end with the publication of the report, but should be contin-
ued into any evaluation work related to the security research project. Depending upon
the scope and scale of the project, additional resources and methods may be available
to evaluate the efficacy of the security research or applied security measure, these activ-
ities should include consideration of societal impact.
2.2.3: The societal impact assessment report
A societal impact assessment will have several key outcomes. For example, a key out-
come will be the identification and overcoming of any negative societal impacts. Anoth-
er outcome will be the benefits of stakeholder interaction and engagement. Yet another
outcome will be the discovery of new knowledge and learning (by the organisation as
well as others). Still another key outcome of this methodology will be a societal impact
assessment report for the security research project. This section of the paper provides
guidance on the contents and purpose of the societal impact report. The report docu-
ments the assessment process, and contains the resulting findings. It acts as a reference
document during the conduct of the project and as part of evaluation work afterwards.
The report can serve both as evidence and a record of the societal impact assessment
process. It can serve as a touchstone during the project for project staff and other par-
ticipants, and as a way to demonstrate commitment to understanding and managing
societal impacts of the project.
This reporting guidance draws upon the structure of privacy and surveillance impact
assessments44, as combined with the influences of CTA and social impact assessments,
and configured for application to societal impacts of security research. This section
describes the structure of the final, completed report; however, several sections of the
report, particularly those related to background, planning and project description,
should be initiated in the planning and preparation stage detailed above.
The report should contain the following sections, details of which are provided below.
44 Wright, David, and Kush Wadhwa, “A step-by-step guide to privacy impact assessment”, presentation paper for the second PIAF Workshop, Sopot, Poland, 24 April 2012. http://www.piafproject.eu/ref/A_step-by-step_guide_to_privacy_impact_assessment-19Apr2012.pdf
32 D3.1 Societal Impact Assessment Manual ASSERT
1. Background and identifying details
2. Introduction and overview of the SIA process
3. Project description
4. Societal Impacts
5. Options and alternatives
6. Design features to manage societal impacts
7. Compliance with laws, regulations, codes and guidelines
8. Stakeholder analysis and result of consultation(s)
9. Recommendations
1. Background and identifying details
The SIA report should state on its cover page at least the following elements:
• Societal impact assessment on [name of the project]
• Name and address of the organisation sponsoring the SIA
• Contact person (the assessor), title and e-mail address
• Date of the SIA report.
The length of the SIA report may justify an executive summary, which should state why the SIA was undertaken, who initiated the SIA and who conducted it. The executive summary should provide a brief description of the security research project or technol-ogy application that was the subject of the SIA. It should say which stakeholders (or stakeholder groups) were consulted. It should identify the principal societal impacts, across the six categories and the alternatives for minimising or avoiding negative im-pacts. The summary should contain the principal recommendations of the SIA report.
The SIA report should include a section that describes how senior management is in-volved in decision-making related to societal impacts of security research. Does the senior management board regularly discuss the impacts of security research or applied security measures? Are there specific office holders whose responsibility includes socie-tal impacts?
This background section should identify any organisational issues that are directly or indirectly implicated by the development of the project. For example, it may become apparent that the development of the project requires putting in place an organisation-al mechanism for ensuring accountability, i.e., that the CEO or her designated senior manager is responsible for ensuring that the development of the project does not nega-tively affect the organisation or stakeholders, and that beneficial societal impacts are maximised. The organisation should have procedures in place whereby funding for the proposed project is tied to completion of a satisfactory SIA beforehand.
2. Introduction and overview of the SIA process
The Introduction should outline the scope of the SIA, when, why and for whom it was performed and by whom. It should provide initial information about the project or se-curity measure assessed. It should introduce the methodology employed in the SIA, which can be drawn from section 2.1 of this paper, or adapted as required. Adaptations
33 D3.1 Societal Impact Assessment Manual ASSERT
should, however, be documented. It should also set out the terms of reference for the assessment.
This section should describe the SIA process undertaken (similarly, this can be drawn from section 2.3) and what was the outcome of each stage of the process. It should de-scribe the scale of the SIA undertaken and why the organisation developing the project and (presumably) sponsoring the SIA felt the scale of the SIA undertaken was appro-priate. It should refer to any stakeholder consultations undertaken and the approach adopted to support these.
This section of the report should also describe any measures that have been undertaken to attempt to increase the role of stakeholders in decision-making processes relating to the research project or security measure implementation.
3. Project description
This section should provide a detailed description of the project including its objectives and justification for the project. This should include initial answers to the societal needs questions from section 2.1 and can assist with project planning.
Details of the project can be added as the SIA progresses, as the “spiral” expands and greater knowledge regarding societal impacts is produced.
This section should include information on the documented societal needs to which the security research or application is addressed, and how the project will address these needs. It should contextualise the project against its theoretical background and core assumptions.
This section should also state the main aims of the project or technology? Why is this research being conducted (or proposed) and the system or technology being estab-lished? What are the principle features of the security measure proposed?
The project description should state who is undertaking the development of the project, when it is expected to be conducted or deployed. It should state the subject of the pro-ject, potential beneficiaries, and stakeholders who might be interested in or affected by the project.
The project description should provide some contextual information about how the project fits in with the organisation’s other services or activities. It should state whether aspects of the project are or will become proprietary. It should indicate the intended outcome of the project (for example, fundamental scientific advances, a new technolo-gy, a demonstrator or a commercial product or service).
4. Societal impacts (risks)
This section should list and describe the societal impacts posed by the project. Thus, the organisation or project manager or assessor should consider the impacts of the proposed project on all six categories of societal impact identified above. This section can set out the responses to the questions from section 2.1. These reflections in initial drafts of the reports can be expanded upon in later and final versions as more infor-mation is produced by investigation and consultation. The assessor should state how she and/or the organisation believe the impacts will affect the project objectives.
34 D3.1 Societal Impact Assessment Manual ASSERT
5. Options and alternatives
The SIA report should include a section that identifies the options and alternatives available to the organisation in order to mitigate, avoid, transfer, eliminate or accept the negative societal impacts identified by the SIA, as well as those that could optimise the realisation of societal benefits. The report should say why particular options or al-ternatives were rejected or discounted and why a particular course of action has been recommended. If the organisation has decided to proceed with the research project or technology implementation despite the SIA raising the risk of negative societal impact, the assessor should say (if she knows) how the organisation justifies these. Opinions and alternatives should be understood broadly to also include alternative policy direc-tions.
6. Design features to manage societal impacts
This section should describe the design features adopted within the project or technol-ogy application to reduce or avoid negative societal impacts and to maximise positive societal impacts, and state what are the implications of these design features (e.g., how they affect the viability of the project).
The report could include a table like the one below:
Table 6: Societal impact response table (source: authors)
Societal
Impact
Category Descrip-
tion
Mitigation /
benefit maxi-
misation
measures
Implications
for the project
Way of life, fears and aspi-
rations
Culture and community
Political systems
Environment
Health and well-being
Personal and property
rights
7. Compliance with laws, regulations, codes and guidelines
35 D3.1 Societal Impact Assessment Manual ASSERT
The SIA report should identify the laws, regulations, codes of conduct and guidelines with which the project complies or should comply.45 This section should also provide an assessment of whether the security research has the potential to generate results that require new legislation to address potential gaps. For example, if a new technology were to drastically improve the surveillance capacity of an existing technology, perhaps by removing any possibility for anonymity in public space, would this require revision of data protection legislation? In addition, this section should state how the organisa-tion monitors compliance with the laws, regulations, codes of conduct and guidelines it has identified.
8. Stakeholder analysis and results of the consultation(s)
The report should identify who are the principal stakeholders interested in or affected by the project, and how the assessor or the organisation arrived at this list. The report should specify what efforts the organisation has made to consult with stakeholders, including the public, to gather their views and ideas about potential societal impacts, how they might be affected by the project (positively and/or negatively) and how nega-tive impacts could be mitigated, avoided, minimised, eliminated, transferred or accept-ed. The assessor should specify which consultation techniques were employed (surveys, interviews, focus groups, workshops, conferences, Delphi, surveys, etc.), when they were undertaken, the results of each consultation exercise and whether differences in opinions were discovered when different techniques were used. If any particular stake-holder consultation tools were used in the process, this should also be noted.
The assessor should state who was consulted and what information materials the or-ganisation provided to stakeholders, including the public. Such materials might be in-cluded as an annex to the report. The assessor should state whether the consultations yielded any new findings and what efforts the organisation has made to take into ac-count stakeholder views and ideas in the design of the project. Did any fundamental changes result from stakeholder consultation? The assessor should state how she views public acceptance of the project (does the public accept the project? Or not? Or is opin-ion divided? Or does anyone care?).
9. Recommendations
The assessor should set out her recommendations, which could be a few or many, de-pending on the case. They could be detailed and specific or high level. The recommen-dations are primarily aimed at reducing or removing negative societal impacts and in-creasing positive societal impacts. Some negative impacts may, on balance, be worth accepting and, if so, the assessor should explain why. The assessor should be clear who
45 Privacy Impact Assessment in the European Union is conducted with reference to the Data Protection Directive 95/46/EC, the E-Privacy Directive 2002/58/EC and the Data Retention Directive 2006/24/EC. Because societal impact assessment takes into ac-count a broader range of potential impacts, it must also take into account a correspond-ingly wider range of legislation.
36 D3.1 Societal Impact Assessment Manual ASSERT
will bear the negative and positive impacts (i.e., will it be society, specific social groups, the organisation conducting the project, stakeholders, suppliers or others?).
The assessor should also set out what further work is necessary or desirable with regard to the societal impact assessment. For example, the assessor should make recommen-dations with regard to the need for independent third-party monitoring of implementa-tion of the recommendations.
The assessor should also make recommendations re whether the SIA report should be made public. The default mode should be to make the SIA report public (e.g., to post it on the organisation’s website), however, there may be circumstances where it might not be appropriate to make the SIA or parts of it public – e.g., there may be security, com-mercial-in-confidence or other competitive reasons. Often the report can be redacted here or there and then made public or sensitive bits can be placed in a confidential an-nex.
2.3: Conclusions and recommendations
A societal impact assessment should be completed for all security research, and lessons
learnt from PIAs, SuIAs, CTA and social impact assessment to ensure high standards
are met in terms of real assessment, and to prevent against SIA becoming a box-ticking
exercise. A coherent methodological approach and structured processes, integrated
with existing research practice, combined with clear reporting should contribute to
minimising negative societal impacts and increasing the positive societal benefits of
security research. The structured methodology presented here provides guidance
through such a process.
Whilst benefits can be achieved by security researchers and security research organisa-
tions conducting SIAs, additional benefits can be achieved by mainstreaming such as-
sessments within wider security research contexts. Options for achieving this include
integrating SIA into research frameworks, gaining the support of research support in-
stitutions and policy-makers, sharing expertise and providing training in societal im-
pact assessment, and collating together evidence of the benefits of societal impact as-
sessment activities.
Societal impact assessment methodologies and guidance could be trialled in the Euro-
pean Commission’s Horizon 2020 Secure Societies work programme (which lists vari-
ous calls for security proposals), mirroring the deployment of the Societal Impact
Working Group checklist. However, as the SIA approach builds upon that checklist, and
adds a structured methodology for assessing and reporting upon societal impact, the
SIWG work could stand a precursor, allowing a more rapid adoption of SIA. Many re-
search actors will have some measure of experience with the preceding approach, upon
which this builds. Deployment of societal impact assessment with large scale research
frameworks will help to achieve policy objectives of secure societies without infringing
upon fundamental values. Such an effort would require the support of the organisations
responsible for such frameworks such as the EU’s Research Executive Agency (REA).
The Social Impact Measurement Working Group identified the following phases, into
which societal impact assessment might be implemented: work programmes and annu-
37 D3.1 Societal Impact Assessment Manual ASSERT
al calls (setting the security research funding framework), proposals (the exercise of
putting together a research proposal and responding to the funding framework), nego-
tiation (between funders and researchers, and involving the alteration of the research
proposal to fit the framework), project execution, and the implementation of a com-
pleted product, system or technique in different contexts.46
Training in this methodology could be supported by institutional funding bodies, re-
search prioritisation agencies, research institutions and other appropriate actors. This
would increase the capability, skill level and experience of security researchers, across a
broad range of organisations, in the conduct of societal impact assessment. This train-
ing could be supported by developing a network of practitioners with expertise in socie-
tal impact assessment. This network would act as a resource and store of experience for
conducting societal impact assessments. It would also facilitate organisations seeking
advice or looking to include societal impact assessment in their research activity, either
on a contractual basis, or through partnerships. The network of practice could also col-
late societal impact assessment reports and examples of good practice in societal im-
pact assessments. This would contribute towards building an evidence base for the abil-
ity of SIA to increase the positive societal impacts of security research and decrease the
negative impacts, as well as providing future research projects with models of how to
achieve this. It would also facilitate analysis of long term security research funding
frameworks as a whole, including the ability to reflect upon the societal impact of agen-
da setting.
46 McCarthy 2012, p. 11
38 D3.1 Societal Impact Assessment Manual ASSERT
3. Review of existing decision support tools
In this section of the report we present the findings of our review of existing decision
support tools and resources. This review was conducted to support the development of
the ASSERT SERIA toolkit. It is critical to understand existing potential solutions, in-
cluding platforms that might be adapted or customised, in order to make the appropri-
ate tool selection and to avoid wasted effort. Our review considered the inherent usabil-
ity of the tool in relation to societal impact assessment, as well as fit with the objectives
of the ASSERT project.
3.1 Decision support tools
Decision support tools generally assist the user in making a structured decision from
amongst a set of available options, to set values to options, and provide choice rules.
These tools are modular and generic, and therefore useable, but do not currently have
inherent support for societal impact assessment in security research. Potential uses for
ASSERT include: 1) as part of, or basis, for SERIA 2) interactive toolkit integrated with
Masterclass, 3) inspiration for own tool specifically designed for security impacts. The
following decision support tools are based around a range of decision support method-
ologies. Is not within the scope of this deliverable to delve too deeply into these meth-
odologies other than to indicate that they are primarily concerned with different ways
to compare a set of relatively comparable options, through shared criteria. A common
example with such software is the decision of which new to buy. In this case common
criteria would include the cost, the fuel economy, the maximum speed, number of pas-
sengers, safety record etc. Each of these criteria could be attributed a value (100mph
for speed), and the criteria could be weighted against each other (for example, I may
express a preference for speed over fuel economy).
3.1.1 D-SIGHT
D-SIGHT in an online multi-criteria decision analysis tool. The software itself supports
a range of methods for the comparison of alternatives. These include pairwise
weighting, ADD, MCDA, and SMART. D-SIGHT has a strong visual presentation, and
the analysis of each alternative is well established. It is Internet based, with a free trial
available, followed by a paid version.
The user specifies criteria for the decision, and can then compare options side-by-side
in interactive graphs and visualisations. The tool has a predefined workflow which is
intended to increase the efficiency of decision making. The tool also provides support
for exporting, recording and communicating the decision made, as well as some collab-
oration tools for collaborative decisions.
Figure 3: D-SIGHT screenshots
39 D3.1 Societal Impact Assessment Manual ASSERT
3.1.2 VISA
Visual Interactive Sensitivity Analysis (VISA) is another Multi-Criteria Decision Analy-
sis software, which provides multiple methodologies for valuing and weighting alterna-
tives in a decision, including SMART, MCDA and ADD. VISA has particular support for
integrating the weightings and valuing of several participants into a shared decision. It
also allows for the use of decision tree methods. VISA is a relatively straightforward
system, with an intuitive interface which is possible for first-time users to understand
with some support. A web based version is also available.
40 D3.1 Societal Impact Assessment Manual ASSERT
Figure 4: VISA screenshots
3.1.3 Web-HIPRE
Web-HIPRE47 is a web-based decision support software offers a wide range of decision-
support methodologies, including value-trees, swing weighting, pairwise comparison,
non-linear value functions, sensitivity analysis, MCDA, ADD, SAT, and SMART. It is
another tool for the evaluation of alternatives. In terms of impact assessment this sug-
gests a requirement for the impact to be already known, at least to some extent. The
methodologies underpinning Web-HIPRE are robust and supported by current best
practices in decision support research in academia, however the tool is less refined that
47 hipre.aalto.fi
41 D3.1 Societal Impact Assessment Manual ASSERT
some of the other options here, and has a basic (if functional) interface, based on Java.
Web-HIPRE also offers support for group decision support.
Figure 5: web-HIPRE screenshots
3.1.4 1000minds
1000minds48 is a web-based decision support software. It is based upon a proprietary
weighting method called PAPRIKA (Potentially All Pairwise RanKings of all possible
Alternatives) which is intended to be a more intuitive and natural pairwise weighting
method. The tool has a strong visual presentation, and the analysis of each alternative
is well established. 1000minds won a Consensus Software Award sponsored by IBM
and Microsoft.
Figure 6: 1000minds screenshots
48 www.1000minds.com
42 D3.1 Societal Impact Assessment Manual ASSERT
43 D3.1 Societal Impact Assessment Manual ASSERT
3.1.5 Transparent Choice
Transparent Choice49 is a collaborative online decision making software intended to
support multi-stakeholder decisions, with features that allow the division of tasks
among stakeholders. It is a decision support tool, intended for the evaluation of alter-
natives with known (or estimated) impacts, and known (or estimated) values for key
criteria. It has an intuitive interface and its collaborative functions work in a manner
similar to dropbox, and stakeholders can be invited to collaborate via email.
Figure 7: Transparent Choice screenshots
49 www.transparentchoice.com
44 D3.1 Societal Impact Assessment Manual ASSERT
3.1.6 SMART
SMART50 is another online decision support software, and is one of the simplest tools
we evaluated in this section. Clear instructions and guidance are given for each step.
Figure 8: SMART screenshots
50 www.smart-decisions.net
45 D3.1 Societal Impact Assessment Manual ASSERT
46 D3.1 Societal Impact Assessment Manual ASSERT
3.2 Learning Support Environments / Learning Management Sys-
tems
This section reviews a set of tools that support learning activities, including the frame-
works for online courses. These tools often allow for multi-media presentation and in-
clude communication tools for collaborative learning. Potential uses for ASSERT in-
clude the creation of a learning environment or online course for assessing social im-
pact of security research, or integration with Masterclass
Many of these products are really very similar in terms of functionality. The all offer
ways of setting up a “course” in a web based online format. Generally, the courses are
composed of a set of content generate or curated by the course provider as you might
find in a traditional offline course or module (often divided up into “Lessons” or “clas-
ses” of appropriately sized learning units), supported with tools to facilitate communi-
cation between teachers and students, and between students, and to facilitate the man-
agement of the class (for example enrolment, setting quizzes or other assessment exer-
cises. These products generally allow the course provider to monitor some activity on
the course (for example, which students have accesses a particular course or assign-
ment) and to restrict access to enrolled students.
3.2.1 Moodle
Moodle51 is a software package for producing Internet-based courses and websites.
Moodle is open source software under the GNU General Public License. Moodle is in-
stalled on a server and can then offer a fully functional learning management system.
Moodle is a mature product, with a large number of users and developed documenta-
tion and support base. It is relatively rapid to set up in a basic configuration, and pro-
vides a large number of education tools (including chat, discussion forum, lessons, da-
tabases, connections to external tools, assignment setting and collection functions).
Moodle has a large number of option features which can be activated to customise the
installation for particular purposes including unusual class formats.
Moodle offers the following features52
• Collaborative tools and activities (forums, chat, wikis, glossaries, activities data-
bases, collaborative publishing)
• Support for various pedagogic approaches (instructor-led, self-paced, blended
learning or online online).
• Calendar
51 https://moodle.org/
52 http://docs.moodle.org/26/en/Features
47 D3.1 Societal Impact Assessment Manual ASSERT
• File management
• Multimedia integration
• Text editors
• Profile management and user access management
• Notifications
• Progress tracking
• Customisable site design
• Support for open standards
• Management of user roles and permissions
• A wide range of plug-ins and plug in management systems
• Reporting and logs
The lesson function seems particularly appropriate for some of the ASSERT features,
for example, a step-by-step impact societal impact assessment:
The lesson activity module enables a teacher to deliver content and/or practice activities in inter-esting and flexible ways. A teacher can use the lesson to create a linear set of content pages or instructional activities that offer a variety of paths or options for the learner. In either case, teachers can choose to increase engagement and ensure understanding by including a variety of questions, such as multiple choice, matching and short answer. Depending on the student's choice of answer and how the teacher develops the lesson, students may progress to the next page, be taken back to a previous page or redirected down a different path entirely.
A lesson may be graded, with the grade recorded in the gradebook.
Lessons may be used
• For self-directed learning of a new topic • For scenarios or simulations/decision-making exercises • For differentiated revision, with different sets of revision questions depending upon an-
swers given to initial questions
3.2.2 WordPress + plugins
WordPress is one of the most popular and commonly used website production and
blogging platforms, which makes it remarkably easy to create a set of web pages, which
can either be hosted by WordPress and be supported by adverts, or can be uploaded
onto a server and hosted by the consortium. WordPress’s functionality can be expanded
through a range of plug ins that provide the educational tools needed. There are several
of these available, as discussed below. There is a large amount of supporting material
available for WordPress, ranging from custom templates up to bespoke website design.
Moodle (integrated into WordPress)
Further to the discussion of Moodle above, Moodle can also be integrated into Word-
Press. The advantage of doing this would be to benefit from the ease of use of the
48 D3.1 Societal Impact Assessment Manual ASSERT
WordPress front-end. The WordPress appearance and templates are more attractive
than those available through Moodle, and using WordPress in this manner would allow
the consortium to have an accessible website with much of the SERIA material that
we’d wish to make public also available on it (without having to log in, create an ac-
count, learn how to navigate the LMS), but also to have all the LMS functionality avail-
able to registered users.
Namaste LMS - http://namaste-lms.org/
Namaste! Also runs as part of a WordPress installation. It offers the following func-
tions:
• Manage unlimited courses and enroll students or let them enroll themselves
• Auto-approve or manually review students who want to enroll in a course.
• Publish the course descriptions on your site to encourage subscribing.
• Charge to enroll in courses - Paypal and Stripe integrations available
• Manage unlimited number of lessons in each course.
• Lessons support rich text, media and plugin contents just like any other Word-Press post or page.
• Define various criteria for lesson completeness - such as completing assign-ment, manual approval, completing exams.
• Define which lessons should be completed to complete a course. Some lessons can be optional.
• Create assignments and approve or reject student's solutions.
• Add notes to student's solutions to help them resolve the problems.
• Manage students in each course and see their to-do lists for every lesson
• Create certificates that will be assigned to students for completing course(s)
• Allow different user roles to administrate and use the learning material
• Connect to exams created with plugins like Watu or WatuPRO
• Run powerful reports using the Namaste! Reports plugin.
• Optionally allow discussion to assignments - powered by the additional Na-maste! Connect plugin
• Receive and send automated email notices about various LMS actions like en-rollments, comments, notes, submitted solutions, etc - via Namaste! Connect.
• Enable a dashboard widget or a page with activity stream showing you latest X days of activity in the system - via Namaste! Connect
LearnDash
LearnDash53 is a plug-in that turns WordPress into a learning management system by
the addition of functions for tools for online lectures, quizzes and checking how stu-
dents are progressing through the material. It provides a simple way of creating an
adaptable custom learning platform. LearnDash introduces an authentication systems
which allows designers to charge for access to particular courses. A demo version is
available, but the full version charges a fee. This fee includes access advice and guid-
ance to individuals setting up learning management programs. LearnDash has good
53 www.learndash.com
49 D3.1 Societal Impact Assessment Manual ASSERT
quality design, but a limited range of functions in comparison to some other LMS
plugins. LearnDash has been used by a number of for-profit education providers.
3.2.3 WordPress + multiple plug-ins
It is possible to create a functional online learning environment through the use of a
WordPress website, with the addition of a larger range of individual plug-ins, which
would replicate the functionality of one of the all-inclusive LMS packages. This would
allow some flexibility and options. However, it would take more effort and resources to
identify and select the appropriate plug-ins and to set them up. There is a greater po-
tential for having something missing that would otherwise be useful, or for conflicts
between the various plugin applications. Given that single plug-ins could provide the
required functionality, this avenue is not explored further.
3.2.4 Big Blue Button
Big Blue Button54 is a plug in for other websites, or hosting on dedicated bigbluebutton
servers. User downloads a desktop client, and it features open source licenses. This has
a tool for setting up education focused online meetings and web conferences, with sup-
port for video and audio communication and presentations.
Perhaps most appropriate for active teaching (video/audio and text chat in real time).
Big Blue Button provides several tools for moderating audio/conversation in real time,
looking at a presentation in real time with students etc. The ASSERT project does not
currently envisage (or has not yet discussed) this form of real-time teaching activity as
part of the SERIA toolkit, and its sustainability is not covered by the ASSERT project
agreement. If the Masterclass concept was to be expanded at a later point, this might a
useful tool for doing that. Big Blue button does not appear to offer all of the functionali-
ty required for ASSERT
3.2.5 OpenMOOC
OpenMOOC55 is an open source massive open online course platform, intended to pro-
vide a highly customisable and free-to-use system for hosting online courses. A MOOC
(massive open online course) is an online course aimed at large-scale interactive partic-
ipation and open access via the web. In addition to traditional course materials such as
videos, readings, and problem sets, MOOCs provide interactive user forums that are
54 http://bigbluebutton.org/
55 http://openmooc.org/
50 D3.1 Societal Impact Assessment Manual ASSERT
intended to help build a community for the students, professors, and teaching assis-
tants (TAs). MOOCs are a recent development in distance education. OpenMOOC is an
open source platform (Apache license 2.0) that implements a fully open MOOC solu-
tion. It is relatively lightweight – for example, rather than using any internal media
hosting, its course videos are hosted on YouTube. Unfortunately, whilst the project was
started in 2012, it is not yet completed, and critically, documents for installation have
not yet been made available. Therefore use of OpenMOOC for ASSERT would require
substantial additional coding effort.
3.2.5 Blackboard
Blackboard56 is an industry standard learning management systems that is used by a
large number of educational institutions. Blackboard’s current offering is a set of five
different platforms with specialist applications for business, education and industry.
This includes the ability to set up a fully functioning custom learning supporting envi-
ronment. Recent additions to the learning platform have included social functionality,
akin to Facebook and Twitter in the form of creating and inviting events, and allowing
for group chat. Blackboard is a very established provider, however it not open source,
nor free to use, but a range of educational and institutional licenses are available.
Key features include a central hub/portal with key information on the course displayed
in a simple place, online assessment functions, rich content editors, a user-centric in-
terface, surveys and course evaluation and tools for collaboration between students.
3.2.6 EdX
EdX57 is a massive open online course (MOOC) platform founded by the Massachusetts
Institute of Technology and Harvard University in May 2012 to host online university-
level courses in a wide range of disciplines to a worldwide audience at no charge and to
conduct research into learning. Potential students can choose from a wide range of
courses, with details of their requirements in terms of commitment and time. Once
registered on the system and on a course, the student has access to the courses. The
courses include automated feedback to quizzes and assessments, and participants can
gain a certificate for completing a course. The set-up of EdX courses, and of similar
MOOCs such as Coursera can provide inspiration for the structure and content of the
SERIA toolkit, but given the complex licensing requirements for courses to be submit-
ted and approved by EdX, currently only extended to a select set of partners, it does not
appear currently possible to host an ASSERT course on the EdX platform. The source
code is available under open source licensing agreements.
56 www.blackboard.com/Sites/International/EMEA/index.html 57 https://www.edx.org
51 D3.1 Societal Impact Assessment Manual ASSERT
3.3 Handbooks and Guides
These tools take the form of a physical or web-based handbook, giving a user access to
best practices and techniques in a particular area. Traditional handbooks are primarily
textual, with some visual material. Online guides provide the opportunity to add mul-
timedia content such as audio, video and animation. Potential use for ASSERT as a
model for what a non-interactive or offline assessment tool might look like – e.g. a
“Handbook for assessing the social impact of security research”.
3.3.1 IDEO HCD toolkit
IDEO58 is a global design and innovation consultancy, which has adopted a research-
informed design methodology to approach a range of problems for its clients. Their
approach, which they term “design thinking” brings together what is possible from a
design point of view with what is desirable from a human perspective. The intent is to
allow people who have not been trained as designers to use the same creative tools to
approach problems.
The Human Centered Design (HCD) Toolkit is a freely downloadable and open source
toolkit designed to help international staff and volunteers with social enterprises and
NGOs understand a community’s needs in new ways, find innovative solutions to meet
those needs, and deliver solutions with financial sustainability in mind.59 The kit
“walks users through the human-centered design process and supports them in activi-
ties such as building listening skills, running workshops, and implementing ideas.”60
IDEO also hosts HCD Connect, an online platform for people taking a human centered
design approach to connect and engage with each other. The IDEO website also con-
tains case studies of successful application of the approach including a project on safe
access to drinking water, and another on children’s eye care in developing countries.
HCD also refers to the suggested approach – Hear, Create, Deliver.
Hear (6 Steps in total) During the hear phase, Design Team will collect stories and
inspiration from people. This includes preparing for and conducting field research.
Create (7 Steps in total) In the Create phase, participants work together in a work-
shop format to translate which is heard from people into frameworks, opportunities,
solutions, and prototypes. During this phase you will move together from concrete to
more abstract thinking in identifying themes and opportunities, and then back to the
concrete with solutions and prototypes.
58 http://www.ideo.com/about/ 59 http://www.ideo.com/work/human-centered-design-toolkit/ 60 Ibid.
52 D3.1 Societal Impact Assessment Manual ASSERT
Deliver (6 Steps in total) The Deliver phase will begin to realize your solutions
through rapid revenue and cost modeling, capability assessment, and implementation
planning61
Figure 9: Human Centered Design Toolkit62
3.3.2 Research Toolkit
The Research Toolkit63 is a website intended to provide tools and guidance for all as-
pects of a clinical research project. The website is organized by phases of a research
project, with resources that have been vetted and curated by the project team.
The Research Toolkit project was funded through two consecutive administrative sup-
plements to the University of Washington's Institute for Translational Health Sciences
Clinical and Translational Sciences Award (CTSA) UL1 RR025014 from the NIH Na-
tional Center for Research Resources.
The toolkit is divided into six sections, each with sub sections.
1. Building Collaborations: A solid foundation is essential to any successful project. The resources and tools found in this section cover fundamental aspects of partnerships, from finding partners to ensur-ing sustained engagement.
2. Developing Proposals: This section provides links to funding sources, budget development aids, and tips on good grant-writing.
61 IDEO, HCD Toolkit. London & New York, IDEO, 2009 62 http://www.ideo.com/work/human-centered-design-toolkit/ 63 http://www.researchtoolkit.org/
53 D3.1 Societal Impact Assessment Manual ASSERT
3. Starting Up a Study: Able to find resources for including contractual agreements, ethical/regulatory approvals, and training study staff, as well as a guide for developing participant-centered study mate-rials.
4. Conducting and Managing Projects: Helpful array of resources to aid the study management process, including procedures for informed consent, operations manuals for clinical research, and ways to keep both research staff and participants engaged over time
5. Disseminating and Closing Research: Sharing research results is both an obligation and an opportuni-ty.
6. Research w/Hispanic and Indigenous Populations: As this is a fundamental and cross-cutting aspect of research, we have compiled a few dozen resources that cover a range of issues from engaging with communities and ensuring research is culturally appropriate, to IRB and regulatory processes and study management.64
The Research Toolkit provides clear, structured research guidance, directed towards
practitioners, and this would have clear transferability to ASSERT in the form of pre-
senting a structured approach to a specific element of a research project. It is primarily
an information resource with no interactive elements.
Figure 10: Research toolkit screenshot
64 http://www.researchtoolkit.org/
54 D3.1 Societal Impact Assessment Manual ASSERT
3.3.3 Crisis Guide: Iran
The US Council on Foreign relations produced an interactive online guide to Iran.65 The
presentation traced Iran’s history, its evolution as an Islamic republic, and its nuclear
programme. It also presented a set of policy options relation to Iran. The guide includes
an animated timeline, details on the regime, nuclear programme and the broader re-
gion, as well as photography, maps and video. This is an example of an attractive and
accessible packaging of information, which makes good use of a combination of ap-
proaches to address a complex subject. Top level information in presented in the intro-
ductory video, whilst exploring the pages provides more detail. This type of presenta-
tion is increasingly common in online information sources, and is build upon common
modular approaches to web development. The key requirements are appropriate con-
tent (in the form of text, maps, video, images and audio material) and determining an
appropriate structure for this material that maximises flow through the site and acces-
sibility of the information. This form of presentation may be most effective upon first
viewing, and its valuable as a reference source or guide to practice may be limited.
However as an introduction to a topic, this approach can be powerful.
Figure 10: Crisis Guide: Iran
65 http://www.cfr.org/interactives/CG_Iran/index.html#/overview/
55 D3.1 Societal Impact Assessment Manual ASSERT
3.4 EU Decision support projects
Several current EU-funded projects include an element of decision support, and the
construction of tools. Some of these also have a specific security dimension. The AS-
SERT project may be able to build upon, expand, customise or incorporate the tools
produced by these projects.
3.4.1 SIAM66
The SIAM project is described by its consortium in the following manner:
“SIAM is an EU-funded research project that provides support to end-users in the as-
sessment process of security measures and technologies (SMT). The overall objective is
to create an Assessment Support Toolkit that takes the complexity of technologies, eco-
nomic aspects, cultural differences and societal dimensions into account. SIAM worked
to help stakeholders to cope with the increasing complexity of assessments by provid-
ing a systematic approach for assessing the potential impact of SMTs.”67
The purpose of the SIAM Assessment Support System is to guide users in their assess-
ment of security measures and technologies. This is realised by means of a set of struc-
tured questions about issues of security, trust, efficiency, and freedom infringement.
These questions have been develop by experts in their respective fields and take into
account social and legal issues. The SIAM system function as a database and stores
store these questions and their associated assessment paths and make them available to
different kinds of users.
The SIAM68 toolkit is best described as a tool for conducting an analysis of the potential
future impacts of security measures and technologies based upon the structured collec-
tion of relevant perspectives, through answers to expert curated questions. It is in a
beta stage with some functions still to be completed, but is functional. It attempts to
help develop a picture of the potential likely impacts of a security intervention by draw-
ing upon the collective knowledge embedded in its multiple participants. Unlike some
of the other tools we have assessed through ASSERT, the SIAM tool is inherently based
upon multiple participants.
The tool is aimed at organisational decision makers seeking to evaluate a potential se-
curity technology intervention (for example, if they should install an abandoned bag-
gage detection system in an airport). The tool identifies a number of roles that should
66 Members of the ASSERT Consortium were invited to participate in the SIAM User Forum II in Berlin 31st October – 1st November, to experience the tool, participate in its testing and to pro-vide feedback on the tool to the SIAM consortium. The following material is based upon that meeting as well as a broader examination of the tool. 67http://www.tu-berlin.de/ztg/menue/forschung/projekte_-_laufend/security_impact_assessment_measures_siam/ 68 http://siam-project.eu/
56 D3.1 Societal Impact Assessment Manual ASSERT
be involved in the assessment process (for example, lawyers, stakeholders, investors,
etc). These actors are invited to participate in the use of the tool. The tool is based
around a large pool of 300 or so questions. These questions have been developed by the
SIAM consortium on the basis of research and literature in areas of law, sociology and
technology, and are intended to get at the key issues in security technology assessment.
actors are presented with a sub-set of the question pool relevant to their role, and with
conditional logic (for example, an initial question might ask if there is any anticipated
risk to human life from a technology, and if the answer is yes, then subsequent ques-
tions will be displayed which probe into this risk in greater detail). The answers to these
questions are collated together from the various participants, and can be used to pro-
duce a final report. There is also a reflexivity score which gives a quantitative evaluation
of how close to inclusive the process was (for example, if it should have involved eight
different roles, but only included four, and of these four some only answered half of the
questions, it would receive a lower reflexivity score). Therefore the SIAM tool can also
be understood as a consultation management tool.
One strength of the tool is the collation of different perspectives. The danger is that
they are currently just reported. The tool needs to help this process reveal diversity, and
potentially develop new approaches/solutions. The tool currently has a static model of
the security intervention being assessed whereas the final intervention might be
changed through this process. The tool needs to identify ways in which the technology
being assessed can be improved (for example by implementing it technologically in a
particular way, by changing elements so that it avoids infringing fundamental rights).
Participants need to be able to suggest such changes or approaches. Explanations and
evidence for decisions are important in this. This also might be part of the value-added
case of why users would make use of this tool.
The tool currently examines a single technology measure at a time, when often the need
will be to comparatively examine a range of options. There will therefore be some re-
dundancy in the tool, with participants being invited several times and having to an-
swer the same question multiple times. Could imagine ways in which parallel cases
could be linked in some ways. Each gets an individual report/case, but could the way
the questions are served reduce the workload of this?
The outputs of this process could with the appropriate legal/institutional/reputational
support be valuable in themselves. A tool like this could potentially be part of (self) cer-
tification approach, or as evidence that best practice in consultation/security impact
assessment has been followed. We discussed at the session the possibility of crypto-
graphically proving that a report generated through the process had not be edited, for
example.
It was suggested of the tool required user training. A requirement for training to answer
questions, introduces an overhead, and a bias towards “professional stakeholders”. The
questions should be answerable by a professional in the relevant field, but without ad-
ditional training.
57 D3.1 Societal Impact Assessment Manual ASSERT
3.4.2 EURO Summer school
The EURO Summer School69, also known as the International Summer School on "Mul-
ti-Criteria Decision Aid": Methods, applications and software. The regular event was
formed by a group of well-known North American and European scientists with the
purpose of promoting the diffusion of the potentialities of multi-criteria models as val-
uable supporting instruments in decision-making. The aim was to provide a continuing
environment for face-to-face networking. Starting in 1983, the Summer School on
MCDA/M is a joint event of the International Society on Multiple Criteria Decision
Making70 and the EURO Working Group on Multi-criteria Decision Aiding.71 The aim
of the school is to give to doctoral students/young researchers a state-of-the-art presen-
tation of multiple criteria methods, applications and software. The school does this
through inviting a combination of researchers intending to develop a thesis in MCDA
alongside industry and government professionals who use such approaches in their
work.
3.4.3 MCDA-RES
The aim of the MCDA-RES project72 is to develop a software decision tool that will ena-
ble multi-criteria decision analysis (MCDA) of renewable energy source investments
and apply it to three case studies. The project concluded in 2004 and had several work
packages were devoted to tool development. The tool was intended to increase coopera-
tion between experts and analysts in the process of making a decision.
Figure 11: MCDA-RES methodology
69 http://www2.hsu-hh.de/logistik/summer-school-2013/history.html 70 International Society on Multiple Criteria Decision Making 71 www.cs.put.poznan.pl/ewgmcda 72 http://www.aegean.gr/environment/energy/mcda/mid.html
58 D3.1 Societal Impact Assessment Manual ASSERT
3.4.4 DESSI
The DESSI project intended to create both a decision making process and a decision
support system for use by end users of security investments. The system is intended to
give insights into the pros and cons of specific security investments. The aim was to
support a transparent and participatory decision making, which is able to take into ac-
count specific contexts and social issues. The targeted audiences were public authori-
ties, developers of security solutions, commercial enterprises and social organisations.
The DESSI tool could be used by decision makers to structure their own assessment
exercises.
The DESSI tool is a planning and decision tool, hosted online with both a security focus
and a reliance upon a participatory methodology. A decision maker could follow the
structured DESSI approach, and in doing so, would take into account a range of key
contextual variables as well as be supported in the conduct of workshops and other
forms of stakeholder engagement. The stakeholder engagement elements are integrated
into the process and form a key part of the process. The tool would keep track of the
process and assist the decision maker. The process is intended to incorporate a set of
dimensions such as security gain/loss, impact on fundamental rights and ethical as-
59 D3.1 Societal Impact Assessment Manual ASSERT
pects, legal framework, social implications, acceptability, political significance and
economy into a security decision making process.
Figure 12: DESSI tool screenshot
3.4.5 DREAM
The DREAM project (Decision support in Real-Time for Efficient Agile Manufactur-
ing)73 aimed to increase the competitiveness of the European manufacturing sector by
developing methods to support real time decision making in manufacturing. The ap-
proach was based upon developing simulation software, particularly for industrial re-
quirements in manufacturing. The project has only started recently and at the time of
writing, no deliverables were available.
3.4.6 FIRST
The FIRST74 project seeks to develop a large scale information extraction and integra-
tion infrastructure for supporting financial decision making. The project had a focus
73 http://dream-simulation.eu/index.php/project-overview 74 http://project-first.eu/
60 D3.1 Societal Impact Assessment Manual ASSERT
upon information gathering and real-time decision support. It intends to create a pro-
cess for integrating information from a knowledge base with real time information
feeds to create models for the detection of financial events, as well as visualisation
tools. The type of knowledge engaged with in this project is highly quantitative and the
approach is heavily mathematical, based upon the challenge of extremely large, highly
dynamics and heterogeneous sources of information. The aim is to increase transpar-
ency by providing a fast, real-time, automatic and more comprehensive information
base can help preventing false decisions. FIRST also seeks to identify insider trading
and other forms of abuse by using large volumes of unstructured financial data incor-
porated into a reputation engine.
Figure 13: FIRST market surveillance
3.4.7 DISASTER
DISASTER (Data Interoperability Solutions at STakeholders Emergencies Reaction)75
aims to increase mutual understanding between international EMS organisations and
so reduce response time by helping create an interoperable solution to solve these is-
sues. Many emergency responders use Emergency Management Systems to facilitate
communication and emergency responses. However there are also problems related to
cultural, legal and linguistic differences. The aim of the project is to increase under-
standing between different participants in a network of emergency actors and thereby
speed up decision making processes in operation environments. This is to be achieved
through the development of a collective ontology, as well as developing techniques for
interoperability between technology systems. The project target outcome is an integra-
tive and modular ontology for establishing a common knowledge structure between all
the first responders involved in an emergency, but being compliant with legacy interna-
75 http://disaster-fp7.eu/
61 D3.1 Societal Impact Assessment Manual ASSERT
tional data formats exchanged in the European Union as long as being seamlessly inte-
grated within current SOA-based Emergency Management Systems. Literally, this is to
make a ‘black box’ that converts the each countries ‘code’ for emergency situation to
other countries one, thus able to handle international emergency situation. The project
user case studies and proof of concept deliverables are available.
3.4.8 ESS
ESS – Emergency Support System76 aims to develop a revolutionary crisis communica-
tion system that will reliably transmit filtered and pre-organized information streams
to the crisis command system, which will provide the relevant information that is actu-
ally needed to make critical decisions. The project is focused on the provision of action-
able information to crisis managers during abnormal events. This information is to
come from the collection and fusion of real time data from deployed sensors. The aim is
for a very large scale process that involves multiple sources of information. Whilst secu-
rity focused, the decisions in these situations are very short term and focused upon
emergency response. This is a very different environment from that of security research
processes, and the information to be processed (number of casualties, locations of peo-
ple and resources) of a markedly different character.
3.5 Analysis and conclusions
Based upon the review of existing tools, we can make the following conclusions
Online decision support tools are potentially useful for making security decisions
(as they may be useful in support of any particular decision) and would require relative-
ly little development time from the ASSERT consortium. However, these tools do not
support the investigation of societal impact assessment if this is currently unknown
(which in many cases it will be). Rather they support the comparative evaluation of
multiple options. Societal impact (or various composite parts of societal impact) can be
attributed a value in these evaluations, and then involved in a weighted comparison
against other considerations (for example, costs, resource requirements, etc). Alterna-
tives do not emerge during the process, but must be identified in advance. Further-
more, using these decision support tools in a societal impact assessment is likely to re-
quire additional information or a paradigm shift to include social impacts well. Indi-
vidual tools might be included as potential tools in handbook or online course, but it is
not suggested that these form the core of a SERIA toolkit.
Many of existing EU projects are either 1) incomplete, 2) quite specific in their focus,
or 3) beyond the scope of ASSERT to replicate. There are, however some lessons that
can be extracted from their experience. The SIAM tool could be a useful way of con-
76 http://www.ess-project.eu/
62 D3.1 Societal Impact Assessment Manual ASSERT
ducting a societal impact assessment for a particular context. It is strongly orientated
towards decision makers looking to implement a particular technology. This may be
appropriate for some forms of security research that are more focused upon application
and demonstrations of particular security technologies. In this case the SIAM tool could
be used to model this assessment process. It would require participants, although in an
appropriately interdisciplinary project these may be the consortium participants. The
danger would be of missing the perspective of external stakeholders at the research
planning stage. In contexts where the intended security technology intervention is not
yet realised (for example if the aim of the project is to develop something, but its fea-
tures are currently unknown), then the SIAM tool may be less useful. It would be worth
linking to the SIAM tool through the ASSERT toolkit when it becomes available. The
ASSERT toolkit should probably provide a paragraph akin to the one above (if not more
detailed) outlining the best way to make use of the SIAM tool, and the contexts in which
its use is appropriate.
Handbooks/Online guides on societal impact assessment would be relatively
straightforward to produce. This could be an enhanced version of the societal impact
assessment methodology presented in section two of this report. The Research Toolkit
provides clear, structured research guidance, directed towards practitioners, and this
would have clear transferability to ASSERT in the form of presenting a structured ap-
proach to a specific element of a research project – in this case societal impact assess-
ment. Similarly, The IDEO toolkit aims support practitioners in investigating a complex
social problem through a range of research methods, an area which has clear parallels
with societal impact assessment. A similar approach for societal impact assessment
would use a structured approach and suitable illustrative content, presented in an ac-
cessible manner to walk people, potentially with little experience of the process,
through a societal impact assessment exercise as set out in section 2. Like the societal
impact assessment methodology, the HCD toolkit promotes high stakeholder involve-
ment, and provides a number of tools for considering the potential social impact of de-
signs or other interventions. The toolkit is non-interactive, although HCD Connect does
provide interaction and communication tools.
Finally, Online learning environments and learning management systems
can provide many of the features of a handbook or online SIA guide, but also potential-
ly link up with supports, and extend the life of the master class concept. Material from
the deliverables and from the master class can be re-purposed to provide content for
such an environment. In addition to providing support to practitioners in security re-
search by presenting structured information on societal impact assessment, such a
learning environment could potentially foster the development of a learning communi-
ty centred on societal impact assessment in security research. Most of these LMS re-
quire customisation for a particular learning goal/objective and context of use. The
services have lots of options that can be activated or deactivated as appropriate. Course
content has to be created and uploaded, and the lessons and tools have to be selected in
a manner that will support the intent of the course. So beyond the choice of tool, the
most important choices will be what material content to use, and how to structure this
within the learning environment. Some tools are available under Open Source software
licenses and can be rapidly adopted.
63 D3.1 Societal Impact Assessment Manual ASSERT
The analysis of existing tools for decision support in this section has supported the de-
cision to develop the SERIA toolkit on the basis of a combination of online guide or
handbook combined with a learning management system.
64 D3.1 Societal Impact Assessment Manual ASSERT
4. The ASSERT online Toolkit
The ASSERT online toolkit is a combination of an online societal impact assessment
“handbook”, with content derived from ASSERT project deliverables (particularly from
WP1), with new material created for WP3 (see section 2), with multimedia content,
structured in an accessible manner, with the additional features provided by a Leaning
Management System that offers additional tools (communication, resource hosting,
user authentication and log-in etc) to registered users. This combination allows the
consortium to support the Masterclass experience with online content, to make infor-
mation on societal impact assessment in security research accessible for a wider range
of users, and also to host the ASSERT expert database created in WP2.
The ASSERT online toolkit is accessible at http://assert.masiondx.com
The Toolkit is intended to meet the following design requirements:
• Accessibility – the toolkit must be straightforward to use, and not present an
unnecessary complication to the user. Information on the toolkit should be pre-
sented in a logical and structured manner. The toolkit should not present barri-
ers to access from a variety of users with a variety of needs. Part of this require-
ment is the need to leverage appropriate social media and web 2.0 tools.
• Appealing – the toolkit should present useful information (relevant to its audi-
ence and purpose) in a visually appealing manner. Part of this requirement is
for interactivity and multimedia (photos, graphics, video and audio) content.
This provides one component of the added value over and above the societal
impact assessment manual.
• Audience – the toolkit should be addressed to researchers who may be interest-
ed in conducting a societal impact assessment as part of a security research pro-
ject, but who may not currently have significant expertise in this area. The sec-
ondary audience for the toolkit should be the evaluators and funders of security
research projects.
• Flexible construction – the tool should be adaptable and changeable over time.
It should be straightforward to add new content, and multiple administrators
should be able to update the tool in order to add a measure of sustainability.
Part of this requirement is the need to build upon existing tools in order to in-
crease the speed of the design process, and ensure familiarity and convenience
for users and content producers.
• User-Authentication – the tool should be able to differentiate between guests
and registered users, and be able to provide different information to these two
groups.
• Support to the Masterclass – the toolkit should be able to support the learning
and networking activities of the ASSERT Masterclasses.
• Host the Expert database – the toolkit should be able to host the ASSERT Ex-
pert database, and provide tools and functions for communicating to and be-
tween this group of users.
• Security and privacy protection – the toolkit should protect the personal infor-
mation of registered users and respect their preferences in terms of communica-
tions.
65 D3.1 Societal Impact Assessment Manual ASSERT
Structurally, the resulting toolkit is a combination of two software packages hosted on
the same server. WordPress is an industry-leading content management and web-
publishing tool. It is free to use and highly customisable, as well as being straightfor-
ward to add and edit content. Similarly, Moodle is a highly-regarded and well-known
learning management system. It is also free to use, and highly customisable. The adapt-
ability of both tools has allowed the ASSERT consortium to create a tool which is ad-
dressed to the specific audience for the tool (security researchers and evaluators), and
that provides the required functionality for the tool. The use of two tools allows for an
accessible (and easily updatable) set of front-end, public facing pages that can be ac-
cessed by anybody with an internet connection, combined with a password-protected
environment accessible to registered users only. The structural arrangement of the two
components is depicted in the following figure. In addition to this core arrangement,
the toolkit also makes use of video content hosted on www.vimeo.com and embedded
in the appropriate WordPress. The server hosting this content is owned by an ASSERT
consortium member which brings the content fully under the control of the consortium.
Figure 14: Structure of toolkit components
The general visitor to www.assert.maisondx.com will encounter the WordPress pages,
which they can navigate in the conventional fashion. On the “online course log-in” page
there is an inline frame which loads content from the Moodle LMS. This allows LMS
users to remain within the general environment of the toolkit, and provides a cohesive
appearance. The graphical appearance of the LMS was customised to be harmonious
with the general appearance of the toolkit.
66 D3.1 Societal Impact Assessment Manual ASSERT
Users can also access the LMS directly at http://www.assert.maisondx.com/lms which
may be preferable for users with smaller displays.
Users of the LMS are registered for one or more “Classes”, which are primarily a way of
structuring access to different types of content. The ASSERT toolkit features three clas-
ses: one dedicated class for each of the Masterclass events and one class which works as
the ASSERT expert database.
The database class is currently the largest group with around 170 registered partici-
pants. The main utility of this class is provided within Moodle by two forums. One of
these is a news forum for posts by the ASSERT consortium under the terms of use pro-
vided to participants. This forum has been configured to operate like a mailing list. The
second forum is an internal forum to which users can subscribe and share messages
between themselves. Email from this forum has been disabled by default in order to
prevent spamming, but users are able to reactive receiving email for their own account
if they wish. Users are able to modify their profile in the LMS and view the profiles of
other participants. By default only basic information is provided.
The Masterclass groups are more fully featured and provide a library of ASSERT doc-
umentation (all the published deliverables, group exercises, additional briefing papers
and reading material for the masterclass, as well as administrative and organisational
documents such as the agenda), chatrooms and forums, as well as user profiles. Users
can send direct messages to other participants. After the Masterclass events, these clas-
ses can persist and host content generated at the Masterclass, including photos, notes,
etc. Users of these classes are divided into two types “teachers” and “students” (the
terminology reflects Moodle’s general use in educational settings). In addition to the
access available to students, teachers are able to edit the content of the class and send
messages to all participants.
Moodle offers a very large range of features and tools for educational purposes, such as
quizzes, assessment and feedback tools, and even gamification “badges” for particular
achievements. Many of these were unnecessary for the purposes of the toolkit and have
been deactivated. Additionally, the guiding principles of the set up of the toolkit are to
provide users with access to important and useful information, but not to overburden
them with unnecessary content, overwhelm them with spam emails, or put their per-
sonal data at risk.
The following figure shows the structure of the public-facing pages created with Word-
Press. This structure is intended to cover the range of topics and material created and
curated by the ASSERT project whilst still being easy to navigate. The website is only
ever one two pages “deep” making it easier to navigate back to the main page.
Figure 15: Structure of public-facing content pages
67 D3.1 Societal Impact Assessment Manual ASSERT
The section “What is societal impact?” is historical and theoretical, providing a sum-
mary of societal impact in past and current research frameworks. This section is in-
tended to provide background. “Conducting a societal impact assessment” is the largest
section and the core of the public facing toolkit. This section draws upon the ASSERT
deliverables, to provide a structured approach to practically incorporating a societal
impact assessment exercise in a security research project. “Case studies” provides con-
crete examples of how a societal impact assessment might work in practice. Many users
will need this section in order to conceptualise how and SIA might work in their own
areas of work. This section is intended to expand over time as more case studies be-
come available. The “Resources” section collates external resources, including tools
created by other FP7 research projects, which can contribute to the conduct of an SIA.
Finally the “ASSERT Masterclass” section contains information about previous and
future ASSERT masterclasses, as well as a web-form to express interest in participating
in these. This section also contains access to the LMS and the registered-user-only con-
tent hosted there.
The key intention with the content in these sections has been to re-structure the infor-
mation produced by the consortium in manner more accessible than the traditional
document-based project deliverables. The sections point towards the texts from which
they are drawn for interested readers, but aim to provide the key information in these.
Video content based upon interviews with security research experts at ASSERT project
workshops is in the process of being produced to add this element to the toolkit.
Figure 16: Screenshot of the front page of the toolkit.
What is Societal
Impact?
History of SIA
Societal impact
under FP7
Societal impact in
H2020
Conducting a
Societal Impact
Assessment
Why assess
societal impact?
Good practice
criteria
Methodology
Planning and
Preparation
Consultation and
analysis
Reporting and
responding
Case Studies
Public Transport
VIC-SIGMA
Resources
Articles and Books
Tools
Stakeholder
engagement and
consultation
ASSERT
Masterclass
Express interest
Online course log-
in
Stirling
Masterclass
Brussels
Masterclass
68 D3.1 Societal Impact Assessment Manual ASSERT
Figure 17: Screenshot of the learning management system
69 D3.1 Societal Impact Assessment Manual ASSERT
Figure 18: Screenshot of the Masterclass courses
70 D3.1 Societal Impact Assessment Manual ASSERT
Figure 19: Screenshot of user profile
The masterclass forum element of the toolkit was user-tested with the participants at
the ASSERT Masterclass event in Stirling in February 2014, and will be further tested
with the participants at the Brussels Masterclass in April 2014.
71 D3.1 Societal Impact Assessment Manual ASSERT
REFERENCES
Becker, Henk A., and Frank Vanclay, The International Handbook of Social Impact
Assessment: Conceptual and Methodological Advances, Cheltenham, Edward
Elgar, 2003.
Bishop, Patrick, and Glyn Davis, “Mapping public participation in policy choices”, Aus-
tralian Journal of Public Administration, Vol.61, No.1, 2002, pp. 14-29.
Braun-Thürmann, Holger, Innovation, Transcript, Bielefield, 2005.
Bruner, J., The Process of Education. The President and Fellows of Harvard College,
Cambridge, MA, 1960.
Buzan, Barry, and Lene Hansen, The Evolution of International Security Studies,
Cambridge University Press, Cambridge UK, 2009.
Buzan, Barry, Ole Weaver and Jaap de Wilde, Security: A New Framework for Analy-
sis, Lynne Rienner, London, 1998.
Chilton, P., Security Metaphors: Cold War Discourse from Containment to Common
House, Peter Lang, New York, 1996.
Deibert, Ron, “Circuits of Power: Security in the Internet Environment” in J.P. Singh
and James N. Rosenau (eds.), Information Technologies and Global Politics: the
Changing Scope of Power and Governance, Suny Press, New York, 2002, pp.
115-142.
Norman Lee and Colin Kirkpatrick, “Evidence-based policy making in Europe: an eval-
uation of European Commission integrated impact assessments”, Impact As-
sessment and Project Appraisal, Vol. 24, No. 1, 2006, pp. 23-33.
Lianos, Michaelis, “Dangerization and the End of Deviance: The Institutional Environ-
ment”, British Journal of Criminology, Vol. 40, No. 2, Spring 2000, pp . 261-278.
http://bjc.oxfordjournals.org/content/40/2.toc
European Commission, “Key procedural steps for the Commission/smart-
regulation/impact”, 20 December 2013. http://ec.europa.eu/smart-
regulation/impact/ia_key/ia_key_en.htm
European Commission, “Secure Societies – protecting freedom and security of Europe
and its citizens”. http://ec.europa.eu/programmes/horizon2020/en/h2020-
section/secure-societies-%E2%80%93-protecting-freedom-and-security-europe-
and-its-citizens
European Commission, “Smart Regulation”, 13 January 2014.
http://ec.europa.eu/smart-regulation/index_en.htm
European Security Research Advisory Board, Meeting the challenge: The European
Security Research Agenda, 2006.
http://ec.europa.eu/enterprise/policies/security/files/esrab_report_en.pdf
European Parliament, Council and Commission, Charter of Fundamental Rights of the
European Union, 2010/C 83/02, OJ, Brussels, 30.3.2010. http://eur-
72 D3.1 Societal Impact Assessment Manual ASSERT
lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:C:2010:083:0389:0403:en:PD
F
Genus, Audley, “Rethinking constructive technology assessment as democratic, reflec-
tive discourse”, Technology Forecasting and Social Change, No. 73, No. 1, 2006,
pp. 13-26. http://www.sciencedirect.com/science/journal/00401625/73/1
Guillemin, Marilys, and Lynn Gillam, “Ethics, Reflexivity and “ethically Important”
moments in research”, Qualitative Inquiry, Vol. 10, No. 2, 2004, pp. 261-280.
Kemp, Deanna, “Understanding the Organizational Context”, in Frank Vanclay and Ana
Maria Esteves, (eds.), New Directions in Social Impact Assessment: Conceptual
and Methodological Advances, Edward Elgar, Cheltenham, 2011, p. 30.
Mason, Jennifer, Qualitative Researching, Sage, London, 1996.
McCarthy, Sabhbh, Report of the Societal Impact Expert Working Group: EC DG
ENTR Report, February 2012.
Müth, Matthias, Verkherspolitik in Metropolen Südostasiens (the politics of traffic in
South East Asian Metropolis areas), Abera, Hamburg, 2003.
Neocleous, Mark, “Security, Liberty and the Myth of Balance: Towards a critique of
security politics”, Contemporary Political Theory, Vol.6, Issue 2, May 2007, pp.
131-149. http://www.palgrave-journals.com/cpt/journal/v6/n2/index.html
Nissenbaum, Helen, Privacy in Context: Technology, Policy, and the Integrity of So-
cial Life, Stanford Law Books, Stanford, 2009.
Organisation for Economic Co-operation and Development (OECD), Stakeholder In-
volvement Techniques, Paris, 2004. http://www.oecd-
nea.org/rwm/reports/2004/nea5418-stakeholder.pdf
O’Faircheallaigh, Ciaran, “Public participation and environmental impact assessment:
Purposes, implications and lessons for public policy making”, Environmental
Impact Assessment Review, Vol. 30, Issue 1, January 2010, pp. 19-27.
Peterman, William, “Advocacy vs. collaboration: comparing inclusionary community
planning models”, Community Development Journal, Vol. 39, No. 3, 2004, pp.
266-276.
Rip, Arie, and Harro van Lente,“Bridging the gap between innovation and ELSA: The
TA program in the Dutch Nano-R&D Program Nano Ned”, Nanoethics, Vol. 7, Is-
sue 1, April 2013, pp. 7-16. http://link.springer.com/journal/11569/7/1/page/1
Rip, Arie, and Johan Schot, “The Past and Future of Constructive Technology Assess-
ment”, Technological Forecasting and Social Change, Vol. 54, Nos. 2-3, 1997, pp.
251-268.
Rip, Arie, and Johan Schot, “Identifying Loci for the influencing the dynamics of tech-
nology development”, in Knut H. Sorensen and Robin Williams (eds.), Shaping
Technology, Guiding Policy: Concepts, Spaces and Tools, Edward Elgar, Chel-
tenham, 2002, pp. 155-172.
73 D3.1 Societal Impact Assessment Manual ASSERT
Schot, Johan, and Frank W. Geels, “Strategic niche management and sustainable inno-
vation journeys: Theory, findings, research agenda and policy”, Technology
Analysis & Strategic Management, Vol. 20, No. 5, 2008. pp. 537-554.
Stoddart, Jennifer, “Auditing Privacy Impact Assessments: The Canadian Experience”,
in David Wright and Paul De Hert (eds.), Privacy Impact Assessment, Springer,
Dordrecht, 2013.
United Nations, Follow-up to General Assembly resolution 64/291 on human security -
Report of the Secretary-General, 5 April 2012.
https://docs.unocha.org/sites/dms/HSU/Publications%20and%20Products/Rep
orts%20of%20the%20Secretary%20General/A-66-763%20English.pdf
Vanclay, Frank, “International Principles for Social Impact Assessment”, Impact As-
sessment and Project Appraisal, Vol. 21, No. 1, 2003, pp. 5-12.
Vanclay, Frank,“International Principles for Social Impact Assessment: their evolu-
tion”, Impact Assessment and Project Appraisal , Vol. 21, No. 1, 2003a, pp. 3-4.
Vanclay, Frank, “Social Impact Assessment: International Principles”, IAIA, Special
publication series No.2, May 2003.
http://www.iaia.org/publicdocuments/special-publications/sp2.pdf
Vanclay, Frank, and Ana M. Esteves (eds.), New Directions in Social Impact Assess-
ment: Conceptual and Methodological Assumptions, Edward Elgar, Cheltenham,
2011.
Wright, David, “The state of the art in privacy impact assessment”, Computer Law &
Security Review, Vol. 28, No. 1, Feb. 2012, pp. 54-61.
http://www.sciencedirect.com/science/journal/02673649.
Wright, David and Paul de Hert, “Introduction to Privacy Impact Assessment”, in David
Wright and Paul de Hert (eds.), Privacy Impact Assessment, Springer, Dordrecht,
2012, pp. 3-33.
Wright, David, and Charles Raab, “Constructing a surveillance impact assessment”,
Computer Law & Security Review, Vol. 28, No. 6, Dec 2012, pp. 613-626.
Wright, David, and Kush Wadhwa, “A step-by-step guide to privacy impact assess-
ment”, presentation paper for the second PIAF Workshop, Sopot, Poland, 24
April 2012. http://www.piafproject.eu/ref/A_step-by-
step_guide_to_privacy_impact_assessment-19Apr2012.pdf
Wright, David, Raphael Gellert, Rocco Bellanova, Serge Gutwirth, Marc Langhenrich,
Michael Freidewald, Dara Hallinan, Silvia Venier and Emilio Mordini,”Privacy
Impact Assessment and Smart Surveillance: A State of the Art Report, Delivera-
ble 3.1, SAPIENT Project, May 2013.