AI MASTER CLASS STRUCTURED LITERATURE REVIEWAnders Kofod-PetersenProfessor, Department of Computer and Information Science, NTNUDeputy Director, The Alexandra Institute, Denmark

AI MASTER CLASS

OVERVIEW OF MASTER CLASS‣ 2018

‣ 24/8 – Introduction

‣ 18/9 – Literature

‣ 21/11 – Writing a scientific paper (ITV454)

‣ 2019

‣ 10-11/1 – DART master student workshop

‣ XX/XX – Research questions, revisited

‣ YY/YY – Reproducibility

‣ ZZ/ZZ – Empirical research

‣ KK/KK – How to write a thesis

‣ 25-26/4 – DART master student conference

THE THESIS‣ You might save the planet. If you don’t know how and why, and can’t

describe it, you won’t get a good grade!

‣ The average student can reproduce knowledge

‣ The above average student can add to knowledge

‣ The good student can reflect on the addition

‣ All of this is in your thesis!

METHOD IS OUR FRIEND

STRUCTURED LITERATURE REVIEW

TODAY‣ How do we know what we know?

‣ SLR

‣ Structure

‣ Planning

‣ Conducting

‣ Reporting

How to do a Structured Literature Review in computer science

Anders Kofod-PetersenVersion 0.2

Contents

1 Introduction 1

2 Structure of a systematic literature review 1

3 Performing a structured literature review 2

3.1 Planning the review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23.2 Conducting the review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

References 7

1 Introduction

How to write a reference list

Doing a systematic literature review is a formal way of synthesising the information avail-able from available primary studies relevant to a set of research questions. The use of sys-tematic literature reviews have traditionally been widespread primarily in medicine (e.g. thewell known Cochrane reviews [1]). Unfortunately it has been used to a much lesser extendin computer science (for an example of how to do reviews in software engineering see: [2]).Systematic literature reviews stand apart from, in computer science the more traditional un-systematic surveys by using a strict methodological framework with a set of well defined stepscarried out in accordance with a predefined protocol.

Using a systematic literature review is in no way a guarantee of finding all relevant lit-erature in a given area. However, there are several advantages in using it: A systematicliterature review can map out existing solutions before a researcher attempts to tackle anarea; it helps researchers in avoiding bias in their work; publishing these reviews also benefitsthe community by allowing others to avoid duplicating the e↵ort; it allows researchers toidentify gaps of knowledge; and it highlights the areas where additional research is required.

If a systematic literature review is conducted thoroughly it fulfils the advantages describedabove and thereby gains scientific value.

This documents attempts to give a short introduction to how to conduct a structuredliterature review within computer science. The examples used are taken from [3].

2 Structure of a systematic literature review

A systematic review has three main phases: i) planning, ii) conducting and iii) reporting.Each of these phases are divided into several steps.

1

HOW DO WE KNOW WHAT WE KNOW?‣ What is your area of research?

‣ Is it interesting?

‣ Is it novel?

‣ Somebody has probably done something similar

‣ Who?

‣ What?

‣ How?

‣ Result?

THE RESEARCH BOX

STRUCTURED LITERATURE REVIEW (SLR)‣ Structure

‣ Planning

‣ Conducting

‣ Reporting

SLR - STRUCTURE‣ Planning

1. Identification of the need for a review

2. Commissioning a review

3. Specifying the research question(s)

4. Developing a review protocol

5. Evaluating the review protocol

WHAT ARE YOUR RESEARCH QUESTIONS?We have a problem (P) that we wish to tackle, under certain constraints(C) to develop a solution (S).

‣ RQ1 What are the existing solutions to P?

‣ RQ2 How does the different solutions found by addressing RQ1 compare to each other with respect to C?

‣ RQ3 What is the strength of the evidence in support of the different solutions?

‣ RQ4 What implications will these findings have when creating S?

THE REVIEW PROTOCOL ‣ The review protocol is what makes your work reproduceable

‣ Begin by making a initial protocol and refine

CONDUCTING A SLR1. Identification of research

2. Selection of primary studies

3. Quality Assessment

4. Data extraction

IDENTIFICATION OF RESEARCH‣ Which sources are to be searched, e.g.

‣ ACM

‣ IEEE

‣ ISI

‣ Science Direct

‣ CiteSeer

‣ SpringerLink

‣ Wiley Inter Science

‣ Engineering Village

‣ Google Scholar

‣ How to search them

‣ Terms

‣ Procedure

This goes into protocol

WHAT TERMS ARE WE SEARCHING FOR?‣ There are many terms that are more or less synonymous

‣ We are looking for the the right combinations of the right synonyms

‣ We are looking for words that have the same semantic meaning (in the domain)

‣ We might be looking for the right combinations of hypernyms and hyponymsTable 1: Search terms

Group 1 Group 2 Group 3 Group 4

Term 1 Synonym1 Synonym2 Synonym3 Synonym4

Term 2 Synonym1 Synonym2 Synonym3 Synonym4

Term 3 Synonym2 Synonym3

Term 4 Synonym3

Term 5 Synonym3

Group 1

Group 2

Group 3

Group 4

Target studies

Figure 1: Relevant studies

4

WHAT TERMS ARE WE SEARCHING FOR?Group 1

Group 2

Group 3

Group 4

Target studies

Search strategy is done by implementing AND, and OR between terms:

([G1, T1] OR [G1,T2]) AND

([G2,T1] OR [G2,T2] OR [G2,T3]) AND

([G3,T1] OR [G3,T2] OR [G3,T3] OR [G3,T4] OR [G3,T5]) AND

([G4,T1] OR [G4,T2)

SELECTION OF PRIMARY STUDIES‣ Searching will generally return far too many results. We can limit the set

by:

1. Remove duplicates (keep the highest ranking source)

2. Remove the same study published in different sources (keep the highest ranking source)

3. Remove studies published before a certain date (or even after)

QUALITY ASSESSMENT

1. Abstract inclusion screening

2. Full text inclusion screening

3. Full text quality screening

The set of papers constructed by applying this search strategy is now ready to go thoughthe selection process.

Step 2: Selection of primary studies

Applying the search described above will most likely return a number of articles far largerthan manageable. The relevant articles should now be selected. The protocol should describedexactly which criteria should be applied in this selection process. However, some points canbe regarded as general and used as removal criteria:

1. Duplicates (keep the highest ranking source),

2. The same study published in di↵erent sources (keep the highest ranking source),

3. Studies published before a certain date (or even after).

Applying this selection now leaves us with a set of relevant studies that can now be filteredwith respect to quality.

Step 3: Study quality assessment

The purpose of this step is to filter away studies that are not thematically relevant to the areachosen. The protocol should define exactly which inclusion (IC) and quality criteria (QC) areemployed (Table 2 gives some examples of criteria).

Table 2: Inclusion and quality criteria

Criteria identification Criteria

IC 1 The study’s main concern is PIC 2 The study is a primary study presenting empirical results

IC 4 The study focuses on CIC 5 The study describes an SQC 1 There is a clear statement of the aim of the researchQC 2 The study is put into context of other studies and research

The criteria can be divided into: primary, secondary and quality screening criteria. In theexample described in Table 2 IC 1 and 2 would be the primary; IC 3 and 4 the secondary;and QC 1 and 2 the quality criteria. The criteria can now be applied in a three stage process:

1. Abstract inclusion criteria screening,

2. Full text inclusion criteria screening,

3. Full text quality screening.

Each step should be thoroughly documented as part of the final protocol. Once the setof studies have gone through this process it is (most likely) further reduced and can now gothough the next step of detailed quality assessment.

5

Spoiler alert: This is for next time!

FINAL QUALITY ASSESSMENT ‣ QC 1 Is there is a clear statement of the aim of the research?

‣ QC 2 Is the study is put into context of other studies and research?

‣ QC 3 Are system or algorithmic design decisions justified?

‣ QC 4 Is the test data set reproducible?

‣ QC 5 Is the study algorithm reproducible?

‣ QC 6 Is the experimental procedure thoroughly explained and reproducible?

‣ QC 7 Is it clearly stated in the study which other algorithms the study's algorithm(s) have been compared with?

‣ QC 8 Are the performance metrics used in the study explained and justified?

‣ QC 9 Are the test results thoroughly analysed?

‣ QC 10 Does the test evidence support the findings presented?

‣ SRQ1 Which single-objective algorithms exist that are suitable for multi-objective extension?

‣ SRQ2 Which multi-objective techniques are used when extending a single-objective algorithm?

‣ SRQ3 Which multi-objective algorithms should be compared with when evaluating an algorithm?

‣ SRQ4 Which test functions are used to test multi-objective algorithms?

‣ SRQ5 Which performance metrics are used to measure multi-objective algorithms?

EXAMPLE (FROM SINGLE-OBJECTIVE TO MULTI-OBJECTIVE)

EXAMPLE (FROM SINGLE-OBJECTIVE TO MULTI-OBJECTIVE)

A. Structured Literature ReviewProcess

This appendix explains the review process of the structured literature review usedto find the related work in Chapter 3.

A.1. Identification of ResearchThis section presents the steps that were required to find the related work.

A.1.1. Search EngineThe majority of articles were found using Google Scholar. Google Scholar aggregatesmost of the prominent digital research literature libraries in one place, making itideal for this task. Additionally, some articles were identified using IEEE Xploresince the site provides better filtering options than Google Scholar and many ofthe related articles are hosted there.

A.1.2. Search StrategyBefore starting the search, a set of search terms had to be made. The search termsare based on the search research questions from Section 3.1.1 and are grouped intothree categories: algorithm, single or multi-objective, and type of contribution.

A visualization using AND (·) between groups and OR (‚) inside groups is shownbelow:

(metaheuristic ‚ optimization algorithm ‚ evolutionary algorithm) ·(multi-objective ‚ single-objective) ·(new ‚ improved ‚ hybrid)

This search resulted in around 400 articles.

95

A. Structured Literature Review Process

A.2. Selection of Primary StudiesThe first sizable cut of articles happened here. Because of the large amount ofarticles, some were cut based on either reading the title, or a quick read of theabstract. The biggest reasons for cutting articles in this step were:

1. Article did not focus on optimization algorithms.

2. Article covered an algorithm that was problem specific.

3. Article focused on improvement but was very old.

After this screening, there were 104 articles left.

A.3. Study Quality AssessmentTo asses the quality of the literature several criteria were used. Based on theinclusion and quality criteria it was decided whether the articles would be removedor not.

A.3.1. Inclusion CriteriaFor a article to be included in the literature review it must cover one or more ofthe following criteria shown in Table A.1. The first three inclusion criteria (IC) arethe primary inclusion criteria, while the two next are secondary. At the end of thetable, the quality criteria (QC) are listed.

Table A.1.: Criteria grouped into primary inclusion criteria, secondary inclusioncriteria and quality criteria.

Criteria identification CriteriaIC 1 Presents a new algorithmIC 2 The study’s main concern is metaheuristicsIC 3 Improves an existing algorithmIC 4 Converts an algorithm from single-objective to multi-objectiveIC 5 The study focuses on multi-objective optimizationQC 1 The study is innovativeQC 2 The algorithm design is justified and reproducible

A.3.2. Abstract Inclusion Criteria ScreeningIn this step, the abstracts of the remaining 104 articles were read and evaluatedbased on the inclusion criteria seen in Table A.1. In certain cases where the abstractdid not give su�cient information about the article, the introduction and conclusion

96

Terms

Selection of primary studies

EXAMPLE (FROM SINGLE-OBJECTIVE TO MULTI-OBJECTIVE)

A. Structured Literature Review Process

A.2. Selection of Primary StudiesThe first sizable cut of articles happened here. Because of the large amount ofarticles, some were cut based on either reading the title, or a quick read of theabstract. The biggest reasons for cutting articles in this step were:

1. Article did not focus on optimization algorithms.

2. Article covered an algorithm that was problem specific.

3. Article focused on improvement but was very old.

After this screening, there were 104 articles left.

A.3. Study Quality AssessmentTo asses the quality of the literature several criteria were used. Based on theinclusion and quality criteria it was decided whether the articles would be removedor not.

A.3.1. Inclusion CriteriaFor a article to be included in the literature review it must cover one or more ofthe following criteria shown in Table A.1. The first three inclusion criteria (IC) arethe primary inclusion criteria, while the two next are secondary. At the end of thetable, the quality criteria (QC) are listed.

Table A.1.: Criteria grouped into primary inclusion criteria, secondary inclusioncriteria and quality criteria.

Criteria identification CriteriaIC 1 Presents a new algorithmIC 2 The study’s main concern is metaheuristicsIC 3 Improves an existing algorithmIC 4 Converts an algorithm from single-objective to multi-objectiveIC 5 The study focuses on multi-objective optimizationQC 1 The study is innovativeQC 2 The algorithm design is justified and reproducible

A.3.2. Abstract Inclusion Criteria ScreeningIn this step, the abstracts of the remaining 104 articles were read and evaluatedbased on the inclusion criteria seen in Table A.1. In certain cases where the abstractdid not give su�cient information about the article, the introduction and conclusion

96

Inclusion Criteria

EXAMPLE (FROM SINGLE-OBJECTIVE TO MULTI-OBJECTIVE)

A. Structured Literature Review Process

Table A.3.: Rating of each article using the quality criteria.Article QC1 QC2 QC3 QC4 QC5 QC6 QC7 QC8 QC9 QC10 QC11 ScoreAMO 1 1 1 1 1 1 0.5 1 1 1 1 10.5DA 1 1 1 1 1 1 1 1 1 1 0.5 10.5EWA 1 1 1 1 1 1 1 1 1 1 1 11LOA 1 1 1 1 1 1 1 1 1 1 1 11MS 1 1 1 1 1 1 1 1 1 1 1 11SOS 1 1 1 1 1 1 1 1 1 1 1 11AMOSA 1 1 1 1 1 1 0.5 1 0.5 1 1 10DPP2 1 1 1 1 1 1 0.5 1 0.5 1 1 10MOALO 1 1 1 1 1 1 1 1 0.5 1 1 10.5(MO)DFA 1 1 1 1 1 1 1 1 0.5 1 1 10.5MOEA/DD 1 1 1 1 1 1 1 1 1 1 1 11MOFPA 1 0.5 1 1 0.5 1 0.5 0.5 1 1 1 9MOGWO 1 1 1 1 1 1 1 1 1 1 1 11MOWOA 1 1 1 0.5 1 1 0.5 1 1 1 1 10NSIWO 1 1 1 1 1 1 0.5 1 0.5 1 1 10NS-MFO 1 1 1 1 1 1 0.5 1 1 1 1 10.5SMPSO 1 1 1 1 1 1 0.5 1 1 1 1 10.5TLBO 1 1 0.5 1 1 1 0 1 1 1 1 9.5

100

EXAMPLE (FROM SINGLE-OBJECTIVE TO MULTI-OBJECTIVE)A.3. Study Quality Assessment

Table A.2.: The final selection of articles for the literature review.Abbrevation Title Author(s)AMO Animal migration optimization: an optimiza-

tion algorithm inspired by animal migrationbehavior

Li et al. [2014]

DA A new dandelion algorithm and optimizationfor extreme learning machine

Gong et al. [2017]

EWA Earthworm optimization algorithm: a bio-inspired metaheuristic algorithm for global op-timization problems

Wang et al. [2015]

LOA Lion optimization algorithm (LOA): a nature-inspired metaheuristic algorithm

Yazdani and Jolai [2016]

MS Moth search algorithm: a bio-inspired meta-heuristic algorithm for global optimization prob-lems

Wang [2016]

SOS Symbiotic Organisms Search: A new meta-heuristic optimization algorithm

Cheng and Prayogo [2014]

AMOSA A simulated annealing-based multiobjective op-timization algorithm: AMOSA

Bandyopadhyay et al. [2008]

DPP2 A modified dual-population approach for solvingmulti-objective problems

Bui et al. [2017]

MOALO Multi-objective ant lion optimizer: a multi-objective optimization algorithm for solving en-gineering problems

Mirjalili et al. [2017]

(MO)DFA Dragonfly algorithm: a new meta-heuristic opti-mization technique for solving single-objective,discrete, and multi-objective problems

Mirjalili [2016]

MOEA/DD An evolutionary many-objective optimizationalgorithm based on dominance and decomposi-tion

Li et al. [2015]

MOFPA Multi-objective Flower Algorithm for Optimiza-tion

Yang et al. [2013]

MOGWO Multi-objective grey wolf optimizer: a novelalgorithm for multi-criterion optimization

Mirjalili et al. [2016]

MOWOA Multi-objective whale optimization Kumawat et al. [2017]NSIWO Multiobjective invasive weed optimization: Ap-

plication to analysis of Pareto improvementmodels in electricity markets

Nikoofard et al. [2012]

NS-MFO Non-dominated sorting moth flame optimiza-tion (NS-MFO) for multi-objective problems

Savsani and Tawhid [2017]

SMPSO SMPSO: A new PSO-based metaheuristic formulti-objective optimization

Nebro et al. [2009a]

TLBO A comparative study of a teaching–learning-based optimization algorithm on multi-objectiveunconstrained and constrained functions

Rao and Waghmare [2014]

99

‣ RQ1: Which AI techniques have been applied to the candidate ranking problem in e-recruitment?

‣ RQ2: Have techniques from semantic web been used to improve candidate ranking in e-recruitment?

‣ RQ3: What is the state of the Art given the solutions from RQ1 and RQ2?

‣ RQ4: What implications will these findings have when creating a design for our system?

‣ RQ5: Given RQ4, what strength of the evidence is supported for the implication?

EXAMPLE (LAZY LEARNED SCREENING FOR EFFICIENT RECRUITMENT )

EXAMPLE (LAZY LEARNED SCREENING FOR EFFICIENT RECRUITMENT )Terms

Selection of primary studies

14 CHAPTER 2. BACKGROUND THEORY AND MOTIVATION

TermID Group1 Group21 Artificial Intelligence Human Resource2 Machine Learning Employee Recruitment3 Reinforcement Learning E-Recruitment4 Deep Learning5 Semantic Web6 Case based reasoning

Table 2.1: Original search terms

(”Artificial Intelligence” OR ”Machine Learning” OR ”Reinforcement Learn-

ing” OR ”Deep Learning” OR ”Semantic Web” OR ”Case based reasoning”)

AND (”Human resource” OR ”Employee recruitment” OR ”E-Recruitment”)

Library URL AssigneeACM digitallibrary

http://dl.acm.org/advsearch.cfm Erik

IEEE Xplore http://ieeexplore.ieee.org/search/advsearch.jsp MagnusWeb of Sci-ence

https://apps.webofknowledge.com Erik

ScienceDirect http://www.sciencedirect.com/science/search MagnusCiteSeerX http://citeseerx.ist.psu.edu/advancedsearch ErikSpringerLink https://link.springer.com/advanced-

searchMagnus

Wiley OnlineLibrary

http://onlinelibrary.wiley.com/advanced/search Erik

Oria https://bibsys-almaprimo.hosted.exlibrisgroup.com

Magnus

GoogleScholar

https://scholar.google.no/ Erik

Table 2.2: Digital archives

2.2.4 Screening and Quality assessment

In order to ensure that time and resources were spent on researching the most

relevant material we applied some selection criteria and performed a quality as-

sessment of the studies. During the quality assessment we have applied a three

step process ”Primary”, ”Secondary” and ”Quality”. During the first step ”Pri-

EXAMPLE (LAZY LEARNED SCREENING FOR EFFICIENT RECRUITMENT )Inclusion Criteria

2.2. STRUCTURED LITERATURE REVIEW PROTOCOL 15

mary” we screen the abstract of papers using a set of primary inclusion criteria.

The papers passing the primary screening move on to the secondary screening.

The secondary screening is done on the full text of the paper, here we use the list

of secondary inclusion criteria. When a paper passes the two screening criteria

we move on to quality assessment.

Primary inclusion criteria

• IC1 The study’s main concern is concerning e-recruiting

• IC2 The study is a primary study presenting empirical results

Secondary inclusion criteria

• IC3 The study focuses on finding and/or ranking candidates.

• IC4 The study has a solution to candidate ranking.

Quality criteria

The following quality criteria were were used to evaluate the studies. We did

this in order to narrow down the amount of studies and get the most relevant

information.

• QC1 Is there a clear statement of the aim of the research?

• QC2 Is the study put into context of other studies and research

• QC3 Are system or algorithmic design decisions justified?

• QC4 Is the test data set reproducible

• QC5 Is the study algorithm reproducible?

• QC6 Is the experimental procedure thoroughly explained and reproducible?

• QC7 Is it clearly stated in the study which other algorithms the study’s

algorithm(s) have been compared with

• QC8Are the performance metrics used in the study explained and justified?

• QC9 Are the test results thoroughly analyzed?

EXAMPLE (LAZY LEARNED SCREENING FOR EFFICIENT RECRUITMENT )

18 CHAPTER 2. BACKGROUND THEORY AND MOTIVATION

Figure 2.5: Applying the quality cirteria, studies marked gray are below thecuto↵ point

EXAMPLE (LAZY LEARNED SCREENING FOR EFFICIENT RECRUITMENT )2.3. MOTIVATION AND RELATED WORK 17

Figure 2.4: Studies passing the inclusion criteria

WHAT’S YOUR POISON?