source deliverable8 6 knowledge database (clean)...d8.4!–!fp7–!313288! 2!...
TRANSCRIPT
Vrije Universiteit Brussel
Database D8.6
Söebech, Olöf
Publication date:2017
Document Version:Final published version
Link to publication
Citation for published version (APA):Söebech, O. (2017). Database D8.6: SOURCE A Virtual Centre of Excellence for Research Support andCoordination on Societal Security. Societal Security Network (SOURCE).
General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal
Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.
Download date: 29. May. 2020
D8.6 DATABASE
This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 313288.
VIRTUAL CENTRE OF EXCELLENCE FOR RESEARCH SUPPORT AND COORDINATION ON SOCIETAL SECURITY
D8.6 [Database]
Abstract:
The SOURCE knowledge database aims to be one stop shop for selected societal security relevant content (other than news articles covered in D8.4) that is freely available. This includes scientific publications, policy documents, research projects, blogs, video interviews and presentations, research projects, reports and links. This is also where users can access all SOURCE related publications, such as our technology trend fact sheets (D8.2). Content is added manually, and the knowledge base is frequently updated by all partners.
Contractual delivery date: 36
Actual delivery date: 37
Version: 1
Total Number of pages:
Author: Olof Soebech
Contributors: Irina van der Vet, Christof Roos
Reviewers:
Dissemination level: PU
Version Log Issue Date Rev. No. Author Change 0.1
0.2
0.3
0.4
0.5
01.01.2014 31.12.2018
Coordinator: PRIO
www.societalsecurity.net
Deliverable submitted in January 2017(M37) in fulfilment of requirements of the FP7 project, SOURCE – Virtual Centre of excellence for research support and coordination on societal security (SEC-‐2012.7.4-‐2: Networking of researchers for a high level multi-‐organisational and cross-‐border collaboration)© Copyright 2014 the SOURCE Consortium (PRIO, FOI, CIES, FhG, TNO, CEPS, VUB, VICESSE, KCL, EOS, TEC, Sciences Po). All rights reserved. This document may change without notice.
Contents Introduction ............................................................................................................................................ 1
1. Purpose of the Knowledgebase .................................................................................................. 2
2. Knowledgebase development .................................................................................................... 2
2.1. Knowledgebase background research ................................................................................... 3
2.2. Knowledgebase Content Organisation and categorisation -‐ Filtering ................................... 4
2.3. Subcontracting ....................................................................................................................... 6
Using the Knowledgebase ....................................................................................................................... 6
3. Technical Features – back end ................................................................................................... 7
3.1. Knowledgebase content input ............................................................................................... 7
4. User experience – front end ..................................................................................................... 12
4.1. Main page: searching the Knowledgebase .......................................................................... 12
5. Next steps ................................................................................................................................. 15
Annex 1 – Letter to Partners ................................................................................................................. 17
Annex 2 – Filters .................................................................................................................................... 19
Annex 3 – Database user stories ........................................................................................................... 22
Annex 4 – Explanatory document for database input and use ............................................................. 25
Annex 5 – Purchase order sprint 1 of document database ................................................................... 28
D8.4 – FP7 – 313288
1
Introduction
As outlined in the Discription of Work (DOW), the primary objective of the work package eight (WP8) is:
• to gather and organize information and research of relevance to research and development on societal security, its conceptualization and implementation.
• to create, collect and sort information of relevance for the study and operationalization of societal security (scholarly, technical, legal and mass media trends).
The secondary objectives include:
• To track and document changes in legal regimes of relevance to societal security in Europe. • To continuously map the state of the play of technological evolution in the security sector. • To create and maintain an index of current and past scholarly research of relevance for societal
security. • To monitor media responses to threats and insecurity. • To maintain a database with a reasoned storage and indexing function.”1
Task 6 of WP8 (D8.6), the Database, in particular, will serve to “gather, catalogue and store raw information on societal security from the widest range of sources for further analysis in use in the activities of the network… The database and website will become a long-‐term research and development tool for scholars, technology developers, lawyers, crisis response units etc.” 2 The information available in the database should continuously grow and provide an overview of the field. It will be a space that connects the fragment of research, as well as a place where the SOURCE network can promote their material.
The VUB as a task leader, along with all SOURCE partners, developed and launched a knowledge database under the Observatory and a document hub on the SOURCE website in 2016. The knowledge base aimed at providing an overview of different items withint the content that were considered to be relevant to societal security by the consortium partners, and eventually by the registered users of the SOURCE network of excellence.
During the first phase of the SOURCE project, work included: background research, analysising similar databases and identification of desirable tools/functions, defining the scope and content of the database and developing a strategy and methodology for collecting relevant data for the content. These tasks were followed by actual construction of the database in collaboration with the subcontracted programmers. During the second phase, the database was made public under the Observatory part of the SOURCE website, and project partners were encouraged to add their content to the database.
1 SOURCE project DOW 2 DOW of the Source Project; part B – p. 22.
D8.4 – FP7 – 313288
2
The Knowledgebase went online in mid 2016 and was opened up for uploads for project partners. Currently the Knowledgebase includes 88 individual inputs: Blogs, websites, research projects, books, policy documents, legislative acts, reports, scientific publications, interviews (sound/video), legal trend cards, technology trend cards, SOURCE publications and other data.
1. Purpose of the Knowledgebase
The knowledge database has been supposed to be a space where the most relevant data sources for societal security can be found in one place. The possibility for filtering and search options, similar to the Mediawatch,3 allows for targeted searching and easier finding of relevant content depending on a person’s interest in specific areas of societal security. Project partners, and eventually registered users of the SOURCE network are able to add content to the database, making it a space for collective intelligence and collaboration by building up a mutual database that serves all users. Thus, the database provides a great opportunity for anyone interested in societal security to use it as a “one stop shop” in order to find good quality information documents, projects links, sound and video in an easy comprehensive way.
2. Knowledgebase development
The first priorities, when thinking of creating a database, were user friendliness, coherent organization of data, ability to make changes and longevity. Work on the Knowledgebase started simultaneously with the Mediawatch development. The idea of categorising and filtering the content in a user-‐friendly way and providing an easily searchable content was meant to serve both the Mediawatch and the database.
There have been four stages in Knowledgebase development. The first stage consisted of a background research: analysing user needs, scanning existing databases and comparing technological solutions. The second stage defined what type of content would be collected and inserted into the database (e.g. scholarly articles, policy documents, videos, blogs, etc.) and what type of information should be collected on each added item (e.g. title, year of publication, author, abstract and keywords). The third stage consisted of building the database itself, with the contribution of the subcontracted developers, and of giving project partners technical permissions to add content and test the database. The fourth and the current stage, focuses on the growth of the database. For the remainder of the SOURCE project, the Knowledgebase is expected to grow in size in terms of ithe number of items in it, and be made available for input from a larger group of users.
The WP9 leaders will also be responsible for disseminating and promoting the use of the database, which, in turn, will benefit to its growth and usefulness. The Knowledgebase will expand and develop if the partners keep on actively contributing to its growth. There is no specific number of content items aimed for. What the database aims to provide is current material, that is being used by experts in the field (e.g. SOURCE partners who practice research on societal security). The goal is for the database to be useful. This can be measured by the end of the project period through google analytics. That is to
3 See D8.4 report Mediawatch on structure and filtering
D8.4 – FP7 – 313288
3
say, if visitors of the SOURCE website use the knowledge database or not. For the database to be revelvant and useful, we aim for regular input (at least 5 new additions per month), and a variety of sources (from scholartly articles, to legal documents and video blogs).
2.1. Knowledgebase background research
The background research consisted of three specific actions:
1. Exploring other online databases, both from technical perspective and from the perspective of their interface.
2. Asking project partners what type of content they would like to see in a database, as they will not only contribute to producing the the input, but also be active users (user analysis)4.
3. Analysing the content, which deemed as relevant by partners to see how to better organise it and display.
Around 30 different websites were scanned by looking at database management systems, applications and technological features, look and feel as well as existing databases to see what functionalities are currently in use.5 Ther selection criteria for being a part of the scan was open: if it included content organisation and data management, we looked at the website. The intention of the Web scan was obviously not to copy other webtools, but to draw inspiration, and test the latest, most effective solutions in the Internet. Special interest was in analysing what type of information was collected, how it was displayed and how it was filtered or organised. It should also be underlined that the relatively low-‐cost SOURCE database reveals some limitations in its applicability and functionality, compared to very elaborate and often very expensive scanned sites. For instance, crawling and adding automated input of data from other systems was deemed implausable. The web scan was nonetheless important for the brainstorming sessions inside the VUB team.
Next, project partners were sent a letter, where they were asked to select 10 documents they found relevant to societal security. These included: European Policy document, National/local policy document, scientific publication, educational material, Source project document, or other publications (the other document types currently available were added later based on the decision to broaden the scope of content types). The letter and a respective form for filling in information about these 10
4 The project partners are also users. They are practicioners and researchers in the field-‐ thus our expert advisers for the time being 5 For example: Database management systems included: http://heuristnetwork.org, https://www.mongodb.com/, http://couchdb.apache.org/, http://ravendb.net/, https://code.google.com/archive/p/terrastore/, https://github.com/danielwertheim/sisodb-‐provider & http://jackrabbit.apache.org/jcr/index.html Databases: https://www.scopus.com/, scp-‐knowledge.eu, https://www.prio.org/Publications/Publications/, https://www.cristin.no, http://www.isn.ethz.ch/Digital-‐Library/Articles, https://scholar.google.be/, https://forschung-‐sachsen-‐anhalt.de/, http://www.cbsm.com/2010/01/12, https://www.xing.com/, Other: https://www.ning.com/, https://www.researchgate.net/, http://sciencestage.com/, http://scorai.org/
D8.4 – FP7 – 313288
4
documents can be found in Annex 1. Letter to partners. Almost 100 documents were received. That helped us in defining the ways to categorise and organise the information in order for the database to be as user-‐friendly and valuable as possible. This information was used both for the database development and the Mediawatch keyword selection.
Then, with an idea of existing databases, and of the type of documents that would be added to the SOURCE Knowledgebase, a dialogue with the technical developers was established. They were subcontracted in order to: find out what options were feasible within the budget, learn from their expertise on database development, and develop the layout and functions of both back end and front end.
Throughout 2016, the database underwent several consultations with project partners to further define what relevant content should be placed there, and how much effort for entering data in the database manually would be required, versus the options of automated operations within the budget constraints. A decision was made to be selective and to choose the content that the partners and experts in the field find relevant for the database, and to continue manual upload. The added value of the manual input is that it allows for using the same system of key concepts, topics and domains as the Mediawatch tool uses. Additionally, cataloguing and indexing becomes better and getting complete information on each content added can be guaranteed. We choose the fields that need to be added and can thereby get fuller information into the database. This would be more difficult if we used scanning, web crawling or other automated input, as that would be dependent on existing information available in other systems. That is to say, the quality of the data input into the database can be ensured at a higher level than with automated uploads. The negative aspect of the manual input is time consumption and dependency on partners actually adding the content to the database. Finally, another interesting aspect of the manual database is that once the website starts allowing users to register, they will have the option of uploading and inserting content into the database, hence, promoting material they have produced or found interesting to share. Additionally, registered users can potentially have their uploads visible on their profile, so that other users can also see those. By making a unique SOURCE Knowledgebase, we are able to add on new functions and features into it over time, allowing for a more organic growth of the tools in the SOURCE Observatory. The VUB receives email notification for evey new input and acts as a moderator. The moderator has the authority to remove or amend content added. Registered users, with the authorisation to add content are also able to flag any content on the database that is considered irrelevant to the knowledge base and to give their explanation for flagging. The moderator (VUB) receives notification for flags and can then decide whether to remove the content input or not.
2.2. Knowledgebase Content Organisation and categorisation -‐ Filtering6
One of the key challenges to building a database that would collect high quality content related to societal security, was to define key terms and concepts relevant to societal security. The goal was to find a way to collect a broad level of content, covering a range of relevant issues, and at the same time,
6 The same filtering system is used for the Knowledgebase as the Mediawatch, except the database keywords are somewhat fewer and added by the person inputting data manually. The report is therefore overlapping with D8.4
D8.4 – FP7 – 313288
5
to organize this content in a way that would be useful to specific fields within societal security. In order to define the most relevant key words, both for the Mediawatch (D8.4) and for the database, a scan of keywords was undertaken.
As a first step for selecting kewords, over 100 societal security related documents were collected with the help of the SOURCE partners. These were then scanned for keywords through a software. Additionally, we asked each partner to suggest keywords that they found relevant to societal security, based on documents they had developed or read. The final list of keywords numbers over 200 (see Annex 2). Based on these keywords as well as the seven societal security domains identified and suggested in deliverable 3.1, we created a first filter system for collection of online news media.
The final system developed was a layered filtering. These keywords are layered into two categories: a) key concept that are more abstract terms related to societal security such as preparedness, resilience and risk and b) topics attributed to the different domains such as refugees, terrorism, natural disasters or economic crisis. For the Mediawatch D8.4, these keywords are used when crawling the news outlets searching for relevant articles. For the database D8.6, a selection of these Topics and Key concepts shows up as a drop down list for those registered users who are adding content. Each content is tagged with at least one key concept and one topic. This allows users of the database to look for content based on predetermined filters, making their search more targeted.
The filtering is further grouped into 8 specific domains that also are applied in the Mediawatch:
-‐ Physical and Personal Security -‐ Socio-‐Economic Security -‐ Cultural Security -‐ Political Security -‐ Radical Uncertainty Security -‐ Financial Security -‐ Environmental Security
Depending on which words from the topic category are identified in the content, it will be grouped into the different domains. These domains are adapted from Source deliverable 3.1: Methodology workshop and review of available empirical sources 7 which suggested 7 domains related to societal security. VUB added financial security to the 7 domains as one specific work package (WP5) deals with financial security and thus it becomes relevant to the partners and the project. Annex 2 lists key concepts and topics that are currently used to scan news articles when selecting them for the Media Watch. A selection of these are used for the database. It was considered inefficient to allow all terms to be included as the person adding content would get lost in the long list of terms. Instead we added a free keyword field to enrich the keyword options. Updates of key concepts and topics continue to take place throughout the project duration, when new terms are found useful.
7 The seven dimentions of Societal Security p.5. http://societalsecurity.net/sites/default/files/D3.1%20Methodology%20workshop%20and%20review%20of%20available%20empirical%20sources.pdf
D8.4 – FP7 – 313288
6
This filtering system is a major aspect of the Observatory (Mediawatch and database) as it is used for firstly identifying relative news articles, and secondly for organising and categorising the database, making it easier for users to find news content that is relevant to their domain of research/interest. There are currently 95 key concepts and 215 topics (see annex 2).
2.3. Subcontracting
With the overlaps of the Mediawatch and the database, and the satisfaction with Pronovix work (the company hired to make the Mediawatch) a decision was made to continue collaborating with the Pronovix team. Pronovix is a Belgian/Hungarian enterprise. They have expertise in Drupal, working in research project environments and creative programmers willing to explore possibilities and opportunities in creating the SOURCE observatory (Mediawatch and Knowledgebase). Pronovix works using agile project management, working in short 1-‐3 week sprints intensely. With these short sprints, certain features are finalised step by step, allowing flexibility in making changes or removing unwanted features. The first version of the Knowledgebase was created in one long sprint, as described in Annex 5. Purchase order sprint 1 of document database. Following this larger sprint, smaller tasks and improvements have been made and Pronovix continues maintenance and security updates of the entire Observatory. Work has continued effectively with the company and they also have been subcontracted for other deliverables in WP8 and have even helped out with WP2 and WP9.
The technical development was conducted in collaboration between VUB and Pronovix, and open source tools used. Annex 3-‐ Database user stories shows partly how the workflow was foreseen.
Using the Knowledgebase
The groundwork for the Knowledgebase took place in 2015. Based also on the layout, of the Mediawatch, the keywords and sorting system, the development and building of the Knowledgebase took place. The database, which we call the Knowledgebase, is available on the SOURCE website: http://societalsecurity.net/Knowledgebase
On December 31 2016, the Knowledgebase included 88 ‘documents’ or rather ‘content items’. These documents were added manually by the SOURCE partners. The type of content collected and inserted into the database are those that are open source, thus freely available. The main requirements for being added to the database are relevance to societal security and quality content. The document types include: Blogs, Websites, Research projects, Books, Policy documents, Legislations, Reports, Scientific publications, Interviews (sound/video), Legal trend Cards, Technology trend cards, SOURCE publications and Other publications. The knowledge base aims to be a “one stop shop” for selected societal security relevant content that is freely available. The Knowledge Base was made public in the summer 2016. The next steps include regular input of content from all project partners, and promotion of the Knowledgebase to potential users. Eventually, registered users of the SOURCE network (extended network) may receive access for adding new content. With the aim of expanding the primary network and their role and engagement within SOURCE, as well as engaging the network of excellence,
D8.4 – FP7 – 313288
7
there is scope to allow more trusted users to add content and become more active participants in the SOURCE online observatory. This process of opening up content input to a broader user group will always require a moderator to ensure quality of content. As stated above, moderator receives email notification on any new content added, recommended as important or flagged as irrelevant. It is then upto the moderator to decide if a content item is relevant for the database or not. The moderator will only delete items that clearly have no relevance for the databse. In cases of questionable relevance, the moderator will contact the person who did the conent input for further clarification. Furthermore, moderator can contact other SOURCE partners for advice, if needed.
3. Technical Features – back end
The Knowledgebase is a manual input database with options to filter input through topics, key concepts and domains. A system scans all attached and uploaded pdf documents also, so an open search will look for the terms also within uploaded documents. Annex 3. Database user stories shows the development of the information input into the database by partners.
3.1. Knowledgebase content input
Project partners and other potentially registered users (from the SOURCE primary network or extended network of excellence) can add new content to the Knowledgebase. They can also mark the content as recommended, and report the content that is not relevant. For every action taken, an email notification is sent to the content administrator (VUB).
In principle, the database is expanding due to to the efforts of experts in the field and potential registered users. For instance, when partners have a new publication, they can add it to the Knowledgebase or if they stumble on a relevant blog, an interesting video, or a new policy document, they can also add it to the Knowledgebase. Therefore, when many experts in the field collectively add their relevant sources to one place, the outcome can become an excellent library of information. Once all partners (and other registered users from the xtended network) become accustomed to adding e.g. 2-‐5 new content items per month, the database will quickly grow.
In order to enhance the quality of the database, the input page has several fields that are obligatory. Image 1 shows the fields that need to be filled in when adding new content to the document database:
D8.4 – FP7 – 313288
8
Image 1. Content input page for Knowledgebase
D8.4 – FP7 – 313288
9
For the field Topic and Key Concept that is a new window that opens that allows user to tick relevant topics and key concepts (Images 2 and 3):
Image 2. Topic box
D8.4 – FP7 – 313288
10
Image 3. Key Concepts box
It was considered very important to have complete data input into the Knowledgebase. Many databases contain partial information, or very limited or even inconsistent information between entries. Therefore, the quality of the Knowledgebase increases due to manual input; the quality and consistency of information added can be ensured which will make the database more meaningful, and the search function more accurate. Also, importing from other databases, or automated uploads would not only result in less consistent input, but would also leave out entirely the filtering system developed for the Mediawatch and implemented within our Knowledgebase. Automated import would additionally risk non-‐ relevant content being added, since it would be imported in batches and uploaders would be less selective than when adding manually. Finally, input would be limited to documents, whereas the Knowledgebase allows for uploads of videos, sound files, links to blogs and other project pages on societal security. In short, the manual input has space for customisation that would not be possible within our budget with automated input. Image 4 shows the different content type inputs that the database supports.
D8.4 – FP7 – 313288
11
Image 4. Content types that can be added to the Knowledgebase
Document type Icon Explanation text
Blog/Website
A website or a blog that focuses on news, articles or information on topics related to societal security.
Research project
Online link to other European, national or international projects related to societal security.
Book
A link to book information or a pdf upload if book is open source.
Policy document
A link to location where policy document can be found, or a pdf upload.
Legislation
A link to online version of legislation or pdf upload.
Report
Government reports, think tank reports, national/EU reports of institutions etc.
Scientific publication
Scholarly articles, scientific publications, relevant academic articles.
Interview/Sound/Video
Video or sound file of an interview, podcast, lecture, presentation, meeting recordings etc.
Legal trend Card
SOURCE project legal trend fact sheets.
D8.4 – FP7 – 313288
12
Technology trend card
SOURCE project technology trend fact sheets
SOURCE publication
Other SOURCE publications, such as deliverables, reports, slides, presentations etc.
Other publication
To be avoided, but if the content fits in no other category, add it here.
4. User experience – front end
The SOURCE Observatory consists of two regularly updated sources of information that will help one in identifying content relevant to Societal Security: The Mediawatch and the Knowledgebase. This includes news, scientific documents, legal and policy information, video interviews, podcasts, blogs and websites. The aim of the Observatory is to create one meaningful space where knowledge is identified, collected and inserted, organised and shared, so that it becomes easy for users to find what they are looking for in order to further their work, whether it is research, policy making, advocating, reporting, or just any interest.
4.1. Main page: searching the Knowledgebase
The database tries to achieve a user-‐friendly ”look and feel”, that is consistent with the “look and feel” of the Mediawatch. It supports that same search functions as the Mediawatch8 where one can filter based on domain, topic, key concept and author. A user can also use the free search box, selecting document type, time frame, location and sorting by either most recent or most recommended results. Image 5 shows how the main page of the Knowledgebase looks like. The search function on the top, list of filters on the right side and the main results in a list below, including an icon and basic content information.
8 See D8.4 Mediawatch report
D8.4 – FP7 – 313288
13
Image 5. Knowledgebase main page
D8.4 – FP7 – 313288
14
The main view shows title of content, the type, link, authors, publishers and the beginning of an abstract, when a user clicks on “see more” s/he remains on the same page but receives further information, including the full abstract, country of publication, year of publication, all the keywords associated with the content and other links or documents (Image 6). The left side icon indicates the type of the file. There is a possibility of recommending the input (thumbs up), and to flag inappropriate content (triangle). Only registered users who can add content can perform these actions.
Image 6 “show more” function
Video and sound content is made available directly on the page when “see more” is clicked on, rather than sending the user to a new page or away from the SOURCE site as seen on image 7.
D8.4 – FP7 – 313288
15
Image 7. Video content
5. Next steps
On one hand D8.6 -‐ Database deliverable is considered completed now. Its frame has been developed, created, tested and it is up and running. The budget for creating the database has been mostly spent. On the other hand, the knowledgebase that the deliverable resulted in is just starting its lifespan and is ready to grow and expand. Project partners keep adding content from occasionaly. Because the system is rather flexible, further developments within the Observatory as a whole can be made. The library of content is expected to grow until the end of the SOURCE project and hopefully beyond that.
Success of the database and its usefulness depend on: a) partners’ contribution by continuing to add content on a regular basis (suggested level of update during the project is 2-‐5 new content items per month), and b) if the extended network is able to add content as well. As a collective effort of sharing relevant resources, the database can become extremely rich in content and useful for the network. Without this effort, the knowledge base will serve little purpose apart from being an example of a database design for future projects. The hope is that all users of the knowledge base can see the added value of this shared resource and thus continue to contribute to it, also after the
D8.4 – FP7 – 313288
16
SOURCE project itself is over. This will be further investigated in WP2, D2.10 Long-‐term business plan for the SOURCE virtual Centre of Excellence.
Collaboration with D2.8 Web-‐based knowledge-‐sharing tool (task leader -‐ TEC) started in 2016. The idea was to explore a user registration page to be added to the Observatory. In that scenario, experts in societal security and those interested, could register to the Observatory. Registered users would be able to see other users (adding a user database /registry of experts that could find each other) and add content to the Knowledgebase. Each user would then have their own page with basic information on them (their expertise), and showing content they had added or recommended to the Knowledgebase. This would further create knowledge and a SOURCE virtual community. Dialogue started among TEC, VUB and Pronovix. However, this is to be further discussed in the framework of D2.8.
WP9, with CEPS as WP leaders, will also take on the role of promoting the Knowledgebase and presenting it further through their networks and through the dissemination process of the SOURCE project.
D8.4 – FP7 – 313288
17
Annex 1 – Letter to Partners
At the start of the task, a letter was sent to the partners along with a form for them to fill out. This was a way to start analysing what type of documents/content would be relevant for the database and for keyword/filtering identification
Deliverable 8.6 Database and website infrastructure
Dear Source partner,
We at the VUB are currently in the process of developing the core aspects of our document database as described in Deliverable 8.6 “Database and website infrastructure.”
The database will serve to “gather, catalogue and store raw information on societal security from the widest range of sources for further analysis in use in the activities of the network… The database and website will become a long-‐term research and development tool for scholars, technology developers, lawyers, crisis response units etc.” The information available on the database should continuously grow and provide an overview of the field. It will be a space that connects a fragmented field of research information, as well as a place where we can promote material created by the Source consortium.
Thus, we are starting to build this database and we would like to ask your help in creating the first round of reference documents. We ask you to select 10 documents you find relevant to societal security. These can include: European Policy document, National/local policy document, scientific publication, educational material, Source project document, or other publications.
We ask you to fill in the attached excel sheet with around ten documents. These documents will serve as our sample for developing the database. They will help us in defining ways to categorise and organise the information in order for the database to be as user friendly and useful as possible. As the information we will collect will be very diverse, it is of critical importance to define keywords and create a system that is relevant for Societal Security research.
Additionally, we ask you to suggest general key words that you would find important for categorising information.
The documents will also become the first data input into the database, so that once ready, it will already contain relevant information, making the database useful from the onset.
The following excel sheet contains the categories to be filled (and includes one example). We ask you to fill it out, and return to us in the first week of September. If you have any questions or comments, don’t hesitate to contact us
All the best
Christof and Olof
D8.4 – FP7 – 313288
18
Excel sheet fields:
Project Partner acronym of project partner
Domain
Tick all that apply: (Physical security, Socio-‐economic security, Political security, Radical uncertainty security, Cultural Security, Environmental security, Cyber security)
Document Type
Tick all that apply: (European Policy document, National/local policy document, scientific publication, educational material, Source project document, country profile, other publication)
Key words choose free keywords -‐ as many as apply Title of Document Full title Author 1 Last name, First Name Author 2 Last name, First Name Author 3 (and 4,5 if existing) Last name, First Name
Publisher (e.g. journal name and number and title, or book publishing company, or European commission)
Year of public. abstract paste whole abstract if possible in Document (link to pdf online or page where further info can be reached )
D8.4 – FP7 – 313288
19
Annex 2 – Filters
Key concepts and Topics are the same as in the Mediawatch
Key concepts Topics Topics (cont.)
aggression acoustic sensors housing
anticipation air pollution human rights
apparatus of security antibiotic-‐resistant bacteria human trafficking
Autonomy anticipatory governance human well being
complexity arms proliferation humanitarian intervention
conflict Art. 29 Working Group identity
conflict management asylum illicit financial flow
conflict prevention aviation illicit substances detection
consistency behaviour immigration
contingency behavioural analysis immigration attitudes
continuity big data income inequality
corruption biological substances detectors information assurance
crisis biometrics institutional habitus
crisis management bio politics INTCEN
crisis prevention bodily privacy intelligence
crisis response border intelligence gathering
critical infrastructure border zones intelligence operation
defence borders intelligence services
democracy catastrophe intelligence support
deprivation CBRNe inter-‐ethnic conflict
detection CCTV inter-‐social conflict
deterrence CEPOL international economic relations
emergence Charlie Hebdo international political economy
enactment chemical substances detection international relations
eu security agencies citizens rights internet of things
external security civil protection intrusion prevention
failure civil society IoT
global security climate change ISIS
global security problematic COSI law enforcement
health security Council JHA law enforcement agency
impact counter terrorism low-‐tech security
information security Crisis Management Centre Finland migration
institutional weakness cross border pollution mobility
integration cultural ideals national identity
internal security cyber-‐attack technologies nationalism
internal security strategy cyber cells natural disaster
D8.4 – FP7 – 313288
20
international security cyber crime natural hazard
justice cyber gangs non-‐military contributions
liberty cyber security Nordfosk
long-‐term stability cyber surveillance OLAF
maritime security cybercrime Ombudsman
mitigation Daesh organised crime
national security Danish Emergency Management Agency organized crime
perceptions of security data protection peace building
precaution data sovereignty peacekeeping
preemption deforestation personal data
preparedness desertification PNR
prevention detectors political culture
privacy deviant behaviour political leadership
privatization of security DG ECHO political priorities
professionals of security DG HOME political science
protests disaster political violence
recovery disaster management pollution
repression disaster risk reduction population growth
resilience drug trafficking poverty
response ecojustice prism
risk economic and financial crises radicalisation
risk society economic crisis radicalization
risks and threats economics radiological and nuclear detectors
rule of law economy refugee
securitisation EDSP refugees
securitization education regionalism
security electronic tagging systems resource depletion
security and development employment RFID based tracing device
security and justice encryption rising oil prices
security and technology encryption algorithms risks in cyberspace
security challenges ENISA satellite navigation based receivers
security devices ENLETS scarce resources
security dilemma environmental disaster scarcity
security experts environmental economy Schengen
security field environmental law sea surveillance
security firms environmental policy secure protocols
security industry environmental refugees seismic sensors
security practices epidemic sensors
security processes ethnic conflict sis ii
D8.4 – FP7 – 313288
21
security products ethnic relations smart cards
security professionals ethnic violence smuggling
security research ethnicity social control
security services EU CTC social engineering
security solutions eu-‐LISA social exclusion
security technologies EURODAC social norms
security-‐industrial complex EUROJUST social unrest
social movement EUROPOL societal acceptance
societal security EUROSUR societal change
society extreme weather standards
sovereignty finance Stockholm programme
speculation financial crisis subprime crisis
stability financial security surveillance
terror attacks first responders surveillance society
threat floods sustainability
tradition food safety Swedish Civil Contingencies Agency
transparency food security technology
uncertainty foreign fighters terrorism
values FRA terrorist group
vulnerability FRONTEX totalitarian regime
fundamental rights trafficking
geographic Information systems transatlantic
global finance transnational field
global governance transport
governmentality trauma
hacktivism violence
hacktivist war
hardware tokens waste dumping
health water pollution
heat wave weak economy
high unemployment wealth distribution
home affairs weapons
welfare state
D8.4 – FP7 – 313288
22
Annex 3 – Database user stories
User stories are a way to describe to programmers how the database could function from the view of different user types
Anonymous User 1. As an anonymous user I can see the 15 most recently added documents. (title, type of
document, domain) 2. As an anonymous user I can filter down the 15 most recently added documents per document
type 3. As an anonymous user, I can do a search for documents by date of publication (range) 4. As an anonymous user I can use the same facets as present in the Media Watch to search 5. As an anonymous user I can search by free search terms. 6. As an anonymous user I can select an article which will show the
● title ● author(s)/ (institution if no specific author) ● publisher ● document type (policy document, scholarly article, SOURCE publication, legal trend
card, scientific trend card, video, audio, other publication) ● domain ● keywords (topics + key concepts + free keywords) ● year of publication ● country of publication (including “European Union”) ● abstract ● link to document (either on our site or external)
Database -‐ Project Partners As a project partner, I can add documents to the database.
Uploading a document
-‐ fields (cf. 1.6) -‐ * title -‐ * author first name + last name and/or institution
-‐ option to add authors -‐ * document type (policy document, scholarly article, SOURCE publication, legal trend
card, scientific trend card, other publication) -‐ * publisher -‐ if scholarly article: journal name and number and volume -‐ select at least one Domain -‐ topics are displayed according to domain(s) chosen.
D8.4 – FP7 – 313288
23
-‐ all key concepts are shown. at least two need to be chosen -‐ free keywords can be entered (not obligatory) -‐ * year of publication -‐ * country of publication (including “European Union”) -‐ * abstract (text box) -‐ add image (optional (align right of text)) -‐ upload document (optional) -‐ link to document (link to external source)
Can edit and delete own documents@
Database -‐ DBAdmin
-‐ can add typography in abstract (editor) -‐ can edit and delete all articles
The same taxonomy is used for both media watch and the database; so changes in this taxonomy will reflect on both those systems.
And we need to discuss database entry page (sorting options and ways to organise e.g. related articles/recent/type/taxonomy)
Additional user stories
Discussed on 31 August with Olof
1. As an anonymous user I can export the search results in an xls document (TBD) 2. As an anonymous user I can suggest content for the Document database via email 3. As an anonymous user I can report a problem on the site by sending a form via email. 4. As an administrator user I get an email notification about new content added by the Project
partners. 5. As an authenticated user (Project partner) I can select multiple Topics easily, without having
to browse through a huge list of Topics in a dropdown select. 6. As an authenticated user I can promote certain Document items to appear as Recommended,
always on top in the list of Documents. 7. As an anonymous user I can filter the Documents by Recommended status, also by Date added
and year of publication. 8. As a system I can fetch documents from external sources (URLs), by downloading the
document to the Drupal file system.
D8.4 – FP7 – 313288
24
9. As a system I can search in the contents of the uploaded PDF documents, making the search results more relevant.
10. As a system I can check URL links in the document nodes and compile a report of broken links. 11. As an authenticated user I can add video and sound media files to the documents by using
YouTube channel and Sound cloud channel links. 12. As an anonymous user I can watch/listen YouTube videos and sound cloud media embedded
in the Document search page.
Questions: -‐ What should the URL path to the Document Database search page? -‐ What should be the Menu title and where should it sit in the Main menu? -‐ Can we use the combined Author / Institution field, assuming that it’s either 1 or more
author(s), or a single Institution? Concern with taxonomies is if the names are represented differently in different docs (J.Doe / J. R. Doe / Doe, John R. / Doe, J.R./etc
-‐ Should we use Author/Institution as facet filter at all? -‐ If yes, can they be added to the same taxonomy vocabulary? -‐ Ok to have Keywords/Topics/Key concept selected tags maximized in 10 in the field? -‐ Can we ask for sample docs with all their meta info? -‐ Buttons on CKeditor for the Abstract. Bold, Italic, Underlined, Bulleted and Numbered list, Link. -‐ Do we need to search in the Related document files, with the user added free keyword?
D8.4 – FP7 – 313288
25
Annex 4 – Explanatory document for database input and use
|Help Text for inserting content into knowledge base and how to use it
Step one: find your invitation to create an account… for some reason, this might be in your junk mail. If you have a problem… send me an email
Step two: Once you have your account and password set, log in
Adding content
Step 3: Top left corner, click on add content => Document
(http://societalsecurity.net/node/add/document)
Step 4: Fill out ALL fields
-‐‑ title (self-‐explanatory) -‐‑ Document type (important!! as this is how the content is grouped… see explanation for types
at the end of this document – or in the guideline link below the field on the page) -‐‑ File type (this is the type of content you are adding. Is it a pdf? Or a link to the website where
you can find the book information/policy document? Is it video file? Or sound cloud. If you have more than one type e.g. a video presentation from an event, and the slides in pdf format, click on video, you can add pdf or url links later in the page
o Then follow by uploading pdf (you can also click on remote url, if the pdf is online to collect the link, then you can avoid the downloading and uploading of the pdf)-‐ linking website, video etc.
-‐‑ Authors/institutions (write first name last name and press ENTER to add (or full name of institution) … you can add multiple names – use institution if author is unknown or the institution is the author)
-‐‑ Publisher/publication (e.g. journal name and number and pages (Journal of Contemporary European Research vol 5 (2), pp. 293-‐311) , or book publishing company, or European Commission)
-‐‑ Year of publication (obvious) -‐‑ Country of publication (European Union is also an option for EU or project documents) -‐‑ Abstract either paste the abstract, using this button! -‐>
It is in the top of the field, and it cleans up the hidden mess in the text. You can also just add a few sentences describing the content.
-‐‑ Topics (important!!! As it is helping people searching and in grouping into domains. Tick at least one, tick as many as apply)
-‐‑ Key concept (Also important for organising the content, so tick at least one, and as many as apply!!)
-‐‑ Free keywords (Optional, it is useful if the main words are not in the list above, because lists above will be adapted based on this. These are also visible to users)
-‐‑ Related links (add a title (or “more information”) and the url of any related material… you can add more than one)
-‐‑ Related documents (same as above, an option to add more help material… however if it is another major document or link, you might consider adding a new entry)
-‐‑ Flags (this is the recommended button-‐ if you think the content you are adding is exceptionally important (or is SOURCE publication) tick the box… Administrator can untag after some time
D8.4 – FP7 – 313288
26
-‐‑ Save (very important! And then you are done)
Using and reviewing knowledge base
http://societalsecurity.net/Knowledgebase
Step five: Look at the knowledge base, search and test,
Top bar allows you to use free search terms, search by document type, time period, and more. The side bar has a list of the topics and key concepts that can help you filter based on your search needs.
Step six: Report or recommend. Only logged in registered users (those with permission to add content) can see these buttons.
If you see an irrelevant content in the database, press “report” and explain in the window that opens why it is inappropriate.
The recommend/not recommend thumb allows you to do that, recommend a document that has not been given that honour. Or to remove the “recommendation” label, if you think the document is not important enough to be considered recommended
I use the term ‘document’, but of course I am talking about any type of content (web, video, sound etc.)
This is it. Please in the first round, add as much content as you can think of. All the sites you use for your work, policy documents, scientific documents, podcasts, blogs, interviews, other projects… We want this to be a rich knowledge base. I would think each of us could in a period of the first month add at least 20 entries
Then, frequently, every 2 weeks, or when you come across something interesting, add it into the knowledge base. This is the only way that the knowledge base remains relevant to you, and to others.
Thank you so much, and do not hesitate to give suggestions for improvements (within practically feasible limits of course)
D8.4 – FP7 – 313288
27
Knowledgebase Help document for choosing the best fitting Document type
Document type Icon Explanation text
Blog/Website
A website or a blog that focuses on news, articles or information on topics related to societal security.
Research project
Online link to other European, national or international projects related to societal security.
Book
A link to book information or a pdf upload if book is open source.
Policy document
A link to location where policy document can be found, or a pdf upload.
Legislation
A link to online version of legislation or pdf upload.
Report
Government reports, think tank reports, national/EU reports of institutions etc.
Scientific publication
Scholarly articles, scientific publications, relevant academic articles.
Interview/Sound/Video
Video or sound file of an interview, podcast, lecture, presentation, meeting recordings etc.
Legal trend Card
SOURCE project legal trend fact sheets.
Technology trend card
SOURCE project technology trend fact sheets
SOURCE publication
Other SOURCE publications, s.a. deliverables, reports, slides, presentations etc.
Other publication
To be avoided, but if the content fits in no other category, add it here.
D8.4 – FP7 – 313288
28
Annex 5 – Purchase order sprint 1 of document database
(Please find it in additional pdf)