tefis d5.4 - cordis · tefis - d5.4.1– connector implementation and documentation for livinglab....
TRANSCRIPT
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Deliverable Title D5.4.1 Connector implementation and documentation for Living Lab
Deliverable Lead: Annika Sällström (LTU-CDT)
Related Work package: WP 5
Author(s): Annika Sällström (LTU-CDT), Santiago Martínez (SQS), Itziar Ormaetxea (SQS), Ainhoa Gracia (SQS), Jonathan Gonzalez Rios (SQS), Gabriele Giammatteo (ENG), Robert Samuelsson (LTU-CDT)
Dissemination level: PU
Due submission date: 31/05/2011
Actual submission: 30/09/2011
Project Number 258142
Instrument: IP
Start date of Project: 01/06/2010
Duration: 30 months
Project coordinator: THALES
Abstract This document summarizes the results for the initial work with the Living Lab connector of TEFIS. This deliverable will be updated when the initial Living Lab use-case is finalized and the Open call experiments have been evaluated.
Page 1 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Project funded by the European Commission under the 7th European Framework Programme for RTD – ICT theme of the Cooperation Programme.
License
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 License.
To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.
Project co-funded by the European Commission within the Seventh Framework Programme (2008-2013)
Copyright by the TEFIS Consortium
Page 2 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Versioning and Contribution History
Version Date Modification reason Modified by
0.1 05/15/2011 Creation Annika Sällström
0.2 06/03/2011 Contribution Santiago Martínez
0.3 06/03/2011 Review Itziar Ormaetxea, Ainhoa Gracia
1.0 - Final version Annika Sällström
2.0 28/09/2011 Updated version Annika Sällström, Itziar Ormaetxea, Ainhoa Gracia, Santiago Martínez, Jonathan Gonzalez Rios, Gabriele Giammatteo, Robert Samuelsson
Page 3 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
TABLE OF CONTENT
Executive Summary ....................................................................................................................................... 5
1 Introduction........................................................................................................................................... 6
2 Living Lab Connector functionality ........................................................................................................ 7
3 Generic specification of the Botnia Living Lab connector workflow..................................................... 7
4 The Living Lab connector and the IMS‐Botnia use‐case........................................................................ 8
5 Connector Usage ................................................................................................................................. 10
5.1 FIRST PHASE – request/Booking of a resource .......................................................................... 10
5.2 SECOND PHASE – Execution order ............................................................................................. 14
5.3 THIRD PHASE – Management of data ........................................................................................ 17
6 Botnia connector implementation ...................................................................................................... 19
6.1 Resource Management.............................................................................................................. 20
6.2 Resources Execution .................................................................................................................. 26
6.3 Data Management ..................................................................................................................... 30
7 Status of the actual implementation of the Botnia Living Lab connector........................................... 32
8 Benefits for the TEFIS Living Lab use‐case of the Living Lab connector .............................................. 32
9 Future Work ........................................................................................................................................ 32
Page 4 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Executive Summary This document summarizes the results for the initial work with the Living Lab connector of TEFIS. For this
implementation we have identified four different Botnia resources of relevance for TEFIS.
They are:
‐ User involvement expertise
‐ Methodology for user involvement
‐ The Botnia Living Lab user database
‐ Business model expertise
To design and implement the first version of the Botnia Living Lab connector we have chosen to
elaborate the connector specification via the IMS‐Botnia use‐case to investigate the potential solution
for the Living Lab connector. The deliverable includes the initial design and workflow of the Botnia
connector to cover Resource management, Resource execution and Data management. In the design we
have taken into account the Botnia infrastructure and the TEFIS architecture to optimize the integration
of Living Lab resources for TEFIS users. This deliverable will be updated when the Living Lab facility has
been integrated into the upcoming experiments of the Open call where the Living Lab resources are
expected to be one of requested kind of resources by the experimenters in combination with resources
from other testbeds of TEFIS.
Page 5 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
1 Introduction The main purpose of the Living Lab connector is to provide different Living Lab resources from Botnia
Living lab for TEFIS access.
There are four main Botnia Living Lab resources identified for TEFIS access based upon the initial use‐
case needs (from the IMS‐Botnia use‐case):
1. User‐involvement expertise One resource of Botnia to be accessible via TEFIS is the expertise in user involvement activities. This resource consists of research expertise in the field of user centred design and evaluation and they support experimenters in setting up, and running user involvement activities. 2. Methodology for user‐involvement For the process of user involvement one resource of Botnia Living Lab is the Form‐IT methodology. This is an iterative and interactive process in several steps for user‐engagement in all phases of the development of an IT‐based service/product – from need finding to beta‐trial and pre‐market launch. Different methods and tools are used to provide professional support for user‐involvement. Often we combine qualitative and quantitative methods for the best results like web‐based questionnaires to investigate specific areas among a bigger user‐group and observations and interviews to go into depth around specific issues and to get answers on why and how. For user‐involvement, it is very important to recruit the right users matching the purpose of your experiment. 3. The Botnia Living Lab Users Data Base This is a database of 6000 creative end‐users (individuals) from 18 years of age and older in Sweden and access to end‐users around the world via 3rd parties. The Botnia user database is currently implemented as a MySQL‐database where End‐user basic data for end‐user involvement are stored.
4. Business model expertise
This resource consists of research expertise in the field of business model development and evaluation.
Their involvement is supported by a business model design scheme to support the business model
development.
The expected results for an experimenter when using Botnia Living Lab resources are:
Redesign of products/services;
Decisions for implementation of new functions;
New target user groups;
New ideas as result of user involvement;
Increased knowledge among experimenters/developers;
Established relations with new business partners;
Faster innovation process(shortened time for development) by support from end‐users for decision making;
Business model evolution
Page 6 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
2 Living Lab Connector functionality
The communication protocol for the Living Lab connector is initially foreseen to be as follows:
The Living Lab connector will be installed within the web infrastructure of Botnia Living Lab as a back‐end application,
Access to resources and result data will be mainly manual considering the resources to get access to including result data cannot be controlled by any software, configured, deployed or monitored by others than the Living Lab providers.
In the long‐term, if “virtualization” would be possible, there might be some possibilities to make some
resources available for direct access and configuration remotely.
3 Generic specification of the Botnia Living Lab connector workflow Here follows a workflow to describe the functionality of the connector in the different phases of an
experiment. In next section we will describe this workflow more in details in relation to the IMS‐Botnia
use‐case.
1. User registration:
Experimenter creates a TEFIS user‐account
Action from Botnia Living Lab connector: None
2, Experiment definition: Experimenter looks for available testing resources in the TEFIS Experimental manager Action from Botnia Living Lab connector: None 3. Experiment Configuration Experimenter specify resources to be used and design the tasks involving Botnia resources Action from Botnia Living Lab connector: None 3. Planning Experimenter makes a request for Botnia Living Lab resources by the Experimental manager. Action from Botnia connector: The Request is sent by e‐mail to the Living Lab manager including the requested configuration for resources and the experiment description. Upon the agreement with the experimenter the final configuration is decided. 5. Execution Tasks are performed using Botnia Living Lab resources. Status of execution of tasks is handled in the Botnia Drupal platform (testplats.com). Action from Botnia Living Lab connector: The connector asks the Drupal platform of the execution and parameter (queued, running and finished) . 6.Reporting Botnia tasks are finalized and results are stored in Drupal(testplats.com) and results are uploaded in iRods.
Page 7 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Page 8 of 32
Action of Botnia Living Lab connector: None (but it might be depending on the actual functionality of iRods) 7. Knowledge sharing Experimenter wants to share experience from this experiment with other experimenters. Information to share is how the experiment was performed and experience of using Botnia resources. Action of Botnia Living Lab connector: None
4 The Living Lab connector and the IMSBotnia usecase To specify and implement the Botnia Living Lab connector the IMS‐Botnia use‐case has been the baseline to identify the different requirements of the connector. This experiment is focused on a mobile application for content sharing and is divided into three different
phases of the service development life‐cycle: concept development, prototype development and
Business model definition. This experiment addresses the three main issues facing mobile applications
today. First, this experiment will explore end‐user feedback to check if the application is suitable for
them. In the second step, the testbed facilities are used as a validation tool, and in the third step, the
correct business model for long‐term sustainability is investigated. In this use‐case we are combining
experimental resources from two different testbeds independent on place: The SQS IMS testbed in
Spain1 and the Botnia Living Lab in Sweden2.
In this experiment the TEFIS testing service is used for designing, planning, management of experimental
workflow, configuration assistance, experimental data management, reporting, and knowledge sharing
with other experimenters. For this purpose the Botnia Living Lab connector is used to create easy
connection between the TEFIS platform and the Living Lab resources.
The access to Botnia Living Lab is through the Tefis Platform and Botnia Connector. Tefis Platform is also
connected with IMS testbed, so the users of Botnia LivingLab will have access directly to IMSTestbed
1 http://www.tefisproject.eu/media/upload/11‐2445‐SQS‐korr_110204_3.pdf 2 http://www.tefisproject.eu/media/upload/11‐2445‐blad‐botnia‐korr_1102041.pdf
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
TEFIS PLATFORM
Figure 1: IMS-Botnia use-case overview
LIVING LAB SQS TESTBED
Phase 1 Phase 2 Phase 3
LIVING LAB + SQS TESTBED
- Number of users - Q-Mobile testing - Max network bandwidth
- Users profiles - Mobile devices - Max network bandwidth by user - Utility - Functional correct
tests - Users profiles
PHASE1: TEFIS as concept evaluation tool
PHASE2: TEFIS as validation tool PHASE3: TEFIS as
business validation tool
Page 9 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
5 Connector Usage The of thuse e connector in TEFIS will be divided in three phases.
5.1 FIRST PHASE – REQUEST/BOOKING OF A RESOURCE In the first phase, once the experimenter has defined an experiment, the Experiment Manager sends the
order to Teagle for booking the resources. Teagle, in turn, sends an order for each resource to the
corresponding connector.
The flow of this job for the IMS‐Botnia use‐case is presented in a schematic way in the next graphic
(Fig.2):
Experimenter ExperimentManager
Teagle(Booking Processor)
CONNECTOR
Botnia Manager
Botnia’s Resources Database
Creation of the
ExperimentBook the resources
Book theresources
TEFIS
Storage the resources and their configurations
Send an email with information about booked resources
Figure 2: First phase flow
Page 10 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
In the figure below (Fig. 3) the experiment definition for the IMS‐Botnia use case is shown. The
experimenter will book the Botnia Living Lab resources to use:
User involvement Methodology;
User database;
User Evaluation Expertise .incl Business Model Creation Expertise;
The IMS testbed’s resources will be booked in the same step.
Figure 3: Advanced Experiment Definition in Tefis Platform
When the Botnia’s connector receives the order to book a resource, it sends an e‐mail to Botnia
Manager. This e‐mail shall contain all the information related to the requested/booked resource with
their multiple and different configurations. Next, the connector saves the data in the resources database,
so it will be possible to have access to the information in the future.
In this use‐case the connector sends an email for each booked resource (User Involvement Methodology, User Database, Business Model creation and Evaluation Expertise) to the Living Lab manager with the correspondent configurations and stores all parameters in the database.
Page 11 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
In Fig.4, the list of experiments can be seen.
Figure 4: Experiment lists in TEFIS Platform (the Experiment manager)
In Fig. 4, the use‐case is represented in TEFIS Platform when an experiment is created.
In the use‐case there are four different tasks in different steps and in each task different resources are
being used.
In this situation the connector will receive the order from TEFIS to book the requested resources.
The three tasks involving Botnia resources are:
Conecpt evaluation task: User Involvement Methodology, User database and User Evaluation Expertise
resources;
Usability testing: User Involvement Methodology resource;
Business Model Selection : ser Involvement Methodology, User Database, Business Model Expertise and
User Evaluation Expertise resources
The connector sends one mail for each booked resource to the Living Lab manager with the resource
name, type and configuration parameters. This information is also stored in a Database in the Living Lab
connector
Page 12 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Figure 5: Resources to book in Tefis Platform
As it is already explained in this document, when an experimenter has requested the resources, the way
the Living Lab connector works is to send an email to the LivingLab Manager acting like a trigger for an
internal mechanisms of Botnia to act on the request.
In those mails, as we can see in Fig. 6, the type of the resource, its name and all parameters of
configuration are included.
This is an example of a typical mail the Botnia Testbed responsible manager will receive when an
experimenter has requested a resource:
From: [email protected] To: [email protected] Subject: A new resource has been booked Mail: Resource type: Botnia User Database Configuration: Name: /userdatabase-UserDataBase_1 Users: 150 Age: 25-30 Gender: Male Language: English
Figure 6: E-mail sent after the requesting of a resource
Through the connector all resources has been booked with the configuration that we want without any
human interaction. But the step for resource requesting is in interaction between experimenter and the
Living Lab provider and the change of availability of each resource will be handled via the connector and
the Drupal/testplats.com to change the status of the request from pending to rejected or accepted or
invalid.
Page 13 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
5.2 SECOND PHASE – EXECUTION ORDER
The second and third phases are closely related.
The second phase is the Execution of the experiment part, and in it, once the experimenter sends the
execution order via TEFIS portal, the Experimenter Manager sends the execution order to the Scheduler.
The Scheduler then sends one order to the Connector for each resource involved in the task
The connector, in turn, sends an update request for Drupal(testplats.com) to change the status of the
experiment. This request is firstly handled as quing until it´s activated manually in testplats.com as
started. This is reported back via the Living Lab connector and the status of the experiment is changed to
running in the connector. (See Figure 7 and 8 below)
The Botnia Manager takes then control of its own resources.
Page 14 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
.
Figure 7: Second phase flow
Page 15 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Below is a screenshot of the form in testplats.com where the actual status of an experiment is updated
manually by the Livivng Lab manager. This information is then being reported back to the Living Lab
connector of TEFIS as described before.
Figure 8: Screenshot from status in testplats.com
Page 16 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
5.3 THIRD PHASE – MANAGEMENT OF DATA The third phase is the management of the data.
The Botnia Living Lab Manager, is responsible for sending the files and data to its resources (input data)
and saves the results of the resources (output data) in the Data Manager. (See Figure 9)
Figure 9: Third phase flow
Page 17 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
As a summary, the whole process can be seen in the next graphic:
Figure 10: Botnia connector involvement in the life cycle of an experiment
Page 18 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
6 Botnia connector implementation The main mission of Botnia Living Lab is to serve as a facility to perform the development of ICT‐based products and services in real‐life contexts with real users. Botnia's objectives include the generation of new knowledge, methods and tools, for open user‐centric research and innovation.
The Botnia Connector, even if was developed to manage a particular facility, has to manage three
different features as any connector:
Resource Management;
Resource Execution;
Data Management;
Botnia’s Living Lab has a different behavior if a comparison is made between its connector and the
connectors of the other testbeds included in TEFIS: it has to manage a human made work. In this context,
and to facilitate the interaction between the testers responsible of reviewing the input provided by the
experimenters, e‐mail alerts are used to provide notifications to the people in charge of the Botnia Living
Lab environment.
Thus, when a resource is instantiated, the connector sends an e‐mail to a specific person so they can
generate the resource and put it in the database with its configuration parameters.
Figure 11: Botnia Living Lab layers
Page 19 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
After that resource is executed, the connector has to send to the contact person the execution order by
e‐mail. At the same time, it generates inside database table that order, assigning pending status.
The execution process management, the update of the execution’s status is completely manual. So, it is
necessary to change manually the status in its field on the database.
For each of these features three different programming modules can be distinguished, which names
have a direct correspondence with them.
The programming language used to develop the connector is Python and MySQL is the database.
For the connector implementation MySQLdb, Smtplib and Re modules of Python have been used. These
modules can be downloaded from Python official website: http://python.org/
Besides this, Linux has been chosen as the system where the connector should be executed and
implanted. Botnia Connector is nowadays operative, running in Ubuntu 10.10. Ubuntu can be
downloaded from this URL: http://www.ubuntu.com/download/ubuntu/download
It is possible to access to the Botnia Connector using this URL: http://62.99.76.22:8010/
Figure: 12 The Botnia Living
6.1 RESOURCE MANAGEMENT
Lab connector instances
“Resources are the main concern of connectors: in fact, the main objective of having testbeds connected
to the TEFIS System is to allow TEFIS users to use the resources provided by testbeds for their own
purposes.
TEFIS includes different types of testbeds (e.g. network‐oriented testbeds, services‐oriented testbeds,
and living labs) that have different concepts of resource. To give an idea of the range of resource types
Page 20 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Page 21 of 32
that TEFIS will integrate, consider that in PACAGrid or PlanetLab a resource is a virtual machine, while in
Botnia a resource is a document, a questionnaire, or even a team of people who are “allocated” for an
experiment. Therefore, the very first requirement for connectors is to homogenize the concept of
resource so that the TEFIS System can deal with those resources in a uniform manner.
The lifecycle of a testbed resource is made of these five steps: i) the resource is required by the system,
ii) the resource is created (or, if it already exists, access is grant to the requester), iii) the resource is
configured, iv) the resource is used, v) the resource is released. This is a general and abstract lifecycle,
but each testbed has its own actual resource lifecycle. Connectors are responsible for mapping the actual
testbed resource lifecycle onto the more general TEFIS resource lifecycle. To do so, connectors should
expose an interface offering a set of functions to manage the various steps of the resource lifecycle.
The TEFIS Middleware and TEFIS Portal will leverage functions exposed by connectors to control the
usage of testbed resources in experiments carried out in TEFIS during their different phases: plan,
request, provision, deploy, run and evaluate (see TEFIS deliverable D6.1.1 for further information on
experiment lifecycle).”3
The implementation for the resources management has five different modules with a class each:
Class bbdd.py: this class manages the negotiation with the database
BtUserDBAdapter.py: which inherits from the first class, only exists the definition of the
functions and calls to the first class. It manages “Users Data Base” resources.
BtUserEEAdapter.py: which inherits from the first class, only exists the definition of the
functions and calls to the first class. It manages “User Evaluation Expertise” resources.
BtUserIMAAdapter.py: which inherits from the first class, only exists the definition of the
functions and calls to the first class. It manages “User Involvement Methodology” resources.
BtBMAdapter.py: which inherits from the first class, only exists the definition of the
functions and calls to the first class. It manages “Business model creation and evaluation
expertise” resources.
3 From Document 5.1.1‐Generic and Specific Connector specification
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
classes structure
In the Botnia specific case, three kind of resources exist. The resource types to be booked from Botnia are:
Users Data Base: It is a Database with End‐users for Future Internet service development and evaluation;
User Evaluation Expertise: It is an Expertise for Human centred design processes;
User Involvement Methodology: It is a Living Lab methodology for service innovation processes by user involvement in all phases from needfinding to pre‐market launch;
Resource Management Module
The functions implemented for the resource management are the following:
list_resources:
Input: the function receives a type of resource
Logic implemented: the purpose of the function is to consult the list of current available
instances that are registered in the database.
Output: it returns all found instances of the specified type of resource defined in the Botnia
Living Lab.
have_resource:
Input: the function receives a type of resource
Logic implemented: the function checks whether or not the specified type of resource exists
Output: it returns True or False
get_resource:
Input: the function receives a name of an instance
Logic implemented: the function checks whether or not the specified instance of a specific type
of resource exists
Output: it returns the name of the instance found
Among others, this function is a complement for other functions considering it is useful
to see configurations.
Page 22 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
add_resource:
Input: name of a new type of resource and configuration
Logic implemented: a new entry in the database for an instance linked to a type of resource is
created with the given name and configuration
Output: an e‐mail is sent to a specific contact person so the Botnia Living Lab Testbed provider is
notified that a new instance has been created.
delete_resource:
Input: name of the instance to be deleted
Logic implemented: the referred instance is deleted from the database
Output: None
get_configuration:
Input: name of the instance
Logic implemented: the function obtains the defined configuration (parameters and current
values) for a specific instance
Output: list of configuration data
get_attribute:
Input: name of the instance and specific attribute (parameter)
Logic implemented: the function obtains the current value contained in specified attribute that is
stored in the database
Output: Current value of the attribute
set_configuration:
Input: list of configuration parameters with their corresponding values and the name of the
instance they are referred to
Logic implemented: the current configuration defined for a specific instance is updated with the
data received
Output: None
Page 23 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
set_attribute:
Input: specific attribute, the value the attribute is intended to have, and the name of the instance it
is referred to
Logic implemented: the attribute or parameter provided is updated with the value received
Output: None
Resource Management Database
The Resources management is based on the database created in the LivingLab. This DataBase will keep
all the information related to the LivingLab resource and any instance booking or execution query.
The Resource Management Database has two tables, one for managing the resources and other one for
managing the number of instances.
The structure of the table to manage the resources module is presented below:
Structure of the table for the resource management
Any type of resource defined for a testbed should have the corresponding code inside a Resource
Adapter class, so it is possible to establish the communication between that resource and the module of
the TEFIS platform that wants to read or provide information from or to the testbed.
In this context, a table should be created in the database to contain different kind of data about the
resource and configuration of instances.
The structure of such table is more detailed below, including the names of the columns defined and the
information they contain.
id
Type of data: integer
Page 24 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
It is the primary key, it is a unique and not null identifier of a specific row of the database. It is
used to have access to a row so searches are possible, and it does not contain useful information
name
Type of data: string
It is the name used to identify a specific instance of the resource. Its structure is:
/Botnia‐Resource_X
common_name
Type of data: string
It is the name given by the user to identify a concrete instance.
age
Type of data: string that identifies ranges, eg. 22‐30
Configuration parameter of the resource that contains the age range of the working people
inscribed in the Living Lab.
gender
Type of data: string
Configuration parameter that represents the gender of the users of the Living Lab.
users
Type of data: integer
Configuration parameter that corresponds to the number of the intended users of the Living Lab.
language
Type of data: string
Configuration parameter that corresponds to the language of the intended users of the Living
Lab.
Resource_type_id
Type of data: integer
Configuration parameter that corresponds to the kind of resource.
Page 25 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
The structure of the table to manage the number of resources module is presented below:
Resource Type
PK id
common_namenumber_of_instances
Structure of the table to manage the number of resources
id
Type of data: integer
It is the primary key, it is a unique and not null identifier of a specific row of the database. It is
used to have access to a row so searches are possible, and it does not contain useful information
common_name
Type of data: string
It is the name given by the user to identify a concrete resource
.
Number_of_instances
Type of data: integer
It is the number of instantiated resources. If the resource has to have infinite instances this
number always will be 0.
6.2 RESOURCES EXECUTION
“A TEFIS experiment is made up of a number of steps that will be executed, in parallel or sequentially,
each one in one of the testbeds connected to the TEFIS System. Of course, testbeds offer heterogeneous
interaction paradigms and usage methods and, therefore, the definition of what a step of an experiment
is is liable to change depending on the testbed in which it is executed. In particular, it is interesting in this
section to analyze the level of execution automation provided by the testbeds. With regards to the
testbeds in the TEFIS project, it is easy to infer that the different testbeds have different levels and types
Page 26 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Page 27 of 32
of automation: from the complete manual execution that takes place in Botnia, to the completely
scriptable PACAGrid executions.”4
The functionality for the current and future resources execution of the Botnia Living Lab consists in
alerting (by email) people in charge of executing the task and status changes in the testplats.com
environment. The actual execution of the previously created and booked resource, and this functionality
is executed manually by the Living lab staff.
In order to achieve this goal, a database table has been created. In this table, it is saved the execution
state and the corresponding used folders with the data of that execution for a given experiment.
This functionality is implemented using a class with two functions in BotniaConnector.py module.
+get_info()+execute()+get_execution_status()
BotniaConnector
Structure of the class for resource execution
Next, the functions implemented in that module are described:
get_info:
Input: nothing
Logic implemented: the function obtains the name and version of the connector
Output: The name and version of the connector are returned
execute:
Input:
a context execution object that could consist on the id of the experiment and the folders
used to take and save the data
the list of resources selected for the experiment
the execution script in case it exists
Logic implemented: the function performs the execution of the experiment for the given context,
with the instances selected that should be configured and it shall be in charge of executing a
script if it is provided
Output: the execution unique identification
get_execution_status:
4 From Document 5.1.1‐Generic and Specific Connector specification
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Input: execution identification
Logic implemented: the function obtains the status of the given execution
Output: the current status of the execution. The status can be:
Running
Failed
Pending
Success
Resource execution database
The resource execution table structure is presented below:
Execution Table
PK id
exec_idexperiment_idinput_data_folderoutput_data_folderexperiment_root_data_foldertask_run_input_data_foldertask_run_output_data_folderexecution_scriptresourcesstatus
The structure of the Execution table is more detailed below, including the names of the columns defined
and the information they contain:
id
Data type: Integer
It is the primary key for each execution. It is a unique and not null identifier of a specific row of
the database. It is used to have access to a row so searches are possible.
exec_id
Data type: Integer
It is the identifier for a specific execution.
Page 28 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
experiment_id
Data type: String
It is part of the context execution object. It represents the identifier of the experiment to be
executed or that has been already executed, that includes a set of resource instances and
configurations
input_data_folder
Data type: String
Folder where the needed or input data, provided among others by the experimenter, that makes
possible the execution of a context is going to be located. It is part then of the context of an
execution object.
output_data_folder
Data type: String
Folder where the result data obtained after the context execution for a set of resource instances
with specific configuration is going to be located. It is part of the context execution object.
experiment_root_data_folder
Data type: String
General folder of the experiment. It is part of the context execution object.
task_run_input_data_folder
Data type: String
Folder with the execution tasks defined by the experimenter. These tasks shall contain a list of
resources and have a specific order of execution. It is part of the context execution object.
task_run_output_data_folder
Data type: String
Folder where the data obtained after the execution of a given task within an experiment will be
saved. It is part of the context execution object.
execution_script
Data type: string
It contains the execution script
resources
Data type: String
Page 29 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
List of resources that is going to be executed
status
Data type: String
Status of the process of execution. The status can be:
Running
Failed
Pending
Success
6.3 DATA MANAGEMENT “TEFIS will have to manage different types of data related to experiments: experiment definition data,
experiment configuration data, experiment input data and experiment output data. An analysis of the
characteristics of that data (type, scope, amount, flows) has been carried out during the initial stages of
the architectural definition summarized in TEFIS deliverable D6.1.1.
The analysis of data that flows in the TEFIS System has highlighted that not all data will be stored in the
TEFIS infrastructure. For instance, data generated within testbeds during experiment runs will be left in
the testbed facilities since the effort to move and store that data in the TEFIS infrastructure is too high.
With this respect, the amount of data generated during an experiment can be very high (for example a
log file for a web server under stress during a performance test).
Considering these issues, the experimental data management system in TEFIS shall be distributed and
there shall be available several data storage services in the system:
The central data service (the Research Platform Repository Service ‐ RPRS) that is deployed in the TEFIS
infrastructure and that communicates with native data storages from testbeds, via the Testbed
Infrastructure Data Service (TIDS).
With this architecture, most of the data generated during the experiment will remain in the data storage
of the testbed and will be accessed by TEFIS directly from there. Connectors must support the
communication between the TEFIS data management system and the testbed data storage offering a
common interface through which the RPRS can interact with the latter to retrieve experimental data.
The fact of being a facility manually managed is, also, an important point on this implementation; the
Botnia Living Lab has not been automated, and the lab providers shall be the intermediates between the
experimenters using the TEFIS platform and the users working on the Living Lab.
It has been decided that any data activity will be managed using get_input_file() and put_output_file()
from Connector.py class:
get_input_data:
Page 30 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Input: id of the experiment, and file to has got.
Logic implemented: the function get the file stored in the data manager.
Output: None
put_output_data:
Input: id of the experiment, and file to has let.
Logic implemented: the function let the result of execution file in the data manager
Output: None
To make it possible to use these functions is necessary install an iRODS server in the same machine as
the connector runs. It can be downloaded from: https://www.irods.org/index.php/Downloads
Also, it is necessary install PyRods, that is a python frameworks to communicate with iRODS server. It can
be downloaded from: http://code.google.com/p/irodspython/downloads/list
The configuration of the Tefis iRODS Server is:
[data‐management]
Irods‐server=tefis6.inria.fr
Irods_port=1247
Irods_zone=TefisTest
Irods_user=tefisUser1
Irods_password=tefisUser1
Page 31 of 32
TEFIS - D5.4.1– Connector implementation and documentation for LivingLab
Page 32 of 32
7 Status of the actual implementation of the Botnia Living Lab connector
After the completion of the first phase, a first implementation of the Living Lab connector is available to be used for the IMS‐BOTNIA use‐case in the first version of the TEFIS platform. A Living Lab use‐case is running and the Botnia Living Lab connector is working into the overall architecture of TEFIS platform and with respect to the Data Manager (with limited functionality, as said), Experiment definition, etc. The Living Lab connector presented in this deliverable is considered to be very different from other TEFIS testbed connectors because it deals with humans and the specific peculiarities of its implementation have been explained. In the near future, it has been considered to implement the first final version of the connector for the Open call experiments and they will be the first beta‐testers of the Living Lab connector.
8 Benefits for the TEFIS Living Lab usecase of the Living Lab connector
To conclude, the Botnia Living Lab connector is really needed for the IMS‐Botnia use‐case described
before as it allows to keep the Living Lab Manager informed of the entire process of the experiment,
sending the relevant work for execution, storing of resource configurations and keep track of execution.
Additionally, the connector will give the experimenter the ability of requesting, booking, configuration
and execution of the Living Lab resources and to support the interaction with the Botnia Living Lab team
and it also makes the Living lab resources to be accessible and used in parallel with other testing
resources.
9 Future Work In future work we plan to investigate the monitoring capabilities for Botnia Living Lab experiments and
further integration between the Botnia Drupal platform (testplats.com), the Living Lab connector and the
Data management functionality of iRods. What also is necessary to develop further is the functionality of
the connector when a task involves resources from different testbeds including Botnia Living Lab. We
also need to make a solution for how to design the functionality of the Experiment manager to handle
parallel tasks and dependencies between tasks and how this might influence next version of the Living
Lab connector design and its implementation.