EUROPEAN COMMISSIONEUROSTAT

Deputy Director-GeneralTask Force Big Data

Draft

Smart Statistics

The members of the DIME/ITDG are invited to comment on this document and provide advice on all strategic aspects of designing, testing, experimenting, prototyping, evaluating and mainstreaming the development of 'trusted' smart statistics. While the initial phase aims at developing a 'proof of concept' for smart statistics, the members of the DIME/ITDG are welcome to propose already at this stage 'use cases' for experimentation.

In this domain, the focus of our experimental research in the next few years should be on the statistical issues rather than whether the technological, legal, or data –access challenges have been resolved or not. When we see smart statistics in the context of an IoT-enabled world, the cognitive process is a fundamental process of how these smart statistics are created. We assume a world where the presence of sensors is ubiquitous while their interconnection is seamless (the technical details of making this happen is fundamental and critical, of course, but should remain out of the scope of study for a statistician).

In a network of seamlessly interconnected sensors (apparently with certain limitations in power consumption ), we foresee the realisation of a cognitive process where each sensor on its own, but mainly all sensors as a network, can perceive current social/economic conditions, plan, decide, act on those conditions, learn from the consequences of their actions, all while following end-to-end (policy) goals. This loop, the cognition loop, senses the environment, plans actions according to input from sensors and policies, decides which scenario fits best its purpose using a reasoning engine, and finally acts on the chosen scenario. The system learns from the past (situations, plans, decisions, actions) and uses this knowledge to improve the decisions in the future.

The above is a 'high level' simplified narrative which can be used as a starting background point for our discussion; such a discussion could include (but not limited to) the following questions:

1. What are the basic components of a system(s) of smart statistics?2. Which are the major statistical issues to address?3. How should a 'Proof of Concept' look like in practice?4. Do we need to develop new data and metadata standards together with IoT

Standardisation bodies like ETSI?5. Should we think of 'privacy by design' already at the development of a POC?6. Whom we should partner with in this path towards smart statistics?7. What could be the preferred scenario for implementation?8. Do smart statistics represent new opportunities and challenges for citizen science

and citizen data?9. Can you propose some promising use cases for experimentation of smart statistics?10. What are the most likely statistical domains for experimentation?

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 2 / 17

1. Background

Our societies and economies are facing an ever increasing degree of digitization that offers unprecedented opportunities for compiling statistics on various domains. The VIP Big Data addresses these opportunities together with challenges emerging from the use of big data sources for official statistics. However, the impact of these innovative technological developments on societies and economies will be by far more fundamental. This development can be expressed by the term "Datafication", which means "taking all aspects of life and turning them into data" (Cukier & Mayer-Schoenberger, 2013)1. Smart devices, electronic networks and constant production of data on all aspects of life and the environment will become an integrative component of how our societies and economies will function. Most if not all data in a decade from now will be "organic", i.e. by-products from activities of people, systems and things (including billions of low-end and affordable smart devices connected to the internet, i.e. the Internet of Things (IoT)).

During the coming decade, we will see a massive proliferation of electronic devices and sensors that are connected to the internet. They will generate and communicate huge amounts of data via this network. According to a study by the EC, IDC expects that the number of IoT2 connections will expand from 1.8 million in 2013 to almost 6 billion in 20203. An entire ecosystem will be developed around the Internet of Things affecting the physical, the digital and the virtual world.

– Figure 1: Internet of Things: knowledge integration4

1 Datafication was coined by Cukier and Mayer-Schönberger in its book "The Rise of Big Data". 2 Internet of Things; The IERC/ITU official definition states that IoT is "A dynamic global network infrastructure with self-configuring capabilities based on standard and interoperable communication protocols where physical and virtual “things” have identities, physical attributes, and virtual personalities and use intelligent interfaces, and are seamlessly integrated into the information network.". Available at: http://www.internet-of-things-research.eu/about_iot.htm3 IDC and TXT Solutions (2014), SMART 2013/0037 Cloud and IoT combination, study for the European Commission. Full text here: http://iotevent.eu/wp-content/uploads/2015/05/DefinitionofaResearchandInnovationPolicyLeveragingCloudComputingandIoTCombination.pdf4 Source: http://www.internet-of-things-research.eu/about_iot.htm

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 3 / 17

The fundamental question for us is how an official statistics organisation should adapt in order to remain relevant within such a data ecosystem? As such, this proposal covers the entire sphere of policy relevant statistics and issues associated with a full-fledged digital society. The use cases could address selective issues from the priorities of the Digital Single Market, Energy Union and Climate, and the Internal Market. Together with achieving a digital single market, the conditions for monitoring this market could be developed. In the near future, smart energy grids together with smart meters will be introduced in order to successfully manage the change to climate friendly energy. This provides opportunities for integrated data productions for the energy sector as well as for the industry and social and environment domain. Datafication of business processes and business relations could contribute substantially to monitoring the development of the internal market.

In addition, the introduction of smart appliances could be widely used to support collection of data on households and individuals. Similar opportunities will emerge for a wide range of policy areas where official statistics need to be re-thought and re-designed.

– Figure 2: Internet of Things: application areas and integration1

The project extends the current work on big data in so far as it is considering the entire ecosystem as an opportunity for intelligent production of relevant data for European Statistics. While the big data project follows a centric approach accessing data in places where they are collected, this project analyses the conditions for using the network and its components for producing relevant statistics, largely instantly and in an automated way.

We can think of Smart Statistics as being the future system of official statistics, where data capturing, processing and analysis will be embedded in the system itself, starting with the digital footprints of the activity. The realisation of such a system will probably go through a very long path. This large-scale study, which will be carried out by means of a competitive tender procedure, constitutes a first input in that

1 Source: https://opentechdiary.wordpress.com/tag/internet-of-things/

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 4 / 17

ambitious pathway. In that respect, and in the absence of any prior similar research in this area, it is important to underline the enormous exploratory nature of this work.

The project should be seen as an integral operational activity of the ESS Vision and the modernisation of the statistical system, while at the same time it tries to sketch ways of extensions in the future, i.e. beyond 2020. It is also aligned to current initiatives on data, information and knowledge management within the Commission, as well as to the data4policy activities (by which, a number of lead DGs, including ESTAT, are committed into building a permanent analytics capability in the Commission for the purposes of designing, implementing, monitoring and evaluating European policies). In this regard, Smart Statistics will provide substantive inputs on a variety of important aspects related to the functioning of data4policy in an ever increasing digitisation and automation of everyday activities. The lead DGs (and other key policy DGs) will be closely consulted in the definition of application areas and policy priorities which will be included in the technical specifications.

2. Scope and Objectives

2.1 Scope

The underlying reference framework of the proposal consists of the ESS Vision 2020 overarching aim of building the future of European Statistics and in particular the harnessing of new data sources. At the same time, the project also addresses other priorities and critical issues which emerge at the intersection of physical and digital worlds such as privacy, ethics, algorithmic transparency, quality, etc. Use cases will be selected in close consultation with users at policy level following the priorities of the Commission. By putting intelligence to all stages of the data lifecycle it is expected to enhance the efficiency of the entire statistical system and enable the ESS to maintain and reinforce its role as a key provider of data4policy in a digital world.

The project will concentrate on the development of the Internet of Things and its various applications, e.g.

mobility, smart cities, smart homes, wearables, smart energy, smart manufacturing smart energy grids, smart farming, …

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 5 / 17

as well as on infrastructural aspects (cloud infrastructures, intelligent gateways, mobile phones, and other controlling devices). In addition, it will analyse horizontal issues, such as security, privacy, algorithmic transparency, or IT.

2.2 Aims and Objectives

The aim of this project is to analyse the technological foundations of a future statistical information system (say 10 years from now), which will operate within an enlarged digitisation of our societies and greater interactions between the web, the web of data and a multitude of smart environments based on the Internet of Things, such as smart cities and Industry 4.0).

The analysis will be done following different dimensions. One dimension will cover architectural aspects. IoT devices, gateways, the network, cloud infrastructures and the business premises could take different roles and perform different activities depending on the purpose of the system, the type of data collected, or the phenomenon to be observed. Data is collected and aggregated by different components of the network. Depending on the amount of intelligence put into these components, decisions may be taken by them autonomously on how to interpret collected data and whether or not to invoke certain actions. In this context data collection and aggregation follows specific objectives. The study will investigate and analyse the possible role of European statistics in such systems. This may include (but it is not limited to) (i) using of third party systems (public or private) under specific agreements or legislation and (ii) developing entirely new data gathering approaches managed by the statistical system itself. A possible scenario for the future could be that statistical offices deploy sensors and devices exclusively for the purpose of generating statistical information. Another scenario would be using existing IoT networks that are deployed for other purposes than statistics in order to extract statistical information. The first approach would correspond to running surveys for statistical purposes while the latter would fit to using secondary data sources, such as administrative or big data. Mixed scenarios are also possible.

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 6 / 17

– Figure 3: IoT Architecture

Source: Gartner (September 2014)

The second dimension consists of examining the implications for official statistics by analysing the issues in terms of type of application domain, e.g.

mobility, smart cities, smart homes, wearables, smart energy, smart manufacturing smart energy grids, smart farming, etc.

Use cases should cover a broad range of the above mentioned application domains.

The third dimension is closely related to the latter, i.e. analysing by policy area. There will be a mapping between policy areas and application domains to identify possible inputs for European policies specifically those belonging to the stated priority areas of the EU.

The main output of the project will be a proof of concept of the implications of the Internet of Things for Official Statistics. The project should analyse latest developments and future perspectives in various domains that will be affected by the IoT and other smart environments, such as energy, transport, employment, health, population, economics, consumption, etc.

It will elaborate and propose a number of use cases that could be followed up in future projects. Wherever possible, the use cases will be mapped against the 10 priorities of the European Commission and their potential for providing (near real

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 7 / 17

time) policy relevant information. Additional criteria, such as delivery of tangible intermediate results could be added.

Particular attention will be paid to issues regarding the potential blending of traditional core survey instruments and sources (LFS, SILC, Health, etc) with data from electronic networks; another priority in this project consists in the design of a comprehensive data and information ecosystem in which statistics, visual analytics and dissemination services for EU policies form integral components of the system.

The project comprises an analysis of IoT and smart systems developments and possible application domains that could serve different policy areas. It requires an extensive consultation of stakeholders involved into the development, deployment, and use of smart systems with official statistics. The final output should include detailed analysis of the opportunities, obstacles, anticipated benefits and costs that occur when embedding statistical intelligence into smart systems. It is expected that this project will serve as input to a possible next phase of pilots following the definition of the use cases.

3. Impact Assessment

3.1 Stakeholder AnalysisThe national statistical offices, the ESS and statistical organisations at UN level are already engaged in projects using big data sources for official statistics. The ESS launched the VIP BIGD in Sep 2015. The UN Global Working Group on Big Data for official statistics was created in 2014. Reasons for engaging in modernisation of official statistics in this area are enlarging the pool of data sources for producing statistics, while facing decreasing willingness of citizens and businesses to reply to questionnaires. At the same time the forming of a digital society and economy changes the phenomenon that should be measured. Additional objectives for engaging into the project are potential for decreasing burden on respondents and increasing efficiency in data production. Considering the characteristics of the data collections systems, quality elements, such as timeliness or relevance could be enhanced.

One of the priority areas of the European Commission is the digital single market. The communication of the European Commission “Towards a thriving data-driven economy” is sketching the features of the future data-driven economy and sets out fields of activity to support the transition to this future economy. In April 2016 the Communication "Digitising European Industry"1 (COM(2016) 180) was published. It is accompanied by a staff working document "Advancing the Internet of Things in Europe"2 that analyses three main pillars for achieving the objective of advancing Europe as leading region in IoT products and services. The currently being prepared Commission Communication on data, information and knowledge management within the Commission calls for sharing data, information and knowledge widely

1 COM(2016) 180 final, see: http://ec.europa.eu/transparency/regdoc/rep/1/2016/EN/1-2016-180-EN-F1-1.PDF

2 see: http://ec.europa.eu/newsroom/dae/document.cfm?doc_id=15276

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 8 / 17

within the Commission and for collaborative working practices as preferred working method. One of the aims of the Communication is also to improve availability and use of data and information for better policy making.

The industry is developing the components of IoT systems and is offering services related to running IoT systems, processing data and doing analysis. Intentions for developing such systems are e.g. preventive maintenance, offering better services in the use of machines and devices related to mobility or health, increasing the efficiency of systems, or reducing impact of services on the environment. Standardisation activities are currently ongoing to ensure communication between the different components of IoT networks. There are currently different initiatives to developing operating systems suitable for the IoT and its different components.

Private entities are important partners in developing components for a statistical system, especially in terms of software development. It is also important to identify win-win situations in cases where statistical offices are using third party infrastructure for the purpose of collection statistical information. In addition, statistical offices could deploy IoT systems for their own purposes (partially) replacing traditional data collection methods.

At the same time enterprises could profit from introducing statistical components into IoT systems by lowering the use of resources for reporting to statistical offices and by their potential utility for the data ecosystems explored by private entities.

The European Central Bank and other central banks are participating in big data activities of the ESS. Joint developments in areas of mutual interest could be proposed to share the burden of developing data collection systems.

Academia use IoT for research purposes and are developing new approaches for data analysis in IoT systems. Existing partnerships should be extended to this new area to accompany the study. Possible synergies should be analysed for exploration at later stage.

The public, on the one hand, will benefit from efficiency gains achieved via IoT systems (energy grid, mobility, smart cities) and will experience better services (smart homes, wearables). On the other hand, increased collection of data and information on the use of devices and related personal data will pose issues of privacy, confidentiality, ethics and misuse of data. Approaches such as privacy by design are trying to avoid the possibility of misusing personal data. Automatic aggregation at certain nodes of the network could as well contribute to preserving privacy of citizens. Official statistics have to analyse these approaches in terms of suitability for the purposes of official statistics. The advantages for citizens would be increased offer and/or reduced burden and at the same time preserving the privacy of the citizens.

It is therefore expected that Smart Statistics will offer a unique opportunity for addressing some fundamental issues relevant to both the industry and society which go beyond official statistics like, for example, algorithmic accountability and transparency.

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 9 / 17

3.2 Project EnvironmentThe project is conducted among other modernisation activities that are currently ongoing. The closest relation is to activities exploring big data for statistical purposes and for improved policy making. While these projects concentrate on specific big data sources and integration of different data, the study here proposed focus on integrated systems, the Internet of Things and its smart applications for their potential to produce statistical information. The current projects are creating valuable input for this second step.

The project will require the collaboration of statisticians, private data owners, researchers, domain administrators, IT specialists and experts from various fields (including on legal issues). The issues to be addressed in the various application domains include - but are not limited to managing and integrating real-time and historical data, robust inference models for semantic statistics, automated / adaptive and scalable spatio-temporal reasoning, analysis and visualisation, embedding existing statistical knowledge (methods, processes, etc) into network appliances and nodes, governance of information and knowledge, privacy, ethics and IT infrastructure.

In order to raise awareness on the importance of IoT developments for official statistics, and highlight the multi-disciplinary character of this work, ESTAT is planning to organise a special topic session entitled 'From Big Data to Smart Statistics' at the next World Statistics Congress, in 2017. Official statisticians, academic researchers, industry players and privacy experts have committed to contributing to this session.

3.3 Cost-Benefit AnalysisDue to the nature of the project, it will be necessary to draw on expertise of specialists in various domains of IoT, which is currently not available, neither in Eurostat nor in other statistical institutes. For this reason the best approach would be a procurement procedure in order to ensure availability of the required mix of expertise.

As it was stated above, the objective of the project is to provide a proof of concept of intelligently integrating statistics into future IoT systems. It will sketch possible future development paths for official statistics, where data capturing, analysis and processing will be embedded in the activity itself, respectively the digital footprints of the activity. Moreover, statistics would be included in future technical systems and networks by design.

The proof of concept should also include estimates of the costs for possible future actions implementing this concept as well as analyse their benefits for European Statistics.

The timing of the project seems to be appropriate as statistical offices are gaining experiences in using big data and because IoT systems are being currently developed.

In case the project would not be conducted there is a risk of missing opportunities in a period of development, where it is easier to influence these developments and take

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 10 / 17

appropriate actions. It will be much more difficult and costly to change existing systems to integrate statistical needs retrospectively. Preliminary discussions with experts from the industry, academia, policy and official statistics suggest that the question is not about whether smart statistics will be realised or not, but rather whether the official statistics community will be pro-actively engaged or … others will definitely take over.

5.4 Risk AnalysisThe risk analysis refers to the execution of the project and to achieving the aim of clearly and adequately defining actionable use cases and a proof of concept.

The project is conducted in a very dynamic environment in which technological solutions are developed in a short time and new hardware and software developments might have systemic effects. This might affect the objective of the project and the definition of possible use cases. In order to minimize this risk it will be necessary to draw on very recent knowledge and to adapt the conceptual level of the conclusions accordingly. The project will not directly implement technical solutions. This would be the purpose of possible follow-up projects

The project requires extensive involvement and consultation of stakeholders including contributions from private sector. It will be necessary to communicate our objectives for conducting this project among stakeholders from the Commission and national authorities in order to ensure participation in the project. In addition, it will be necessary to clearly stress the importance of stakeholder participation and enumerate the type of stakeholders in the technical descriptions of the call in order to ensure appropriate preparations by the bidders.

There is always a risk that the defined use cases might not be perceived as being feasible in the context (and constraints) in which statistical offices are operating. In addition, the relevance of new statistical products might be difficult to assess, in particular because they may refer to completely new statistical products. The contractor is required to have deep knowledge on the role, objectives and activities of statistical offices, the conditions in which they operate, including relations with other major national stakeholders, and expectations of key users of the statistical products.

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 11 / 17

Annex 1 – Stakeholder Analysis

Stakeholder NeedsExternal Users Eurostat Internal

UsersPartners/Suppliers

Com

miss

ion

polic

y DG

s

NSI

s

Inte

rnati

onal

Acad

emic

s

Data

Pro

tecti

on

Auth

oriti

es

Gene

ral P

ublic

Subj

ect m

atter

un

its

Lega

l Uni

t / q

ualit

y

Met

hodo

logy

/ IT

DG JR

C / D

IGIT

Acad

emia

/ Re

sear

ch Indu

stry

Relevant information for policy design, implementation, monitoring and evaluation

X X

Legal and ethical conditions are met X X XHarness new data sources X X X X XIncrease efficiency X X X X XIncrease quality X X X X X XScientific support (X) X X X XResponse burden X X Xgain clients XAccess to data for research X XEstablish partnerships X X XStandardisation X X X

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 13 / 17

Stakeholder Future ExpectationExternal Users Eurostat Internal

UsersPartners/Suppliers

Com

miss

ion

polic

y DG

s

NSI

s

Inte

rnati

onal

Acad

emic

s

Data

Pro

tecti

on

Auth

oriti

es

Subj

ect m

atter

un

its

Lega

l Uni

t / q

ualit

y

Met

hodo

logy

/ IT

DG JR

C / D

IGIT

Acad

emia

/ Re

sear

ch Indu

stry

Flexible response to new user needs X X XSupport in data selection and analysis; Statistical expertise

X X X X

Design and planning of IT infrastructure X X XDevelopment of new skills X X XConstant improvement of data availability and quality

X X X

Exploring new data sources X X XJoint developments X X XExtension of client base X X XExploration of new business opportunities XBenchmarking data XExplore legal conditions using systems for statistical purposes

X

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 14 / 17

Annex 2 - Risk Analysis

I (Impact) x L(Likelihood) = RL (Risk Level)

1 (low) - 5 (very high) * 1 (low) - 3(very high) = 1(low) – 25 (very

high)

Risk No.

RiskIf "risk" happens then…

I L RL Mitigation: Proposals to eliminate / minimise the risk (What you will do in any case to reduce/manage the risk)

Contingency: What you will do to respond (manage the impact) if the risk occurs

(What you already do when it happens)

1 Technology disruption 4 4 16 Put special emphasis on expertise in the domain

adapt the conceptual level of the conclusions

n Insufficient expertise 5 2 10 clearly stating the requirements and providing information on the application areas

Reject offer

Insufficient stakeholder involvement 4 2 8 communicate objectives of study among stakeholders from the Commission and national authorities in order to ensure participation in the project.Clearly state importance of stakeholder participation and enumerate the type of stakeholders in the technical descriptions of the call in order to ensure preparations by the bidders.Plan well in advance workshops

Re-contact stakeholders in case of consultationsConsult stakeholders in case of non-participationPhone, video or face to face contactEnlarge pool of stakeholders in order to be able to replace missing stakeholders

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 15 / 17

Contact stakeholders to prepare for participation

Insufficient involvement of NSIs 4 1 4 Inform well in advance at appropriate level

Enlarge group of NSIs; have more than one contact person

Proposed solutions not appropriate for ESS

4 2 8 Consult NSIs; ensure involvement -

Delays in project execution 3 2 6 Analyse execution periodically; review with contractor; set priorities

Reschedule activities

IoT system differ by country 4 3 12 Include different countries into analysis; make overview by country

-

Legal environments differ by country 4 2 8 Include different countries into analysis; make overview by country

Adjust use cases

Changes in legislation (impacting on IoT applications, data protection)

4 2 8 Monitor changes Evaluate before implementing use cases

I (Impact) x L (Likelihood) = RL (Risk Magnitude)

PM² Simplified Template V.0.3 Date: 12.08.2016 Version: 0.1 16 / 17

17