BigFoot project

GRIDPOCKET 2012 1

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Filip GLUSZAK filip.gluszak@gridpocket.com

+33 6 79 73 90 52

Paris - Sophia Antipolis 20/03/2012

Contents �  SmartGrid applications for BigData

�  Grid monitoring �  Smart metering �  Intelligent home

�  BigFoot research project �  Consortium �  Platform �  Objectives

�  Conclusions

GRIDPOCKET 2009 CONFIDENTIAL 2

3

Provider of value added services platforms for energy

•  GRID = ‘electric network’, POCKET = ‘personal’,

•  established in 2009, 10 employees

About GridPocket

3

GridPocket Systems SA R&D Center Poland, Gdansk

Head-Quarters France, Sophia-Antipolis

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

4

« kWh » to « Energy Services » paradigm shift

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

New needs and opportunities:

Data Services for Energy Environmental

constraints

Market deregulation

Technology disruption

GRIDPOCKET 2012

Smart Grid data analytics market will generate $11.3 billion cumulative from

2011 to 2014 (Pike Research 2011)

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Data sources in SmartGrid

� Grid monitoring

� Smart Meters �  Intelligent home / M2M

5 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

6 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Synchrophasors data �  Phaser Measurement Units provide

phaser information (magnitude and phase angle) in real time

�  Reporting rate of up to 60 frames per second to Phaser Data Concetrators

7 Source : Siemens.com

Source : Siemens.com

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Example projects �  Tennessee Valley Authority

(TVA) Hadoop �  OpenPDC (data from Phaser

Measurement Units) standard �  project done by Claudera

�  Data volumes : �  Serving 9 million consumers �  Planning 1000 PMDs �  30 readings per second �  4.2 billion samples per day �  Need 500TB storage

�  Application �  MapReduce oscillation scan :

detection of power grid anomalies, creating power grid maps and evaluating power consumption history

8

Source : claudera.com GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

9 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

10

Smart metering deployments

Electricity, Gas and Water markets 270 millions smart meters in 2013 (ABI, Gartner)

$200 billion investment by 2015 (Pike)

Smart metering projects world-wide (Engage Consulting)

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Regulations supportin smart metering �  EU energy-climate package

20/20/20 �  EU directive supporting smart

metering (2009/72/CE) – 80% in 2020

�  Directive EU – obligation for energy reduction of 1,5% per year

�  Building energy performance directive (2002/91/EG)

�  RT 2012 – new constructions regulation

11 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Commercial Billing �  Valuation, Estimation and Editing (VEE) – prepare data for

billing �  Application of complex pricing methods, rate schedules

�  Time of use (TOU), Critical peak pricing (CPP), Peak day pricing (PDP)

�  Production of billing data, settlement statements, transaction reports, production and distribution billing

12 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Consumer applications �  Energy consumption visualization �  Bill analysis (where the usage comes from, reduce call center

load) �  “Rebound effect” management, behavioral influencing �  Social applications

13

0 10 20 30 40

Indirect

Direct display

PC displays

Ambient (alerts)

Social impact

Variable pricing

% d’économie d’énergie Source: Oxford 2006 report based

on peaks

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Demand-side management �  Grid assets usage analysis and forecasting and backcasting �  Load research and trend analysis �  Remote load control – demand – response mangement �  Outage management and distribution optimization

14

http://www.moorsydeactiongroup.org.uk/windpower.html Source; Eurpean Energy Exchange (28 Dec 2009) http://www.eex.com/en/Market%20Data/Trading%20Data/Power/Intraday%20|%20Spot/Intraday%20Chart%20|%20Spot/spot-intra-chart/2009-12-28/1d

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Revenue management �  De-regulation of energy markets

○  High utilities consumers churn at 10-15% annual levels �  Product differentiation, customers segmentation �  Customer retention (loyalty) tools, behavioral analysis �  Revenue loss detection (customer accounts analysis)

15

Source Poweo financial report Sept 2009

Source CACI Scottish and Southern Energy Case 2008

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Volumes of data � For 35 million meters (France), data

reading every 10,30, or 60 minutes � Generates : 40-200 TB per year (depending

on reading frequency and format) � Gas and water grid under deployment

16 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Source : ERDF

!Source : ERDF

Example projects with metering data �  Ontario Power Authority (OPA) experimentation planned 4.5

million meter collecting data every hour – system wide analysis

�  Southern California Edison with Teradata – 100TB warehousing for user metering data

�  Itron partners with IBM, SAP and Teradata to form Active Smart Grid Analytics – enabling applications such as Settlements, Network Loss, and Revenue Protection

�  Opower - 56 billions meter reads per year, planning to use Hadoop

17

External data sources

OperationalDatabases

Interactive Query Engine

DistributedData Store

DistributedFileSystem

Batch Analytics Engine

Dashboard

Q Q Q

Ad hoc Queries

Data MiningAlgorithms

StatisticalPatternsAnalysis

Applications:* Front-end* Data analysis

BigFoot System:* DataStores* Data Engines

Data Sources

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

18 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Intelligent home applications

19

•  Security  (detec,on,  camera)  

•  Smart  metering  

•  Ligh,ng  control  •  Smart  appliances  

•  Comfort  control  /  HVAC  

•  Mul,media  

•  And  more…  

GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Volumes of data �  Multiple standards for sensors inter-

connectivity �  ETSI M2M framework �  For 35 million meters,

�  4-20PB per year in the future (100 sensors per HH)

20 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

21 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

BIG Data Analytics of Digital FOOTprints

BigFoot project consortium

�  Eurecom (France) Project leader �  Ecole Polytechnique Fedérale de

Lausanne �  TU Berlin / Deutsche Telecom Lab

(T-Lab) (Germany) �  Symantec (Ireland) �  GridPocket (France)

�  Projet Proposal to EU FP7 Call

22 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Eurecom experimental platform

�  100 nodes �  single ARM v5, 1.2GHz,

512MB RAM, 64GB

�  Power efficient �  5 watts per node

�  Simple network �  GB Ethernet

23 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

Eurecom platform - research results �  Benchmarking platform for performance evaluation

and comparative analysis of the inner components in the DFS and parallel processing framework

�  System optimization for map reduce, using synthetic workloads (eg. load similar to Facebook jobs or others)

�  2 papers published �  « ACM SIGOPS LADIS : Hbase, bulk data input »

- resultas integrated in the Hbase source (v0.9.x) �  Data analysis for a telecom operator (A1 Austria)

– log files (billing and anomaly detection) algorithm

24 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

BigFoot fundamental objectives �  Cross-layer approach system

optimization �  Service-oriented query engine - transform

high level declarative query language into parallel programs

�  Novel data placement mechanisms for storage layer optimization - physical optimization (layout, typically data is sorted in time, optimize the way data is written)

�  Go beyond best-effort on virtualized clusters (Performance guarantee) �  Currently virtualization servers are

applications agnostic, (eg. If several VM share same I/O there is no benefit)

GRIDPOCKET 2009 CONFIDENTIAL 25

VIRTUALIZATION

APPLICATIONS: DATA ANALYTICS SCALABLE ALGORITHM DESIGN

HIGH-PERFORMANCE BATCH ANALYTICS ENGINE

DATA STORES

HIGH-LEVEL QUERY LANGUAGE

LATENCY-SENSITIVE QUERY ENGINE

Distributed File System

Distributed Data Base

BigFoot experimental objectives �  Experimental test beds

�  Fine-grained measurements on all system layers

�  Design, implementation and evaluation of algorithms for large data processing �  IT Logs Cyber Attacks analysis

(Symantec) �  Grid data analysis (GridPocket)

GRIDPOCKET 2009 CONFIDENTIAL 26

BigFoot open-source objectives

� Contribution to the existing open-source projects (eg. Apache Hadoop)

� Establishment of new open-source projects for BigFoot when relevant

GRIDPOCKET 2009 CONFIDENTIAL 27

BigFoot - SmartGrid data processing tasks �  Data analysis tasks apply, for example, to the

following cases: �  Consumer billing: a monthly scan of all customer data to

produce consumption reports; �  Consumer Dashboard Web applications: analysis of the

whole consumption data to create personalized customer reports;

�  Load analysis: non intrusive detection of the loads mix, detection of anomalies, consumer behavioral analysis

�  Consumer segmentation: execution of complex algorithms to classify consumers based on their consumption patterns and produce, for example, personalized contract;

�  Assets usage analysis: analysis of geographical consumption patterns and design of predictive algorithms to help operators in provisioning of their electric network.

28 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

BigFoot SmartGrid scenario

29 GRIDPOCKET 2012

BigFoot Platform SmartGrid Apps

Conclusions �  Smart Grid and big data

�  Generates significant amounts of data �  New needs and applications are coming

�  Hadoop technology �  Promising, however still need to gain on maturity �  Complex setup and maintenance �  Interoperability �  Limited skills and support available in Europe

�  Next steps �  BigFoot and other BigData projects w �  GridPocket and BigFoot are open for cooperation �  Looking for comments and suggestions

30 GRIDPOCKET 2012

GRIDPOCKET P E R S O N A L S M A R T G R I D S O L U T I O N S !

31 GRIDPOCKET 2012