Towards a Methodology for Benchmarking Edge Processing Frameworks
Pedro Silva, Alexandru Costan, Gabriel AntoniuInria, IRISA
France
Invited Talk, BenchCouncil’19, Denver, November 2019
Data Shifts to the EdgeBy 2022 Gartner predicts that 75% of enterprise-generated data will be created and processed outside of the data center and cloud infrastructures compared with 10% today.
Source: Smarter with Gartner, What Edge Computing Means for Infrastructure and Operations, October 3, 2018Extract from: BullSequana Edge positioning paper (Atos) 2
Why Edge Processing?
Advantagesq Easier access to dataq Bandwidth savingq Privacyq High potential parallelism
EDGE
DATA
CLOUD / DC
DATA
FOG
5
Edge Processing Tools
qCustom softwareqGeneric frameworks
qApache EdgentqAmazon GreengrassqAzure Stream AnalyticsqIBM Watson IoTqIntel IoTqOracle Edge Analyticsq…
EDGE
DATA
CLOUD / DC
DATA
FOG
6
How Great?EDGE
DATA
CLOUD / DC
DATA
FOGWhat is their performance?
Under which conditions?
Do they integrate well with my app?
8
Benchmarking: Questionsq Are the cost models precise?
q What is the impact of networking on the performance?
q How do my algorithms react to real-time scenarios?
q How does my hybrid approach compare to a fullycentralized solution? FOG
EDGE
CLOUD
q SILVA, P., COSTAN A. and ANTONIU, G., Towards a Methodology for BenchmarkingEdge Processing Frameworks. 1st Workshop on Parallel AI and Systems for the Edge (PAISE workshop collocated with IPDPS 2019).
10
Benchmarking Platform: Objectives
q Benchmark complete scenarios
q Control network characteristics
q Control framework configuration parameters
q Control Edge, Fog and Cloud infrastructures
11
Benchmarking Edge Processing: Related Work
q TPCx-IoTq Created for hardware benchmarkingq Fog oriented
q Academic benchmarksq Difficult to reproduceq Lack of a clear methodology (metrics, workloads,
parameters)q Not focused on the tools
12
Benchmarking Edge Processing Tools
met
rics
q Edge/Fog data processing toolsq Processing performanceq Supported programming languagesq Connectivityq Development easiness
q Use casesq Overall application performanceq Viability on different infrastructure configurations
workload
datatransmission
processing
…13
Benchmarking Edge Processing Tools : Parameters
Edge Fog
Cloud
…
…
…
Workloads:CCTV NYC TaxiEEW
Network:BandwidthLossLatency
Network:BandwidthLossLatency
Edge:Processingtools
Fog:MQTT server+ processingtools
Cloud:Kafka + Flink
15
Benchmarking Edge Processing Tools: Metrics
Edge Fog
Cloud
…
…
…
Throughput
Latency Edge to Fog Latency Fog to Cloud Processing Latency
Throughput
Each component has a resource utilization log.
16
Benchmarking Platform: Implementationq Experiment managerq Configures the infrastructureq Deploys frameworks/toolsq Deploys applications and manages their
executionsq Monitors resource usageq Gathers metrics and logs
q Edge+Fog+Cloud processing managementq Wrappers/interfaces
q Metric generation, configuration, connection
Experim
ent Manager
Infrastructure
VMs / Containers Bare Metal
Edge Fog Cloud
Python /Execo / EnosLib
Grid5K
enoslib
appstack
17
Earthquake Early Warning Systems (EEW)
Warning broadcaster
Seismometer
Data center
Data upload
P-wave
18
Earthquake Early Warning Systems (EEW)
ScientificInstruments
Intermediate machines with computing capabilities
…
Centralized data center Broadcasting users
… …
Data
Warning
q Deem: hierarchical and distributed ML algorithm
q Enables the usage of multiple types of sensors
q Enables the deployment on less powerful networks
q Enables local decisionmaking.
Deem: local decision
Deem: global decision
q FAUVEL, K. ; BALOUEK-THOMERT, D. ; MELGAR, D. ; SILVA, P., SIMONET, A. ; ANTONIU G. ; COSTAN, A ; MASSON, V ; PARASHAR, M. ; RODERO, I. ; TERMIER, A. A Distributed Multi-Sensor Machine Learning Approach to Earthquake Early Warning. Just accepted at AAAI 2020.
q SILVA, P., BALOUEK-THOMERT, D.; FAUVEL, K. ; MELGAR, D. ; SIMONET, A. ; ANTONIU G. ; COSTAN, A ; MASSON, V ; PARASHAR, M. ; RODERO, I. ; TERMIER, A A hybrid Fog and Cloud computing based approach for Earthquake Early Warning Systems. (In preparation.)
19
EEW: Fog-Based Infrastructure
q Thousands of producers
q High load on Fog and Cloud
q Objectivesq Reduction of network costsq Reduction of Cloud costsq Easier network reconfiguration (intelligent fog nodes)
20